Skip to content

OProfile: Profile Your System

OProfile

Introduction

Profiling

OProfile is a tool used for profiling.

Profiling is the process of examining data from various existing data sources (databases, files, etc.) and collecting statistics and information about this data. It is very similar to data analysis.

Profiling objectives include:

  • Identifying reusable data for other purposes
  • Getting measurements on data quality and compliance with company standards
  • Evaluating risks generated by integrating this data into new applications
  • Evaluating if metadata correctly describes source data
  • Having a good understanding of the impact of source data on upcoming projects to anticipate surprises. Late discovery of data problems can lead to delays and budget overruns (e.g., having to modify a code format in hundreds of places across projects, having to rebuild associated references, having to rediscuss and revalidate contractual documents, etc.)
  • Having a global view of the data to allow reference data management or data governance to enhance data quality

OProfile

OProfile can profile:

  • Interrupts
  • Applications and shared libraries
  • Complete system performance
  • Events at regular intervals

One big advantage is that it uses minimal system resources.

OProfile disadvantages:

  • One major drawback of OProfile is that it can miss captures
  • You need to install many development packages to use it, which isn't always feasible in production environments

To better understand how OProfile works, here's a diagram:

Oprofile

OProfile aims to be as unintrusive as possible. OProfile loads a kernel driver at boot time that manages hardware performance based on processor counting. The /dev/oprofile device serves as an interface with user mode. The daemon reads this device and stores information in /var/lib/oprofile/samples.

Installation

Red Hat

On Red Hat, here's what we'll install:

1
yum install oprofile oprofile-gui

To ensure OProfile will work properly, let's check that it's present in the kernel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
> grep -i PROFIL /boot/config-`uname -r`
CONFIG_EVENT_PROFILE=y
CONFIG_PROFILING=y
CONFIG_OPROFILE=m
CONFIG_OPROFILE_EVENT_MULTIPLEX=y
CONFIG_HAVE_OPROFILE=y
# GCOV-based kernel profiling
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
CONFIG_FUNCTION_PROFILER=y

Only the first two highlighted lines are essential; the others can be useful but are not mandatory.

Now we need the kernel debug tools:

1
yum install kernel-debuginfo

Note: kernel-debug is different from kernel-debuginfo

Usage

Setup

Before using OProfile, you need to define what type of application you want to profile.

  • If you want to profile the kernel, we need to make OProfile start at system boot. Add this line (/etc/rc.local):
1
opcontrol --setup --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux
  • If you want to profile an application, then execute this:
1
opcontrol --setup --no-vmlinux

It's possible to reduce the profiling scope. Here's an example:

1
2
opcontrol --list-events
opcontrol --events=CPU_CLK_UNHALTED:400000 --event=DATA_MEM_REFS:10000

Test

Let's run a test. First, we'll reset the current data:

1
opcontrol --reset

Then we'll start profiling:

1
opcontrol --start

Now run the desired tests, and OProfile metrics will be periodically saved under /var/lib/oprofile/samples/. To force this:

1
opcontrol --dump

Then we'll stop profiling:

1
opcontrol --stop

View Results

For all applications:

1
opreport > ~/oprofile.data

If we look at the oprofile.data file, we find this interesting information (~/oprofile.data):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
CPU: CPU with timer interrupt, speed 863.195 MHz (estimated)
Profiling through timer interrupt
          TIMER:0|
  samples|     %|
------------------
  1698146 99.5540 no-vmlinux
     1178  0.0691 libc-2.12.so
      957  0.0561 libpixman-1.so.0.18.4
      839  0.0492 libxul.so
      665  0.0390 libpython2.6.so.1.0
      569  0.0334 libsqlite3.so
      323  0.0189 libglib-2.0.so.0.2200.5
...
  • samples: Shows how many times the executable was called
  • %: The percentage of CPU time spent executing each executable

If you want to profile just one application:

1
opreport -l /bin/bash > ~/oprofile.data

FAQ

Why is the CPU time at 0%?

This is because APIC is not enabled. Make sure idle=pool is in grub. Then check that it's enabled in /proc/cmdline.

Resources

Red Hat documentation: /usr/share/doc/oprofile-*/oprofile.html