Stephane Eranian wrote:
Will,
Thanks for posting this message. I think we do need to come
up with a strategy as to how we validate the kernel
interface and the library. Keep in mind that the library
is optional, you do not need the library to use the kernel
interface. Tools can replace it with whatever they like, they
may even have everything hardcoded.
On Tue, Aug 01, 2006 at 03:03:56PM -0400, William Cohen wrote:
Having confidience that the code works as expect would be a really
good thing for perfmon2. Currently, there has been a significant
developement on perform2 in the kernel, the user-space libraries, and
example programs demonstrating how to use it. Software packages such
as gdb and gcc have dejagnu test suites that check for previous
problems and check that the code functions as expected. There should
be similar testing available for perfmon2. These tests should be be
able to run unattended.
I have never used dejagnu myself. So I don't know how easy this is.
I have written some dejagnu tests for systemtap and oprofile. It
wouldn't be that hard to code up the framework with some tests and make
it fairly easy for people to add tests to it.
The perfmon2 kernel additions, shared libraries, and differences in
performance monitoring hardware make it a bit more difficult to use
some of the existing tools. Testing of user-space programs can catch
cases where the program crashes, but testing of the kernel may cause a
crash of the system and require reboot. Valgrind intercepts the
systemcalls to model the kernels actions on userspace and the
additional systemcalls for perfmon2 are not recognized by valgrind
(some patches developed to address this Valgrind issue). Some
programming tools such as cgov are not designed to work with shared
libraries. Finally, each processor implementation has its own
implentation of performance monitoring hardware, the testsuite need to
be designed to allow for the variations between processor performance
monitoring hardware.
Testing the Kernel code
Want to check the following in kernel:
-Exercise the various sections of perfmon kernel code
-Invalid arguments to system calls handled gracefully
-No problems with locks
Functional testing is something we can do in a portable way. I have
written a long long time ago some simple tests. They do not require
knowledge about a particular PMU. It gets more difficult when you are
trying to verify that measurement results are correct. You cannot
simply count cycles and expect the same value over and over. You have to
use an event for which you know the answer in advance. In the past, on IA-64,
I have used the event that counts misaligned loads retired. This is very
convenient because those do not happen frequently and you can fairly easily
create a test where you know the answer. I suspect on x86 and others we
may be able to construct such test cases, maybe using MMX events.
There are a number of things in pfmon/tests and libpfm/examples to
exercise the code and provide particular cases. Is there some comments
on what the goal of some of them are, e.g. what arguments to use with
pfmon when running them.
OProfile has a similar problem with the events being machine specific
and the measurments are likely to vary from run to run. pfmon has
similar problems. There can still be "smoke tests" to make sure that the
event is being measured in various situations. It would be nice to make
sure that every event measures, but that is probably not reasonable.
But there are lots of things that can be tested without looking at the
counts, just looking at permissions for instance.
Much of the kernel testing could probably be done as part of the
libpfm testing.
Well, it depends on what you are trying to test. Functional testing
does not require the library.
True functional testing doesn't require the library. It was a
suggestions to minimize the number of separate testsuites need to be
maintained. Kind of a pre-test to make sure that things work before
testing the library.
Testing the shared libraries, examples, and pfmon
Want to check the following in shared libraries:
-Exercise the various sections of library and example programs
-Invalid arguments to library handled gracefully
-Check for use of uninitialized memory access past array bounds
The library interface is divded into two sections: generic and model-specific.
The geenric interface is published in pfmlib.h and works for all PMU
models. Then in each PMU model-specific header file you may have function
specific to a PMU model. For instance, on IA-64, pfm_ita2_get_event_group()
is specific to Itanium2. Those would have to be tested as well, yet they
are not as critical as the generic ones. The key generic function
is pfm_dispatch_events(), and unfortunately, it is difficult to use it
in a generic manner, i.e., you need to know the answer which is always
model specific. For instance to verify that is does the correct job
for LOADS_RETIRED on Itanium2, you need to know this event can only
be measured by PMC5. That means that some of the tests for this function
would have to be model specific even though this is a generic function.
The oprofile testing looked at information provided by the
/dev/oprofile/cpu_type. Similar information for the testing could be
obtained from /proc/cpuinfo to determine which things should be tested
for perfmon2. Is there a case where there would be a choice of perfmon
support installed? For example oprofile could use timer mode or perfmon
hardware support. Is there ever a choice of which kernel module for
perfmon support would be installed?
At OLS2006 the use of GCC's mudflap option on libpfm example code was
demonstrated and confirmed an array bounds problems found earlier in
libpfm. Patches for valgrind have also been created to handle the new
perfmon2 systemcalls this should allow checking for problems with
memory allocation. The main limitations of these tools is the code
cannot be statically linked because they replace some of the memory
allocation routines. Currently, the libpfm examples are statically
linked.
Yes, mudflap is an interesting option to gcc. Yet when I used it after OLS I had
some difficulty decoding the output. I could not find a good documentation
explaining how to interepret the messages.
There are a couple papers on mudflap. The gcc summit paper has some
disccusion of the output messages.
http://gcc.gnu.org/wiki/Mudflap_Pointer_Debugging
http://gcc.fyxm.net/summit/2003/mudflap.pdf
It would be nice if valgrind had stripped down instrumentation to do
code coverage. It would be easy to get code coverage of shared
libraries then. Some people have used cachegrind to do this, but this
has significant slowdown due to the additional overhead to track
memory references and simulate the cache. The following URL outlines
what is required to make a stripped down code coverage skin, but
doesn't address mapping the data back to source code:
wwwse.inf.tu-dresden.de/data/courses/ss06/SFT/exercise/13_slides.pdf
Probably want to have several additional rules in the libpfm and pfmon
make file to run the tests, e.g. "check", "check_syscall",
"check_library", and "check_examples".
Yes, we could certainly integrate the testing into the source tree and
make empire for libpfm.
Will, do you intend to take a stab at some of the tests you are proposing
here? And if so, which one would you do first?
Thanks.
Yes, I do intend to take a stab at the testing code. As far as order I
see the kernel systemcalls and systemcall error handling first. Would
like to improve confidence that all the system calls are bulletproof.
The generic libpfm library calls would be next.
I really wished that valgrind had an instrumentation skin that did the
equivalent to gcov builtin. That would be a really useful for thist (and
other projects).
-Will
_______________________________________________
perfmon mailing list
[email protected]
http://www.hpl.hp.com/hosted/linux/mail-archives/perfmon/
