Gary, We just went through the same exercise with hpctoolkit's call path profiler regarding system. Below I point you to fresh code that solves our problem that can be adapted to solve yours.
The problem is that the call to system involves forking (as you know). When forking using PAPI and the perfmon drivers, bad things happen. PAPI should take care of this, but it doesn't. In the fullness of time, PAPI will get fixed. In the meantime, we have a workaround. To deal with system in our call stack profiler, we fork off another process to execute system calls on our behalf. We fork off this server process early before monitoring gets turned on. That way we don't need to fork processes for system while monitoring. See the following: the system server: https://hpc.svn.rice.edu/r/hpctoolkit/trunk/src/hpctoolkit/csprof/system_server.c its .h file: https://hpc.svn.rice.edu/r/hpctoolkit/trunk/src/hpctoolkit/csprof/system_server.h a client that initializes it, uses it and finalizes it: https://hpc.svn.rice.edu/r/hpctoolkit/trunk/src/hpctoolkit/csprof/fnbounds/fnbounds_dynamic.c While this works well for us, there are a few things to worry about: 1. the process that gets forked off is under control of "monitor" (it has our monitoring code preloaded and in control). Thus, we need to be careful to sidestep monitor's involvement explicitly by calling unwrapped versions of exit, fork, and system. (Note hpcrun and csprof use different versions of "monitor"). hpcrun uses an older version that doesn't wrap as many things. Thus you can't use the code I pointed you at directly in hpcrun for that reason. e.g., you won't find a version of monitor_real_system available, nor monitor_real_fork. You can call probably call the libc routines directly to get around monitor. On our systems, that can be done by invoking __libc_system or __libc_fork, etc. 2. if you have user's system commands executed by another process, you need to make sure that anything they expect in the client makes it there, e.g. environment variables, pipes. That is what I meant about having problems doing this in general. You can override setenv an application and pass environment variable settings to the system server process. However, env can be manipulated directly by user apps. Most applications won't do this. You can probably code something that will work for all but pathological applications. I hope this helps. Regards, John M-C On May 13, 2008, at 7:07 PM, [EMAIL PROTECTED] wrote: > > Hi Mark > > I have written a test case that consistently demonstrates this > problem. > > Using this program: > >> cat ForkTest.cpp > > #include <stdio.h> > #include <stdlib.h> > > main (int argc, char **argv) { > int i, j, k; > printf ("Start test\n"); > > system("ls -al"); > > for (i=0 ; i<1000 ; i++) { > if ((i % 100) == 0) { > printf ("Doing loop %d\n", i); > } > for (j=0 ; j<1000000 ; j++) { > k = j * i; > } > } > printf ("End test\n"); > } > > I compile it with: > > gcc -g -o ./FortTest. ./FortTest.cpp > > I then run HPCToolkit and pfmon on it with the commands: > > hpcrun -e CPU_OP_CYCLES_ALL:32767 -o ./hpcrun.data -- ./FortTest >./ > hpcrun > 2>./hpcrun.debug > hpcprof --debug 5 ./FortTest ./hpcrun.data/* >./hpcprof 2>./ > hpcprof.debug > pfmon --debug --follow-fork -v -e CPU_OP_CYCLES_ALL ./FortTest >pfmon > 2>pfmon.debug > > The hpcrun data files created are: > > -rw-r--r-- 1 hpctk users 1666 May 13 16:06 > bash.CPU_OP_CYCLES_ALL.molson.19755.0x4d2b > -rw-r--r-- 1 hpctk users 1164 May 13 16:06 > ForkTest.CPU_OP_CYCLES_ALL.molson.19754.0x4d2a > -rw-r--r-- 1 hpctk users 2200 May 13 16:06 > ls.CPU_OP_CYCLES_ALL.molson.19755.0x4d2b > > The results in the file ./hpcprof show : > > Columns correspond to the following events [event:period (events/ > sample)] > CPU_OP_CYCLES_ALL:32767 - CPU Operating Cycles -- All CPU cycles > counted > (35 samples) > CPU_OP_CYCLES_ALL:32767 - CPU Operating Cycles -- All CPU cycles > counted > (0 samples) > CPU_OP_CYCLES_ALL:32767 - CPU Operating Cycles -- All CPU cycles > counted > (38 samples) > CPU_OP_CYCLES_ALL (min):32767 - CPU Operating Cycles -- All CPU > cycles > counted (The minimum for events of this type.) (0 samples) [not shown] > CPU_OP_CYCLES_ALL (max):32767 - CPU Operating Cycles -- All CPU > cycles > counted (The maximum for events of this type.) (63 samples) [not > shown] > CPU_OP_CYCLES_ALL (sum):32767 - CPU Operating Cycles -- All CPU > cycles > counted (Summed over all events of this type.) (73 samples) [not > shown] > > The multiplication of samples times period (73 * 32767) give me: > > 2,391,991 Cpu cycles used > > But the results in the file ./pfmon show: > > 26,045,577,230 CPU_OP_CYCLES_ALL > > The test actually runs about 20 seconds on my system which makes the > number > reported by pfmon appear to be accurate. > > > If I comment out the line (to rerun the test without the fork): > system("ls -al"); > > Then recompile and rerun the test: > > The only hpcrun data file created is: > > -rw-r--r-- 1 hpctk users 1392 May 13 16:09 > ForkTest.CPU_OP_CYCLES_ALL.molson.19814.0x4d66 > > The results in the file ./hpcprof show : > > Columns correspond to the following events [event:period (events/ > sample)] > CPU_OP_CYCLES_ALL:32767 - CPU Operating Cycles -- All CPU cycles > counted > (795988 samples) > > Which represents: > > 26,082,138,796 Cpu cycles used > > The pfmon results for this test run reports: > > 26,045,163,278 CPU_OP_CYCLES_ALL > > It is clear from these test results that hpcrun does not work > correctly for > this type of executable and the problem is now easily reproducable. > > I do not know if the workaround you have considered is the best way > to fix > this or if the PAPI fix is a better way to go. But if you guys can > point > me > in the right direction and give me an idea what needs to be changed, > I am > willing to try and implement the fix. I just want to come up with a > correction > that someone with the authority to commit changes to the CVS/SVN > would be > willing to put into a future release. > > Thanks for your time, > > Gary > > _______________________________________________ > Ptools-perfapi mailing list > [EMAIL PROTECTED] > http://lists.cs.utk.edu/listinfo/ptools-perfapi > > !DSPAM:482a2dae24087422695342! -- John Mellor-Crummey Associate Professor Dept of Computer Science Rice University email: [EMAIL PROTECTED] phone: 713-348-5179 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ perfmon2-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/perfmon2-devel
