* Goetz, George ([email protected]) wrote: > Hello, > > We are trying to use the Linux Trace Toolkit (LTT) for the first time to > perform some performance modeling on an application we are developing. > > We are using the real time TimeSys Linux kernel. The vendor recommends the > LTT tool but does not offer direct support for it. We are using Version > 2.6.22.1-rt4 to patch the TimeSys kernel (Recommended by the vendor). This > patch constrains us to > 0.8.83 version of the LTT Viewer and 0.42 for LTT Control. We have > successfully built all the kernel and tools. Several files had to be patched > manually. We linked in all the modules statically so did not do any of the > modprobe commands. We have been able to generate some trace data but have not > > It appears as if everything started up ok as indicated in the following > excerpt from dmesg : > > io scheduler noop registered > io scheduler anticipatory registered > io scheduler deadline registered > io scheduler cfq registered (default) > LTT : ltt-facilities init > LTT : Facility core registered with id 0 > LTT : Facility fs registered with id 225 > LTT : Facility kernel_arch registered with id 230 > LTT : Facility kernel registered with id 211 > LTT : Facility list registered with id 145 > LTT : Facility mm registered with id 59 > LTT : Facility net registered with id 179 > LTT : Facility locking registered with id 178 > LTT : ltt-relay init > ltt-control init > LTT : ltt-facility-statedump init > Boot video device is 0000:00:02.0 > pci_hotplug: PCI Hot Plug PCI Core version: 0.5 > ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3]) > ACPI: Processor [CPU1] (supports 2 throttling states) > ACPI Exception (processor_core-0782): AE_NOT_FOUND, Processor Device is not > present [20070126] > Real Time Clock Driver v1.12ac > > When using the following command : > > -bash-2.05b# lttctl -n TryAgain -c -d -l /mnt/debugfs/ltt -t /usr/trace > > The output to the terminal looks like everything is starting up correctly: > > Linux Trace Toolkit Trace Control 0.42-16072007 > > Controlling trace : TryAgain > > Linux Trace Toolkit Trace Daemon 0.42-16072007 > > Reading from debugfs directory : /mnt/debugfs/ltt/TryAgain > Writing to trace directory : /usr/trace > > Creating supplementary trace files > Appending facility file core.xml > Appending facility file fs.xml > Appending facility file kernel.xml > Appending facility file kernel_arch_arm.xml > Appending facility file kernel_arch_c2.xml > Appending facility file kernel_arch_i386.xml > Appending facility file kernel_arch_mips.xml > Appending facility file kernel_arch_powerpc.xml > Appending facility file kernel_arch_ppc.xml > Appending facility file kernel_arch_x86_64.xml > Appending facility file locking.xml > Appending facility file mm.xml > Appending facility file net.xml > Appending facility file stack_arch_i386.xml > Appending facility file stack_arch_x86_64.xml > Appending facility file list.xml > Appending facility file user_generic.xml > Appending facility file xen.xml > Appending facility file compact.xml > > The directory /mnt/debugfs/ initially has directories kprobes, ltt, and > usbmon. The usbmon has files 0s, 0t, and 0u but all are empty. The directory > kprobes has files enabled and list but both are empty. The ltt directory is > empty. After the command the directory TryAgain is created under directory > ltt. TryAgain is created with file cpu_0 and directory 'control' under it. > All the files are empty. > > The messages in dmesg again appear to indicate things worked > > ACPI: Power Button (FF) [PWRF] > input: Power Button (CM) as /class/input/input1 > ACPI: Power Button (CM) [PWRB] > device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: > [email protected] > eth0: link up, 100Mbps, full-duplex > NET: Registered protocol family 10 > lo: Disabled Privacy Extensions > eth0: no IPv6 routers present > ltt-control ltt_control_input > ltt_control : trace TryAgain > Creating trace TryAgain > ltt-control ltt_control_input > ltt_control : trace TryAgain > Start tracing TryAgain > Dumping facility core > Dumping facility mm > Dumping facility list > Dumping facility locking > Dumping facility net > Dumping facility kernel > Dumping facility fs > Dumping facility kernel_arch > ltt_statedump_start > do_ltt_statedump > do_ltt_statedump end > > When I stop the trace with > > -bash-2.05b# lttctl -n TryAgain -q > > The following output is produced > > Linux Trace Toolkit Trace Control 0.42-16072007 > > Controlling trace : TryAgain > > The following messages were added to dmesg > > ltt-control ltt_control_input > ltt_control : trace TryAgain > Stop tracing TryAgain > > The /usr/trace directory contains cpu_0 and two directories - control > and eventdefs. The file cpu_0 contains the majority of the data. The > eventdefs contains XML descriptions of events. These are created when > the command is entered. There are 5 files under the control directory > - facilities_0, interrupts_0, modules_0, network_0, and processes_0. > The only file which contains data is processes_0. The others are > empty.
Clearly, there is a problem here. facilities_0 should contain the event metadata. Please look into lttctl --help to see if -q was the appropriate way to stop tracing _and_ end the trace session. My first guess is that you are just stopping the trace. Make sure that the /mnt/debugfs/ltt directory is empty after you stop tracing and end the trace session. It looks like there is still data sitting in your buffers in memory. > > We are using a separate computer to analyze the data so I copy the > entire trace directory to the remote computer. The Viewer does not > recognize the trace data. If I invoke the viewer with the command > > lttv -m batchAnalysis -t trace > > It states the obvious - the newtwork_0, interrupts_0, modules_0, and > faciltities_0 do not contain a trace. It also gives an error > > **ERROR** Trace /mnt/trace1 has no facility tracefile > > And aborts. > > I have several basic questions: > > > 1. Should all the files under control have data when invoked with > the above command line? Yes, at least a buffer header. > 2. Is the facilities_0 file the one the viewer is complaining about > in the above error message? If not what does the error message mean? > How do I generate the missing information? See response above. > 3. Is there some way to look at or analyze the data in processes_0 > and cpu_0 using the Viewer if some of the files are empty? No. facilities_0 contains the metainfo, which is necessary to map between event IDs and the event xml description. > 4. Is there a web site with documentation beyond that on LTTng > explaining the Viewer? Everything we have is available through lttng.org. But it may not match your old version though. > > We have also tried to run LTT using the non real-time version of Linux > (2.6.20 from TimeSys) with the same results. In this case the patch > (2.6.20-lttng-0.9.0.tar.bz2) applied cleanly and no manual editing was > required. The Viewer (0.8.79-02032007.tar.gz) was also rebuilt. > > Thanks in advance for any help and information. No problem. I hope this will be helpful to you. Thanks for the detailed problem report. Mathieu > > George Goetz > > _______________________________________________ > ltt-dev mailing list > [email protected] > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev -- Mathieu Desnoyers Operating System Efficiency Consultant EfficiOS Inc. http://www.efficios.com _______________________________________________ ltt-dev mailing list [email protected] http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
