------- Comment From tmri...@de.ibm.com 2019-07-22 05:04 EDT------- During execution of command 'perf top' the error message:
Not enough memory for annotating '__irf_end' symbol!) is emitted from this call sequence: __cmd_top perf_top__mmap_read perf_top__mmap_read_idx perf_event__process_sample hist_entry_iter__add hist_iter__top_callback perf_top__record_precise_ip hist_entry__inc_addr_samples symbol__inc_addr_samples symbol__get_annotation symbol__alloc_hist In this function the size of symbol __irf_end is calculated. The size of a symbol is the difference between its start and end address. When the symbol was read the first time, it was: symbol__new: __irf_end 0xe954d0-0xe954d0 which is correct and maps with /proc/kallsyms: root@s8360046:~/linux-4.15.0/tools/perf# fgrep _irf_end /proc/kallsyms 0000000000e954d0 t __irf_end root@s8360046:~/linux-4.15.0/tools/perf# In function symbol__alloc_hist() the end of symbol __irf_end is symbol__alloc_hist sym:__irf_end start:0xe954d0 end:0x3ff80045a8 which is identical with the first module entry in /proc/kallsyms This results in a symbol size of __irf_req for histogram analyses of 70334140059072 bytes and a malloc() for this requested size fails. The root cause of this is function __dso__load_kallsyms() +-> symbols__fixup_end() Function symbols__fixup_end() enlarges the last symbol in the kallsyms map # fgrep __irf_end /proc/kallsyms 0000000000e954d0 t __irf_end # to the start address of the first module: # cat /proc/kallsyms | sort | egrep ' [tT] ' .... 0000000000e952d0 T __security_initcall_end 0000000000e954d0 T __initramfs_size 0000000000e954d0 t __irf_end 000003ff800045a8 T fc_get_event_number [scsi_transport_fc] 000003ff800045d0 t store_fc_vport_disable [scsi_transport_fc] 000003ff800046a8 T scsi_is_fc_rport [scsi_transport_fc] 000003ff800046d0 t fc_target_setup [scsi_transport_fc] On s390 the kernel is located around memory address 0x200, 0x10000 or 0x100000, depending on linux version. Modules however start some- where around 0x3ff xxxx xxxx. This is different than x86 and produces a large gap for which histogram allocation fails. On x86 modules simply follow the kernel and the gap is minor, just some pages to adjust to a page or segment boundary. This kernel mapping is identical when I run # ./perf record -- true # ./perf report -D | fgrep MAP 0 0xe8 [0x50]: PERF_RECORD_MMAP -1/0: [0x200(0x3ff800043a8) @ 0x200]: x [kernel.kallsyms]_text where the kernel map is extremely large. I will post a patch to the linux kernel mailling list for discussion and fixing the issue today. When it is upstream we can backport it to 4.18 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1828166 Title: perf top problem on z with Ubuntu 18.04 Status in Ubuntu on IBM z Systems: Confirmed Status in linux package in Ubuntu: Confirmed Status in linux source package in Bionic: In Progress Status in linux source package in Cosmic: Invalid Bug description: SRU Justification: ================== [Impact] * The perf top tool hangs and shows error messages, like 'Not enough memory for annotating' [Fix] * edeb0c90df3581b821a764052d185df985f8b8dc edeb0c9 "perf tools: Stop fallbacking to kallsyms for vdso symbols lookup" [Test Case] * start a benchmark (mem_alloc, but it doesn't really matter what) * execute perf top in a second terminal * the output of perf top is correct * now stop the benchmark * and perf top shows an error message, like "Not enough memory for annotating '__irf_end' symbol!)" * and perf top can't be exited anymore [Regression Potential] * The regression potential can be considered as low since this happens only while using the perf top tool * and it is known that the commit (above) fixes the problem * and the fix is upstream since 4.19 [Other Info] * current disco and eoan kernels don't show that problem * bisecting result points to above commit * applies cleanly on cosmic, but has a little conflict on bionic (both master-next) _________________________ perf top hangs and shows error messages ---uname output--- Linux weather 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:32:27 UTC 2019 s390x s390x s390x GNU/Linux ---Steps to Reproduce--- I start a benchmark (mem_alloc, but it really doesn't matter) and then issue perf top in a second terminal, the output from perf top is correct. Now I stop the benchmark: perf top shows a error message (Not enough memory for annotating '__irf_end' symbol!) and I can't quit from perf top anymore Following analyse took place: No problem with current kernel . Bi-Secting of perf tool took place and following commit was found: commit edeb0c90df3581b821a764052d185df985f8b8dc (HEAD, refs/bisect/bad) Author: Arnaldo Carvalho de Melo <a...@redhat.com> Date: Tue Oct 16 17:08:29 2018 -0300 perf tools: Stop fallbacking to kallsyms for vdso symbols lookup When you apply this patch the issue is gone, however it is contained in these versions: git tag --contains edeb0c90df3581b821 v4.19 v4.20 .... The level I was debugging was kernel 4.15 which does not contain this patch. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1828166/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
