------- Comment From [email protected] 2019-07-15 09:59 EDT------- I added: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0c9108b083706330cd5484d121fbb0ad67e8f647
in addition to: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?h=next&id=7ccc4fe5ff9e3a134e863beed0dba18a5e511659 It ran a lot longer - hour instead of minutes, but then ended up with this: [18734.191331] perf: interrupt took too long (1054 > 1051), lowering kernel.perf_event_max_sample_rate to 7500 [18736.617855] perf: Dynamic interrupt throttling disabled, can hang your system! [18751.191062] perf: interrupt took too long (2317 > 1333), lowering kernel.perf_event_max_sample_rate to 3250 [18753.006339] perf: interrupt took too long (2218 > 1), lowering kernel.perf_event_max_sample_rate to 3500 [18754.156398] perf: Dynamic interrupt throttling disabled, can hang your system! [18775.067223] perf: interrupt took too long (2227 > 1), lowering kernel.perf_event_max_sample_rate to 3500 [18779.532549] perf: Dynamic interrupt throttling disabled, can hang your system! [18834.315583] perf: Dynamic interrupt throttling disabled, can hang your system! [18851.090933] Watchdog CPU:102 Hard LOCKUP [18851.090936] Modules linked in: kvm_hv kvm vmx_crypto crct10dif_vpmsum ast drm_kms_helper ttm ofpart cmdlinepart drm fb_sys_fops ipmi_powernv at24 syscopyarea ipmi_devintf powernv_flash sysfillrect ipmi_msghandler opal_prd mtd ibmpowernv sysimgblt i2c_algo_bit uio_pdrv_genirq uio sch_fq_codel ip_tables x_tables autofs4 mlx5_core ahci mlxfw crc32c_vpmsum tg3 libahci devlink [18851.090995] CPU: 102 PID: 0 Comm: swapper/102 Tainted: G L 4.15.0-54-generic #58 [18851.090997] NIP: c000000000100740 LR: c00000000010058c CTR: c0000000000fe770 [18851.091000] REGS: c000000007ad3d80 TRAP: 0900 Tainted: G L (4.15.0-54-generic) [18851.091001] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28002882 XER: 00000000 [18851.091014] CFAR: c00000000000deb8 SOFTE: 0 GPR00: c000000000100584 c00020397b423850 c00000000170b800 c000003f80925000 GPR04: 000000007fffffff c00020397b4238b0 c000203994486958 c0002039944868b8 GPR08: 0000000000000004 0000000000000000 0000000000000001 0000000000000000 GPR12: c0002039944868f8 c000000007a56200 c00020397b423f90 0000000000000000 GPR16: 0000000000000000 c00000000004ad60 c00000000004ad30 c0000000011d5380 GPR20: 0000000000000800 c000000001742494 0000000000000066 0000000000000001 GPR24: 0000000000000198 0000000080000000 0000000000000000 0000000000000006 GPR28: 0000000000000000 c0000000018d1808 0000000006004010 c0002039944868a0 [18851.091058] NIP [c000000000100740] power_pmu_enable+0x4f0/0x600 [18851.091060] LR [c00000000010058c] power_pmu_enable+0x33c/0x600 [18851.091061] Call Trace: [18851.091065] [c00020397b423850] [c000000000100584] power_pmu_enable+0x334/0x600 (unreliable) [18851.091071] [c00020397b423930] [c0000000002c9dbc] ctx_resched+0xec/0x150 [18851.091075] [c00020397b423970] [c0000000002ca014] __perf_install_in_context+0x1f4/0x280 [18851.091079] [c00020397b4239c0] [c0000000002bf7d0] remote_function+0x40/0x90 [18851.091083] [c00020397b4239f0] [c0000000001db9dc] flush_smp_call_function_queue+0xac/0x1d0 [18851.091087] [c00020397b423a70] [c00000000004bf7c] smp_ipi_demux_relaxed+0x9c/0x110 [18851.091092] [c00020397b423ab0] [c000000000047948] doorbell_exception+0xa8/0xe0 [18851.091096] [c00020397b423ae0] [c000000000009ad4] h_doorbell_common+0x114/0x120 [18851.091102] --- interrupt: e81 at replay_interrupt_return+0x0/0x4 LR = arch_local_irq_restore+0x74/0x90 [18851.091106] [c00020397b423dd0] [0000004000000000] 0x4000000000 (unreliable) [18851.091111] [c00020397b423df0] [c000000000acea80] cpuidle_enter_state+0xf0/0x450 [18851.091116] [c00020397b423e50] [c00000000017852c] call_cpuidle+0x4c/0x90 [18851.091119] [c00020397b423e70] [c000000000178940] do_idle+0x2b0/0x330 [18851.091122] [c00020397b423ec0] [c000000000178bf8] cpu_startup_entry+0x38/0x40 [18851.091125] [c00020397b423ef0] [c00000000004d280] start_secondary+0x4f0/0x510 [18851.091129] [c00020397b423f90] [c00000000000ab6c] start_secondary_prolog+0x10/0x14 [18851.091131] Instruction dump: [18851.091135] 7fa9f000 419d001c 7d29c850 7d244b78 f93801b8 4bfffe44 60000000 60000000 [18851.091147] 39200000 38800000 f93801b8 4bfffe2c <eae10098> 4bfffb98 60000000 60000000 [18854.550395] Watchdog CPU:102 became unstuck [18873.055963] perf: interrupt took too long (329 > 1), lowering kernel.perf_event_max_sample_rate to 48500 [18873.965492] perf: interrupt took too long (420 > 411), lowering kernel.perf_event_max_sample_rate to 38000 <snip> [18898.926809] perf: Dynamic interrupt throttling disabled, can hang your system! [18933.376330] perf: interrupt took too long (1534 > 1481), lowering kernel.perf_event_max_sample_rate to 5000 [18942.748893] perf: interrupt took too long (1589 > 1), lowering kernel.perf_event_max_sample_rate to 5000 [18961.230616] perf: interrupt took too long (1190 > 1188), lowering kernel.perf_event_max_sample_rate to 13250 [18961.241736] perf: interrupt took too long (1495 > 1487), lowering kernel.perf_event_max_sample_rate to 10500 [18961.281641] perf: interrupt took too long (2010 > 1868), lowering kernel.perf_event_max_sample_rate to 7750 [18967.384573] perf: Dynamic interrupt throttling disabled, can hang your system! [18982.547548] perf: interrupt took too long (2517 > 1290), lowering kernel.perf_event_max_sample_rate to 3000 [19000.662645] perf: interrupt took too long (1378 > 1), lowering kernel.perf_event_max_sample_rate to 5750 [19003.153641] perf: interrupt took too long (1576 > 1), lowering kernel.perf_event_max_sample_rate to 5000 [19020.801440] perf: interrupt took too long (2984 > 2000), lowering kernel.perf_event_max_sample_rate to 2500 [19021.901595] perf: interrupt took too long (3748 > 3730), lowering kernel.perf_event_max_sample_rate to 2000 [19023.757417] perf: Dynamic interrupt throttling disabled, can hang your system! Anju - we missing something else? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1752002 Title: [P9,POwer NV][WSP][DD2.1][Ubuntu 1804][Perf fuzzer] : Call trace is seen while running perf fuzzer (perf:) Status in The Ubuntu-power-systems project: Incomplete Status in linux package in Ubuntu: In Progress Status in linux source package in Bionic: In Progress Bug description: == Comment: #0 - Shriya R. Kulkarni <[email protected]> - 2018-02-02 01:21:36 == Problem Description : ============= Warn on message is seen while running perf fuzzer tests. Machine details : ========== Hardware : Witherspoon (wsp12) + DD2.1 OS : Ubuntu 1804 uname -a : 4.13.0-32-generic #35~lp1746225 ( Kernel from the bug : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 ) Steps to reproduce : ============ Build Kernel : -------------------- To avoid the kernel crash due to Perf fuzzer , use the kernel mentioned in the link : https://bugzilla.linux.ibm.com/show_bug.cgi?id=164107#c7 #! /bin/bash set -x git clone https://github.com/deater/perf_event_tests.git cd perf_event_tests/include mkdir asm cd asm wget http://9.114.13.132/repo/shriya/perf_regs.h cd ../../lib make sleep 10 cd ../fuzzer make sleep 10 echo 0 > /proc/sys/kernel/nmi_watchdog echo 2 > /proc/sys/kernel/perf_event_paranoid echo 100000 > /proc/sys/kernel/perf_event_max_sample_rate ./perf_fuzzer -r 1492143527 Call trace : ======= [ 329.228031] ------------[ cut here ]------------ [ 329.228039] WARNING: CPU: 43 PID: 9088 at /home/jsalisbury/bugs/lp1746225/ubuntu-artful/kernel/events/core.c:3038 perf_pmu_sched_task+0x170/0x180 [ 329.228040] Modules linked in: ofpart at24 uio_pdrv_genirq uio cmdlinepart powernv_flash mtd ipmi_powernv vmx_crypto ipmi_devintf ipmi_msghandler ibmpowernv opal_prd crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 crc32c_vpmsum lpfc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm tg3 nvmet_fc drm ahci nvmet nvme_fc libahci nvme_fabrics mlxfw nvme_core devlink scsi_transport_fc [ 329.228068] CPU: 43 PID: 9088 Comm: perf_fuzzer Not tainted 4.13.0-32-generic #35~lp1746225 [ 329.228070] task: c000003f776ac900 task.stack: c000003f77728000 [ 329.228071] NIP: c000000000299b70 LR: c0000000002a4534 CTR: c00000000029bb80 [ 329.228073] REGS: c000003f7772b760 TRAP: 0700 Not tainted (4.13.0-32-generic) [ 329.228073] MSR: 900000000282b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> [ 329.228079] CR: 24008822 XER: 00000000 [ 329.228080] CFAR: c000000000299a70 SOFTE: 0 GPR00: c0000000002a4534 c000003f7772b9e0 c000000001606200 c000003fef858908 GPR04: c000003f776ac900 0000000000000001 ffffffffffffffff 0000003fee730000 GPR08: 0000000000000000 0000000000000000 c0000000011220d8 0000000000000002 GPR12: c00000000029bb80 c000000007a3d900 0000000000000000 0000000000000000 GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000000 0000000000000000 c000003f776ad090 c000000000c71354 GPR24: c000003fef716780 0000003fee730000 c000003fe69d4200 c000003f776ad330 GPR28: c0000000011220d8 0000000000000001 c0000000014c6108 c000003fef858900 [ 329.228098] NIP [c000000000299b70] perf_pmu_sched_task+0x170/0x180 [ 329.228100] LR [c0000000002a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228101] Call Trace: [ 329.228102] [c000003f7772b9e0] [c0000000002a0678] perf_iterate_sb+0x158/0x2a0 (unreliable) [ 329.228105] [c000003f7772ba30] [c0000000002a4534] __perf_event_task_sched_in+0xc4/0x230 [ 329.228107] [c000003f7772bab0] [c0000000001396dc] finish_task_switch+0x21c/0x310 [ 329.228109] [c000003f7772bb60] [c000000000c71354] __schedule+0x304/0xb80 [ 329.228111] [c000003f7772bc40] [c000000000c71c10] schedule+0x40/0xc0 [ 329.228113] [c000003f7772bc60] [c0000000001033f4] do_wait+0x254/0x2e0 [ 329.228115] [c000003f7772bcd0] [c000000000104ac0] kernel_wait4+0xa0/0x1a0 [ 329.228117] [c000003f7772bd70] [c000000000104c24] SyS_wait4+0x64/0xc0 [ 329.228121] [c000003f7772be30] [c00000000000b184] system_call+0x58/0x6c [ 329.228121] Instruction dump: [ 329.228123] 3beafea0 7faa4800 409eff18 e8010060 eb610028 ebc10040 7c0803a6 38210050 [ 329.228127] eb81ffe0 eba1ffe8 ebe1fff8 4e800020 <0fe00000> 4bffffbc 60000000 60420000 [ 329.228131] ---[ end trace 8c46856d314c1811 ]--- [ 375.755943] hrtimer: interrupt took 31601 ns == Comment: #4 - SEETEENA THOUFEEK <[email protected]> - 2018-02-05 06:34:09 == == Comment: #5 - SEETEENA THOUFEEK <[email protected]> - 2018-02-05 06:36:12 == We have similar issue reported on different distro where Anju Provided the patch. Patch attached above. . Will check with her if that patch got accepted upstream == Comment: #14 - SEETEENA THOUFEEK <[email protected]> - 2018-02-23 01:52:50 == To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1752002/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
