Happy to hear that. ///jon
> -----Original Message----- > From: [email protected] <[email protected]> > Sent: 2-Nov-18 05:05 > To: Jon Maloy <[email protected]> > Cc: [email protected]; [email protected] > Subject: Re: [tipc-discussion] TIPC scalability viewpoints > > Hello Jon, > > I tested this with 4.18.10 and it passes the fine-tuned test I have. > I'll run it for a while to be sure, but looks good. > > Thanks, > -- > Juhamatti > to 1. marrask. 2018 klo 2.15 Jon Maloy ([email protected]) kirjoitti: > > > > Hi Juhamatti, > > The second is probably the one that was fixed by commit d3092b2efca1c > ("tipc: fix unsafe rcu locking when accessing publication list") which was > delivered to 4.18 around a month ago. Not sure how far back this has been > applied, but if you look at the very latest 4.15 upstream you may be lucky. > > > > Unfortunately I don't recognize the first one, -it seems to be a timer > interrupt deleting a subscription object that is still locked while sending an > event. What I can say, is that the whole topology server was redesigned in > Linux 4.17, among other things because this kind of problems never seemed > to end. > > > > Do you have any possibility to upgrade your kernel to 4.19 and try again? > There has been a few bug fixes added during 4.18/4.19 that you may benefit > from. Otherwise I may have to look closer at this one, and see what can be > done. > > > > Regards > > ///jon > > > > > > > -----Original Message----- > > > From: [email protected] <[email protected]> > > > Sent: 31-Oct-18 10:23 > > > To: [email protected] > > > Cc: [email protected] > > > Subject: Re: [tipc-discussion] TIPC scalability viewpoints > > > > > > Hello again, > > > > > > Here is another issue found in the same setup, looks a bit different. > > > Please see below. > > > > > > BR, > > > -- > > > Juhamatti > > > > > > [ 6276.191737] Unable to remove publication from failed node [ > > > 6276.191737] (type=18888, lower=43818, node=0x100136c, > > > ref=936542876, > > > key=936542877) [ 6276.205086] BUG: unable to handle kernel NULL > > > pointer dereference at 0000000000000088 [ 6276.213016] IP: > > > __call_rcu.constprop.69+0x25/0x210 > > > [ 6276.217837] PGD 0 P4D 0 > > > [ 6276.220405] Oops: 0002 [#1] SMP NOPTI [ 6276.224103] Modules linked > in: > > > tipc ip6_udp_tunnel udp_tunnel xt_CHECKSUM iptable_mangle > > > ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat > nf_conntrack_ipv4 > > > nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp > > > llc veth iptable_filter ipmi_ssif > > > nls_iso8859_1 intel_rapl pnd2_edac x86_pkg_temp_thermal > > > intel_powerclamp coretemp kvm_intel asix kvm usbnet input_leds > > > irqbypass joydev intel_cstate mii intel_rapl_perf mac_hid ipmi_si > > > ipmi_devintf shpchp ipmi_msghandler sch_fq_codel ib_iser rdma_cm > > > iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi > > > scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress > > > raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor > > > async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear > > > hid_generic usbhid hid crct10dif_pclmul crc32_pclmul [ 6276.295019] > > > ghash_clmulni_intel pcbc ast ttm aesni_intel drm_kms_helper > > > aes_x86_64 syscopyarea crypto_simd glue_helper sysfillrect sysimgblt > > > cryptd ixgbe fb_sys_fops igb i40e drm ahci dca i2c_algo_bit mdio > > > libahci ptp i2c_ismt pps_core [ 6276.316114] CPU: 15 PID: 100 Comm: > > > ksoftirqd/15 Not tainted 4.15.0-38-generic #41-Ubuntu [ 6276.324259] > Hardware name: Supermicro Super Server/A2SDV-16C-TLN5F, BIOS 1.0b > 12/12/2017 [ 6276.332413] RIP: > > > 0010:__call_rcu.constprop.69+0x25/0x210 > > > [ 6276.337773] RSP: 0018:ffffb215c34cbbd0 EFLAGS: 00010246 [ > > > 6276.343033] > > > RAX: 0000000000000076 RBX: ffff9a5046cdd110 RCX: 0000000000000001 [ > > > 6276.350213] RDX: ffffffffbe4a9200 RSI: 0000000000000080 RDI: > > > 0000000000000000 [ 6276.357402] RBP: ffffb215c34cbc08 R08: > > > 0000000000000000 R09: 0000000000001550 [ 6276.364595] R10: > > > 00000000000012df R11: 00000000ffffffff R12: ffff9a4fe3bbfc80 [ > > > 6276.371778] > > > R13: ffff9a5221a54740 R14: ffffffffbe4a9200 R15: 0000000000000080 [ > > > 6276.378963] FS: 0000000000000000(0000) GS:ffff9a561f5c0000(0000) > > > knlGS:0000000000000000 > > > [ 6276.387096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ > > > 6276.392878] CR2: 0000000000000088 CR3: 00000006f4818000 CR4: > > > 00000000003406e0 [ 6276.400057] Call Trace: > > > [ 6276.402544] kfree_call_rcu+0x1a/0x20 [ 6276.406256] > > > tipc_publ_notify+0xae/0x190 [tipc] [ 6276.410840] > > > tipc_node_write_unlock+0xf3/0x100 [tipc] [ 6276.415979] > > > tipc_node_link_down+0x99/0x140 [tipc] [ 6276.420858] > > > tipc_node_timeout+0x126/0x130 [tipc] [ 6276.425644] ? > > > tipc_node_calculate_timer.isra.13+0x60/0x60 [tipc] [ 6276.431795] > > > call_timer_fn+0x30/0x130 [ 6276.435487] > > > run_timer_softirq+0x3fb/0x450 [ 6276.439614] ? > __switch_to_asm+0x34/0x70 [ 6276.443573] ? > > > __switch_to+0xad/0x500 [ 6276.447262] ? __switch_to_asm+0x40/0x70 [ > > > 6276.451219] ? __switch_to_asm+0x34/0x70 [ 6276.455178] ? > > > __switch_to_asm+0x40/0x70 [ 6276.459141] __do_softirq+0xe4/0x2bb [ > > > 6276.462750] run_ksoftirqd+0x22/0x60 [ 6276.466364] > > > smpboot_thread_fn+0xfc/0x170 [ 6276.470397] kthread+0x121/0x140 [ > > > 6276.473645] ? sort_range+0x30/0x30 [ 6276.477153] ? > > > kthread_create_worker_on_cpu+0x70/0x70 > > > [ 6276.482237] ret_from_fork+0x35/0x40 [ 6276.485835] Code: b7 f8 > > > ff 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 41 > > > 57 41 56 41 55 41 54 49 89 ff 53 49 89 d6 48 83 ec 10 83 e7 07 0f 85 > > > dd 00 00 00 <49> 89 77 08 49 c7 07 00 00 00 00 9c 58 0f 1f 44 00 00 > > > 49 > > > 89 c5 > > > [ 6276.504818] RIP: __call_rcu.constprop.69+0x25/0x210 RSP: > > > ffffb215c34cbbd0 [ 6276.511642] CR2: 0000000000000088 [ 6276.515000] > > > ---[ end trace f418a9afa145bb5d ]--- [ 6276.528130] Kernel panic - not > syncing: > > > Fatal exception in interrupt [ 6276.534610] Kernel Offset: > > > 0x3c000000 from > > > 0xffffffff81000000 (relocation range: > > > 0xffffffff80000000-0xffffffffbfffffff) > > > [ 6276.553092] ---[ end Kernel panic - not syncing: Fatal exception > > > in interrupt [ 6276.560285] WARNING: CPU: 15 PID: 100 at > > > /build/linux-CuEUJv/linux-4.15.0/kernel/sched/core.c:1192 > > > set_task_cpu+0x158/0x160 > > > [ 6276.571459] Modules linked in: tipc ip6_udp_tunnel udp_tunnel > > > xt_CHECKSUM iptable_mangle ipt_MASQUERADE > nf_nat_masquerade_ipv4 > > > iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat > > > nf_conntrack xt_tcpudp bridge stp llc veth iptable_filter ipmi_ssif > > > nls_iso8859_1 intel_rapl pnd2_edac x86_pkg_temp_thermal > > > intel_powerclamp coretemp kvm_intel asix kvm usbnet input_leds > > > irqbypass joydev intel_cstate mii intel_rapl_perf mac_hid ipmi_si > > > ipmi_devintf shpchp ipmi_msghandler sch_fq_codel ib_iser rdma_cm > > > iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi > > > scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress > > > raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor > > > async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear > > > hid_generic usbhid hid crct10dif_pclmul crc32_pclmul [ 6276.642267] > > > ghash_clmulni_intel pcbc ast ttm aesni_intel drm_kms_helper > > > aes_x86_64 syscopyarea crypto_simd glue_helper sysfillrect sysimgblt > > > cryptd ixgbe fb_sys_fops igb i40e drm ahci dca i2c_algo_bit mdio libahci > ptp i2c_ismt pps_core > > > [ 6276.663306] CPU: 15 PID: 100 Comm: ksoftirqd/15 Tainted: G D > > > 4.15.0-38-generic #41-Ubuntu > > > [ 6276.672734] Hardware name: Supermicro Super > > > Server/A2SDV-16C-TLN5F, BIOS 1.0b 12/12/2017 [ 6276.683335] RIP: > > > 0010:set_task_cpu+0x158/0x160 [ 6276.690288] RSP: > > > 0018:ffff9a561f5c3dc8 EFLAGS: 00010006 [ 6276.698025] > > > RAX: 0000000000000200 RBX: ffff9a50a7565900 RCX: 0000000000000000 [ > > > 6276.707673] RDX: 0000000000000000 RSI: 0000000000000000 RDI: > > > ffff9a50a7565900 [ 6276.717305] RBP: ffff9a561f5c3de8 R08: > > > 0000000000000000 R09: 000000000000ffff [ 6276.726926] R10: > > > ffff9a561f5c3c90 > > > R11: 0000000000000000 R12: ffff9a50a756642c [ 6276.736527] R13: > > > 0000000000000000 R14: 0000000000000000 R15: 0000000000022880 [ > > > 6276.746122] FS: 0000000000000000(0000) GS:ffff9a561f5c0000(0000) > > > knlGS:0000000000000000 > > > [ 6276.756700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ > > > 6276.764891] CR2: 0000000000000088 CR3: 00000006f4818000 CR4: > > > 00000000003406e0 [ 6276.774418] Call Trace: > > > [ 6276.779188] <IRQ> > > > [ 6276.783446] try_to_wake_up+0x159/0x480 [ 6276.789458] > > > default_wake_function+0x12/0x20 [ 6276.795834] > > > autoremove_wake_function+0x12/0x40 > > > [ 6276.802444] __wake_up_common+0x73/0x130 [ 6276.808426] > > > __wake_up_common_lock+0x80/0xc0 [ 6276.814730] > > > __wake_up+0x13/0x20 [ 6276.819987] > > > wake_up_klogd_work_func+0x40/0x60 [ 6276.826444] > > > irq_work_run_list+0x52/0x80 [ 6276.832376] irq_work_run+0x2c/0x40 [ > > > 6276.837876] flush_smp_call_function_queue+0x66/0xf0 > > > [ 6276.844845] generic_smp_call_function_single_interrupt+0x13/0x30 > > > [ 6276.852952] smp_call_function_interrupt+0x3e/0xd0 > > > [ 6276.859743] call_function_interrupt+0x84/0x90 [ 6276.866171] > > > </IRQ> [ 6276.870234] RIP: 0010:panic+0x1fe/0x244 [ 6276.876021] RSP: > > > 0018:ffffb215c34cb918 EFLAGS: 00000286 ORIG_RAX: > > > ffffffffffffff03 > > > [ 6276.885572] RAX: 0000000000000041 RBX: 0000000000000000 RCX: > > > 0000000000000006 [ 6276.894699] RDX: 0000000000000000 RSI: > > > 0000000000000092 RDI: ffff9a561f5d6490 [ 6276.903823] RBP: > > > ffffb215c34cb990 R08: 0000000000000000 R09: 000000000000157e [ > > > 6276.912957] R10: 0000000000000008 R11: 00000000ffffffff R12: > > > 0000000000000000 [ 6276.922071] R13: 0000000000000000 R14: > > > 0000000000000009 R15: 0000000000000002 [ 6276.931172] ? > > > panic+0x1f7/0x244 [ 6276.936358] oops_end+0xb6/0xd0 [ 6276.941463] > > > no_context+0x17c/0x440 [ 6276.946934] ? __const_udelay+0x41/0x50 [ > > > 6276.952753] __bad_area_nosemaphore+0xf3/0x1b0 [ 6276.959186] > > > bad_area_nosemaphore+0x14/0x20 [ 6276.965378] > > > __do_page_fault+0xd4/0x4d0 [ 6276.971226] ? up+0x32/0x50 [ > > > 6276.976035] > > > do_page_fault+0x2e/0xe0 [ 6276.981632] page_fault+0x25/0x50 [ > > > 6276.986954] RIP: 0010:__call_rcu.constprop.69+0x25/0x210 > > > [ 6276.994290] RSP: 0018:ffffb215c34cbbd0 EFLAGS: 00010246 [ > > > 6277.001543] > > > RAX: 0000000000000076 RBX: ffff9a5046cdd110 RCX: 0000000000000001 [ > > > 6277.010723] RDX: ffffffffbe4a9200 RSI: 0000000000000080 RDI: > > > 0000000000000000 [ 6277.019901] RBP: ffffb215c34cbc08 R08: > > > 0000000000000000 R09: 0000000000001550 [ 6277.029015] R10: > > > 00000000000012df R11: 00000000ffffffff R12: ffff9a4fe3bbfc80 [ > > > 6277.038070] > > > R13: ffff9a5221a54740 R14: ffffffffbe4a9200 R15: 0000000000000080 [ > > > 6277.047129] kfree_call_rcu+0x1a/0x20 [ 6277.052708] > > > tipc_publ_notify+0xae/0x190 [tipc] [ 6277.059161] > > > tipc_node_write_unlock+0xf3/0x100 [tipc] [ 6277.066123] > > > tipc_node_link_down+0x99/0x140 [tipc] [ 6277.072792] > > > tipc_node_timeout+0x126/0x130 [tipc] [ 6277.079348] ? > > > tipc_node_calculate_timer.isra.13+0x60/0x60 [tipc] [ 6277.087259] > > > call_timer_fn+0x30/0x130 [ 6277.092670] > > > run_timer_softirq+0x3fb/0x450 [ 6277.098438] ? > __switch_to_asm+0x34/0x70 [ 6277.103961] ? > > > __switch_to+0xad/0x500 [ 6277.109137] ? __switch_to_asm+0x40/0x70 [ > > > 6277.114493] ? __switch_to_asm+0x34/0x70 [ 6277.119765] ? > > > __switch_to_asm+0x40/0x70 [ 6277.125006] __do_softirq+0xe4/0x2bb [ > > > 6277.129861] run_ksoftirqd+0x22/0x60 [ 6277.134699] > > > smpboot_thread_fn+0xfc/0x170 [ 6277.139965] kthread+0x121/0x140 [ > > > 6277.144439] ? sort_range+0x30/0x30 [ 6277.149163] ? > > > kthread_create_worker_on_cpu+0x70/0x70 > > > [ 6277.155452] ret_from_fork+0x35/0x40 [ 6277.160258] Code: 32 ff > > > ff ff 80 8b 64 08 00 00 04 e9 4e ff ff ff 0f 0b e9 f2 fe ff ff f7 43 > > > 5c fd ff ff ff 0f 84 fc fe ff ff 0f 0b e9 > > > f5 fe ff ff <0f> 0b e9 ff fe ff ff 90 0f 1f 44 00 00 55 48 89 e5 41 > > > 56 > > > 41 55 > > > [ 6277.181837] ---[ end trace f418a9afa145bb5e ]--- [ 6277.187806] > > > ------------[ cut here ]------------ [ 6277.193765] sched: > > > Unexpected reschedule of offline CPU#0! > > > [ 6277.200584] WARNING: CPU: 15 PID: 100 at > > > /build/linux-CuEUJv/linux-4.15.0/arch/x86/kernel/smp.c:128 > > > native_smp_send_reschedule+0x3a/0x40 > > > [ 6277.214247] Modules linked in: tipc ip6_udp_tunnel udp_tunnel > > > xt_CHECKSUM iptable_mangle ipt_MASQUERADE > nf_nat_masquerade_ipv4 > > > iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat > > > nf_conntrack xt_tcpudp bridge stp llc veth iptable_filter ipmi_ssif > > > nls_iso8859_1 intel_rapl pnd2_edac x86_pkg_temp_thermal > > > intel_powerclamp coretemp kvm_intel asix kvm usbnet input_leds > > > irqbypass joydev intel_cstate mii intel_rapl_perf mac_hid ipmi_si > > > ipmi_devintf shpchp ipmi_msghandler sch_fq_codel ib_iser rdma_cm > > > iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi > > > scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress > > > raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor > > > async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear > > > hid_generic usbhid hid crct10dif_pclmul crc32_pclmul [ 6277.291150] > > > ghash_clmulni_intel pcbc ast ttm aesni_intel drm_kms_helper > > > aes_x86_64 syscopyarea crypto_simd glue_helper sysfillrect sysimgblt > > > cryptd ixgbe fb_sys_fops igb i40e drm ahci dca i2c_algo_bit mdio libahci > ptp i2c_ismt pps_core > > > [ 6277.315672] CPU: 15 PID: 100 Comm: ksoftirqd/15 Tainted: G D W > > > 4.15.0-38-generic #41-Ubuntu > > > [ 6277.326902] Hardware name: Supermicro Super > > > Server/A2SDV-16C-TLN5F, BIOS 1.0b 12/12/2017 [ 6277.336859] RIP: > > > 0010:native_smp_send_reschedule+0x3a/0x40 > > > [ 6277.344224] RSP: 0018:ffff9a561f5c3de8 EFLAGS: 00010086 [ > > > 6277.351330] > > > RAX: 0000000000000000 RBX: ffff9a50a7565900 RCX: 0000000000000006 [ > > > 6277.360358] RDX: 0000000000000007 RSI: 0000000000000092 RDI: > > > ffff9a561f5d6490 [ 6277.369378] RBP: ffff9a561f5c3de8 R08: > > > 0000000000000000 R09: 00000000000015cd [ 6277.378405] R10: > > > ffff9a561f5c3dc8 R11: 00000000ffffffff R12: ffff9a50a756642c [ > > > 6277.387428] > > > R13: 0000000000000004 R14: 0000000000000046 R15: 0000000000022880 [ > > > 6277.396489] FS: 0000000000000000(0000) GS:ffff9a561f5c0000(0000) > > > knlGS:0000000000000000 > > > [ 6277.406505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ > > > 6277.414171] CR2: 0000000000000088 CR3: 00000006f4818000 CR4: > > > 00000000003406e0 [ 6277.423241] Call Trace: > > > [ 6277.427619] <IRQ> > > > [ 6277.431550] try_to_wake_up+0x3c6/0x480 [ 6277.437319] > > > default_wake_function+0x12/0x20 [ 6277.443503] > > > autoremove_wake_function+0x12/0x40 > > > [ 6277.449960] __wake_up_common+0x73/0x130 [ 6277.455811] > > > __wake_up_common_lock+0x80/0xc0 [ 6277.462004] > > > __wake_up+0x13/0x20 [ 6277.467146] > > > wake_up_klogd_work_func+0x40/0x60 [ 6277.473510] > > > irq_work_run_list+0x52/0x80 [ 6277.479353] irq_work_run+0x2c/0x40 [ > > > 6277.484753] flush_smp_call_function_queue+0x66/0xf0 > > > [ 6277.491642] generic_smp_call_function_single_interrupt+0x13/0x30 > > > [ 6277.499674] smp_call_function_interrupt+0x3e/0xd0 > > > [ 6277.506405] call_function_interrupt+0x84/0x90 [ 6277.512787] > > > </IRQ> [ 6277.516805] RIP: 0010:panic+0x1fe/0x244 [ 6277.522558] RSP: > > > 0018:ffffb215c34cb918 EFLAGS: 00000286 ORIG_RAX: > > > ffffffffffffff03 > > > [ 6277.532076] RAX: 0000000000000041 RBX: 0000000000000000 RCX: > > > 0000000000000006 [ 6277.541160] RDX: 0000000000000000 RSI: > > > 0000000000000092 RDI: ffff9a561f5d6490 [ 6277.550232] RBP: > > > ffffb215c34cb990 R08: 0000000000000000 R09: 000000000000157e [ > > > 6277.559296] R10: 0000000000000008 R11: 00000000ffffffff R12: > > > 0000000000000000 [ 6277.568356] R13: 0000000000000000 R14: > > > 0000000000000009 R15: 0000000000000002 [ 6277.577422] ? > > > panic+0x1f7/0x244 [ 6277.582578] oops_end+0xb6/0xd0 [ 6277.587644] > > > no_context+0x17c/0x440 [ 6277.593041] ? __const_udelay+0x41/0x50 [ > > > 6277.598779] __bad_area_nosemaphore+0xf3/0x1b0 [ 6277.605124] > > > bad_area_nosemaphore+0x14/0x20 [ 6277.611212] > > > __do_page_fault+0xd4/0x4d0 [ 6277.616958] ? up+0x32/0x50 [ > > > 6277.621667] > > > do_page_fault+0x2e/0xe0 [ 6277.627168] page_fault+0x25/0x50 [ > > > 6277.632417] RIP: 0010:__call_rcu.constprop.69+0x25/0x210 > > > [ 6277.639666] RSP: 0018:ffffb215c34cbbd0 EFLAGS: 00010246 [ > > > 6277.646845] > > > RAX: 0000000000000076 RBX: ffff9a5046cdd110 RCX: 0000000000000001 [ > > > 6277.655951] RDX: ffffffffbe4a9200 RSI: 0000000000000080 RDI: > > > 0000000000000000 [ 6277.665065] RBP: ffffb215c34cbc08 R08: > > > 0000000000000000 R09: 0000000000001550 [ 6277.674176] R10: > > > 00000000000012df R11: 00000000ffffffff R12: ffff9a4fe3bbfc80 [ > > > 6277.683298] > > > R13: ffff9a5221a54740 R14: ffffffffbe4a9200 R15: 0000000000000080 [ > > > 6277.692426] kfree_call_rcu+0x1a/0x20 [ 6277.698025] > > > tipc_publ_notify+0xae/0x190 [tipc] [ 6277.704432] > > > tipc_node_write_unlock+0xf3/0x100 [tipc] [ 6277.711361] > > > tipc_node_link_down+0x99/0x140 [tipc] [ 6277.718027] > > > tipc_node_timeout+0x126/0x130 [tipc] [ 6277.724591] ? > > > tipc_node_calculate_timer.isra.13+0x60/0x60 [tipc] [ 6277.732516] > > > call_timer_fn+0x30/0x130 [ 6277.737940] > > > run_timer_softirq+0x3fb/0x450 [ 6277.743701] ? > __switch_to_asm+0x34/0x70 [ 6277.749216] ? > > > __switch_to+0xad/0x500 [ 6277.754386] ? __switch_to_asm+0x40/0x70 [ > > > 6277.759732] ? __switch_to_asm+0x34/0x70 [ 6277.764993] ? > > > __switch_to_asm+0x40/0x70 [ 6277.770220] __do_softirq+0xe4/0x2bb [ > > > 6277.775071] run_ksoftirqd+0x22/0x60 [ 6277.779899] > > > smpboot_thread_fn+0xfc/0x170 [ 6277.785156] kthread+0x121/0x140 [ > > > 6277.789619] ? sort_range+0x30/0x30 [ 6277.794333] ? > > > kthread_create_worker_on_cpu+0x70/0x70 > > > [ 6277.800617] ret_from_fork+0x35/0x40 [ 6277.805419] Code: 2d 61 > > > 60 01 73 17 48 8b 05 84 21 16 01 be fd 00 > > > 00 00 48 8b 40 30 e8 36 b6 ba 00 5d c3 89 fe 48 c7 c7 60 fe 0b be e8 > > > 06 48 03 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 83 > > > ec > > > 20 65 > > > [ 6277.826959] ---[ end trace f418a9afa145bb5f ]--- ke 31. lokak. > > > 2018 klo 14.39 [email protected] kirjoitti: > > > > > > > > Hello, > > > > > > > > Thanks Ying! While running a server with 250 LXCs and 2500 TIPC > > > > sockets running with modified connection demo, I am seeing the > > > > problem described below. The kernel is stock Ubuntu1804 > > > > 4.15.0-38-generic with > > > > 16 cores. It is fairly easy to reproduce, after starting up > > > > containers it will happen within an hour or so. > > > > > > > > Please let me know if there is already a patch for it, I am happy > > > > to try. I can also try fix candidates quite easily. > > > > > > > > BR, > > > > -- > > > > Juhamatti > > > > > > > > [ 3195.231191] BUG: unable to handle kernel paging request at > > > > 0000000200001dc7 [ 3195.238220] IP: _raw_spin_lock_bh+0x1b/0x30 [ > > > > 3195.242432] PGD 0 P4D 0 [ 3195.244988] Oops: 0002 [#1] SMP NOPTI > > > > [ 3195.248668] Modules linked in: binfmt_misc tipc ip6_udp_tunnel > > > > udp_tunnel xt_CHECKSUM iptable_mangle ipt_MASQUERADE > > > > nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 > > > > nf_defrag_ipv4 > > > > nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc veth > > > > iptable_filter ipmi_ssif nls_iso8859_1 intel_rapl pnd2_edac > > > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm > > > irqbypass > > > > asix intel_cstate usbnet intel_rapl_perf joydev input_leds mii > > > > ipmi_si ipmi_devintf ipmi_msghandler mac_hid shpchp sch_fq_codel > > > > ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp > > > > libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs > > > > zstd_compress > > > > raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor > > > > async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear > > > > hid_generic usbhid hid crct10dif_pclmul [ 3195.319457] > > > > crc32_pclmul ghash_clmulni_intel pcbc ast aesni_intel ttm > > > > aes_x86_64 drm_kms_helper crypto_simd glue_helper cryptd > > > > syscopyarea sysfillrect sysimgblt ixgbe igb fb_sys_fops i40e drm > > > > ahci dca i2c_algo_bit libahci mdio i2c_ismt ptp pps_core [ > > > > 3195.341632] CPU: 1 PID: 348 Comm: kworker/u32:7 Not tainted 4.15.0- > 38-generic #41-Ubuntu [ 3195.349754] Hardware name: > > > > Supermicro Super Server/A2SDV-16C-TLN5F, BIOS 1.0b 12/12/2017 [ > > > > 3195.357891] Workqueue: tipc_send tipc_send_work [tipc] [ > > > > 3195.363061] > > > > RIP: 0010:_raw_spin_lock_bh+0x1b/0x30 [ 3195.367788] RSP: > > > > 0018:ffff90d3df243e90 EFLAGS: 00010246 [ 3195.373032] RAX: > > > > 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08 [ > > > > 3195.380203] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI: > > > > 0000000200001dc7 [ 3195.387393] RBP: ffff90d3df243e90 R08: > > > > ffff90d3df243f10 R09: ffff90d3df25a780 [ 3195.394561] R10: > > > > ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7 [ > > > > 3195.401729] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15: > > > > ffff90d1317a6c18 [ 3195.408897] FS: 0000000000000000(0000) > > > > GS:ffff90d3df240000(0000) > > > > knlGS:0000000000000000 > > > > [ 3195.417025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ > > > > 3195.422798] CR2: 0000000200001dc7 CR3: 00000007711f8000 CR4: > > > > 00000000003406e0 [ 3195.429965] Call Trace: > > > > [ 3195.432429] <IRQ> > > > > [ 3195.434467] tipc_subscrp_timeout+0x25/0x90 [tipc] [ > > > > 3195.439311] > > > > call_timer_fn+0x30/0x130 [ 3195.442994] > > > > run_timer_softirq+0x3fb/0x450 [ 3195.447113] ? > lapic_next_deadline+0x26/0x30 [ 3195.451404] ? > > > > clockevents_program_event+0xd8/0x110 > > > > [ 3195.456304] __do_softirq+0xe4/0x2bb [ 3195.459903] > > > > do_softirq_own_stack+0x2a/0x40 [ 3195.464107] </IRQ> [ > > > > 3195.466219] > > > > do_softirq.part.17+0x50/0x60 [ 3195.470250] > > > > __local_bh_enable_ip+0x50/0x60 [ 3195.474456] > > > > _raw_spin_unlock_bh+0x1e/0x20 [ 3195.478576] > > > > tipc_sk_rcv+0x15a/0x440 [tipc] [ 3195.482782] > > > > tipc_node_xmit+0x147/0x190 [tipc] [ 3195.487254] > __tipc_sendstream+0x1a9/0x2d0 [tipc] [ 3195.491987] ? > > > > aa_sk_perm+0x48/0x1a0 [ 3195.495587] tipc_sendstream+0x35/0x50 > > > > [tipc] [ 3195.499976] tipc_send_packet+0x17/0x20 [tipc] [ > > > > 3195.504454] > > > > sock_sendmsg+0x3e/0x50 [ 3195.507967] kernel_sendmsg+0x2b/0x30 > [ > > > > 3195.511663] tipc_send_to_sock+0x12f/0x290 [tipc] [ 3195.516395] ? > > > > __switch_to_asm+0x40/0x70 [ 3195.520338] ? > > > __switch_to_asm+0x34/0x70 > > > > [ 3195.524280] ? __switch_to_asm+0x34/0x70 [ 3195.528222] ? > > > > __switch_to_asm+0x40/0x70 [ 3195.532174] ? > > > __switch_to_asm+0x34/0x70 > > > > [ 3195.536133] tipc_send_work+0x23/0x40 [tipc] [ 3195.540433] > > > > process_one_work+0x1de/0x410 [ 3195.544468] > > > worker_thread+0x32/0x410 > > > > [ 3195.548147] kthread+0x121/0x140 [ 3195.551403] ? > > > > process_one_work+0x410/0x410 [ 3195.555611] ? > > > > kthread_create_worker_on_cpu+0x70/0x70 > > > > [ 3195.560686] ? do_syscall_64+0x73/0x130 [ 3195.564545] ? > > > > SyS_exit_group+0x14/0x20 [ 3195.568404] ret_from_fork+0x35/0x40 [ > > > > 3195.571998] Code: 00 5d c3 31 c0 5d c3 66 0f 1f 84 00 00 00 00 00 > > > > 0f 1f 44 00 00 55 65 81 05 8f 8a c7 73 00 02 00 00 48 89 e5 31 c0 > > > > ba > > > > 01 00 00 00 <f0> 0f b1 17 85 c0 75 02 5d c3 89 c6 e8 24 12 74 ff > > > > 66 90 5d c3 [ 3195.590981] RIP: _raw_spin_lock_bh+0x1b/0x30 RSP: > > > > ffff90d3df243e90 [ 3195.599011] CR2: 0000000200001dc7 [ > > > > 3195.604403] ---[ end trace 4ce4054b7f724f81 ]--- [ 3195.619914] > > > > Kernel panic - not > > > > syncing: Fatal exception in interrupt [ 3195.628264] Kernel Offset: > > > > 0xaa00000 from 0xffffffff81000000 (relocation range: > > > > 0xffffffff80000000-0xffffffffbfffffff) > > > > [ 3195.649393] ---[ end Kernel panic - not syncing: Fatal > > > > exception in interrupt [ 3195.658447] WARNING: CPU: 1 PID: 348 at > > > > /build/linux-CuEUJv/linux-4.15.0/kernel/sched/core.c:1192 > > > > set_task_cpu+0x158/0x160 > > > > [ 3195.671428] Modules linked in: binfmt_misc tipc ip6_udp_tunnel > > > > udp_tunnel xt_CHECKSUM iptable_mangle ipt_MASQUERADE > > > > nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 > > > > nf_defrag_ipv4 > > > > nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc veth > > > > iptable_filter ipmi_ssif nls_iso8859_1 intel_rapl pnd2_edac > > > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm > > > irqbypass > > > > asix intel_cstate usbnet intel_rapl_perf joydev input_leds mii > > > > ipmi_si ipmi_devintf ipmi_msghandler mac_hid shpchp sch_fq_codel > > > > ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp > > > > libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs > > > > zstd_compress > > > > raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor > > > > async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear > > > > hid_generic usbhid hid crct10dif_pclmul [ 3195.749731] > > > > crc32_pclmul ghash_clmulni_intel pcbc ast aesni_intel ttm > > > > aes_x86_64 drm_kms_helper crypto_simd glue_helper cryptd > > > > syscopyarea sysfillrect sysimgblt ixgbe igb fb_sys_fops i40e drm > > > > ahci dca i2c_algo_bit libahci mdio i2c_ismt ptp pps_core > > > > [ 3195.775727] CPU: 1 PID: 348 Comm: kworker/u32:7 Tainted: G D > > > > 4.15.0-38-generic #41-Ubuntu [ 3195.787072] Hardware name: > > > > Supermicro Super Server/A2SDV-16C- > > > TLN5F, > > > > BIOS 1.0b 12/12/2017 [ 3195.797128] Workqueue: tipc_send > > > > tipc_send_work [tipc] [ 3195.804212] RIP: > > > > 0010:set_task_cpu+0x158/0x160 [ 3195.810603] RSP: > > > > 0018:ffff90d3df243808 EFLAGS: 00010006 [ 3195.817773] RAX: > > > > 0000000000000200 RBX: ffff90d2c5b11640 RCX: 0000000000000000 [ > > > > 3195.826856] RDX: 0000000000000000 RSI: 0000000000000000 RDI: > > > > ffff90d2c5b11640 [ 3195.835936] RBP: ffff90d3df243828 R08: > > > > 0000000000000000 R09: 000000000000ffff [ 3195.845016] R10: > > > > 0000000000000010 R11: 0000000000000000 R12: ffff90d2c5b1216c [ > > > > 3195.854101] R13: 0000000000000000 R14: 0000000000000000 R15: > > > > 0000000000022880 [ 3195.863174] FS: 0000000000000000(0000) > > > > GS:ffff90d3df240000(0000) > > > > knlGS:0000000000000000 > > > > [ 3195.873202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ > > > > 3195.880878] CR2: 0000000200001dc7 CR3: 00000007711f8000 CR4: > > > > 00000000003406e0 [ 3195.889943] Call Trace: > > > > [ 3195.894313] <IRQ> > > > > [ 3195.898250] try_to_wake_up+0x159/0x480 [ 3195.904009] > > > > default_wake_function+0x12/0x20 [ 3195.910199] > > > > autoremove_wake_function+0x12/0x40 > > > > [ 3195.916660] __wake_up_common+0x73/0x130 [ 3195.922508] > > > > __wake_up_common_lock+0x80/0xc0 [ 3195.928695] > > > __wake_up+0x13/0x20 [ > > > > 3195.933831] wake_up_klogd_work_func+0x40/0x60 [ 3195.940185] > > > > irq_work_run_list+0x52/0x80 [ 3195.946026] ? > > > > tick_sched_do_timer+0x50/0x50 [ 3195.952206] > > > > irq_work_tick+0x3b/0x50 [ 3195.957696] > > > > update_process_times+0x42/0x60 [ 3195.963797] > > > > tick_sched_handle+0x26/0x70 [ 3195.969635] ? > > > > tick_sched_do_timer+0x50/0x50 [ 3195.975827] > > > > tick_sched_timer+0x39/0x80 [ 3195.981576] > > > > __hrtimer_run_queues+0xdf/0x230 [ 3195.987756] > > > > hrtimer_interrupt+0xa0/0x1d0 [ 3195.993676] > > > > smp_apic_timer_interrupt+0x6f/0x130 > > > > [ 3196.000208] apic_timer_interrupt+0x84/0x90 [ 3196.006312] RIP: > > > > 0010:panic+0x1fe/0x244 [ 3196.012062] RSP: 0018:ffff90d3df243bd8 > > > > EFLAGS: 00000286 ORIG_RAX: > > > > ffffffffffffff11 > > > > [ 3196.021578] RAX: 0000000000000041 RBX: 0000000000000000 RCX: > > > > 0000000000000006 [ 3196.030668] RDX: 0000000000000000 RSI: > > > > 0000000000000092 RDI: ffff90d3df256490 [ 3196.039773] RBP: > > > > ffff90d3df243c50 R08: 0000000000000000 R09: 000000000000113b [ > > > > 3196.048892] R10: 0000000000000008 R11: 00000000ffffffff R12: > > > > 0000000000000000 [ 3196.058026] R13: 0000000000000000 R14: > > > > 0000000000000009 R15: 0000000000000002 [ 3196.067174] ? > > > > panic+0x1f7/0x244 [ 3196.072419] oops_end+0xb6/0xd0 [ > > > > panic+3196.077569] > > > > no_context+0x17c/0x440 [ 3196.083061] > > > > __bad_area_nosemaphore+0xf3/0x1b0 [ 3196.089520] > > > > bad_area_nosemaphore+0x14/0x20 [ 3196.095714] > > > > __do_page_fault+0xd4/0x4d0 [ 3196.101560] ? > > > > update_load_avg+0x57f/0x6e0 [ 3196.107657] > > > > do_page_fault+0x2e/0xe0 [ 3196.113242] page_fault+0x25/0x50 [ > 3196.118560] RIP: > > > > 0010:_raw_spin_lock_bh+0x1b/0x30 [ 3196.125207] RSP: > > > > 0018:ffff90d3df243e90 EFLAGS: 00010246 [ 3196.132299] RAX: > > > > 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08 [ > > > > 3196.141292] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI: > > > > 0000000200001dc7 [ 3196.150282] RBP: ffff90d3df243e90 R08: > > > > ffff90d3df243f10 R09: ffff90d3df25a780 [ 3196.159277] R10: > > > > ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7 [ > > > > 3196.168262] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15: > > > > ffff90d1317a6c18 [ 3196.177238] ? > > > > tipc_subscrb_release_cb+0x30/0x30 [tipc] [ 3196.184271] ? > > > > tipc_subscrb_release_cb+0x30/0x30 [tipc] [ 3196.191262] > > > > tipc_subscrp_timeout+0x25/0x90 [tipc] [ 3196.197751] > > > > call_timer_fn+0x30/0x130 [ 3196.203029] > > > > run_timer_softirq+0x3fb/0x450 [ 3196.208673] ? > lapic_next_deadline+0x26/0x30 [ 3196.214423] ? > > > > clockevents_program_event+0xd8/0x110 > > > > [ 3196.220707] __do_softirq+0xe4/0x2bb [ 3196.225616] > > > > do_softirq_own_stack+0x2a/0x40 [ 3196.231114] </IRQ> [ > > > > 3196.234491] > > > > do_softirq.part.17+0x50/0x60 [ 3196.239759] > > > > __local_bh_enable_ip+0x50/0x60 [ 3196.245196] > > > > _raw_spin_unlock_bh+0x1e/0x20 [ 3196.250555] > > > > tipc_sk_rcv+0x15a/0x440 [tipc] [ 3196.255986] > > > > tipc_node_xmit+0x147/0x190 [tipc] [ 3196.261677] > __tipc_sendstream+0x1a9/0x2d0 [tipc] [ 3196.267615] ? > > > > aa_sk_perm+0x48/0x1a0 [ 3196.272420] tipc_sendstream+0x35/0x50 > > > > [tipc] [ 3196.278014] tipc_send_packet+0x17/0x20 [tipc] [ > > > > 3196.283673] > > > > sock_sendmsg+0x3e/0x50 [ 3196.288359] kernel_sendmsg+0x2b/0x30 > [ > > > > 3196.293212] tipc_send_to_sock+0x12f/0x290 [tipc] [ 3196.299094] ? > > > > __switch_to_asm+0x40/0x70 [ 3196.304205] ? > > > __switch_to_asm+0x34/0x70 > > > > [ 3196.309301] ? __switch_to_asm+0x34/0x70 [ 3196.314379] ? > > > > __switch_to_asm+0x40/0x70 [ 3196.319446] ? > > > __switch_to_asm+0x34/0x70 > > > > [ 3196.324510] tipc_send_work+0x23/0x40 [tipc] [ 3196.329931] > > > > process_one_work+0x1de/0x410 [ 3196.335103] > > > worker_thread+0x32/0x410 > > > > [ 3196.339919] kthread+0x121/0x140 [ 3196.344299] ? > > > > process_one_work+0x410/0x410 [ 3196.349640] ? > > > > kthread_create_worker_on_cpu+0x70/0x70 > > > > [ 3196.355853] ? do_syscall_64+0x73/0x130 [ 3196.360851] ? > > > > SyS_exit_group+0x14/0x20 [ 3196.365841] ret_from_fork+0x35/0x40 [ > > > > 3196.370567] Code: 32 ff ff ff 80 8b 64 08 00 00 04 e9 4e ff ff ff > > > > 0f 0b e9 f2 fe ff ff f7 43 5c fd ff ff ff 0f 84 fc fe ff ff 0f 0b > > > > e9 > > > > f5 fe ff ff <0f> 0b e9 ff fe ff ff 90 0f 1f 44 00 00 55 48 89 e5 > > > > 41 56 > > > > 41 55 > > > > [ 3196.391999] ---[ end trace 4ce4054b7f724f82 ]--- [ 3196.397922] > > > > ------------[ cut here ]------------ [ 3196.403838] sched: > > > > Unexpected reschedule of offline CPU#0! > > > > [ 3196.410627] WARNING: CPU: 1 PID: 348 at > > > > /build/linux-CuEUJv/linux-4.15.0/arch/x86/kernel/smp.c:128 > > > > native_smp_send_reschedule+0x3a/0x40 > > > > [ 3196.424161] Modules linked in: binfmt_misc tipc ip6_udp_tunnel > > > > udp_tunnel xt_CHECKSUM iptable_mangle ipt_MASQUERADE > > > > nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 > > > > nf_defrag_ipv4 > > > > nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc veth > > > > iptable_filter ipmi_ssif nls_iso8859_1 intel_rapl pnd2_edac > > > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm > > > irqbypass > > > > asix intel_cstate usbnet intel_rapl_perf joydev input_leds mii > > > > ipmi_si ipmi_devintf ipmi_msghandler mac_hid shpchp sch_fq_codel > > > > ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp > > > > libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs > > > > zstd_compress > > > > raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor > > > > async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear > > > > hid_generic usbhid hid crct10dif_pclmul [ 3196.500686] > > > > crc32_pclmul ghash_clmulni_intel pcbc ast aesni_intel ttm > > > > aes_x86_64 drm_kms_helper crypto_simd glue_helper cryptd > > > > syscopyarea sysfillrect sysimgblt ixgbe igb fb_sys_fops i40e drm > > > > ahci dca i2c_algo_bit libahci mdio i2c_ismt ptp pps_core > > > > [ 3196.526181] CPU: 1 PID: 348 Comm: kworker/u32:7 Tainted: G D W > > > > 4.15.0-38-generic #41-Ubuntu [ 3196.537321] Hardware name: > > > > Supermicro Super Server/A2SDV-16C- > > > TLN5F, > > > > BIOS 1.0b 12/12/2017 [ 3196.547206] Workqueue: tipc_send > > > > tipc_send_work [tipc] [ 3196.554142] RIP: > > > > 0010:native_smp_send_reschedule+0x3a/0x40 > > > > [ 3196.561423] RSP: 0018:ffff90d3df243790 EFLAGS: 00010082 [ > > > > 3196.568445] RAX: 0000000000000000 RBX: 0000000000000000 RCX: > > > > 0000000000000006 [ 3196.577395] RDX: 0000000000000007 RSI: > > > > 0000000000000086 RDI: ffff90d3df256490 [ 3196.586344] RBP: > > > > ffff90d3df243790 R08: 0000000000000000 R09: 000000000000119e [ > > > > 3196.595297] R10: 000000000000025f R11: 00000000ffffffff R12: > > > > ffff90d3df222880 [ 3196.604262] R13: ffff90d2c5b11640 R14: > > > > ffff90d3df243848 R15: ffff90d3df222880 [ 3196.613232] FS: > > > > 0000000000000000(0000) GS:ffff90d3df240000(0000) > > > > knlGS:0000000000000000 > > > > [ 3196.623178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ > > > > 3196.630768] CR2: 0000000200001dc7 CR3: 00000007711f8000 CR4: > > > > 00000000003406e0 [ 3196.639761] Call Trace: > > > > [ 3196.644061] <IRQ> > > > > [ 3196.647923] resched_curr+0x5d/0xc0 [ 3196.653276] > > > > check_preempt_curr+0x7a/0x90 [ 3196.659130] > > > ttwu_do_wakeup+0x1e/0x140 > > > > [ 3196.664745] ttwu_do_activate+0x77/0x80 [ 3196.670433] > > > > try_to_wake_up+0x1d6/0x480 [ 3196.676117] > > > > default_wake_function+0x12/0x20 [ 3196.682237] > > > > autoremove_wake_function+0x12/0x40 > > > > [ 3196.688617] __wake_up_common+0x73/0x130 [ 3196.694389] > > > > __wake_up_common_lock+0x80/0xc0 [ 3196.700494] > > > __wake_up+0x13/0x20 [ > > > > 3196.705549] wake_up_klogd_work_func+0x40/0x60 [ 3196.711831] > > > > irq_work_run_list+0x52/0x80 [ 3196.717598] ? > > > > tick_sched_do_timer+0x50/0x50 [ 3196.723707] > > > > irq_work_tick+0x3b/0x50 [ 3196.729118] > > > > update_process_times+0x42/0x60 [ 3196.735141] > > > > tick_sched_handle+0x26/0x70 [ 3196.740900] ? > > > > tick_sched_do_timer+0x50/0x50 [ 3196.747014] > > > > tick_sched_timer+0x39/0x80 [ 3196.752684] > > > > __hrtimer_run_queues+0xdf/0x230 [ 3196.758789] > > > > hrtimer_interrupt+0xa0/0x1d0 [ 3196.764640] > > > > smp_apic_timer_interrupt+0x6f/0x130 > > > > [ 3196.771101] apic_timer_interrupt+0x84/0x90 [ 3196.777130] RIP: > > > > 0010:panic+0x1fe/0x244 [ 3196.782802] RSP: 0018:ffff90d3df243bd8 > > > > EFLAGS: 00000286 ORIG_RAX: > > > > ffffffffffffff11 > > > > [ 3196.792240] RAX: 0000000000000041 RBX: 0000000000000000 RCX: > > > > 0000000000000006 [ 3196.801259] RDX: 0000000000000000 RSI: > > > > 0000000000000092 RDI: ffff90d3df256490 [ 3196.810293] RBP: > > > > ffff90d3df243c50 R08: 0000000000000000 R09: 000000000000113b [ > > > > 3196.819342] R10: 0000000000000008 R11: 00000000ffffffff R12: > > > > 0000000000000000 [ 3196.828408] R13: 0000000000000000 R14: > > > > 0000000000000009 R15: 0000000000000002 [ 3196.837484] ? > > > > panic+0x1f7/0x244 [ 3196.842654] oops_end+0xb6/0xd0 [ > > > > panic+3196.847725] > > > > no_context+0x17c/0x440 [ 3196.853142] > > > > __bad_area_nosemaphore+0xf3/0x1b0 [ 3196.859519] > > > > bad_area_nosemaphore+0x14/0x20 [ 3196.865635] > > > > __do_page_fault+0xd4/0x4d0 [ 3196.871401] ? > > > > update_load_avg+0x57f/0x6e0 [ 3196.877434] > > > > do_page_fault+0x2e/0xe0 [ 3196.882942] page_fault+0x25/0x50 [ > 3196.888121] RIP: > > > > 0010:_raw_spin_lock_bh+0x1b/0x30 [ 3196.894624] RSP: > > > > 0018:ffff90d3df243e90 EFLAGS: 00010246 [ 3196.901641] RAX: > > > > 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08 [ > > > > 3196.910595] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI: > > > > 0000000200001dc7 [ 3196.919541] RBP: ffff90d3df243e90 R08: > > > > ffff90d3df243f10 R09: ffff90d3df25a780 [ 3196.928459] R10: > > > > ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7 [ > > > > 3196.937309] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15: > > > > ffff90d1317a6c18 [ 3196.946097] ? > > > > tipc_subscrb_release_cb+0x30/0x30 [tipc] [ 3196.952912] ? > > > > tipc_subscrb_release_cb+0x30/0x30 [tipc] [ 3196.959636] > > > > tipc_subscrp_timeout+0x25/0x90 [tipc] [ 3196.965850] > > > > call_timer_fn+0x30/0x130 [ 3196.970861] > > > > run_timer_softirq+0x3fb/0x450 [ 3196.976282] ? > lapic_next_deadline+0x26/0x30 [ 3196.981856] ? > > > > clockevents_program_event+0xd8/0x110 > > > > [ 3196.988004] __do_softirq+0xe4/0x2bb [ 3196.992839] > > > > do_softirq_own_stack+0x2a/0x40 [ 3196.998277] </IRQ> [ > > > > 3197.001619] > > > > do_softirq.part.17+0x50/0x60 [ 3197.006875] > > > > __local_bh_enable_ip+0x50/0x60 [ 3197.012300] > > > > _raw_spin_unlock_bh+0x1e/0x20 [ 3197.017639] > > > > tipc_sk_rcv+0x15a/0x440 [tipc] [ 3197.023057] > > > > tipc_node_xmit+0x147/0x190 [tipc] [ 3197.028715] > __tipc_sendstream+0x1a9/0x2d0 [tipc] [ 3197.034618] ? > > > > aa_sk_perm+0x48/0x1a0 [ 3197.039384] tipc_sendstream+0x35/0x50 > > > > [tipc] [ 3197.044919] tipc_send_packet+0x17/0x20 [tipc] [ > > > > 3197.050539] > > > > sock_sendmsg+0x3e/0x50 [ 3197.055207] kernel_sendmsg+0x2b/0x30 > [ > > > > 3197.060044] tipc_send_to_sock+0x12f/0x290 [tipc] [ 3197.065927] ? > > > > __switch_to_asm+0x40/0x70 [ 3197.071048] ? > > > __switch_to_asm+0x34/0x70 > > > > [ 3197.076162] ? __switch_to_asm+0x34/0x70 [ 3197.081257] ? > > > > __switch_to_asm+0x40/0x70 [ 3197.086345] ? > > > __switch_to_asm+0x34/0x70 > > > > [ 3197.091425] tipc_send_work+0x23/0x40 [tipc] [ 3197.096854] > > > > process_one_work+0x1de/0x410 [ 3197.102030] > > > worker_thread+0x32/0x410 > > > > [ 3197.106854] kthread+0x121/0x140 [ 3197.111239] ? > > > > process_one_work+0x410/0x410 [ 3197.116576] ? > > > > kthread_create_worker_on_cpu+0x70/0x70 > > > > [ 3197.122788] ? do_syscall_64+0x73/0x130 [ 3197.127786] ? > > > > SyS_exit_group+0x14/0x20 [ 3197.132784] ret_from_fork+0x35/0x40 [ > > > > 3197.137508] Code: 2d 61 60 01 73 17 48 8b 05 84 21 16 01 be fd 00 > > > > 00 00 48 8b 40 30 e8 36 b6 ba 00 5d c3 89 fe 48 c7 c7 60 fe ab 8c > > > > e8 > > > > 06 48 03 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 > > > > 83 ec > > > > 20 65 > > > > [ 3197.158946] ---[ end trace 4ce4054b7f724f83 ]--- [ 3197.164963] > > > > unchecked MSR access error: WRMSR to 0x83f (tried to write > > > > 0x00000000000000f6) at rIP: 0xffffffff8ba6c1a8 > > > > (native_write_msr+0x8/0x30) > > > > [ 3197.179360] Call Trace: > > > > [ 3197.183165] <IRQ> > > > > [ 3197.186532] native_apic_msr_write+0x2b/0x40 [ 3197.192143] > > > > x2apic_send_IPI_self+0x20/0x30 [ 3197.197660] > > > > arch_irq_work_raise+0x2a/0x40 [ 3197.203080] > > > irq_work_queue+0x8d/0xa0 > > > > [ 3197.208058] dbs_update_util_handler+0x80/0xa0 [ 3197.213822] > > > > update_load_avg+0x5c5/0x6e0 [ 3197.219068] > > > enqueue_entity+0x5e/0x6b0 > > > > [ 3197.224124] enqueue_task_fair+0x6c/0x7f0 [ 3197.229449] ? > > > > __update_load_avg_blocked_se.isra.37+0xd1/0x150 > > > > [ 3197.236608] activate_task+0x54/0xc0 [ 3197.241510] ? > > > > sched_clock+0x9/0x10 [ 3197.246338] ttwu_do_activate+0x49/0x80 [ > > > > 3197.251514] try_to_wake_up+0x1d6/0x480 [ 3197.256684] > > > > default_wake_function+0x12/0x20 [ 3197.262288] > > > > autoremove_wake_function+0x12/0x40 > > > > [ 3197.268162] __wake_up_common+0x73/0x130 [ 3197.273428] > > > > __wake_up_common_lock+0x80/0xc0 [ 3197.279026] > > > __wake_up+0x13/0x20 [ > > > > 3197.283562] wake_up_klogd_work_func+0x40/0x60 [ 3197.289321] > > > > irq_work_run_list+0x52/0x80 [ 3197.294545] ? > > > > tick_sched_do_timer+0x50/0x50 [ 3197.300116] > > > > irq_work_tick+0x3b/0x50 [ 3197.304987] > > > > update_process_times+0x42/0x60 [ 3197.310455] > > > > tick_sched_handle+0x26/0x70 [ 3197.315672] ? > > > > tick_sched_do_timer+0x50/0x50 [ 3197.321233] > > > > tick_sched_timer+0x39/0x80 [ 3197.326351] > > > > __hrtimer_run_queues+0xdf/0x230 [ 3197.331909] > > > > hrtimer_interrupt+0xa0/0x1d0 [ 3197.337205] > > > > smp_apic_timer_interrupt+0x6f/0x130 > > > > [ 3197.343108] apic_timer_interrupt+0x84/0x90 [ 3197.348574] RIP: > > > > 0010:panic+0x1fe/0x244 [ 3197.353674] RSP: 0018:ffff90d3df243bd8 > > > > EFLAGS: 00000286 ORIG_RAX: > > > > ffffffffffffff11 > > > > [ 3197.362537] RAX: 0000000000000041 RBX: 0000000000000000 RCX: > > > > 0000000000000006 [ 3197.370972] RDX: 0000000000000000 RSI: > > > > 0000000000000092 RDI: ffff90d3df256490 [ 3197.379405] RBP: > > > > ffff90d3df243c50 R08: 0000000000000000 R09: 000000000000113b [ > > > > 3197.387843] R10: 0000000000000008 R11: 00000000ffffffff R12: > > > > 0000000000000000 [ 3197.396291] R13: 0000000000000000 R14: > > > > 0000000000000009 R15: 0000000000000002 [ 3197.404743] ? > > > > panic+0x1f7/0x244 [ 3197.409283] oops_end+0xb6/0xd0 [ > > > > panic+3197.413730] > > > > no_context+0x17c/0x440 [ 3197.418521] > > > > __bad_area_nosemaphore+0xf3/0x1b0 [ 3197.424273] > > > > bad_area_nosemaphore+0x14/0x20 [ 3197.429763] > > > > __do_page_fault+0xd4/0x4d0 [ 3197.434912] ? > > > > update_load_avg+0x57f/0x6e0 [ 3197.440318] > > > > do_page_fault+0x2e/0xe0 [ 3197.445203] page_fault+0x25/0x50 [ > 3197.449827] RIP: > > > > 0010:_raw_spin_lock_bh+0x1b/0x30 [ 3197.455862] RSP: > > > > 0018:ffff90d3df243e90 EFLAGS: 00010246 [ 3197.462432] RAX: > > > > 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08 [ > > > > 3197.470940] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI: > > > > 0000000200001dc7 [ 3197.479454] RBP: ffff90d3df243e90 R08: > > > > ffff90d3df243f10 R09: ffff90d3df25a780 [ 3197.487973] R10: > > > > ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7 [ > > > > 3197.496495] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15: > > > > ffff90d1317a6c18 [ 3197.505043] ? > > > > tipc_subscrb_release_cb+0x30/0x30 [tipc] [ 3197.511689] ? > > > > tipc_subscrb_release_cb+0x30/0x30 [tipc] [ 3197.518319] > > > > tipc_subscrp_timeout+0x25/0x90 [tipc] [ 3197.524511] > > > > call_timer_fn+0x30/0x130 [ 3197.529573] > > > > run_timer_softirq+0x3fb/0x450 [ 3197.535073] ? > lapic_next_deadline+0x26/0x30 [ 3197.540684] ? > > > > clockevents_program_event+0xd8/0x110 > > > > [ 3197.546832] __do_softirq+0xe4/0x2bb [ 3197.551680] > > > > do_softirq_own_stack+0x2a/0x40 [ 3197.557086] </IRQ> [ > > > > 3197.560362] > > > > do_softirq.part.17+0x50/0x60 [ 3197.565561] > > > > __local_bh_enable_ip+0x50/0x60 [ 3197.570937] > > > > _raw_spin_unlock_bh+0x1e/0x20 [ 3197.576228] > > > > tipc_sk_rcv+0x15a/0x440 [tipc] [ 3197.581602] > > > > tipc_node_xmit+0x147/0x190 [tipc] [ 3197.587245] > __tipc_sendstream+0x1a9/0x2d0 [tipc] [ 3197.593144] ? > > > > aa_sk_perm+0x48/0x1a0 [ 3197.597909] tipc_sendstream+0x35/0x50 > > > > [tipc] [ 3197.603461] tipc_send_packet+0x17/0x20 [tipc] [ > > > > 3197.609101] > > > > sock_sendmsg+0x3e/0x50 [ 3197.613767] kernel_sendmsg+0x2b/0x30 > [ > > > > 3197.618609] tipc_send_to_sock+0x12f/0x290 [tipc] [ 3197.624497] ? > > > > __switch_to_asm+0x40/0x70 [ 3197.629601] ? > > > __switch_to_asm+0x34/0x70 > > > > [ 3197.634697] ? __switch_to_asm+0x34/0x70 [ 3197.639781] ? > > > > __switch_to_asm+0x40/0x70 [ 3197.644846] ? > > > __switch_to_asm+0x34/0x70 > > > > [ 3197.649909] tipc_send_work+0x23/0x40 [tipc] [ 3197.655317] > > > > process_one_work+0x1de/0x410 [ 3197.660470] > > > worker_thread+0x32/0x410 > > > > [ 3197.665274] kthread+0x121/0x140 [ 3197.669636] ? > > > > process_one_work+0x410/0x410 [ 3197.674955] ? > > > > kthread_create_worker_on_cpu+0x70/0x70 > > > > [ 3197.681146] ? do_syscall_64+0x73/0x130 [ 3197.686118] ? > > > > SyS_exit_group+0x14/0x20 [ 3197.691094] ret_from_fork+0x35/0x40 > > > > pe 26. lokak. 2018 klo 6.38 Ying Xue ([email protected]) kirjoitti: > > > > > > > > > > Hi Juhamatti, > > > > > > > > > > This is a good test plan indeed. If you encounter any problem > > > > > during the testing, please share it here. > > > > > > > > > > From my view, TIPC had fully supported networking nemespace at > > > > > least three years ago and its sockets could be fully isolated > > > > > between > > > containers. > > > > > > > > > > Thanks, > > > > > Ying > > > > > > > > > > On 10/26/2018 11:12 AM, [email protected] wrote: > > > > > > Hello, > > > > > > > > > > > > I'm planning to test TIPC scalability in more detail, > > > > > > especially regarding the binding table and topology (service > > > > > > tracking) service with large number of nodes (~1k) and sockets > > > > > > (100-1k). Is it enough to do it with LXC containers with > > > > > > isolated namespaces to get realistic results? My concern here > > > > > > is that if TIPC implementation would not provide full isolation to > sockets e.g. > > > > > > for performance reasons inside the same kernel, then this may > > > > > > not provide results matching to real environments. VMs are the > > > > > > other choice, but they tend to be heavy. Are the namespaces > > > > > > fully isolated > > > inside the kernel? > > > > > > > > > > > > Thanks for the info, > > > > > > -- > > > > > > Juhamatti > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > tipc-discussion mailing list > > > > > > [email protected] > > > > > > https://lists.sourceforge.net/lists/listinfo/tipc-discussion > > > > > > > > > > > > > > > _______________________________________________ > > > tipc-discussion mailing list > > > [email protected] > > > https://lists.sourceforge.net/lists/listinfo/tipc-discussion _______________________________________________ tipc-discussion mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/tipc-discussion
