Hello,
Thanks Ying! While running a server with 250 LXCs and 2500 TIPC
sockets running with modified connection demo, I am seeing the problem
described below. The kernel is stock Ubuntu1804 4.15.0-38-generic with
16 cores. It is fairly easy to reproduce, after starting up containers
it will happen within an hour or so.
Please let me know if there is already a patch for it, I am happy to
try. I can also try fix candidates quite easily.
BR,
--
Juhamatti
[ 3195.231191] BUG: unable to handle kernel paging request at 0000000200001dc7
[ 3195.238220] IP: _raw_spin_lock_bh+0x1b/0x30
[ 3195.242432] PGD 0 P4D 0
[ 3195.244988] Oops: 0002 [#1] SMP NOPTI
[ 3195.248668] Modules linked in: binfmt_misc tipc ip6_udp_tunnel
udp_tunnel xt_CHECKSUM iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc veth
iptable_filter ipmi_ssif nls_iso8859_1 intel_rapl pnd2_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
asix intel_cstate usbnet intel_rapl_perf joydev input_leds mii ipmi_si
ipmi_devintf ipmi_msghandler mac_hid shpchp sch_fq_codel ib_iser
rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
hid_generic usbhid hid crct10dif_pclmul
[ 3195.319457] crc32_pclmul ghash_clmulni_intel pcbc ast aesni_intel
ttm aes_x86_64 drm_kms_helper crypto_simd glue_helper cryptd
syscopyarea sysfillrect sysimgblt ixgbe igb fb_sys_fops i40e drm ahci
dca i2c_algo_bit libahci mdio i2c_ismt ptp pps_core
[ 3195.341632] CPU: 1 PID: 348 Comm: kworker/u32:7 Not tainted
4.15.0-38-generic #41-Ubuntu
[ 3195.349754] Hardware name: Supermicro Super Server/A2SDV-16C-TLN5F,
BIOS 1.0b 12/12/2017
[ 3195.357891] Workqueue: tipc_send tipc_send_work [tipc]
[ 3195.363061] RIP: 0010:_raw_spin_lock_bh+0x1b/0x30
[ 3195.367788] RSP: 0018:ffff90d3df243e90 EFLAGS: 00010246
[ 3195.373032] RAX: 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08
[ 3195.380203] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI: 0000000200001dc7
[ 3195.387393] RBP: ffff90d3df243e90 R08: ffff90d3df243f10 R09: ffff90d3df25a780
[ 3195.394561] R10: ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7
[ 3195.401729] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15: ffff90d1317a6c18
[ 3195.408897] FS: 0000000000000000(0000) GS:ffff90d3df240000(0000)
knlGS:0000000000000000
[ 3195.417025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3195.422798] CR2: 0000000200001dc7 CR3: 00000007711f8000 CR4: 00000000003406e0
[ 3195.429965] Call Trace:
[ 3195.432429] <IRQ>
[ 3195.434467] tipc_subscrp_timeout+0x25/0x90 [tipc]
[ 3195.439311] call_timer_fn+0x30/0x130
[ 3195.442994] run_timer_softirq+0x3fb/0x450
[ 3195.447113] ? lapic_next_deadline+0x26/0x30
[ 3195.451404] ? clockevents_program_event+0xd8/0x110
[ 3195.456304] __do_softirq+0xe4/0x2bb
[ 3195.459903] do_softirq_own_stack+0x2a/0x40
[ 3195.464107] </IRQ>
[ 3195.466219] do_softirq.part.17+0x50/0x60
[ 3195.470250] __local_bh_enable_ip+0x50/0x60
[ 3195.474456] _raw_spin_unlock_bh+0x1e/0x20
[ 3195.478576] tipc_sk_rcv+0x15a/0x440 [tipc]
[ 3195.482782] tipc_node_xmit+0x147/0x190 [tipc]
[ 3195.487254] __tipc_sendstream+0x1a9/0x2d0 [tipc]
[ 3195.491987] ? aa_sk_perm+0x48/0x1a0
[ 3195.495587] tipc_sendstream+0x35/0x50 [tipc]
[ 3195.499976] tipc_send_packet+0x17/0x20 [tipc]
[ 3195.504454] sock_sendmsg+0x3e/0x50
[ 3195.507967] kernel_sendmsg+0x2b/0x30
[ 3195.511663] tipc_send_to_sock+0x12f/0x290 [tipc]
[ 3195.516395] ? __switch_to_asm+0x40/0x70
[ 3195.520338] ? __switch_to_asm+0x34/0x70
[ 3195.524280] ? __switch_to_asm+0x34/0x70
[ 3195.528222] ? __switch_to_asm+0x40/0x70
[ 3195.532174] ? __switch_to_asm+0x34/0x70
[ 3195.536133] tipc_send_work+0x23/0x40 [tipc]
[ 3195.540433] process_one_work+0x1de/0x410
[ 3195.544468] worker_thread+0x32/0x410
[ 3195.548147] kthread+0x121/0x140
[ 3195.551403] ? process_one_work+0x410/0x410
[ 3195.555611] ? kthread_create_worker_on_cpu+0x70/0x70
[ 3195.560686] ? do_syscall_64+0x73/0x130
[ 3195.564545] ? SyS_exit_group+0x14/0x20
[ 3195.568404] ret_from_fork+0x35/0x40
[ 3195.571998] Code: 00 5d c3 31 c0 5d c3 66 0f 1f 84 00 00 00 00 00
0f 1f 44 00 00 55 65 81 05 8f 8a c7 73 00 02 00 00 48 89 e5 31 c0 ba
01 00 00 00 <f0> 0f b1 17 85 c0 75 02 5d c3 89 c6 e8 24 12 74 ff 66 90
5d c3
[ 3195.590981] RIP: _raw_spin_lock_bh+0x1b/0x30 RSP: ffff90d3df243e90
[ 3195.599011] CR2: 0000000200001dc7
[ 3195.604403] ---[ end trace 4ce4054b7f724f81 ]---
[ 3195.619914] Kernel panic - not syncing: Fatal exception in interrupt
[ 3195.628264] Kernel Offset: 0xaa00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 3195.649393] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[ 3195.658447] WARNING: CPU: 1 PID: 348 at
/build/linux-CuEUJv/linux-4.15.0/kernel/sched/core.c:1192
set_task_cpu+0x158/0x160
[ 3195.671428] Modules linked in: binfmt_misc tipc ip6_udp_tunnel
udp_tunnel xt_CHECKSUM iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc veth
iptable_filter ipmi_ssif nls_iso8859_1 intel_rapl pnd2_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
asix intel_cstate usbnet intel_rapl_perf joydev input_leds mii ipmi_si
ipmi_devintf ipmi_msghandler mac_hid shpchp sch_fq_codel ib_iser
rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
hid_generic usbhid hid crct10dif_pclmul
[ 3195.749731] crc32_pclmul ghash_clmulni_intel pcbc ast aesni_intel
ttm aes_x86_64 drm_kms_helper crypto_simd glue_helper cryptd
syscopyarea sysfillrect sysimgblt ixgbe igb fb_sys_fops i40e drm ahci
dca i2c_algo_bit libahci mdio i2c_ismt ptp pps_core
[ 3195.775727] CPU: 1 PID: 348 Comm: kworker/u32:7 Tainted: G D
4.15.0-38-generic #41-Ubuntu
[ 3195.787072] Hardware name: Supermicro Super Server/A2SDV-16C-TLN5F,
BIOS 1.0b 12/12/2017
[ 3195.797128] Workqueue: tipc_send tipc_send_work [tipc]
[ 3195.804212] RIP: 0010:set_task_cpu+0x158/0x160
[ 3195.810603] RSP: 0018:ffff90d3df243808 EFLAGS: 00010006
[ 3195.817773] RAX: 0000000000000200 RBX: ffff90d2c5b11640 RCX: 0000000000000000
[ 3195.826856] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff90d2c5b11640
[ 3195.835936] RBP: ffff90d3df243828 R08: 0000000000000000 R09: 000000000000ffff
[ 3195.845016] R10: 0000000000000010 R11: 0000000000000000 R12: ffff90d2c5b1216c
[ 3195.854101] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000022880
[ 3195.863174] FS: 0000000000000000(0000) GS:ffff90d3df240000(0000)
knlGS:0000000000000000
[ 3195.873202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3195.880878] CR2: 0000000200001dc7 CR3: 00000007711f8000 CR4: 00000000003406e0
[ 3195.889943] Call Trace:
[ 3195.894313] <IRQ>
[ 3195.898250] try_to_wake_up+0x159/0x480
[ 3195.904009] default_wake_function+0x12/0x20
[ 3195.910199] autoremove_wake_function+0x12/0x40
[ 3195.916660] __wake_up_common+0x73/0x130
[ 3195.922508] __wake_up_common_lock+0x80/0xc0
[ 3195.928695] __wake_up+0x13/0x20
[ 3195.933831] wake_up_klogd_work_func+0x40/0x60
[ 3195.940185] irq_work_run_list+0x52/0x80
[ 3195.946026] ? tick_sched_do_timer+0x50/0x50
[ 3195.952206] irq_work_tick+0x3b/0x50
[ 3195.957696] update_process_times+0x42/0x60
[ 3195.963797] tick_sched_handle+0x26/0x70
[ 3195.969635] ? tick_sched_do_timer+0x50/0x50
[ 3195.975827] tick_sched_timer+0x39/0x80
[ 3195.981576] __hrtimer_run_queues+0xdf/0x230
[ 3195.987756] hrtimer_interrupt+0xa0/0x1d0
[ 3195.993676] smp_apic_timer_interrupt+0x6f/0x130
[ 3196.000208] apic_timer_interrupt+0x84/0x90
[ 3196.006312] RIP: 0010:panic+0x1fe/0x244
[ 3196.012062] RSP: 0018:ffff90d3df243bd8 EFLAGS: 00000286 ORIG_RAX:
ffffffffffffff11
[ 3196.021578] RAX: 0000000000000041 RBX: 0000000000000000 RCX: 0000000000000006
[ 3196.030668] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff90d3df256490
[ 3196.039773] RBP: ffff90d3df243c50 R08: 0000000000000000 R09: 000000000000113b
[ 3196.048892] R10: 0000000000000008 R11: 00000000ffffffff R12: 0000000000000000
[ 3196.058026] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000002
[ 3196.067174] ? panic+0x1f7/0x244
[ 3196.072419] oops_end+0xb6/0xd0
[ 3196.077569] no_context+0x17c/0x440
[ 3196.083061] __bad_area_nosemaphore+0xf3/0x1b0
[ 3196.089520] bad_area_nosemaphore+0x14/0x20
[ 3196.095714] __do_page_fault+0xd4/0x4d0
[ 3196.101560] ? update_load_avg+0x57f/0x6e0
[ 3196.107657] do_page_fault+0x2e/0xe0
[ 3196.113242] page_fault+0x25/0x50
[ 3196.118560] RIP: 0010:_raw_spin_lock_bh+0x1b/0x30
[ 3196.125207] RSP: 0018:ffff90d3df243e90 EFLAGS: 00010246
[ 3196.132299] RAX: 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08
[ 3196.141292] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI: 0000000200001dc7
[ 3196.150282] RBP: ffff90d3df243e90 R08: ffff90d3df243f10 R09: ffff90d3df25a780
[ 3196.159277] R10: ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7
[ 3196.168262] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15: ffff90d1317a6c18
[ 3196.177238] ? tipc_subscrb_release_cb+0x30/0x30 [tipc]
[ 3196.184271] ? tipc_subscrb_release_cb+0x30/0x30 [tipc]
[ 3196.191262] tipc_subscrp_timeout+0x25/0x90 [tipc]
[ 3196.197751] call_timer_fn+0x30/0x130
[ 3196.203029] run_timer_softirq+0x3fb/0x450
[ 3196.208673] ? lapic_next_deadline+0x26/0x30
[ 3196.214423] ? clockevents_program_event+0xd8/0x110
[ 3196.220707] __do_softirq+0xe4/0x2bb
[ 3196.225616] do_softirq_own_stack+0x2a/0x40
[ 3196.231114] </IRQ>
[ 3196.234491] do_softirq.part.17+0x50/0x60
[ 3196.239759] __local_bh_enable_ip+0x50/0x60
[ 3196.245196] _raw_spin_unlock_bh+0x1e/0x20
[ 3196.250555] tipc_sk_rcv+0x15a/0x440 [tipc]
[ 3196.255986] tipc_node_xmit+0x147/0x190 [tipc]
[ 3196.261677] __tipc_sendstream+0x1a9/0x2d0 [tipc]
[ 3196.267615] ? aa_sk_perm+0x48/0x1a0
[ 3196.272420] tipc_sendstream+0x35/0x50 [tipc]
[ 3196.278014] tipc_send_packet+0x17/0x20 [tipc]
[ 3196.283673] sock_sendmsg+0x3e/0x50
[ 3196.288359] kernel_sendmsg+0x2b/0x30
[ 3196.293212] tipc_send_to_sock+0x12f/0x290 [tipc]
[ 3196.299094] ? __switch_to_asm+0x40/0x70
[ 3196.304205] ? __switch_to_asm+0x34/0x70
[ 3196.309301] ? __switch_to_asm+0x34/0x70
[ 3196.314379] ? __switch_to_asm+0x40/0x70
[ 3196.319446] ? __switch_to_asm+0x34/0x70
[ 3196.324510] tipc_send_work+0x23/0x40 [tipc]
[ 3196.329931] process_one_work+0x1de/0x410
[ 3196.335103] worker_thread+0x32/0x410
[ 3196.339919] kthread+0x121/0x140
[ 3196.344299] ? process_one_work+0x410/0x410
[ 3196.349640] ? kthread_create_worker_on_cpu+0x70/0x70
[ 3196.355853] ? do_syscall_64+0x73/0x130
[ 3196.360851] ? SyS_exit_group+0x14/0x20
[ 3196.365841] ret_from_fork+0x35/0x40
[ 3196.370567] Code: 32 ff ff ff 80 8b 64 08 00 00 04 e9 4e ff ff ff
0f 0b e9 f2 fe ff ff f7 43 5c fd ff ff ff 0f 84 fc fe ff ff 0f 0b e9
f5 fe ff ff <0f> 0b e9 ff fe ff ff 90 0f 1f 44 00 00 55 48 89 e5 41 56
41 55
[ 3196.391999] ---[ end trace 4ce4054b7f724f82 ]---
[ 3196.397922] ------------[ cut here ]------------
[ 3196.403838] sched: Unexpected reschedule of offline CPU#0!
[ 3196.410627] WARNING: CPU: 1 PID: 348 at
/build/linux-CuEUJv/linux-4.15.0/arch/x86/kernel/smp.c:128
native_smp_send_reschedule+0x3a/0x40
[ 3196.424161] Modules linked in: binfmt_misc tipc ip6_udp_tunnel
udp_tunnel xt_CHECKSUM iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc veth
iptable_filter ipmi_ssif nls_iso8859_1 intel_rapl pnd2_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
asix intel_cstate usbnet intel_rapl_perf joydev input_leds mii ipmi_si
ipmi_devintf ipmi_msghandler mac_hid shpchp sch_fq_codel ib_iser
rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
hid_generic usbhid hid crct10dif_pclmul
[ 3196.500686] crc32_pclmul ghash_clmulni_intel pcbc ast aesni_intel
ttm aes_x86_64 drm_kms_helper crypto_simd glue_helper cryptd
syscopyarea sysfillrect sysimgblt ixgbe igb fb_sys_fops i40e drm ahci
dca i2c_algo_bit libahci mdio i2c_ismt ptp pps_core
[ 3196.526181] CPU: 1 PID: 348 Comm: kworker/u32:7 Tainted: G D W
4.15.0-38-generic #41-Ubuntu
[ 3196.537321] Hardware name: Supermicro Super Server/A2SDV-16C-TLN5F,
BIOS 1.0b 12/12/2017
[ 3196.547206] Workqueue: tipc_send tipc_send_work [tipc]
[ 3196.554142] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 3196.561423] RSP: 0018:ffff90d3df243790 EFLAGS: 00010082
[ 3196.568445] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006
[ 3196.577395] RDX: 0000000000000007 RSI: 0000000000000086 RDI: ffff90d3df256490
[ 3196.586344] RBP: ffff90d3df243790 R08: 0000000000000000 R09: 000000000000119e
[ 3196.595297] R10: 000000000000025f R11: 00000000ffffffff R12: ffff90d3df222880
[ 3196.604262] R13: ffff90d2c5b11640 R14: ffff90d3df243848 R15: ffff90d3df222880
[ 3196.613232] FS: 0000000000000000(0000) GS:ffff90d3df240000(0000)
knlGS:0000000000000000
[ 3196.623178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3196.630768] CR2: 0000000200001dc7 CR3: 00000007711f8000 CR4: 00000000003406e0
[ 3196.639761] Call Trace:
[ 3196.644061] <IRQ>
[ 3196.647923] resched_curr+0x5d/0xc0
[ 3196.653276] check_preempt_curr+0x7a/0x90
[ 3196.659130] ttwu_do_wakeup+0x1e/0x140
[ 3196.664745] ttwu_do_activate+0x77/0x80
[ 3196.670433] try_to_wake_up+0x1d6/0x480
[ 3196.676117] default_wake_function+0x12/0x20
[ 3196.682237] autoremove_wake_function+0x12/0x40
[ 3196.688617] __wake_up_common+0x73/0x130
[ 3196.694389] __wake_up_common_lock+0x80/0xc0
[ 3196.700494] __wake_up+0x13/0x20
[ 3196.705549] wake_up_klogd_work_func+0x40/0x60
[ 3196.711831] irq_work_run_list+0x52/0x80
[ 3196.717598] ? tick_sched_do_timer+0x50/0x50
[ 3196.723707] irq_work_tick+0x3b/0x50
[ 3196.729118] update_process_times+0x42/0x60
[ 3196.735141] tick_sched_handle+0x26/0x70
[ 3196.740900] ? tick_sched_do_timer+0x50/0x50
[ 3196.747014] tick_sched_timer+0x39/0x80
[ 3196.752684] __hrtimer_run_queues+0xdf/0x230
[ 3196.758789] hrtimer_interrupt+0xa0/0x1d0
[ 3196.764640] smp_apic_timer_interrupt+0x6f/0x130
[ 3196.771101] apic_timer_interrupt+0x84/0x90
[ 3196.777130] RIP: 0010:panic+0x1fe/0x244
[ 3196.782802] RSP: 0018:ffff90d3df243bd8 EFLAGS: 00000286 ORIG_RAX:
ffffffffffffff11
[ 3196.792240] RAX: 0000000000000041 RBX: 0000000000000000 RCX: 0000000000000006
[ 3196.801259] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff90d3df256490
[ 3196.810293] RBP: ffff90d3df243c50 R08: 0000000000000000 R09: 000000000000113b
[ 3196.819342] R10: 0000000000000008 R11: 00000000ffffffff R12: 0000000000000000
[ 3196.828408] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000002
[ 3196.837484] ? panic+0x1f7/0x244
[ 3196.842654] oops_end+0xb6/0xd0
[ 3196.847725] no_context+0x17c/0x440
[ 3196.853142] __bad_area_nosemaphore+0xf3/0x1b0
[ 3196.859519] bad_area_nosemaphore+0x14/0x20
[ 3196.865635] __do_page_fault+0xd4/0x4d0
[ 3196.871401] ? update_load_avg+0x57f/0x6e0
[ 3196.877434] do_page_fault+0x2e/0xe0
[ 3196.882942] page_fault+0x25/0x50
[ 3196.888121] RIP: 0010:_raw_spin_lock_bh+0x1b/0x30
[ 3196.894624] RSP: 0018:ffff90d3df243e90 EFLAGS: 00010246
[ 3196.901641] RAX: 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08
[ 3196.910595] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI: 0000000200001dc7
[ 3196.919541] RBP: ffff90d3df243e90 R08: ffff90d3df243f10 R09: ffff90d3df25a780
[ 3196.928459] R10: ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7
[ 3196.937309] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15: ffff90d1317a6c18
[ 3196.946097] ? tipc_subscrb_release_cb+0x30/0x30 [tipc]
[ 3196.952912] ? tipc_subscrb_release_cb+0x30/0x30 [tipc]
[ 3196.959636] tipc_subscrp_timeout+0x25/0x90 [tipc]
[ 3196.965850] call_timer_fn+0x30/0x130
[ 3196.970861] run_timer_softirq+0x3fb/0x450
[ 3196.976282] ? lapic_next_deadline+0x26/0x30
[ 3196.981856] ? clockevents_program_event+0xd8/0x110
[ 3196.988004] __do_softirq+0xe4/0x2bb
[ 3196.992839] do_softirq_own_stack+0x2a/0x40
[ 3196.998277] </IRQ>
[ 3197.001619] do_softirq.part.17+0x50/0x60
[ 3197.006875] __local_bh_enable_ip+0x50/0x60
[ 3197.012300] _raw_spin_unlock_bh+0x1e/0x20
[ 3197.017639] tipc_sk_rcv+0x15a/0x440 [tipc]
[ 3197.023057] tipc_node_xmit+0x147/0x190 [tipc]
[ 3197.028715] __tipc_sendstream+0x1a9/0x2d0 [tipc]
[ 3197.034618] ? aa_sk_perm+0x48/0x1a0
[ 3197.039384] tipc_sendstream+0x35/0x50 [tipc]
[ 3197.044919] tipc_send_packet+0x17/0x20 [tipc]
[ 3197.050539] sock_sendmsg+0x3e/0x50
[ 3197.055207] kernel_sendmsg+0x2b/0x30
[ 3197.060044] tipc_send_to_sock+0x12f/0x290 [tipc]
[ 3197.065927] ? __switch_to_asm+0x40/0x70
[ 3197.071048] ? __switch_to_asm+0x34/0x70
[ 3197.076162] ? __switch_to_asm+0x34/0x70
[ 3197.081257] ? __switch_to_asm+0x40/0x70
[ 3197.086345] ? __switch_to_asm+0x34/0x70
[ 3197.091425] tipc_send_work+0x23/0x40 [tipc]
[ 3197.096854] process_one_work+0x1de/0x410
[ 3197.102030] worker_thread+0x32/0x410
[ 3197.106854] kthread+0x121/0x140
[ 3197.111239] ? process_one_work+0x410/0x410
[ 3197.116576] ? kthread_create_worker_on_cpu+0x70/0x70
[ 3197.122788] ? do_syscall_64+0x73/0x130
[ 3197.127786] ? SyS_exit_group+0x14/0x20
[ 3197.132784] ret_from_fork+0x35/0x40
[ 3197.137508] Code: 2d 61 60 01 73 17 48 8b 05 84 21 16 01 be fd 00
00 00 48 8b 40 30 e8 36 b6 ba 00 5d c3 89 fe 48 c7 c7 60 fe ab 8c e8
06 48 03 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec
20 65
[ 3197.158946] ---[ end trace 4ce4054b7f724f83 ]---
[ 3197.164963] unchecked MSR access error: WRMSR to 0x83f (tried to
write 0x00000000000000f6) at rIP: 0xffffffff8ba6c1a8
(native_write_msr+0x8/0x30)
[ 3197.179360] Call Trace:
[ 3197.183165] <IRQ>
[ 3197.186532] native_apic_msr_write+0x2b/0x40
[ 3197.192143] x2apic_send_IPI_self+0x20/0x30
[ 3197.197660] arch_irq_work_raise+0x2a/0x40
[ 3197.203080] irq_work_queue+0x8d/0xa0
[ 3197.208058] dbs_update_util_handler+0x80/0xa0
[ 3197.213822] update_load_avg+0x5c5/0x6e0
[ 3197.219068] enqueue_entity+0x5e/0x6b0
[ 3197.224124] enqueue_task_fair+0x6c/0x7f0
[ 3197.229449] ? __update_load_avg_blocked_se.isra.37+0xd1/0x150
[ 3197.236608] activate_task+0x54/0xc0
[ 3197.241510] ? sched_clock+0x9/0x10
[ 3197.246338] ttwu_do_activate+0x49/0x80
[ 3197.251514] try_to_wake_up+0x1d6/0x480
[ 3197.256684] default_wake_function+0x12/0x20
[ 3197.262288] autoremove_wake_function+0x12/0x40
[ 3197.268162] __wake_up_common+0x73/0x130
[ 3197.273428] __wake_up_common_lock+0x80/0xc0
[ 3197.279026] __wake_up+0x13/0x20
[ 3197.283562] wake_up_klogd_work_func+0x40/0x60
[ 3197.289321] irq_work_run_list+0x52/0x80
[ 3197.294545] ? tick_sched_do_timer+0x50/0x50
[ 3197.300116] irq_work_tick+0x3b/0x50
[ 3197.304987] update_process_times+0x42/0x60
[ 3197.310455] tick_sched_handle+0x26/0x70
[ 3197.315672] ? tick_sched_do_timer+0x50/0x50
[ 3197.321233] tick_sched_timer+0x39/0x80
[ 3197.326351] __hrtimer_run_queues+0xdf/0x230
[ 3197.331909] hrtimer_interrupt+0xa0/0x1d0
[ 3197.337205] smp_apic_timer_interrupt+0x6f/0x130
[ 3197.343108] apic_timer_interrupt+0x84/0x90
[ 3197.348574] RIP: 0010:panic+0x1fe/0x244
[ 3197.353674] RSP: 0018:ffff90d3df243bd8 EFLAGS: 00000286 ORIG_RAX:
ffffffffffffff11
[ 3197.362537] RAX: 0000000000000041 RBX: 0000000000000000 RCX: 0000000000000006
[ 3197.370972] RDX: 0000000000000000 RSI: 0000000000000092 RDI: ffff90d3df256490
[ 3197.379405] RBP: ffff90d3df243c50 R08: 0000000000000000 R09: 000000000000113b
[ 3197.387843] R10: 0000000000000008 R11: 00000000ffffffff R12: 0000000000000000
[ 3197.396291] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000002
[ 3197.404743] ? panic+0x1f7/0x244
[ 3197.409283] oops_end+0xb6/0xd0
[ 3197.413730] no_context+0x17c/0x440
[ 3197.418521] __bad_area_nosemaphore+0xf3/0x1b0
[ 3197.424273] bad_area_nosemaphore+0x14/0x20
[ 3197.429763] __do_page_fault+0xd4/0x4d0
[ 3197.434912] ? update_load_avg+0x57f/0x6e0
[ 3197.440318] do_page_fault+0x2e/0xe0
[ 3197.445203] page_fault+0x25/0x50
[ 3197.449827] RIP: 0010:_raw_spin_lock_bh+0x1b/0x30
[ 3197.455862] RSP: 0018:ffff90d3df243e90 EFLAGS: 00010246
[ 3197.462432] RAX: 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08
[ 3197.470940] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI: 0000000200001dc7
[ 3197.479454] RBP: ffff90d3df243e90 R08: ffff90d3df243f10 R09: ffff90d3df25a780
[ 3197.487973] R10: ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7
[ 3197.496495] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15: ffff90d1317a6c18
[ 3197.505043] ? tipc_subscrb_release_cb+0x30/0x30 [tipc]
[ 3197.511689] ? tipc_subscrb_release_cb+0x30/0x30 [tipc]
[ 3197.518319] tipc_subscrp_timeout+0x25/0x90 [tipc]
[ 3197.524511] call_timer_fn+0x30/0x130
[ 3197.529573] run_timer_softirq+0x3fb/0x450
[ 3197.535073] ? lapic_next_deadline+0x26/0x30
[ 3197.540684] ? clockevents_program_event+0xd8/0x110
[ 3197.546832] __do_softirq+0xe4/0x2bb
[ 3197.551680] do_softirq_own_stack+0x2a/0x40
[ 3197.557086] </IRQ>
[ 3197.560362] do_softirq.part.17+0x50/0x60
[ 3197.565561] __local_bh_enable_ip+0x50/0x60
[ 3197.570937] _raw_spin_unlock_bh+0x1e/0x20
[ 3197.576228] tipc_sk_rcv+0x15a/0x440 [tipc]
[ 3197.581602] tipc_node_xmit+0x147/0x190 [tipc]
[ 3197.587245] __tipc_sendstream+0x1a9/0x2d0 [tipc]
[ 3197.593144] ? aa_sk_perm+0x48/0x1a0
[ 3197.597909] tipc_sendstream+0x35/0x50 [tipc]
[ 3197.603461] tipc_send_packet+0x17/0x20 [tipc]
[ 3197.609101] sock_sendmsg+0x3e/0x50
[ 3197.613767] kernel_sendmsg+0x2b/0x30
[ 3197.618609] tipc_send_to_sock+0x12f/0x290 [tipc]
[ 3197.624497] ? __switch_to_asm+0x40/0x70
[ 3197.629601] ? __switch_to_asm+0x34/0x70
[ 3197.634697] ? __switch_to_asm+0x34/0x70
[ 3197.639781] ? __switch_to_asm+0x40/0x70
[ 3197.644846] ? __switch_to_asm+0x34/0x70
[ 3197.649909] tipc_send_work+0x23/0x40 [tipc]
[ 3197.655317] process_one_work+0x1de/0x410
[ 3197.660470] worker_thread+0x32/0x410
[ 3197.665274] kthread+0x121/0x140
[ 3197.669636] ? process_one_work+0x410/0x410
[ 3197.674955] ? kthread_create_worker_on_cpu+0x70/0x70
[ 3197.681146] ? do_syscall_64+0x73/0x130
[ 3197.686118] ? SyS_exit_group+0x14/0x20
[ 3197.691094] ret_from_fork+0x35/0x40
pe 26. lokak. 2018 klo 6.38 Ying Xue ([email protected]) kirjoitti:
>
> Hi Juhamatti,
>
> This is a good test plan indeed. If you encounter any problem during the
> testing, please share it here.
>
> From my view, TIPC had fully supported networking nemespace at least
> three years ago and its sockets could be fully isolated between containers.
>
> Thanks,
> Ying
>
> On 10/26/2018 11:12 AM, [email protected] wrote:
> > Hello,
> >
> > I'm planning to test TIPC scalability in more detail, especially
> > regarding the binding table and topology (service tracking) service
> > with large number of nodes (~1k) and sockets (100-1k). Is it enough to
> > do it with LXC containers with isolated namespaces to get realistic
> > results? My concern here is that if TIPC implementation would not
> > provide full isolation to sockets e.g. for performance reasons inside
> > the same kernel, then this may not provide results matching to real
> > environments. VMs are the other choice, but they tend to be heavy. Are
> > the namespaces fully isolated inside the kernel?
> >
> > Thanks for the info,
> > --
> > Juhamatti
> >
> >
> > _______________________________________________
> > tipc-discussion mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/tipc-discussion
> >
_______________________________________________
tipc-discussion mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tipc-discussion
