Hi,
In an arp flooding network environment, One server running ovs crushed with
soft lockup bug.
Attached is the system log.
ovs version: 2.1.3
Linux kernel version:3.10.0-123.el7.x86_64, Centos7
Thanks,
-Fan
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: irq 116: nobody cared (try booting
with the "irqpoll" option)
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: CPU: 0 PID: 0 Comm: swapper/0 Not
tainted 3.10.0-123.el7.x86_64 #1
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: Hardware name: Powerleader
PR2510R/S26D3-F, BIOS 3.0a 04/16/2014
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: ffff88065adce100 c8248df6471e4986
ffff880667c03e40 ffffffff815e19ba
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: ffff880667c03e68 ffffffff810f96e2
ffff88065adce100 0000000000000074
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: 0000000000000000 ffff880667c03ea8
ffffffff810f9b02 c8248df6471e4986
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: Call Trace:
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: <IRQ> [<ffffffff815e19ba>]
dump_stack+0x19/0x1b
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff810f96e2>]
__report_bad_irq+0x32/0xd0
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff810f9b02>]
note_interrupt+0x132/0x1f0
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff810f7221>]
handle_irq_event_percpu+0xe1/0x1e0
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff810f735d>]
handle_irq_event+0x3d/0x60
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff810f9fe7>]
handle_edge_irq+0x77/0x130
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff81014c3f>]
handle_irq+0xbf/0x150
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff815ed78a>] ?
atomic_notifier_call_chain+0x1a/0x20
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff815f434f>] do_IRQ+0x4f/0xf0
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff815e94ad>]
common_interrupt+0x6d/0x6d
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: <EOI> [<ffffffff814834e2>] ?
cpuidle_enter_state+0x52/0xc0
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff81483615>]
cpuidle_idle_call+0xc5/0x200
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff8101bc7e>]
arch_cpu_idle+0xe/0x30
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff810b4725>]
cpu_startup_entry+0xf5/0x290
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff815c3927>]
rest_init+0x77/0x80
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff81a06fa7>]
start_kernel+0x429/0x44a
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff81a06987>] ?
repair_env_string+0x5c/0x5c
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff81a06120>] ?
early_idt_handlers+0x120/0x120
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff81a065ee>]
x86_64_start_reservations+0x2a/0x2c
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffff81a06742>]
x86_64_start_kernel+0x152/0x175
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: handlers:
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: [<ffffffffa07ed4c0>] e1000_msix_other
[e1000e]
Mar 2 15:02:21 ml01-cloud-kvm012 kernel: Disabling IRQ #116
Mar 2 15:02:22 ml01-cloud-kvm012 sh: abrt-dump-oops: Found oopses: 1
Mar 2 15:02:22 ml01-cloud-kvm012 sh: abrt-dump-oops: Creating problem
directories
Mar 2 15:02:23 ml01-cloud-kvm012 abrt-dump-oops: Reported 1 kernel oopses to
Abrt
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: BUG: soft lockup - CPU#4 stuck for
22s! [flow_dumper:6610]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: Modules linked in: nfnetlink_queue
nfnetlink_log bluetooth rfkill nf_conntrack_netlink nfnetlink ip_vs_rr ip_vs
xt_nat xt_REDIRECT iptable_raw veth openvswitch vxlan ip_tunnel gre fuse btrfs
zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 binfmt_misc
ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle ip6table_filter
ip6_tables iptable_filter ip_tables ebtable_nat ebtables bridge stp llc sg
coretemp mxm_wmi iTCO_wdt iTCO_vendor_support e1000e kvm_intel kvm
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw
sb_edac gf128mul glue_helper ablk_helper cryptd edac_core ptp wmi i2c_i801
pcspkr pps_core lpc_ich ioatdma mperf dm_mirror dm_region_hash dm_log shpchp
dm_mod
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: ipmi_devintf mfd_core mei_me mei dca
ipmi_si ipmi_msghandler nfsd auth_rpcgss nfs_acl lockd sunrpc xfs libcrc32c
sd_mod crc_t10dif crct10dif_common mgag200 syscopyarea sysfillrect sysimgblt
i2c_algo_bit drm_kms_helper ttm isci drm libsas ahci libahci scsi_transport_sas
libata i2c_core
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: CPU: 4 PID: 6610 Comm: flow_dumper
Not tainted 3.10.0-123.el7.x86_64 #1
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: Hardware name: Powerleader
PR2510R/S26D3-F, BIOS 3.0a 04/16/2014
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: task: ffff88065ac7b8e0 ti:
ffff8803465b8000 task.ti: ffff8803465b8000
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: RIP: 0010:[<ffffffff815e90ea>]
[<ffffffff815e90ea>] _raw_spin_lock+0x3a/0x50
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: RSP: 0018:ffff880667d03b30 EFLAGS:
00000206
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: RAX: 00000000000035a6 RBX:
0000000000000000 RCX: 0000000000001614
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: RDX: 0000000000001616 RSI:
0000000000001616 RDI: ffff880bd91fa158
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: RBP: ffff880667d03b30 R08:
ffff880bca0dac18 R09: ffff880667d03a78
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: R10: 0000000000000010 R11:
ffff88054516af20 R12: ffff880667d03aa8
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: R13: ffffffff815f2d9d R14:
ffff880667d03b30 R15: ffff880bd91fa140
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: FS: 00007f5429ffb700(0000)
GS:ffff880667d00000(0000) knlGS:0000000000000000
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: CR2: 00007fb2d6df1650 CR3:
0000000bdc5fc000 CR4: 00000000001407e0
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: Stack:
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: ffff880667d03b60 ffffffffa073329f
ffff8805bfd67300 ffffe8f9e7d06800
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: ffff8805e71df780 ffff880667d03b98
ffff880667d03c48 ffffffffa07329a4
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: 00000002048c6000 ffff880667d03be8
ffff880667d03bc8 ffff88053f9d2100
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: Call Trace:
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: <IRQ>
Mar 2 15:02:37 ml01-cloud-kvm012 kernel:
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa073329f>]
ovs_flow_stats_update+0x4f/0xd0 [openvswitch]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa07329a4>]
ovs_dp_process_received_packet+0x84/0x120 [openvswitch]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa073901a>]
ovs_vport_receive+0x2a/0x30 [openvswitch]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa0739f21>]
netdev_frame_hook+0xc1/0x120 [openvswitch]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814cf9c2>]
__netif_receive_skb_core+0x282/0x870
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff8101a0d9>] ?
read_tsc+0x9/0x20
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814cffc8>]
__netif_receive_skb+0x18/0x60
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814d0050>]
netif_receive_skb+0x40/0xd0
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814d0aa8>]
napi_gro_receive+0x58/0x80
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa07e854f>]
e1000_receive_skb+0x7f/0xe0 [e1000e]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa07e9b6a>]
e1000_clean_rx_irq+0x24a/0x400 [e1000e]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa07f17ac>]
e1000e_poll+0xbc/0x330 [e1000e]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814d041a>]
net_rx_action+0x15a/0x250
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff81067047>]
__do_softirq+0xf7/0x290
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff815f3a5c>]
call_softirq+0x1c/0x30
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff81014d25>]
do_softirq+0x55/0x90
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff810673e5>]
irq_exit+0x115/0x120
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff815f4358>] do_IRQ+0x58/0xf0
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff815e94ad>]
common_interrupt+0x6d/0x6d
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: <EOI>
Mar 2 15:02:37 ml01-cloud-kvm012 kernel:
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff812dab90>] ?
__nla_put+0x20/0x30
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff815e90c2>] ?
_raw_spin_lock+0x12/0x50
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa0736a8d>] ?
ovs_nla_put_flow+0x29d/0x6e0 [openvswitch]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa0733453>]
ovs_flow_stats_get+0x133/0x180 [openvswitch]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa07307b7>]
ovs_flow_cmd_fill_info+0x1c7/0x320 [openvswitch]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffffa0730999>]
ovs_flow_cmd_dump+0x89/0xf0 [openvswitch]
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814fb5ae>]
netlink_dump+0x7e/0x230
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814fba80>]
netlink_recvmsg+0x320/0x3e0
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814ffee1>] ?
genl_rcv_msg+0x91/0xd0
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814b807f>]
sock_recvmsg+0xbf/0x100
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff812b8655>] ?
cpumask_next_and+0x35/0x50
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814b838e>]
___sys_recvmsg+0x11e/0x2b0
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff810c29f2>] ?
do_futex+0x172/0x5e0
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814b8f41>]
__sys_recvmsg+0x51/0x90
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff814b8f92>]
SyS_recvmsg+0x12/0x20
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: [<ffffffff815f2119>]
system_call_fastpath+0x16/0x1b
Mar 2 15:02:37 ml01-cloud-kvm012 kernel: Code: 0f c1 07 89 c2 c1 ea 10 66 39
c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 83
e8 01 74 0a 0f b7 0f <66> 39 ca 75 f1 5d c3 0f 1f 80 00 00 00 00 eb da 66 0f 1f
44 00
Mar 2 15:02:37 ml01-cloud-kvm012 sh: abrt-dump-oops: Found oopses: 1
Mar 2 15:02:37 ml01-cloud-kvm012 sh: abrt-dump-oops: Creating problem
directories
Mar 2 15:02:37 ml01-cloud-kvm012 abrt-server: Lock file
'/var/tmp/abrt/post-create.lock' is locked by process 16820
_______________________________________________
discuss mailing list
[email protected]
http://openvswitch.org/mailman/listinfo/discuss
