Re: [Qemu-devel] vhost-net thread getting stuck ?
On 01/10/2013 04:25 AM, Chegu Vinod wrote: Hello, 'am running into an issue with the latest bits. [ Pl. see below. The vhost thread seems to be getting stuck while trying to memcopy...perhaps a bad address?. ] Wondering if this is a known issue or some recent regression ? Hi: Looks like the issue has been fixed in following commits, does you tree contain these? 499744209b2cbca66c42119226e5470da3bb7040 and 76fe45812a3b134c39170ca32dfd4b7217d33145. They have been merged in to Linus 3.8-rc tree. Thanks 'am using the latest qemu (from qemu.git) and the latest kvm.git kernel on the host. Started the guest using the following command line /usr/local/bin/qemu-system-x86_64 \ -enable-kvm \ -cpu host \ -smp sockets=8,cores=10,threads=1 \ -numa node,nodeid=0,cpus=0-9,mem=64g \ -numa node,nodeid=1,cpus=10-19,mem=64g \ -numa node,nodeid=2,cpus=20-29,mem=64g \ -numa node,nodeid=3,cpus=30-39,mem=64g \ -numa node,nodeid=4,cpus=40-49,mem=64g \ -numa node,nodeid=5,cpus=50-59,mem=64g \ -numa node,nodeid=6,cpus=60-69,mem=64g \ -numa node,nodeid=7,cpus=70-79,mem=64g \ -m 524288 \ -mem-path /dev/hugepages \ -name vm2 \ -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm2.monitor,server,now ait \ -drive file=/dev/libvirt_lvm2/vm2,if=none,id=drive-virtio-disk0,format=raw,cache =none,aio=native \ -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=v irtio-disk0,bootindex=1 \ -monitor stdio \ -net nic,model=virtio,macaddr=52:54:00:71:01:02,netdev=nic-0 \ -netdev tap,id=nic-0,ifname=tap0,script=no,downscript=no,vhost=on \ -vnc :4 Was just doing a basic kernel build in the guest when it hung with the following in the dmesg of the host. Thanks Vinod BUG: soft lockup - CPU#46 stuck for 23s! [vhost-135220:135231] Modules linked in: kvm_intel kvm fuse ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc autofs4 sunrpc pcc_cpufreq ipv6 vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp crc32c_intel ghash_clmulni_intel microcode pcspkr mlx4_core be2net lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core sg netxen_nic ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul pata_acpi ata_generic ata_piix hpsa lpfc scsi_transport_fc scsi_tgt radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: kvm] CPU 46 Pid: 135231, comm: vhost-135220 Not tainted 3.7.0+ #1 HP ProLiant DL980 G7 RIP: 0010:[8147bab0] [8147bab0] skb_flow_dissect+0x1b0/0x440 RSP: 0018:881ffd131bc8 EFLAGS: 0246 RAX: 8a1f7dc70c00 RBX: RCX: 7fa0 RDX: RSI: 881ffd131c68 RDI: 8a1ff1bd6c80 RBP: 881ffd131c58 R08: 881ffd131bf8 R09: 8a1ff1bd6c80 R10: 0010 R11: 0004 R12: 8a1ff1bd6c80 R13: 000b R14: 8147330b R15: 881ffd131b58 FS: () GS:8a1fff98() knlGS: CS: 0010 DS: ES: CR0: 8005003b CR2: 003d5c810dc0 CR3: 009f77c04000 CR4: 27e0 DR0: DR1: DR2: DR3: DR6: 0ff0 DR7: 0400 Process vhost-135220 (pid: 135231, threadinfo 881ffd13, task 881ffcb754c0) Stack: 881ffd131c18 81477b90 00e2 2b289bcc58ce 881ffd131ce4 00a2 00a2 00a2 00a2 881ffd131c88 937e754e Call Trace: [81477b90] ? memcpy_fromiovecend+0x90/0xd0 [8147f3ca] __skb_get_rxhash+0x1a/0xe0 [a03c90f8] tun_get_user+0x468/0x660 [tun] [81090010] ? __sdt_alloc+0x80/0x1a0 [a03c934d] tun_sendmsg+0x5d/0x80 [tun] [a0468e8a] handle_tx+0x34a/0x680 [vhost_net] [a04691f5] handle_tx_kick+0x15/0x20 [vhost_net] [a0466dfc] vhost_worker+0x10c/0x1c0 [vhost_net] [a0466cf0] ? vhost_attach_cgroups_work+0x30/0x30 [vhost_net] [a0466cf0] ? vhost_attach_cgroups_work+0x30/0x30 [vhost_net] [8107ecfe] kthread+0xce/0xe0 [8107ec30] ? kthread_freezable_should_stop+0x70/0x70 [815537ac] ret_from_fork+0x7c/0xb0 [8107ec30] ? kthread_freezable_should_stop+0x70/0x70 Code: b6 50 06 48 89 ce 48 c1 ee 20 31 f1 41 89 0e 48 8b 48 20 48 33 48 18 48 89 c8 48 c1 e8 20 31 c1 41 89 4e 04 e9 35 ff ff ff 66 90 0f b6 50 09 e9 1a ff ff ff 0f 1f 80 00 00 00 00 41 8b 44 24 68 [root@hydra11 kvm_rik]# Message from syslogd@hydra11 at Jan 9 13:06:58 ... kernel:BUG: soft lockup - CPU#46 stuck for 22s! [vhost-135220:135231] -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to
Re: [Qemu-devel] vhost-net thread getting stuck ?
On 1/9/2013 8:35 PM, Jason Wang wrote: On 01/10/2013 04:25 AM, Chegu Vinod wrote: Hello, 'am running into an issue with the latest bits. [ Pl. see below. The vhost thread seems to be getting stuck while trying to memcopy...perhaps a bad address?. ] Wondering if this is a known issue or some recent regression ? Hi: Looks like the issue has been fixed in following commits, does you tree contain these? 499744209b2cbca66c42119226e5470da3bb7040 and 76fe45812a3b134c39170ca32dfd4b7217d33145. They have been merged in to Linus 3.8-rc tree. I was using kvm.git kernel (as of today morning)looks like the fixes aren't there yet. Will try the Linus's 3.8-rc tree. Thanks! Vinod Thanks 'am using the latest qemu (from qemu.git) and the latest kvm.git kernel on the host. Started the guest using the following command line /usr/local/bin/qemu-system-x86_64 \ -enable-kvm \ -cpu host \ -smp sockets=8,cores=10,threads=1 \ -numa node,nodeid=0,cpus=0-9,mem=64g \ -numa node,nodeid=1,cpus=10-19,mem=64g \ -numa node,nodeid=2,cpus=20-29,mem=64g \ -numa node,nodeid=3,cpus=30-39,mem=64g \ -numa node,nodeid=4,cpus=40-49,mem=64g \ -numa node,nodeid=5,cpus=50-59,mem=64g \ -numa node,nodeid=6,cpus=60-69,mem=64g \ -numa node,nodeid=7,cpus=70-79,mem=64g \ -m 524288 \ -mem-path /dev/hugepages \ -name vm2 \ -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm2.monitor,server,now ait \ -drive file=/dev/libvirt_lvm2/vm2,if=none,id=drive-virtio-disk0,format=raw,cache =none,aio=native \ -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=v irtio-disk0,bootindex=1 \ -monitor stdio \ -net nic,model=virtio,macaddr=52:54:00:71:01:02,netdev=nic-0 \ -netdev tap,id=nic-0,ifname=tap0,script=no,downscript=no,vhost=on \ -vnc :4 Was just doing a basic kernel build in the guest when it hung with the following in the dmesg of the host. Thanks Vinod BUG: soft lockup - CPU#46 stuck for 23s! [vhost-135220:135231] Modules linked in: kvm_intel kvm fuse ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc autofs4 sunrpc pcc_cpufreq ipv6 vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp crc32c_intel ghash_clmulni_intel microcode pcspkr mlx4_core be2net lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core sg netxen_nic ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul pata_acpi ata_generic ata_piix hpsa lpfc scsi_transport_fc scsi_tgt radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: kvm] CPU 46 Pid: 135231, comm: vhost-135220 Not tainted 3.7.0+ #1 HP ProLiant DL980 G7 RIP: 0010:[8147bab0] [8147bab0] skb_flow_dissect+0x1b0/0x440 RSP: 0018:881ffd131bc8 EFLAGS: 0246 RAX: 8a1f7dc70c00 RBX: RCX: 7fa0 RDX: RSI: 881ffd131c68 RDI: 8a1ff1bd6c80 RBP: 881ffd131c58 R08: 881ffd131bf8 R09: 8a1ff1bd6c80 R10: 0010 R11: 0004 R12: 8a1ff1bd6c80 R13: 000b R14: 8147330b R15: 881ffd131b58 FS: () GS:8a1fff98() knlGS: CS: 0010 DS: ES: CR0: 8005003b CR2: 003d5c810dc0 CR3: 009f77c04000 CR4: 27e0 DR0: DR1: DR2: DR3: DR6: 0ff0 DR7: 0400 Process vhost-135220 (pid: 135231, threadinfo 881ffd13, task 881ffcb754c0) Stack: 881ffd131c18 81477b90 00e2 2b289bcc58ce 881ffd131ce4 00a2 00a2 00a2 00a2 881ffd131c88 937e754e Call Trace: [81477b90] ? memcpy_fromiovecend+0x90/0xd0 [8147f3ca] __skb_get_rxhash+0x1a/0xe0 [a03c90f8] tun_get_user+0x468/0x660 [tun] [81090010] ? __sdt_alloc+0x80/0x1a0 [a03c934d] tun_sendmsg+0x5d/0x80 [tun] [a0468e8a] handle_tx+0x34a/0x680 [vhost_net] [a04691f5] handle_tx_kick+0x15/0x20 [vhost_net] [a0466dfc] vhost_worker+0x10c/0x1c0 [vhost_net] [a0466cf0] ? vhost_attach_cgroups_work+0x30/0x30 [vhost_net] [a0466cf0] ? vhost_attach_cgroups_work+0x30/0x30 [vhost_net] [8107ecfe] kthread+0xce/0xe0 [8107ec30] ? kthread_freezable_should_stop+0x70/0x70 [815537ac] ret_from_fork+0x7c/0xb0 [8107ec30] ? kthread_freezable_should_stop+0x70/0x70 Code: b6 50 06 48 89 ce 48 c1 ee 20 31 f1 41 89 0e 48 8b 48 20 48 33 48 18 48 89 c8 48 c1 e8 20 31 c1 41 89 4e 04 e9 35 ff ff ff 66 90 0f b6 50 09 e9 1a ff ff ff 0f 1f 80 00 00 00 00 41 8b 44 24 68 [root@hydra11 kvm_rik]# Message from syslogd@hydra11 at Jan 9 13:06:58 ... kernel:BUG: soft lockup - CPU#46 stuck for 22s! [vhost-135220:135231] .
