[Bug 231117] I/O lockups inside bhyve vms
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231117 Mateusz Kwiatkowski changed: What|Removed |Added CC||kwia...@panic.pl --- Comment #19 from Mateusz Kwiatkowski --- I have very similar problem as described in this issue: I/O in guests hangs. I've experienced this with FreeBSD 11.2, 12.0 (both with ZFS inside) and Ubuntu 18.04 (ext4) guests. This started happening after migrating guests from old hypervisor running 12.0 on Xeon, to the new on running CURRENT (r347183) on AMD Epyc. On old hypervisor these VMs were running stable for couple of months. On the new hypervisor there's plenty of free resources. Swap is disabled. Mem: 3761M Active, 1636M Inact, 5802M Wired, 51G Free ARC: 4000M Total, 487M MFU, 3322M MRU, 3456K Anon, 129M Header, 59M Other 2228M Compressed, 3202M Uncompressed, 1.44:1 Ratio vfs.zfs.arc_min: 8215740416 vfs.zfs.arc_max: 52582796492 Procstat from bhyve provess: root@utgard:~ # procstat -kk 95379 PIDTID COMMTDNAME KSTACK 95379 101075 bhyve mevent mi_switch+0x174 sleepq_switch+0x110 sleepq_catch_signals+0x3e7 sleepq_wait_sig+0xf _sleep+0x2d0 kqueue_kevent+0xa94 kern_kevent_fp+0x95 kern_kevent+0x9f kern_kevent_generic+0x70 sys_kevent+0x61 amd64_syscall+0x276 fast_syscall_common+0x101 95379 101258 bhyve blk-4:0-0 mi_switch+0x174 sleepq_switch+0x110 sleepq_catch_signals+0x3e7 sleepq_wait_sig+0xf _sleep+0x2d0 umtxq_sleep+0x153 do_wait+0x206 __umtx_op_wait_uint_private+0x7e amd64_syscall+0x276 fast_syscall_common+0x101 95379 101259 bhyve blk-4:0-1 mi_switch+0x174 sleepq_switch+0x110 sleepq_catch_signals+0x3e7 sleepq_wait_sig+0xf _sleep+0x2d0 umtxq_sleep+0x153 do_wait+0x206 __umtx_op_wait_uint_private+0x7e amd64_syscall+0x276 fast_syscall_common+0x101 95379 101260 bhyve blk-4:0-2 mi_switch+0x174 sleepq_switch+0x110 sleepq_catch_signals+0x3e7 sleepq_wait_sig+0xf _sleep+0x2d0 umtxq_sleep+0x153 do_wait+0x206 __umtx_op_wait_uint_private+0x7e amd64_syscall+0x276 fast_syscall_common+0x101 95379 101261 bhyve blk-4:0-3 mi_switch+0x174 sleepq_switch+0x110 sleepq_catch_signals+0x3e7 sleepq_wait_sig+0xf _sleep+0x2d0 umtxq_sleep+0x153 do_wait+0x206 __umtx_op_wait_uint_private+0x7e amd64_syscall+0x276 fast_syscall_common+0x101 95379 101262 bhyve blk-4:0-4 mi_switch+0x174 sleepq_switch+0x110 sleepq_catch_signals+0x3e7 sleepq_wait_sig+0xf _sleep+0x2d0 umtxq_sleep+0x153 do_wait+0x206 __umtx_op_wait_uint_private+0x7e amd64_syscall+0x276 fast_syscall_common+0x101 95379 101263 bhyve blk-4:0-5 mi_switch+0x174 sleepq_switch+0x110 sleepq_catch_signals+0x3e7 sleepq_wait_sig+0xf _sleep+0x2d0 umtxq_sleep+0x153 do_wait+0x206 __umtx_op_wait_uint_private+0x7e amd64_syscall+0x276 fast_syscall_common+0x101 95379 101264 bhyve blk-4:0-6 mi_switch+0x174 sleepq_switch+0x110 sleepq_catch_signals+0x3e7 sleepq_wait_sig+0xf _sleep+0x2d0 umtxq_sleep+0x153 do_wait+0x206 __umtx_op_wait_uint_private+0x7e amd64_syscall+0x276 fast_syscall_common+0x101 95379 101265 bhyve blk-4:0-7 mi_switch+0x174 sleepq_switch+0x110 sleepq_catch_signals+0x3e7 sleepq_wait_sig+0xf _sleep+0x2d0 umtxq_sleep+0x153 do_wait+0x206 __umtx_op_wait_uint_private+0x7e amd64_syscall+0x276 fast_syscall_common+0x101 95379 101266 bhyve vtnet-5:0 txmi_switch+0x174 sleepq_switch+0x110 sleepq_catch_signals+0x3e7 sleepq_wait_sig+0xf _sleep+0x2d0 umtxq_sleep+0x153 do_wait+0x206 __umtx_op_wait_uint_private+0x7e amd64_syscall+0x276 fast_syscall_common+0x101 95379 101267 bhyve vcpu 0 mi_switch+0x174 sleepq_switch+0x110 sleepq_timedwait+0x4f msleep_spin_sbt+0x144 vm_run+0x970 vmmdev_ioctl+0x7ea devfs_ioctl+0xca VOP_IOCTL_APV+0x63 vn_ioctl+0x124 devfs_ioctl_f+0x1f kern_ioctl+0x28a sys_ioctl+0x15d amd64_syscall+0x276 fast_syscall_common+0x101 95379 101268 bhyve vcpu 1 mi_switch+0x174 sleepq_switch+0x110 sleepq_timedwait+0x4f msleep_spin_sbt+0x144 vm_run+0x970 vmmdev_ioctl+0x7ea devfs_ioctl+0xca VOP_IOCTL_APV+0x63 vn_ioctl+0x124 devfs_ioctl_f+0x1f kern_ioctl+0x28a sys_ioctl+0x15d amd64_syscall+0x276 fast_syscall_common+0x101 I will be happy to provide more information to help solving this issue. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Bug 231117] I/O lockups inside bhyve vms
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231117 --- Comment #20 from Mateusz Kwiatkowski --- Output of procstat -kk -a | grep zfs: # procstat -kk -a | grep zfs 0 100403 kernel zfsvfs mi_switch+0x174 sleepq_switch+0x110 sleepq_wait+0x43 _sleep+0x2da taskqueue_thread_loop+0xe1 fork_exit+0x84 fork_trampoline+0xe 0 100771 kernel zfs_vn_rele_taskq mi_switch+0x174 sleepq_switch+0x110 sleepq_wait+0x43 _sleep+0x2da taskqueue_thread_loop+0xe1 fork_exit+0x84 fork_trampoline+0xe 46 100365 zfskern solthread 0xfff mi_switch+0x174 sleepq_switch+0x110 sleepq_timedwait+0x4f _cv_timedwait_sbt+0x185 zthr_procedure+0x12c fork_exit+0x84 fork_trampoline+0xe 46 100366 zfskern solthread 0xfff mi_switch+0x174 sleepq_switch+0x110 sleepq_timedwait+0x4f _cv_timedwait_sbt+0x185 zthr_procedure+0x12c fork_exit+0x84 fork_trampoline+0xe 46 100367 zfskern arc_dnlc_evicts_thr mi_switch+0x174 sleepq_switch+0x110 sleepq_wait+0x43 _cv_wait+0x15b arc_dnlc_evicts_thread+0x133 fork_exit+0x84 fork_trampoline+0xe 46 100369 zfskern dbuf_evict_thread mi_switch+0x174 sleepq_switch+0x110 sleepq_timedwait+0x4f _cv_timedwait_sbt+0x185 dbuf_evict_thread+0x296 fork_exit+0x84 fork_trampoline+0xe 46 100402 zfskern l2arc_feed_thread mi_switch+0x174 sleepq_switch+0x110 sleepq_timedwait+0x4f _cv_timedwait_sbt+0x185 l2arc_feed_thread+0x1d5 fork_exit+0x84 fork_trampoline+0xe 46 100698 zfskern trim zroot mi_switch+0x174 sleepq_switch+0x110 sleepq_timedwait+0x4f _cv_timedwait_sbt+0x185 trim_thread+0xab fork_exit+0x84 fork_trampoline+0xe 46 100812 zfskern txg_thread_entermi_switch+0x174 sleepq_switch+0x110 sleepq_wait+0x43 _cv_wait+0x15b txg_thread_wait+0x9a txg_quiesce_thread+0x2a6 fork_exit+0x84 fork_trampoline+0xe 46 100813 zfskern txg_thread_entermi_switch+0x174 sleepq_switch+0x110 sleepq_timedwait+0x4f _cv_timedwait_sbt+0x185 txg_thread_wait+0x8d txg_sync_thread+0x53b fork_exit+0x84 fork_trampoline+0xe 46 100814 zfskern solthread 0xfff mi_switch+0x174 sleepq_switch+0x110 sleepq_wait+0x43 _cv_wait+0x15b zthr_procedure+0xeb fork_exit+0x84 fork_trampoline+0xe 46 100815 zfskern solthread 0xfff mi_switch+0x174 sleepq_switch+0x110 sleepq_wait+0x43 _cv_wait+0x15b zthr_procedure+0xeb fork_exit+0x84 fork_trampoline+0xe -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Bug 231117] bhyve: I/O lockups inside guests
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231117 Kubilay Kocak changed: What|Removed |Added Summary|I/O lockups inside bhyve|bhyve: I/O lockups inside |vms |guests Status|New |Open See Also||https://bugs.freebsd.org/bu ||gzilla/show_bug.cgi?id=2296 ||14 Keywords||performance -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Bug 231117] bhyve: I/O lockups inside guests
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231117 Rodney W. Grimes changed: What|Removed |Added CC||rgri...@freebsd.org Keywords|performance | --- Comment #21 from Rodney W. Grimes --- (In reply to Mateusz Kwiatkowski from comment #19) > On the new hypervisor there's plenty of free resources. Swap is disabled. > > Mem: 3761M Active, 1636M Inact, 5802M Wired, 51G Free > ARC: 4000M Total, 487M MFU, 3322M MRU, 3456K Anon, 129M Header, 59M Other > 2228M Compressed, 3202M Uncompressed, 1.44:1 Ratio > > vfs.zfs.arc_min: 8215740416 > vfs.zfs.arc_max: 52582796492 How much memory are your VM's using? Ie, what is the total of all the VM's you have running at any time? How much total memory does the system have? Your arc max is 52Gbytes, 4G is in use now, so it could try to use another 48G bytes, your system has 51G free under the above conditions so the arc could drive your system to 3G of free memory. I am concerned your host work load my at times drive your system into memory starvation unless these numbers are actually with all the system under load. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Bug 231117] bhyve: I/O lockups inside guests
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231117 --- Comment #22 from Mateusz Kwiatkowski --- (In reply to Rodney W. Grimes from comment #21) > How much memory are your VM's using? Ie, what is the total of all the VM's > you have > running at any time? How much total memory does the system have? VMs may use 8GBs of RAM in total. Hypervisor has 64GB in total. So VMs and ARC could create OOM condition but the statistics I provided were taken at the moment when one of the VMs hang. I'll reduce arc_max anyway to 8GB to make sure this isn't the cause. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
[Bug 231117] bhyve: I/O lockups inside guests
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231117 --- Comment #23 from Rodney W. Grimes --- (In reply to Mateusz Kwiatkowski from comment #22) > VMs may use 8GBs of RAM in total. Hypervisor has 64GB in total. So VMs and > ARC could create OOM condition but the statistics I provided were taken at > themoment when one of the VMs hang. I'll reduce arc_max anyway to 8GB to make > sure this isn't the cause. Not sure you need to cut arc_max to 8GB, but just make sure that arc_max + total(all VM's) + host usage is < than 64GB. I have been pointed at: If the guest(s) are using virtio-block, it might be interesting to check the status of the IO queues for those devices. The reported issues could be a case of https://smartos.org/bugview/OS-7613. (Posted as https://reviews.freebsd.org/D19501) I have pinged the D19501 review and it looks like some forward motion has happened there. You *may* if you can want to try the patch that is there now, knowing that it is about to be revised. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
sysutils/docker-freebsd ?
Dear Colleagues, Do you know if there are plans to revive sysutils/docker-freebsd? It has been "broken" for 2 months, and the FreeBSD Wiki says that docker-freebsd is the recommended way of running docker apps. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ signature.asc Description: PGP signature
Re: sysutils/docker-freebsd ?
yes there are plans, there is some coordinating going on at ircs://chat.freenode.net/#freebsd-docker but it's nothing impressive right now just a few users starting Sent from my a tiny pocket computer. > On May 7, 2019, at 22:22, Victor Sudakov wrote: > > Dear Colleagues, > > Do you know if there are plans to revive sysutils/docker-freebsd? It has > been "broken" for 2 months, and the FreeBSD Wiki says that > docker-freebsd is the recommended way of running docker apps. > > -- > Victor Sudakov, VAS4-RIPE, VAS47-RIPN > 2:5005/49@fidonet http://vas.tomsk.ru/ ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
