bhyve: corrupting zfs pools?
I am doing a simple experiment. I get FreeBSD image from here: ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz Then I run in bhyve with two additional "disks" created with truncate -s 4g: $ bhyveload -m 1G -d ~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test $ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img -s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s 3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l com2,/dev/nmdm0A -c 2 -m 1g test Note sectorsize=512/4096 options. Not sure if it's them that cause the trouble. Then, in the VM: $ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2 $ zfs create -p l2arc-test/ROOT/initial $ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f - Afterwards, zpool status -v reports no problem. But then I run zpool scrub and get the following in the end: $ zpool status -v pool: l2arc-test state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun 2 13:58:17 2015 config: NAMESTATE READ WRITE CKSUM l2arc-test ONLINE 0 0 9 mirror-0 ONLINE 0 018 vtbd1 ONLINE 0 025 vtbd2 ONLINE 0 023 errors: Permanent errors have been detected in the following files: /l2arc-test/ROOT/initial/usr/bin/svnlitesync /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz /l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a The same issue is reproducible with ahci-hd. My host system is a recent amd64 CURRENT as well. The hardware platform is AMD. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
bhyve: bhyveload, bhyve, bhyvectl --destroy
I am very new to bhyve, so sorry if I am asking something silly or obvious. I am using bhyve to speed up my testing and it seems that each time I need to restart a VM I need to go through the cycle of destroying it with bhyvectl --destroy, then re-loading a kernel with bhyveload and then actually booting the VM with bhyve. It seems that I have to do this even if I don't change th kernel between reboots. My first naive impression was that the point of bhyveload was to load the kernel once. Seems it ain't so? -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
Hi Andriy, I am very new to bhyve, so sorry if I am asking something silly or obvious. I am using bhyve to speed up my testing and it seems that each time I need to restart a VM I need to go through the cycle of destroying it with bhyvectl --destroy, then re-loading a kernel with bhyveload and then actually booting the VM with bhyve. It seems that I have to do this even if I don't change th kernel between reboots. My first naive impression was that the point of bhyveload was to load the kernel once. Seems it ain't so? bhyveload does the job of what BIOS/boot0/1/2/loader would do on real h/w, so it has to be executed each time on restart. One optimization to the cycle you mentioned is that bhyvectl --destroy only has to be done when the VM is no longer needed i.e you can loop with bhyveload/bhyve. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On 2015-06-02 07:20, Andriy Gapon wrote: > > I am very new to bhyve, so sorry if I am asking something silly or obvious. > I am using bhyve to speed up my testing and it seems that each time I need to > restart a VM I need to go through the cycle of destroying it with bhyvectl > --destroy, then re-loading a kernel with bhyveload and then actually booting > the > VM with bhyve. It seems that I have to do this even if I don't change th > kernel > between reboots. My first naive impression was that the point of bhyveload > was > to load the kernel once. Seems it ain't so? > Are you running -CURRENT, 10-STABLE, or 10.1? I think there was talk recently of removing the requirement to destroy the VM before running bhyveload again. -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
Are you running -CURRENT, 10-STABLE, or 10.1? I think there was talk recently of removing the requirement to destroy the VM before running bhyveload again. That's been the case in 10.1/STABLE and CURRENT for quite a while (the original change was r267216). later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On 02/06/2015 17:51, Peter Grehan wrote: > Hi Andriy, > >> I am very new to bhyve, so sorry if I am asking something silly or obvious. >> I am using bhyve to speed up my testing and it seems that each time I need to >> restart a VM I need to go through the cycle of destroying it with bhyvectl >> --destroy, then re-loading a kernel with bhyveload and then actually booting >> the >> VM with bhyve. It seems that I have to do this even if I don't change th >> kernel >> between reboots. My first naive impression was that the point of bhyveload >> was >> to load the kernel once. Seems it ain't so? > > bhyveload does the job of what BIOS/boot0/1/2/loader would do on real h/w, so > it has to be executed each time on restart. > > One optimization to the cycle you mentioned is that bhyvectl --destroy only > has > to be done when the VM is no longer needed i.e you can loop with > bhyveload/bhyve. I see now. Thank you very much! BTW, and probably you are already aware of this, the documentation could use some work :-) I noticed at least the following: - bhyvectl is not documented - bhyve(8) and bhyveload(8) refer to vmm(4), which does not exist - bhyveload(8) does not mention that -d can be used multiple times - bhyve(8) has at least one incorrect reference to bhyveload(*4*) - bhyve(8) mentions vmnet - does that exist? But these are very minor things. bhyve rules :) -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On 2015-06-02 12:16, Andriy Gapon wrote: On 02/06/2015 17:51, Peter Grehan wrote: Hi Andriy, I am very new to bhyve, so sorry if I am asking something silly or obvious. I am using bhyve to speed up my testing and it seems that each time I need to restart a VM I need to go through the cycle of destroying it with bhyvectl --destroy, then re-loading a kernel with bhyveload and then actually booting the VM with bhyve. It seems that I have to do this even if I don't change th kernel between reboots. My first naive impression was that the point of bhyveload was to load the kernel once. Seems it ain't so? bhyveload does the job of what BIOS/boot0/1/2/loader would do on real h/w, so it has to be executed each time on restart. One optimization to the cycle you mentioned is that bhyvectl --destroy only has to be done when the VM is no longer needed i.e you can loop with bhyveload/bhyve. I see now. Thank you very much! BTW, and probably you are already aware of this, the documentation could use some work :-) I noticed at least the following: - bhyvectl is not documented - bhyve(8) and bhyveload(8) refer to vmm(4), which does not exist - bhyveload(8) does not mention that -d can be used multiple times - bhyve(8) has at least one incorrect reference to bhyveload(*4*) - bhyve(8) mentions vmnet - does that exist? But these are very minor things. bhyve rules :) vmnet does exist, it is a type of tap, I'll get it hooked up as an alias to the tap man page. -- Allan Jude ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: corrupting zfs pools?
On 02/06/2015 14:14, Andriy Gapon wrote: > > I am doing a simple experiment. > > I get FreeBSD image from here: > ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz > > Then I run in bhyve with two additional "disks" created with truncate -s 4g: > $ bhyveload -m 1G -d > ~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test > $ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s > 3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img > -s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s > 3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l > com2,/dev/nmdm0A -c 2 -m 1g test > > Note sectorsize=512/4096 options. Not sure if it's them that cause the > trouble. > > Then, in the VM: > $ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2 > $ zfs create -p l2arc-test/ROOT/initial > $ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f - > > Afterwards, zpool status -v reports no problem. > But then I run zpool scrub and get the following in the end: > $ zpool status -v > pool: l2arc-test > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. >see: http://illumos.org/msg/ZFS-8000-8A > scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun 2 13:58:17 2015 > config: > > NAMESTATE READ WRITE CKSUM > l2arc-test ONLINE 0 0 9 > mirror-0 ONLINE 0 018 > vtbd1 ONLINE 0 025 > vtbd2 ONLINE 0 023 > > errors: Permanent errors have been detected in the following files: > > /l2arc-test/ROOT/initial/usr/bin/svnlitesync > /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz > /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz > > /l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a > > > The same issue is reproducible with ahci-hd. > > My host system is a recent amd64 CURRENT as well. The hardware platform is > AMD. > I used the following monstrous command line to reproduce the test in qemu: $ qemu-system-x86_64 -smp 2 -m 1024 -drive file=/tmp/livecd2/R2.img,format=raw,if=none,id=bootd -device virtio-blk-pci,drive=bootd -drive file=/tmp/l2arc-test/hdd1,if=none,id=hdd1,format=raw -device virtio-blk-pci,drive=hdd1,logical_block_size=4096 -drive file=/tmp/l2arc-test/hdd2,id=hdd2,if=none,format=raw -device virtio-blk-pci,drive=hdd2,logical_block_size=4096 -drive file=/tmp/l2arc-test/ssd,id=ssd,if=none,format=raw -device virtio-blk-pci,drive=ssd,logical_block_size=4096 ... And several other variations of logical_block_size and physical_block_size. The tests a re very slow, but there are no checksum errors. So, I suspect guest memory corruption caused by bhyve. Perhaps the problem is indeed specific to AMD-V. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 14:20 +0300: > I am very new to bhyve, so sorry if I am asking something silly or obvious. > I am using bhyve to speed up my testing and it seems that each time I need to > restart a VM I need to go through the cycle of destroying it with bhyvectl > --destroy, then re-loading a kernel with bhyveload and then actually booting > the > VM with bhyve. It seems that I have to do this even if I don't change th > kernel > between reboots. My first naive impression was that the point of bhyveload > was > to load the kernel once. Seems it ain't so? Hmm... I'm not seeing that here... I just scp a new kernel into the vm, install it, and run shutdown -r now which drops bhyve back to loader, and loads the new kernel... I've been doing this quite successfully over the last few months... I am running a month old HEAD though... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On 02/06/2015 19:49, John-Mark Gurney wrote: > Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 14:20 +0300: >> I am very new to bhyve, so sorry if I am asking something silly or obvious. >> I am using bhyve to speed up my testing and it seems that each time I need to >> restart a VM I need to go through the cycle of destroying it with bhyvectl >> --destroy, then re-loading a kernel with bhyveload and then actually booting >> the >> VM with bhyve. It seems that I have to do this even if I don't change th >> kernel >> between reboots. My first naive impression was that the point of bhyveload >> was >> to load the kernel once. Seems it ain't so? > > Hmm... I'm not seeing that here... I just scp a new kernel into the > vm, install it, and run shutdown -r now which drops bhyve back to > loader, and loads the new kernel... I've been doing this quite > successfully over the last few months... > > I am running a month old HEAD though... > I guess you are running bhyve through the shell script vmrun.sh? I am doing everything by hand. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: corrupting zfs pools?
Hi Andriy, On Tue, Jun 2, 2015 at 9:45 AM, Andriy Gapon wrote: > On 02/06/2015 14:14, Andriy Gapon wrote: >> >> I am doing a simple experiment. >> >> I get FreeBSD image from here: >> ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz >> >> Then I run in bhyve with two additional "disks" created with truncate -s 4g: >> $ bhyveload -m 1G -d >> ~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test >> $ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s >> 3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img >> -s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s >> 3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l >> com2,/dev/nmdm0A -c 2 -m 1g test >> >> Note sectorsize=512/4096 options. Not sure if it's them that cause the >> trouble. >> >> Then, in the VM: >> $ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2 >> $ zfs create -p l2arc-test/ROOT/initial >> $ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f - >> >> Afterwards, zpool status -v reports no problem. >> But then I run zpool scrub and get the following in the end: >> $ zpool status -v >> pool: l2arc-test >> state: ONLINE >> status: One or more devices has experienced an error resulting in data >> corruption. Applications may be affected. >> action: Restore the file in question if possible. Otherwise restore the >> entire pool from backup. >>see: http://illumos.org/msg/ZFS-8000-8A >> scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun 2 13:58:17 2015 >> config: >> >> NAMESTATE READ WRITE CKSUM >> l2arc-test ONLINE 0 0 9 >> mirror-0 ONLINE 0 018 >> vtbd1 ONLINE 0 025 >> vtbd2 ONLINE 0 023 >> >> errors: Permanent errors have been detected in the following files: >> >> /l2arc-test/ROOT/initial/usr/bin/svnlitesync >> /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz >> /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz >> >> /l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a >> >> >> The same issue is reproducible with ahci-hd. >> >> My host system is a recent amd64 CURRENT as well. The hardware platform is >> AMD. >> > > I used the following monstrous command line to reproduce the test in qemu: > $ qemu-system-x86_64 -smp 2 -m 1024 -drive > file=/tmp/livecd2/R2.img,format=raw,if=none,id=bootd -device > virtio-blk-pci,drive=bootd -drive > file=/tmp/l2arc-test/hdd1,if=none,id=hdd1,format=raw -device > virtio-blk-pci,drive=hdd1,logical_block_size=4096 -drive > file=/tmp/l2arc-test/hdd2,id=hdd2,if=none,format=raw -device > virtio-blk-pci,drive=hdd2,logical_block_size=4096 -drive > file=/tmp/l2arc-test/ssd,id=ssd,if=none,format=raw -device > virtio-blk-pci,drive=ssd,logical_block_size=4096 ... > > And several other variations of logical_block_size and physical_block_size. > The tests a re very slow, but there are no checksum errors. > > So, I suspect guest memory corruption caused by bhyve. Perhaps the problem is > indeed specific to AMD-V. > Perhaps, but I wasn't able to repro this. I tried your recipe to repro on two systems running -current at r283917. - Intel Sandybridge server: Xeon E52650 with 8 cores/16 threads and 64GB memory - hdd1/hdd2 on UFS - hdd1/hdd2 on ZFS - AMD Opteron server: Opteron 6230 with 8 cores and 16GB memory - hdd1/hdd2 on UFS Can you provide some more details about your setup? I can then try to repro on a system similar to your setup. - uname -a - sysctl hw.model - sysctl hw.ncpu - sysctl hw.physmem - sysctl hw.vmm - host filesystem underlying hdd1 and hdd2 best Neel > -- > Andriy Gapon > ___ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300: > On 02/06/2015 19:49, John-Mark Gurney wrote: > > Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 14:20 +0300: > >> I am very new to bhyve, so sorry if I am asking something silly or obvious. > >> I am using bhyve to speed up my testing and it seems that each time I need > >> to > >> restart a VM I need to go through the cycle of destroying it with bhyvectl > >> --destroy, then re-loading a kernel with bhyveload and then actually > >> booting the > >> VM with bhyve. It seems that I have to do this even if I don't change th > >> kernel > >> between reboots. My first naive impression was that the point of > >> bhyveload was > >> to load the kernel once. Seems it ain't so? > > > > Hmm... I'm not seeing that here... I just scp a new kernel into the > > vm, install it, and run shutdown -r now which drops bhyve back to > > loader, and loads the new kernel... I've been doing this quite > > successfully over the last few months... > > > > I am running a month old HEAD though... > > I guess you are running bhyve through the shell script vmrun.sh? > I am doing everything by hand. Correct: sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0 It's nice.. shutdown -r now and shutdown -p now both work exactly as you'd expect them to... :) -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: bhyveload, bhyve, bhyvectl --destroy
On 2015-06-02 19:04, John-Mark Gurney wrote: Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300: On 02/06/2015 19:49, John-Mark Gurney wrote: Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 14:20 +0300: I am very new to bhyve, so sorry if I am asking something silly or obvious. I am using bhyve to speed up my testing and it seems that each time I need to restart a VM I need to go through the cycle of destroying it with bhyvectl --destroy, then re-loading a kernel with bhyveload and then actually booting the VM with bhyve. It seems that I have to do this even if I don't change th kernel between reboots. My first naive impression was that the point of bhyveload was to load the kernel once. Seems it ain't so? Hmm... I'm not seeing that here... I just scp a new kernel into the vm, install it, and run shutdown -r now which drops bhyve back to loader, and loads the new kernel... I've been doing this quite successfully over the last few months... I am running a month old HEAD though... I guess you are running bhyve through the shell script vmrun.sh? I am doing everything by hand. Correct: sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0 It's nice.. shutdown -r now and shutdown -p now both work exactly as you'd expect them to... :) yes, vmrun.sh puts bhyve in a while loop. -- Allan Jude ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
