RE: Cannot run FreeBSD 11 as a vm-bhyve guest under FreeBSD 12
Hello. I'd like to run a FreeBSD 11.3 VM under FreeBSD 12.1. As I'm already using vm-bhyve to run a Windows guest, I think the best (or possibly only) choice is to use bhyve for the FreeBSD guest too. I've got the following conf file: > guest="freebsd" > loader="bhyveload" > cpu=1 > memory=512M > network0_type="virtio-net" > network0_switch="public" > disk0_type="virtio-blk" > disk0_name="disk0" > disk0_dev="sparse-zvol" Running "vm install -f f11b FreeBSD-11.3-RELEASE-amd64-disc1.iso", will boot the guest, but it will hang after a while with: > /boot/kernel/kernel text=0x1564b08 data=0x145330+0x4cdf30 > syms=[0x8+0x16daf0+0x8+0x186a43] Booting... > | "vm stop" will stop it (through an ACIP shutdown) after a while. So I tried with UEFI; here's the config file: > guest="freebsd" > loader="uefi" > > cpu=1 > memory=512M > network0_type="virtio-net" > network0_switch="public" > disk0_type="virtio-blk" > disk0_name="disk0" > disk0_dev="sparse-zvol" > disk1_type="ahci-cd" > disk1_dev="custom" > disk1_name="/zroot/vm/.iso/FreeBSD-11.3-RELEASE-amd64-disc1.iso" > graphics="yes" > xhci_mouse="yes" > graphics_listen="192.168.xxx.1" Starting it with "vm start" and connecting through VNC, it goes through the same stages as above, but ends up with a blank screen with a white block cursor in the first column of the first row. >Any hint on how to get past this? This is likely caused by an issue in the way vm-bhyve handled stdio, which only worked correctly with older builds of bhyve. I would recommend installing vm-bhyve from ports (you should get version 1.4.2), which hopefully will boot using bhyveload with no issues. Matt > bye & Thanks > av. ___ >>freebsd-virtualization@freebsd.org mailing list >>https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >To unsubscribe, send any mail to >"freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
System (ps/top) & bhyvectl memory usage differences
Hello, Is there a reason why the resident memory used by a bhyve guest is quite different when comparing ps/top & bhyvectl? Does bhyvectl take in account something in kernel space that top/ps doesn't see? USER PID %CPU %MEM VSZRSS TT STAT STARTED TIME COMMAND root 12670 0.1 1.4 2120984 951764 1 SC+ 22Jan20 157:29.22 bhyve: smtp-a (bhyve) # bhyvectl --get-stats --vm=smtp-a | grep Res Resident memory 1439875072 1.4G vs 925M - Another thing I've just remembered, which is probably unrelated to the above but still concerning memory. I have a guest with 2G memory allocated, and dmesg lists 2048MB real memory. The real & avail figures are also quite close which resembles the output I generally expect on real hardware. Hypervisor: Origin = "bhyve bhyve " real memory = 2147483648 (2048 MB) avail memory = 2043318272 (1948 MB) However, I have a guest with 5G allocated, and get the following in dmesg - Hypervisor: Origin = "bhyve bhyve " real memory = 6442450944 (6144 MB) avail memory = 5141663744 (4903 MB) bhyveload -m 5G ... bhyve -c 2 -m 5G -AHP ... I haven't tested where it seems to change. My only theory is that it could possibly be something to do with crossing the old 32bit limit? Regards, Matt Churchyard ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: bhyve issues on Dell C6220 node
> Hi, > >2016 is slow (even slower doing windows updates). 2019 is much better. A > >tip I have found is a minimum of 1 cpu, 2 cores and 4 threads to get decent > >speed from that OS (more CPUs in bhyve >tends to make performance worse - in > >my observations - in 2016). Also, use the Virtio collection from RedHat for > >vionet and viostor. We are currently using 0.1.171 without issue. The ahci > >>emulation in itself is extremely slow. NVMe and virtio is really the only > >way to go. > > Is there anything else I can check here? I haven’t got round to testing > networking yet but I’m using nvme for the disk. > It’s basically unusable and there is no way I could put anything production > on it. Just highlighting an icon on the desktop takes several seconds. >Windows is slow when running on Intel CPUs that don't support APICv. >That are (nearly?) all desktop CPUs, all Xeons before Sandy Bridge and some >Xeons after it. The problem is that Bhyve doesn't implement TPR shadowing. I'm >currently working on it. The review can >be found >here: https://reviews.freebsd.org/D22942 The speedup is about factor 6! >I've received some feedback in a private mail, a second version that adds TPR >thresholds can be found in my Github branch here: > https://github.com/Yamagi/freebsd/commits/wip/tpr_shadowing >A backport to 12.1 (the branch also includes the Intel SpeedShift patches from >https://reviews.freebsd.org/D18028) is here: >https://github.com/Yamagi/freebsd/commits/production/12.1 >I've applied it to 2 of my production servers about 4 hours ago. Looks good so >far. I'll update the review when I'm sure that it doesn't break anything, >maybe early next week. I've been running the patch for a few days now and not seen any problems. It's a complete difference compared to running stock 12.1. Thanks for your response and awesome work. Don't know if any more work on it is required but it would be nice if this made it into 12.2 as it's much easier for me to track the binary releases than have to rebuild the entire system. This fairly small patch makes a massive difference to Windows performance on something that doesn't have APICv I also found the fix for my original console issue (actually someone else fixed it, I just hadn't seen their pull request). I'm using vm-bhyve which originally redirected bhyve output to a log file to catch any error output. Changes to the way bhyve opens stdin & stderr caused this to stop working correctly. Regards, Matt >Regards, >Yamagi -- Homepage: https://www.yamagi.org Github: https://github.com/yamagi GPG: 0x1D502515 ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve issues on Dell C6220 node
> On 9 Jan 2020, at 18:58, Yamagi Burmeister wrote: > > Hi, > >>> 2016 is slow (even slower doing windows updates). 2019 is much better. A >>> tip I have found is a minimum of 1 cpu, 2 cores and 4 threads to get decent >>> speed from that OS (more CPUs in bhyve >tends to make performance worse - >>> in my observations - in 2016). Also, use the Virtio collection from RedHat >>> for vionet and viostor. We are currently using 0.1.171 without issue. The >>> ahci >emulation in itself is extremely slow. NVMe and virtio is really the >>> only way to go. >> Is there anything else I can check here? I haven’t got round to testing >> networking yet but I’m using nvme for the disk. >> It’s basically unusable and there is no way I could put anything production >> on it. Just highlighting an icon on the desktop takes several seconds. > > Windows is slow when running on Intel CPUs that don't support APICv. > That are (nearly?) all desktop CPUs, all Xeons before Sandy Bridge > and some Xeons after it. The problem is that Bhyve doesn't implement > TPR shadowing. I'm currently working on it. The review can be found > here: https://reviews.freebsd.org/D22942 The speedup is about factor > 6! > > I've received some feedback in a private mail, a second version that > adds TPR thresholds can be found in my Github branch here: > https://github.com/Yamagi/freebsd/commits/wip/tpr_shadowing > > A backport to 12.1 (the branch also includes the Intel SpeedShift > patches from https://reviews.freebsd.org/D18028) is here: > https://github.com/Yamagi/freebsd/commits/production/12.1 > > I've applied it to 2 of my production servers about 4 hours ago. Looks > good so far. I'll update the review when I'm sure that it doesn't break > anything, maybe early next week. I’ve had a system with the TPR changes applied for a few hours now and it seems to be ok. The install took less than 10 minutes compared to 40+ minutes before and is much more useable. I’ll leave it running over the weekend and see how it goes. Regards Matt > > Regards, > Yamagi > > -- > Homepage: https://www.yamagi.org > Github: https://github.com/yamagi > GPG: 0x1D502515 ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: bhyve issues on Dell C6220 node
Hi, > >2016 is slow (even slower doing windows updates). 2019 is much better. A > >tip I have found is a minimum of 1 cpu, 2 cores and 4 threads to get decent > >speed from that OS (more CPUs in bhyve >tends to make performance worse - in > >my observations - in 2016). Also, use the Virtio collection from RedHat for > >vionet and viostor. We are currently using 0.1.171 without issue. The ahci > >>emulation in itself is extremely slow. NVMe and virtio is really the only > >way to go. > > Is there anything else I can check here? I haven’t got round to testing > networking yet but I’m using nvme for the disk. > It’s basically unusable and there is no way I could put anything production > on it. Just highlighting an icon on the desktop takes several seconds. >Windows is slow when running on Intel CPUs that don't support APICv. >That are (nearly?) all desktop CPUs, all Xeons before Sandy Bridge and some >Xeons after it. The problem is that Bhyve doesn't implement TPR shadowing. I'm >>currently working on it. The review can be found >here: https://reviews.freebsd.org/D22942 The speedup is about factor 6! >I've received some feedback in a private mail, a second version that adds TPR >thresholds can be found in my Github branch here: > https://github.com/Yamagi/freebsd/commits/wip/tpr_shadowing >A backport to 12.1 (the branch also includes the Intel SpeedShift patches from >https://reviews.freebsd.org/D18028) is here: >>https://github.com/Yamagi/freebsd/commits/production/12.1 >I've applied it to 2 of my production servers about 4 hours ago. Looks good so >far. I'll update the review when I'm sure that it doesn't break anything, >maybe >early next week. Thanks Yamagi, That's brilliant and provides me some hope that I may not have to completely abandon windows guests after all... These servers are just sat next to my desk in testing at the moment so I'll see if I can have a go at applying the patches and testing it in the morning. Regards, Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: bhyve issues on Dell C6220 node
On Thu, 9 Jan 2020 at 08:37, Rodney W. Grimes mailto:freebsd-...@gndrsh.dnsmgr.net>> wrote: > > Loading kernel... > /boot/kernel/kernel text=0x168fdf1 data=0x1d0a68+0x768d80 > syms=[0x8+0x178bc0+0x8+0x1969d5] > Loading configured modules... > can't find '/boot/entropy' > / > " > > I get either a "|" or "/" character, then nothing else. My experience has been when I see this it is a "wrong console" issue, ie the kernel has decided to use something else for a console and your output is going there. It might be your setup using a UEFI with a fb console up to the end of the loader, then the kernel decides it is using a serial console. Or vice versa. >Rod is correct. I came across this issue in 12.0 and probably is also in 11.3 >by now. See here: https://wiki.freebsd.org/bhyve/UEFI for some tips that may >help you work out where your console is >redirecting. >I use UEFI for all guest operating systems with a custom UEFI loader that has >been fixed to handle OpenBSD (as you know :-) ). Thanks guys. I have managed to get FreeBSD to boot using the UEFI loader with boot_serial=no. Shame that I can’t get it to boot with bhyveload as it seems a more streamlined boot process and allowed me to easily access the main console directly from ssh. One thing I have now noticed is that it will boot when accessing the console via an nmdm device on com1. However, if I try and boot bhyve in the foreground using stdio (I usually just run it in a tmux window as I find it far more flexible than nmdm/cu), it stops, so clearly does seem to be a console issue. I can’t however find any loader setting that will fix it. > > I've also tested a windows server 2016 guest, and while that will actually > install via UEFI, it is noticeably slow - over a minute to boot to the login > screen and everything crawls along > >2016 is slow (even slower doing windows updates). 2019 is much better. A tip >I have found is a minimum of 1 cpu, 2 cores and 4 threads to get decent speed >from that OS (more CPUs in bhyve >tends to make performance worse - in my >observations - in 2016). Also, use the Virtio collection from RedHat for >vionet and viostor. We are currently using 0.1.171 without issue. The ahci >>emulation in itself is extremely slow. NVMe and virtio is really the only >way to go. Is there anything else I can check here? I haven’t got round to testing networking yet but I’m using nvme for the disk. It’s basically unusable and there is no way I could put anything production on it. Just highlighting an icon on the desktop takes several seconds. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
bhyve issues on Dell C6220 node
Hello, I've recently got hold of some Dell C6220 systems (2 servers in a single chassis) while I was hoping to use for bhyve The basic spec of a single server is as follows - Xeon(R) CPU E5-2670 (VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID) 64GB DDR3 LSI MegaRaid 9265-8i FreeBSD 12.1 I had a bit of trouble getting the disks configured and booting (**probably unrelated but see below for the details). However, my main problem at the moment is that booting either 11.3 or 12.1 in a bhyve guest (using bhyveload) gets as far as starting to boot from the install ISO, then just hangs " Loading kernel... /boot/kernel/kernel text=0x168fdf1 data=0x1d0a68+0x768d80 syms=[0x8+0x178bc0+0x8+0x1969d5] Loading configured modules... can't find '/boot/entropy' / " I get either a "|" or "/" character, then nothing else. I've also tested a windows server 2016 guest, and while that will actually install via UEFI, it is noticeably slow - over a minute to boot to the login screen and everything crawls along I'm at a loss at the moment. My test machine for years has been an old i3-2100 system and that has always booted freebsd & windows guests fairly well I'd be grateful if anyone has any ideas or insight, or whether I just need to scrap the whole idea and either switch to a completely different hypervisor (which I'd really rather not do as I know FreeBSD very well and was planning on making use of ZFS for backups/migration/etc) or find some different systems **Regarding the disks, I didn't really want a raid controller and have ended up creating a jbod (RAID0) volume for each disk, which appears as mfidX. This seems to work fine, although at first I tried to create a single pool of 2 mirrored pairs (4 disks). This failed to boot due to zfs block i/o errors on boot. In the end I had to create a separate boot mirror across 2 partitions, the create a second pool for actual data storage. Regards, Matt Churchyard ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: VM Switch broken after update on 20190507
>I just updated my bhyve server running CURRENT up to r349133 and can't start >my network switch. I suspect that this problem has a root cause in the kernel >configuration change made on 20190507 that created if_tuntap to replace the >tunnel and tap devices. My vm switch has been in use for over a year and >worked fine. When booting I get the message: > /boot/kernel/if_tap.ko - unsupported file type. > I can't remove or destroy my vm switch to create a new one. I get the > message: > vm switch remove public re0 - unable to locate switch id vm destroy public - > failed to remove virtual switch > My vm_bhyve port is version 1.3.0 > I tried to locate any config files that were generated by the vm switch > create utility. I was not successful. I wanted to see if I could just > manually edit something to change if_tap to if_tuntap. What do I need to > make my vm switch work again? Hi Tom, I haven't tested this myself but there's a recent PR that was provided to fix this that hasn't made it into ports yet. https://github.com/churchers/vm-bhyve/pull/305/files Matt >Tom -- Public Keys: PGP KeyID = 0x5F22FDC1 GnuPG KeyID = 0x620836CF ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop
Victor Sudakov wrote on 2019-04-22 19:43:
...
>> And the implementation is pretty brutal:
>> # 'vm stopall'
>> # stop all bhyve instances
>> # note this will also stop instances not started by vm-bhyve #
>> core::stopall(){
>> local _pids=$(pgrep -f 'bhyve:')
>>
>> echo "Shutting down all bhyve virtual machines"
>> killall bhyve
>> sleep 1
>> killall bhyve
>> wait_for_pids ${_pids}
>> }
> yow.
>>
>> I wonder what the effect of the second kill is, that seems odd.
>
> Indeed.
> the first killall will cause each client OS to see a soft shutdown signal.
> the sleep 1 gives them some time to flush their buffers. the second killall
> says, time's up, just stop.
Both of these killall commands send a soft shutdown signal and I've never seen
an instance in my own testing where the second has caused an instant poweroff.
I've just double checked this, and a FreeBSD guest stopped with the existing
code fully shuts itself down, ending with "acpi0: Powering system off"
The main reason for having both is that in my initial testing, I could not get
Windows to respond to a single SIGINT. 100% of the time it would simply act
like nothing had happened. Sending two however triggered a shutdown. I only
have a single test machine though so maybe it's something strange about my
particular system that no one else experiences...
Matt
> i think this is worse than brutal, it's wrong. consider freebsd's own work
> flow when trying to comply with the first soft shutdown it got:
> https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220
> this has bitten me more than once, because using "pageins" as a proxy for "my
> server processes are busy trying to synchronize their user mode state" is
> inaccurate. i think _any_ continuing I/O should > be reason to wait the full
> 60 seconds.
> and so i think the "sleep 1" above should be a "sleep 65".
> What is needed in vm-bhyve is the feature that if ACPI does not stop
> the guest for a predefined period of time, the guest is powered off.
> i agree with this.
--
> P Vixie
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
RE: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor
> > > > > 1. Does ARC actually cache zfs volumes (not files/datasets)? > > > > Yes it does. > > I find this distinction between volumes/files/etc and what is cached > causes confusion (as well as "volumes not datasets"). > > Both ZVOLs and Z file systems are types of dataset. A dataset stores > data in records (usually up to 128kb in size). It's these records > that are cached (and that most ZFS functions such as > compression/raidz/zil/etc work with) > > As far as the ZFS lower levels are concerned, there is no difference > between a volume and a file system. >Thank you Matt, this was very instructive. > > > 2. If ARC does cache volumes, does this cache make sense on a > > > hypervisor, because guest OSes will probably have their own disk cache > > > anyway. > > > > IMHO not much, because the guest OS is relying on the fact that when > > it writes it’s own cached data out to „disk“, it will be committed > > to stable storage. > > Maybe I've missed something but I don't quite get the link between > read cache (ARC) and guest writes here? >Maybe there was a confusion between read and write caches, but my question >still stands: >Does it make sense to cache the same data (for reading too) twice: one time in >host's RAM (ZFS ARC) and the other time in guest's RAM (whatever fs the guest >uses, all modern OSes have disk caches)? >What do VMWare or VirtualBox do for this situation? Do they ever cache their >volumes in the hypervisor's RAM? Virtualbox would be no different to bhyve in that it doesn't care what storage system you are using or how it is configured, that's up to the system admin. I believe VMFS is more akin to other "traditional" file systems, and doesn't do RAM caching to anywhere near the extent of ZFS. I do think you can use SSD/NVMe/etc as cache in VMWare. My initial instinct would be to keep cache on, but reduce the limit to allocate the majority of the RAM for guests. (I'd still want at least 4GB as an absolute minimum though, probably more on systems with 100GB+ total). Of course you could probably test with cache set to all/metadata and see what effect it has. Adding L2ARC may be useful if the main pool is spinning disks, but then I've heard there's a rule of thumb for requiring X amount of ARC for Y amount of L2ARC, but I'm not sure what that rule is. I'd also be intrigued to know what the logic in FreeNAS is for it. It is simply a case of "(arc = total_ram - guest_allocated)"? Is there a lower limit based on a percentage or total RAM, and/or a hard lower limit? Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor
Patrick M. Hausen wrote: > > > 1. Does ARC actually cache zfs volumes (not files/datasets)? > > Yes it does. I find this distinction between volumes/files/etc and what is cached causes confusion (as well as "volumes not datasets"). Both ZVOLs and Z file systems are types of dataset. A dataset stores data in records (usually up to 128kb in size). It's these records that are cached (and that most ZFS functions such as compression/raidz/zil/etc work with) As far as the ZFS lower levels are concerned, there is no difference between a volume and a file system. > > > 2. If ARC does cache volumes, does this cache make sense on a > > hypervisor, because guest OSes will probably have their own disk cache > > anyway. > > IMHO not much, because the guest OS is relying on the fact that when > it writes it’s own cached data out to „disk“, it will be committed to > stable storage. Maybe I've missed something but I don't quite get the link between read cache (ARC) and guest writes here? > This is an important point. > > 3. Would it make sense to limit vfs.zfs.arc_max to 1/8 or even less > > of total RAM, so that most RAM is available to guest machines? > > Yes if you build your own solution on plain FreeBSD. No if you are > running FreeNAS which already tries to autotune the ARC size according > to the memory committed to VMs. > > > 4. What other zfs tuning measures can you suggest for a bhyve > > hypervisor? > > e.g. > zfs set sync=always zfs/vm > > if zfs/vm is the dataset under which you create the ZVOLs for your > emulated disks. >Well, bhyve already has an option for this: >The block-device-options are: >nocache Open the file with O_DIRECT. >directOpen the file using O_SYNC. >roForce the file to be opened read-only. >I think something like >"-s 4:0,virtio-blk,/dev/zvol/zroot/vm/mail/disk0,direct" >would do the same? > > I’m using this for all my VM „disks“ and have added a 16 GB SLOG > device to my spinning disk pool - seems to work great. This is on a home > system. > Is SLOG also used by zfs volumes? As above, the core of ZFS doesn't care what type of dataset it is working with. ARC/ZIL/etc all work exactly the same. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: NVMe and Bhyve
Michael Reifenberger wrote: > Hi, > first I tried to install windows10 ltsc 2019 onto a nvme disk. > This failed, the windows installer did not find a disk to install. > > Then I tried the following setup: > ... > disk0_type="ahci-hd" > disk0_dev="zvol" > disk0_name="disk0" > disk1_type="nvme" > disk1_dev="zvol" > disk1_name="disk1" > disk1_type="ahci-cd" > disk1_name="disk2.img" >If this is exact copy/paste, shouldn't the 2 lines above use disk2 prefix >instead? There's definitely an issue with the configuration file, but vm-bhyve retrieves config settings by looking through the file (which is loaded into a variable) line by line and returning the first match. As such, it will likely be ignoring the ahci-cd & disk2.img settings. Strange why the last disk is a CD and pointing at a .img file though. More useful would be the actual bhyve command from /path/to/guest/vm-bhyve.log, and maybe also confirmation that the zvol path that appears in that command line correctly points to an existing zvol. Matt > ... > > And Installed windows on disk0. > Disk0 is found as expected, disk1 is not. > > The devicemanager reports an error: > https://imgur.com/a/zrHx23y > > Shouldn't nvme be supported in behyve? ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: bhyve /Xen
On Sun, Jan 15, 2017 at 10:16:33PM -0500, Jeremy Faulkner wrote: > > > On 2017-01-15 5:45 PM, The Doctor wrote: > >>> 2) Windows 2016 and bhyve > >>> > >>> > >>> Why can Widnows 2016 rpc not work with bhyve? > >> > >> Can you describe the case that isn't working ? (to get a repro) > > > > Under VNC you can log in but not under rpc. > > > > Anyone else running Win 2016 in FreeBSD 11 bhyve? > > > > You are using RDP to the guest's ip not the FreeBSD bhyve host's ip > address correct? Bhyve provides VNC on the bhyve host's ip, no RDP. > AFAIK both IPs are the smae. Bhyve provides a VNC server which allows you to access the guest console. This listens on all interfaces/IP addresses on the bhyve host by default (unless you specify an address in the bhyve command line). This has nothing to do with RDP. If you want to connect via RDP you'll need to make sure networking is configured and working in the Windows guest, and that RDP in enabled in Windows system settings. You then RDP to the IP address of the Windows machine. The Windows guest will not have the same IP address as the bhyve host. If you're using NAT on the bhyve host then it's a bit more complicated as you'll need to decide on a way of getting access to the guest (most obvious way is by forwarding the RDP port on the bhyve host). Also note that RDP and RPC are two different things Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Windows 2016
On Tue, Nov 1, 2016 at 8:58 AM, Matt Churchyard via freebsd-virtualization mailto:freebsd-virtualization@freebsd.org>> wrote: On Tue, Nov 01, 2016 at 03:16:12PM +0100, Daniel Tihanyi wrote: > On Tue, 2016-11-01 at 08:02 -0600, The Doctor wrote: > > On Tue, Nov 01, 2016 at 09:02:57AM +, Matt Churchyard wrote: > > > > > > > > > > > I was able to install the Windows 2016 using the GUI method. > > > > > > > > One problem. > > > > > > > > It is assigned an IP but > > > > > > > > the netmask and the default route is not showing up. > > > > > > > > Also??The virtual Windows 2016 box does not see a network > > > > interface. > > > This makes very little sense. You usually specify the netmask and > > > default route when you assign Windows an IP address. > > > And how have you assigned the guest an IP address if it has no > > > network interface? > > > > > The taps are allocated an IP address. > > > > I fully concur. > > > > In the install at one point, I declare that the IP address, netmask > > and gateway are 'declared' at the 'MB ' BIOS interface. > > > > > > > > > > > > > > > What must I do to convince this box that is it on a switch > > > On the bhyve host it should be configured just like any other > > > bhyve virtual machine. The guest should have a virtio-net device, > > > which is linked to a tap interface on the host. That tap interface > > > should be bridged with whichever physical network adapter you want > > > the guest connected to. > > > > > > However, I think I mentioned in a previous message that Windows > > > does not have the virtio-net drivers by default. You need to boot > > > the guest with the virtio-net driver ISO attached and install the > > > driver. You should see the interface in Device Manager flagged as > > > not installed/working. > > > > > > > That is correct . > > > > I do have an iso of the MB drivers, > > > > namely a Supermicro??X10DRW-i MB? > > > > and? > > > > from Intel I did obtain the .exe file for the i350GbE drivers. > > I script a 2 phase installation as follows: > > > > 1) > > > > > > /usr/sbin/bhyve -c 2 -m 4G -w -H -s 3,ahci-cd,./.iso/14393.0.160715- > > 1616.RS1_RELEASE_SERVER_EVAL_X64FRE_EN-US.ISO -s 4,ahci- > > hd,windows2016.img -s 5,virtio-net,tap15 -s > > 29,fbuf,tcp=0.0.0.0:5900<http://0.0.0.0:5900>,w=800,h=600,wait -s > > 30,xhci,tablet -s > > 31,lpc -l com1,stdio -l > > bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd > > windows2016guest > > > > 2) > > > > > > /usr/sbin/bhyve -c 2 -m 4G -w -H -s > > 3,ahci-cd,./.iso/null-install.iso -s 4,ahci-hd,windows2016.img -s > > 5,virtio-net,tap15 -s > > 29,fbuf,tcp=0.0.0.0:5900<http://0.0.0.0:5900>,w=1024,h=768,wait -s > > 30,xhci,tablet -s 31,lpc -l com1,/dev/nmdm15A -l > > bootrom,/usr/local/share/uefi- firmware/BHYVE_UEFI.fd > > windows2016guest & > > > > As I said, I have picked up an ISO from the MB manufacturer > > > > let's call it manufacturer.iso . > > > > Where do I place said iso in the 'phase' of things? > > You use virtio network adapters, you have to install the virtio > drivers and NOT the drivers for the physical interface. Then configure > the IP Address, gateway, etc. in Windows. >All right, let me repeat what I am up aginst. >I do set the IP configuration in the virtual BIOS. >However when the Win2016 VM boots it sees >1) the localhost >2) its assigned IP address >3) A yellow triangle where the ethernet adaptor is. I'm not sure what people find so difficult about this - Just treat it like it was a real Windows machine 1) Do not assign the guest's IP address on the host 2) Do not set the IP in the guest BIOS (I'm not even sure what you mean by this) As you have found, the virtio-net network adapter has a yellow triangle in Windows, because it is not installed. You need to run bhyve but replace the Windows ISO with the virtio-net driver ISO, which you can download off the Internet; Then install the network driver through VNC. The guest has a "virtio-net" interface which is created by bhyve - This has nothing to do with your motherboard and you do not need to install any of the motherboard or physical interface drivers in the guest. Once you have the virtio driver installed, you will have a "Redhat VirtIO" network adapter in Windows, and you
RE: Windows 2016
On Tue, Nov 01, 2016 at 03:16:12PM +0100, Daniel Tihanyi wrote: > On Tue, 2016-11-01 at 08:02 -0600, The Doctor wrote: > > On Tue, Nov 01, 2016 at 09:02:57AM +, Matt Churchyard wrote: > > > > > > > > > > > I was able to install the Windows 2016 using the GUI method. > > > > > > > > One problem. > > > > > > > > It is assigned an IP but > > > > > > > > the netmask and the default route is not showing up. > > > > > > > > Also??The virtual Windows 2016 box does not see a network > > > > interface. > > > This makes very little sense. You usually specify the netmask and > > > default route when you assign Windows an IP address. > > > And how have you assigned the guest an IP address if it has no > > > network interface? > > > > > The taps are allocated an IP address. > > > > I fully concur. > > > > In the install at one point, I declare that the IP address, netmask > > and gateway are 'declared' at the 'MB ' BIOS interface. > > > > > > > > > > > > > > > What must I do to convince this box that is it on a switch > > > On the bhyve host it should be configured just like any other > > > bhyve virtual machine. The guest should have a virtio-net device, > > > which is linked to a tap interface on the host. That tap interface > > > should be bridged with whichever physical network adapter you want > > > the guest connected to. > > > > > > However, I think I mentioned in a previous message that Windows > > > does not have the virtio-net drivers by default. You need to boot > > > the guest with the virtio-net driver ISO attached and install the > > > driver. You should see the interface in Device Manager flagged as > > > not installed/working. > > > > > > > That is correct . > > > > I do have an iso of the MB drivers, > > > > namely a Supermicro??X10DRW-i MB? > > > > and? > > > > from Intel I did obtain the .exe file for the i350GbE drivers. > > I script a 2 phase installation as follows: > > > > 1) > > > > > > /usr/sbin/bhyve -c 2 -m 4G -w -H -s 3,ahci-cd,./.iso/14393.0.160715- > > 1616.RS1_RELEASE_SERVER_EVAL_X64FRE_EN-US.ISO -s 4,ahci- > > hd,windows2016.img -s 5,virtio-net,tap15 -s > > 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait -s 30,xhci,tablet -s > > 31,lpc -l com1,stdio -l > > bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd > > windows2016guest > > > > 2) > > > > > > /usr/sbin/bhyve -c 2 -m 4G -w -H -s > > 3,ahci-cd,./.iso/null-install.iso -s 4,ahci-hd,windows2016.img -s > > 5,virtio-net,tap15 -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait -s > > 30,xhci,tablet -s 31,lpc -l com1,/dev/nmdm15A -l > > bootrom,/usr/local/share/uefi- firmware/BHYVE_UEFI.fd > > windows2016guest & > > > > As I said, I have picked up an ISO from the MB manufacturer > > > > let's call it manufacturer.iso . > > > > Where do I place said iso in the 'phase' of things? > > You use virtio network adapters, you have to install the virtio > drivers and NOT the drivers for the physical interface. Then configure > the IP Address, gateway, etc. in Windows. >All right, let me repeat what I am up aginst. >I do set the IP configuration in the virtual BIOS. >However when the Win2016 VM boots it sees >1) the localhost >2) its assigned IP address >3) A yellow triangle where the ethernet adaptor is. I'm not sure what people find so difficult about this - Just treat it like it was a real Windows machine 1) Do not assign the guest's IP address on the host 2) Do not set the IP in the guest BIOS (I'm not even sure what you mean by this) As you have found, the virtio-net network adapter has a yellow triangle in Windows, because it is not installed. You need to run bhyve but replace the Windows ISO with the virtio-net driver ISO, which you can download off the Internet; Then install the network driver through VNC. The guest has a "virtio-net" interface which is created by bhyve - This has nothing to do with your motherboard and you do not need to install any of the motherboard or physical interface drivers in the guest. Once you have the virtio driver installed, you will have a "Redhat VirtIO" network adapter in Windows, and you can assign an IP/Netmask/Gateway to this just as you normally would in Windows. Forget about the Linux machines. They have the virtio-net driver built in, so you can just configure eth0 out-of-the-box like normal. If you've configured their IP addresses on the bhyve host then that's not really the correct way to configure things, even if it doesn't actually break anything; The IP address should be configured inside the guest OS. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Windows 2016
>I was able to install the Windows 2016 using the GUI method. >One proble. >It is assigned an IP but >the netmask and the default route is not showing up. >Also The virtual Windows 2016 box does not see a network interface. This makes very little sense. You usually specify the netmask and default route when you assign Windows an IP address. And how have you assigned the guest an IP address if it has no network interface? >What must I do to convince this box that is it on a switch On the bhyve host it should be configured just like any other bhyve virtual machine. The guest should have a virtio-net device, which is linked to a tap interface on the host. That tap interface should be bridged with whichever physical network adapter you want the guest connected to. However, I think I mentioned in a previous message that Windows does not have the virtio-net drivers by default. You need to boot the guest with the virtio-net driver ISO attached and install the driver. You should see the interface in Device Manager flagged as not installed/working. Just one other thing to add to this. I'm not sure if this is the case here, but it's something I've seen a few times now. When configuring IP addresses, etc for a guest, this should be done -inside- the guest. Do not assign a guest's IP address to the host, or to the tap interface. Bhyve works just like any other hypervisor. IP settings are done in the guest as if it was a real machine, the host just acts like a switch. Matt >192.168.0.60 >Thar is handling the default routing to the interent. >For argument sake , >the Windows server is IP as 192.168.0.68 >I have an ubuntu up at 192.168.0.57 >a Centos at 192.168.0.54 >and >Fedora at 192.168.0.53 >all running successfully. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Windows 2016
>I was able to install the Windows 2016 using the GUI method. >One proble. >It is assigned an IP but >the netmask and the default route is not showing up. >Also The virtual Windows 2016 box does not see a network interface. This makes very little sense. You usually specify the netmask and default route when you assign Windows an IP address. And how have you assigned the guest an IP address if it has no network interface? >What must I do to convince this box that is it on a switch On the bhyve host it should be configured just like any other bhyve virtual machine. The guest should have a virtio-net device, which is linked to a tap interface on the host. That tap interface should be bridged with whichever physical network adapter you want the guest connected to. However, I think I mentioned in a previous message that Windows does not have the virtio-net drivers by default. You need to boot the guest with the virtio-net driver ISO attached and install the driver. You should see the interface in Device Manager flagged as not installed/working. Matt >192.168.0.60 >Thar is handling the default routing to the interent. >For argument sake , >the Windows server is IP as 192.168.0.68 >I have an ubuntu up at 192.168.0.57 >a Centos at 192.168.0.54 >and >Fedora at 192.168.0.53 >all running successfully. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Installing SQL Server on Windows 2012 bhyve guests
-Original Message- From: owner-freebsd-virtualizat...@freebsd.org [mailto:owner-freebsd-virtualizat...@freebsd.org] On Behalf Of Peter Grehan Sent: 25 October 2016 19:19 To: Randy Terbush Cc: freebsd-virtualization@freebsd.org Subject: Re: Installing SQL Server on Windows 2012 bhyve guests >Hi Randy, > I've followed the info provided here > https://wiki.freebsd.org/bhyve/Windows > to create a Windows Server 2012 image that I successfully installed on > the bhyve hypervisor. After getting a working Win2012 image running, > applying updates, etc. I attempted to install SQL Server 2014. In the > late stages of that install, the process attempts to start the SQL server > engine and fails. > There have been some reports of this previously. What might be happening is > that SQL Server has stricter requirements on the underlying block >size than > NTFS itself has. > Since you are using a file-backed image, the reported block size from the > emulated storage controller may be 8KB or even higher, depending on >the > underlying filesystem type. > A suggestion is to force the block size to 4KB (has to be done during > install as well), and if the problem persists, try 512 bytes. This is done by > using >the 'sectorsize=' parameter to the disk configuration e.g. > -s 4,ahci-hd,/path/to/disk.img,sectorsize=4096 Just to add, if you're using vm-bhyve you should be able to add this with the diskX_opts template/config option disk0_opts="sectorsize=4096" I may adjust the sample templates to include a sectorsize option for Windows guests, although I'm not 100% clear on what exact OS versions or applications require specific settings. I do know that SQL Server has been reported to suddenly fall over unless a valid block size is used (although I believe it did install and run), but again I'm not sure exactly what the "correct" settings are. Also note that you shouldn't need any grub_* template options if you're using UEFI. Those options only apply to grub-bhyve. Matt >later, >Peter. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Windows 2016 Server
>On Mon, Oct 24, 2016 at 08:07:13AM +, Matt Churchyard wrote: > On Fri, Oct 21, 2016 at 03:21:10PM -0700, Peter Grehan wrote: > > >> [Windows could not parse or process unattend answer file > > >> [D:\autounattend.xml] for pass [windowsPE]. The answer file is > > >> invalid.] > > > > > > Something is wrong with your autounattend.xml file. > > > > Still working on the 2k16 unattend file. Unfortunately things have > > changed from TP5 in a non-obvious way :( > > > > A workaround is to do a GUI install. > > > > later, > > > > Peter. > > > Please explain what you mean by a GUI install. > > Not sure if anyone's replied directly to you - > > 1) Make sure you are running at least 11-RELEASE > > 2) Use an "off-the-shelf" Windows install CD > > 3) Add the following to your bhyve command > > -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait > -s 30,xhci,tablet > > 4) Run bhyve and then use a VNC client to connect and install using > the normal Windows GUI > >Well all right did follow https://wiki.freebsd.org/bhyve/UEFI >My next question is after the initial install is done, how do I followup on >the next step , i.e. to see if the VM is booting. As long as you have the fbuf device specified you can connect to the IP address of the bhyve host using VNC to view the "physical" guest console and watch it boot. Usually with Windows you'd then use the VNC console to log in and set up permanent RDP access. One other issue is that Windows doesn't support the virtio-net network device by default. The easiest way to get this working is to boot the guest with the virtio driver ISO attached instead of the Windows install disk (once Windows is installed and working). You can then install the driver for the network interface from the CD using the VNC console. Depending on what you are doing you may find it easier to use something like iohyve/chyves/vm-bhyve/vmrc that handles all the raw bhyve commands for you. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Windows 2016 Server
On Fri, Oct 21, 2016 at 03:21:10PM -0700, Peter Grehan wrote: > >> [Windows could not parse or process unattend answer file > >> [D:\autounattend.xml] for pass [windowsPE]. The answer file is > >> invalid.] > > > > Something is wrong with your autounattend.xml file. > > Still working on the 2k16 unattend file. Unfortunately things have > changed from TP5 in a non-obvious way :( > > A workaround is to do a GUI install. > > later, > > Peter. > Please explain what you mean by a GUI install. Not sure if anyone's replied directly to you - 1) Make sure you are running at least 11-RELEASE 2) Use an "off-the-shelf" Windows install CD 3) Add the following to your bhyve command -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait -s 30,xhci,tablet 4) Run bhyve and then use a VNC client to connect and install using the normal Windows GUI Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Bhyve tests and findings
> [..] > > You just need the vm name (win) on the end: > > # bhyve [options] vm-name > >Sorry, I was being stupid. With the vm name on the end, it just exits with >error code 1 and without printing anytning. >I have recompiled bhyve itself and the libvmmapi, maybe it's not enough? The >kernel module? It still remains that from 10.3 Definitely sounds like something not right with bhyve; It shouldn't exit like that. You can try building vmm from 11, although I'm not really a kernel dev and can't help much with trying to merge 11 code into 10. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Bhyve tests and findings
> > > > > > > > > > > > > uefi="yes" > > > > > > > graphics="yes" > > > > > > > > > > > > Dear Colleagues, > > > > > > > > > > > > Can I enjoy and test all those nice "UEFI-GOP" features on a > > > > > > 10.3-RELEASE system, or do I have to install CURRENT for that? > > > > > > > > > > AFAIK you can checkout usr.sbin/bhyve from the 11 branch and > > > > > compile it on 10.3. > > > > > > > > I'm not sure how to compile something from a different branch > > > > without compiling the whole world. make what? > > > > > > For bhyve it's easy. Check out the sources, go to usr.sbin/bhyve > > > and run > > > > > > make depend; make obj; make; make install. > > > > >Unfortunately, after all my efforts, with libvmmapi and bhyve > > >compiled from CURRENT sources: > > > > >Aug 09 12:46:52: warning; UEFI graphics is only available in > > >FreeBSD 11 and newer Aug 09 12:46:52: booting > > > > >Bummer! > > > > It's possible that you do have a bhyve system with graphics support. > > Unfortunately vm-bhyve isn't aimed at users who have manually merged > > changes, and will blindly tell anyone that isn't running 11+ that they > > don't have graphics support. > > > > You have a couple of options - > > > > 1) Run bhyve manually rather than using vm-bhyve > > 2) Upgrade to the 11 beta > > 3) Modify vm-bhyve to not automatically skip graphics support if the > > version is less than 11 > > >Aha! I've commented out the version check from vm-run, and now when > >trying to install, I get: > > >авг 09 17:01:05: initialising > >авг 09 17:01:05: [loader: none] > >авг 09 17:01:05: [uefi: yes] > >авг 09 17:01:05: [cpu: 1] > >авг 09 17:01:05: [memory: 1G] > >авг 09 17:01:05: [hostbridge: standard] авг 09 17:01:05: [com > >ports: com1] авг 09 17:01:05: [uuid: > >093f9134-5e18-11e6-9502-5404a6b49a66] > >авг 09 17:01:05: [utctime: no] > >авг 09 17:01:05: [debug mode: no] > >авг 09 17:01:05: [primary disk: disk0.img] авг 09 17:01:05: > >[primary disk dev: file] авг 09 17:01:05: generated static mac > >58:9c:fc:03:77:3b (based on 'win:0:1470736865:0') авг 09 17:01:05: > >initialising network device tap5 авг 09 17:01:05: adding tap5 -> > >bridge1 (isolated) авг 09 17:01:05: dynamically allocated port 5900 > >for vnc connections авг 09 17:01:05: booting авг 09 17:01:05: [bhyve > >options: -c 1 -m 1G -Hwl bootrom,/d02/vm/.config/BHYVE_UEFI.fd -U > >093f9134-5e18-11e6-9502-5404a6b49a66] > >авг 09 17:01:05: [bhyve devices: -s 0,hostbridge -s 31,lpc -s > >4:0,ahci-hd,/d02/vm/win/disk0.img -s > >5:0,virtio-net,tap5,mac=58:9c:fc:03:77:3b -s > >>6:0,fbuf,tcp=0.0.0.0:5900,wait] авг 09 17:01:05: [bhyve console: -l > >com1,/dev/nmdm0A] авг 09 17:01:05: [bhyve iso device: -s > >3:0,ahci-cd,/d02/vm/.iso/bhyve_win7_x64.iso] > >авг 09 17:01:05: starting bhyve (run 1) авг 09 17:01:05: bhyve exited > >with status 1 авг 09 17:01:05: destroying network device tap5 авг 09 > >17:01:05: stopped > > Looks like bhyve is exiting immediately. Easiest thing is to probably add > debug="yes" to the guest config file then try and run it again. Any output > from the bhyve command will be written to bhyve.log (note that's a different > file to vm-bhyve.log). > Tried that. root@vas:/d02/vm/win # file bhyve.log bhyve.log: empty root@vas:/d02/vm/win # more win.conf guest="windows" uefi="yes" graphics="yes" cpu=1 memory=1G network0_type="virtio-net" network0_switch="isolated" disk0_type="ahci-hd" disk0_name="disk0.img" debug="yes" uuid="093f9134-5e18-11e6-9502-5404a6b49a66" network0_mac="58:9c:fc:03:77:3b" root@vas:/d02/vm/win # >I have tried replicating manually all the options vm is passing to bhyve, like >bhyve -c 1 -m 1G -Hwl bootrom,/d02/vm/.config/BHYVE_UEFI.fd \ >-U 093f9134-5e18-11e6-9502-5404a6b49a66 \ >>-s 0,hostbridge -s 31,lpc -s 4:0,ahci-hd,/d02/vm/win/disk0.img \ >-s 5:0,virtio-net,tap5,mac=58:9c:fc:03:77:3b -s > 6:0,fbuf,tcp=0.0.0.0:5900,wait \ >-l com1,stdio \ > -s 3:0,ahci-cd,/d02/vm/.iso/bhyve_win7_x64.iso > But bhyve just spits out the "usage" help (that's why it exits immediately). > But I cannot find out what's incorrect with the options. You just need the vm name (win) on the end: # bhyve [options] vm-name Matt -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Bhyve tests and findings
> > > > > > > > > > > > uefi="yes" > > > > > > graphics="yes" > > > > > > > > > > Dear Colleagues, > > > > > > > > > > Can I enjoy and test all those nice "UEFI-GOP" features on a > > > > > 10.3-RELEASE system, or do I have to install CURRENT for that? > > > > > > > > AFAIK you can checkout usr.sbin/bhyve from the 11 branch and compile it > > > > on 10.3. > > > > > > I'm not sure how to compile something from a different branch without > > > compiling the whole world. make what? > > > > For bhyve it's easy. Check out the sources, go to usr.sbin/bhyve and run > > > > make depend; make obj; make; make install. > > >Unfortunately, after all my efforts, with libvmmapi and bhyve compiled from > >CURRENT sources: > > >Aug 09 12:46:52: warning; UEFI graphics is only available in FreeBSD 11 and > >newer > >Aug 09 12:46:52: booting > > >Bummer! > > It's possible that you do have a bhyve system with graphics support. > Unfortunately vm-bhyve isn't aimed at users who have manually merged changes, > and will blindly tell anyone that isn't running 11+ that they don't have > graphics support. > > You have a couple of options - > > 1) Run bhyve manually rather than using vm-bhyve > 2) Upgrade to the 11 beta > 3) Modify vm-bhyve to not automatically skip graphics support if the version > is less than 11 >Aha! I've commented out the version check from vm-run, and now when >trying to install, I get: >авг 09 17:01:05: initialising >авг 09 17:01:05: [loader: none] >авг 09 17:01:05: [uefi: yes] >авг 09 17:01:05: [cpu: 1] >авг 09 17:01:05: [memory: 1G] >авг 09 17:01:05: [hostbridge: standard] >авг 09 17:01:05: [com ports: com1] >авг 09 17:01:05: [uuid: 093f9134-5e18-11e6-9502-5404a6b49a66] >авг 09 17:01:05: [utctime: no] >авг 09 17:01:05: [debug mode: no] >авг 09 17:01:05: [primary disk: disk0.img] >авг 09 17:01:05: [primary disk dev: file] >авг 09 17:01:05: generated static mac 58:9c:fc:03:77:3b (based on >'win:0:1470736865:0') >авг 09 17:01:05: initialising network device tap5 >авг 09 17:01:05: adding tap5 -> bridge1 (isolated) >авг 09 17:01:05: dynamically allocated port 5900 for vnc connections >авг 09 17:01:05: booting >авг 09 17:01:05: [bhyve options: -c 1 -m 1G -Hwl >bootrom,/d02/vm/.config/BHYVE_UEFI.fd -U 093f9134-5e18-11e6-9502-5404a6b49a66] >авг 09 17:01:05: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >4:0,ahci-hd,/d02/vm/win/disk0.img -s 5:0,virtio-net,tap5,mac=58:9c:fc:03:77:3b >-s >6:0,fbuf,tcp=0.0.0.0:5900,wait] >авг 09 17:01:05: [bhyve console: -l com1,/dev/nmdm0A] >авг 09 17:01:05: [bhyve iso device: -s >3:0,ahci-cd,/d02/vm/.iso/bhyve_win7_x64.iso] >авг 09 17:01:05: starting bhyve (run 1) >авг 09 17:01:05: bhyve exited with status 1 >авг 09 17:01:05: destroying network device tap5 >авг 09 17:01:05: stopped Looks like bhyve is exiting immediately. Easiest thing is to probably add debug="yes" to the guest config file then try and run it again. Any output from the bhyve command will be written to bhyve.log (note that's a different file to vm-bhyve.log). Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Bhyve tests and findings
>Lars Engels wrote: > > > > > > > > > > uefi="yes" > > > > > graphics="yes" > > > > > > > > Dear Colleagues, > > > > > > > > Can I enjoy and test all those nice "UEFI-GOP" features on a > > > > 10.3-RELEASE system, or do I have to install CURRENT for that? > > > > > > AFAIK you can checkout usr.sbin/bhyve from the 11 branch and compile it > > > on 10.3. > > > > I'm not sure how to compile something from a different branch without > > compiling the whole world. make what? > > For bhyve it's easy. Check out the sources, go to usr.sbin/bhyve and run > > make depend; make obj; make; make install. >Unfortunately, after all my efforts, with libvmmapi and bhyve compiled from >CURRENT sources: >Aug 09 12:46:52: warning; UEFI graphics is only available in FreeBSD 11 and >newer >Aug 09 12:46:52: booting >Bummer! It's possible that you do have a bhyve system with graphics support. Unfortunately vm-bhyve isn't aimed at users who have manually merged changes, and will blindly tell anyone that isn't running 11+ that they don't have graphics support. You have a couple of options - 1) Run bhyve manually rather than using vm-bhyve 2) Upgrade to the 11 beta 3) Modify vm-bhyve to not automatically skip graphics support if the version is less than 11 Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Bhyve tests and findings
> Zitat von Peter Grehan : > Hi Mike, > >> - Windows 8, 8.1 and 10 installs and runs in graphical mode flawlessly. > > Have you had any issues with the XHCI mouse on 8/8.1 ? > >> - I was able to graphically Restore/Reconfigure a Acronis >> Windows-Backup into a Bhyve instance using the Acronis Restore-CD >> (Converting a BIOS Win8.1 to UEFI Win8.1) > > Very nice ! > >> - Only vnclient from FreeBSD can connect to the bhyve VNC Server. >> I havn't found any vncviewer running on Windows which where able to >> work (tried UltraVNC, RealVNC, ...) > > Some VNC clients refuse to connect when only null-auth is advertized > by the server. There is a patch to bhyve to support VNC password-auth, > which may fix the issue with these clients. > >Yes, that sounds reasonable. >> - in VNC only most basic Keys work most special characters like (*\@) >> (and of course no german localization) but at least a usual US-kbd >> would be helpful. >> (Is there a way to debug the keystrokes or duplicate a localized VNC >> kbd from some VNC server) > > Nothing outside of modifying the source, but it seems useful enough > to add a parameter for this. > >> - For the SAP-Systems it seems that only 4 disks get used when the >> disk type is virtio-blk. >> (Is this intentionally or a feature of vm-bhyve? How to provide more >> disks) > > I'll let Matt comment on that. There's no limitation with guests that > support MSI interrupts for adapters. Unfortunately, Windows guests > require legacy interrupts for the AHCI controller, which is where the > restriction originates. > >I'd like to use a 6 disks setup with Centos7. >Centos7 on XEN PVM has no issue with supporting 6 paravirtulized disks. >Thats the config (for vm-bhyve) where only the first 4 disks are used for the >guest (Centos7): >uefi="yes" >cpu=1 >memory=2G >network0_type="virtio-net" >network0_switch="public" >disk0_name="root" >disk0_type="virtio-blk" >disk0_dev="zvol" >disk1_name="swap" >disk1_type="virtio-blk" >disk1_dev="zvol" >disk2_name="sapmnt" >disk2_type="virtio-blk" >disk2_dev="zvol" >disk3_name="usrsap" >disk3_type="virtio-blk" >disk3_dev="zvol" >disk4_name="db" >disk4_type="virtio-blk" >disk4_dev="zvol" >disk5_name="log" >disk5_type="virtio-blk" >disk5_dev="zvol" >graphics="yes" >graphics_port="5903" >graphics_listen="0.0.0.0" >graphics_res="1600x900" >graphics_wait="no" >xhci_mouse="yes" Sorry I didn't actually spot that you were using my vm-bhyve code to run the guest the first time. The UEFI code in vm-bhyve is currently limited to using slots 4/5/6 for disks so you should only see 3... (I put a CD device on slot 3) If you look in the /path/to/guest/vm-bhyve.log log file you'll probably see something like "ending disks as disk2 due to UEFI limitations". I've never been 100% on which guests have the limitation and which don't, so in UEFI mode I played it safe and just used the 3 slots. If you run the guest with the grub loader instead of UEFI you should get all the disks. If you want, you can comment out the limitation in the vm-bhyve source and see if the guest will pick up the extra disks. It's the 4 lines after the "can't go past slot 6" comment around line 364 of lib/vm-run. If it works it may be worth me adding a configuration option to toggle this limit. As mentioned in my last email FreeBSD CURRENT (12) now supports up to 32 disks per ahci device, and I've successfully had 8 disks in Windows using just 1 pci slot. This won't be in 11.0 unfortunately (as far as I'm aware) but I hope it gets merged into a future 11 release. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: A couple of newbie questions
Colleagues, >I like bhyve very much, and have sucessfully run FreeBSD and Ubuntu >16.04 server in FreeBSD 10.3 bhyve, with vm-bhyve as a shell. >Now I am trying to boot Windows 7 but have not succeeded so far. >However there are things I don't quite understand. A couple of questions, if >you allow. >1. Why is it that for some guest systems, there are two stages: first >bhyveload or grub2-bhyve and then bhyve itself. And for UEFI systems there is >>only one stage. Does it mean that eventually bhyveload and grub2-bhyve will >become totally obsolete and the one-stage VM startup procedure >will become >the universal method? Originally the quickest and easiest way to get guests running on bhyve was to load the guest into memory "manually", then run it. This started with just bhyveload to load FreeBSD guests, then expanded to supporting various other guests with grub-bhyve. UEFI was required to get Windows running (and makes bhyve act a bit more like other hypervisors) but took a lot of effort. UEFI does seem to be the best way to run most guests. Linux guests have always been a bit finicky unless they have grub2 installed, and the ability to get remote access to the console makes things like a bhyve web frontend feasible. Bhyveload is still a pretty quick and useful way to run FreeBSD guests though, and I don't think that or grub-bhyve will go anywhere. >2. All this fbuf/VNC stuff looks cool, but I don't quite understand. You can >see the guest OS's console in VNC, like the Windows desktop, or only >the UEFI >shell, and then you have to access the guest OS via RDP/ssh etc ? With the frame buffer enabled you see the full guest OS in vnc, same as you would in Virtualbox/VMWare/etc. Matt >TIA for explanations. >-- >Victor Sudakov, VAS4-RIPE, VAS47-RIPN >sip:suda...@sibptus.tomsk.ru ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Bhyve tests and findings
> > uefi="yes" > graphics="yes" > Dear Colleagues, > Can I enjoy and test all those nice "UEFI-GOP" features on a 10.3-RELEASE > system, or do I have to install CURRENT for that? I believe you should also be able to use the 11 beta. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
FW: Bhyve tests and findings
(sent this to the wrong list...) -Original Message- From: Matt Churchyard Sent: 04 August 2016 12:57 To: 'Michael Reifenberger' Cc: freebsd...@freebsd.org Subject: RE: Bhyve tests and findings >Hi, >after waiting for UEFI-GOP and using bhyve (with vm-bhyve as a convenient >tool) an a new E3-1225 v5 based system, the following are my findings >so far. >(BTW: > Currently my rational for running bhyve is twofold: > - Run a Windows 8.1 or 10 instance for accessing different remote locations > via different VPN solutions > - Run Centos7/RHEL7 instances with SAP Systems on it >) >But first a huge Thanks to all who worked on bhyve and made it usable in its >current state. >That alone is quite impressive. >What works so far: >- Windows 8, 8.1 and 10 installs and runs in graphical mode flawlessly. >- Centos7 installs and runs too using the UEFI-GOP Image (Yeah, no more >Grub fiddling :-) >- I was able to graphically Restore/Reconfigure a Acronis Windows-Backup into >a Bhyve instance > using the Acronis Restore-CD (Converting a BIOS Win8.1 to UEFI >Win8.1) > What doesn't: >- Only vnclient from FreeBSD can connect to the bhyve VNC Server. > I havn't found any vncviewer running on Windows which where able to >work (tried UltraVNC, RealVNC, ...) I've been using TightVNC on Windows (first free client I came across) since GOP support came out and have not had any problems with it. > - in VNC only most basic Keys work most special characters like (*\@) > (and of course no german localization) but at least a usual US-kbd would be > helpful. It's not clear whether you're saying that those special character have a problem or not? I'm in the UK and these keys seem to work fine in the VNC client I'm using. > (Is there a way to debug the keystrokes or duplicate a localized VNC kbd > from some VNC server) >- For the SAP-Systems it seems that only 4 disks get used when the disk type >is virtio-blk. > (Is this intentionally or a feature of vm-bhyve? How to provide more >disks) I know that the UEFI firmware only connects slots 3/4/5/6, but I don't know if some guests can support non-boot disks on other slots. In my bhyve manager I used to limit Windows to 3 disks (+1 cd), however I have tested 8 disks in Server2012 with a FreeBSD 12 host, which allows up to 32 disks per ahci controller. >- It seems to miss a way to add an ISO CD/DVD without booting from it >automatically. > Also ISO's seem to miss a hot-plug feature (f.e. for inserting driver CD's > after installation. I've just tried adding a bootable CD to bhyve as a second device and yes, it does appear to try and boot from it even though the hdd is bootable. I'm sure I've seen a UEFI cli at some point but I don't know if there's any way to configure boot order (devs?) Obviously hot-plug CD would be nice and I expect is a feature that would come eventually once the basics are finished. >Some additional questions: >- Can one over-provisioning/ballooning guest memory's ? >- Is it (speed-wise) better to use ZFS-zvol's or files in regular >ZFS-directories? >- Are the virtio-blk or ahci-hd disks having the same overhead? >- Can ahci-hd be used paravirtualized in Centos? These are probably all questions for the devs... Matt >Thanks in advance! >All in all it looks quite promising! >Greetings >--- Mike > >Gruß >--- >Michael Reifenberger >___ >freebsd-virtualization@freebsd.org mailing list >https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >To unsubscribe, send any mail to >"freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Understanding Bhyve shutdown
As I understand it 1) shutdown from guest 2) 'kill ' -> pressing the power button once. 3) --force-poweroff -> holding power button in 4) --force-reset -> pressing the reset button 5) bhyvectl --destroy -> same as 3? (although vmm is destroyed as well) 1 or 2 are obviously preferred. I've found however that some guests don't respond that well to the apci event. CentOS 6 seems to need apci installing and enabling?!, and my Win2012R2 machine will only shutdown if I send the signal twice. The rest are all hard shutdowns that should not be seen as a way to stop the guest, just a last resort if it can't be stopped correctly. I don't know the practical difference between poweroff&destroy vs just destroy. I see no reason why the documentation suggests reboot rather than shutdown. Bhyve exits either way and the only difference is the exit code. When using one of the bhyve management tools that support reboots (such as vmrun.sh/vm-bhyve/iohyve) however, a "restart" exit code (0) will cause the guest to restart automatically; If a guest is locked up for example, a --force-reset will cause it to immediately reboot. >From a host, the only safe choice is 'kill ' (possibly multiple times) >and wait for bhyve process to (hopefully) exit. If that doesn't work either >the acpi issue needs fixing or ideally the guest needs to be stopped using its >built-in shutdown function. The devs will have to confirm why they're implemented the way they are. My instinct is that bhyvectl works on the vmm device, and reset/poweroff are just functions that are possible via that interface, and so have been exposed. The ACPI shutdown event likely needs to be initiated by the bhyve process itself, hence using a signal to it. /end-speculation I think most users will want to use a bhyve tool so the raw specifics of the bhyve/bhyvectl commands are glossed over, although that doesn't mean the handbook documentation of the base commands shouldn't be as complete/correct as possible of course. Matt -Original Message- From: owner-freebsd-virtualizat...@freebsd.org [mailto:owner-freebsd-virtualizat...@freebsd.org] On Behalf Of Roman Bogorodskiy Sent: 13 April 2016 11:55 To: freebsd-virtualization@FreeBSD.org Subject: Understanding Bhyve shutdown Hi, I was trying to get better understanding of how to properly shutdown VMs in bhyve, but unfortunately the documentation does not provide much details on that. Specifically, handbook [I] suggests to reboot a machine and then run bhyvectl --destroy on it. The bhyvectl(8) manpage mentions the '--force-reset' and '--force-poweroff' switches, but does not give details on those. I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html I tried all the options I know and wrote down the results. I also have some questions, hopefully you'll be able to answer some of them. 1. bhyvectl --vm=$name --destroy * looks like hard poweroff in the guest * the corresponding bhyve(8) process goes away * /dev/vmm/ entry goes away In my experience, it's a dangerous way to shutdown a VM because sometimes it appears it damages the image and VM fails to boot with something like this: --- Starting devd. mode = 0100600, inum = 170269, fs = / panic: ffs_valloc: dup alloc cpuid = 0 KDB: stack backtrace: #0 0x80984e30 at kdb_backtrace+0x60 #1 0x809489e6 at vpanic+0x126 #2 0x809488b3 at panic+0x43 #3 0x80b74a6e at ffs_valloc+0x84e #4 0x80bb60ad at ufs_makeinode+0x7d #5 0x80bb24fd at ufs_create+0x2d #6 0x80e71841 at VOP_CREATE_APV+0xa1 #7 0x809cd9e6 at uipc_bindat+0x346 #8 0x809c5488 at kern_bindat+0x108 #9 0x809c52a7 at sys_bind+0x77 #10 0x80d4b3f7 at amd64_syscall+0x357 #11 0x80d30adb at Xfast_syscall+0xfb Uptime: 3s Dump failed. Partition too small. --- 2. kill -SIGTERM $bhyve_pid If guest supports ACPI shutdown: * guest shuts down cleanly * the corresponding bhyve(8) process terminates * /dev/vmm entry is still here, need bhyvectl --destroy for complete cleanup If guest does not support ACPI shutdown (such as doing sysctl hw.acpi.power_button_state=NONE): * Nothing happens Q1: Is there a way to know if a guest reacted to power button but waiting for the bhyve process to terminate? Q2: Why it's not done via bhyvectl (it seems that it's easier for users + don't have to overload a useful SIGTERM signal) 3. bhyvectl --vm=$name --force-poweroff * looks like hard poweroff in the guest * the corresponding bhyve(8) process goes away * /dev/vmm entry is still here, need bhyvectl --destroy for complete cleanup Q: what's the practical difference with just doing --destroy right away? 4. bhyvectl --vm=$name --force-reset Looks very similar to item #3 with just different exit code (reboot appears to be using 0, while shutdown and halt use 1 and 2). Q: what's the practical use of it? Would greatly appreciate if somebody could provide more de
RE: adding diskspace to a bhyve instance
On 19/11/15 12:20 PM, John wrote: > Hello list, > > What's the best way of increasing the space of a bhyve guest instance? > Would it be via growfs? Inside or outside of the vm? Or would it be > better to truncate another chunk of space and refer to it in /etc/fstab? > > thanks, > Hello John, > > - Do you use virtio-blk (a file created with 'truncate') or ahci-hd (A > ZFS filesystem, for example) for your guest disk? You have to increase the > space of the virtual disk that bhyve uses. I see no reason to associate truncated files with virtio, and ZVOL with ahci-hd. You could just as easily use virtio with a ZVOL and ahci-hd with a truncated file. The choice between the two emulation modes really just comes down to which is supported better by the guest, or the one that performs better in your environment. (For example Windows doesn't support virtio-blk, so you need to use ahci-hd regardless of whether you're using ZFS/ZVOL storage or a sparse file). > - If you used 'truncate', my guess is that you can use truncate to > create a new file of larger size. Then, boot into a livecd in bhyve with both > > disks and do a 'dd' from one disk to the other. I have tried to 'dd' > between files on the host but that didn't seem to work. > - If it is a ZFS filesystem, create a new filesystem with 'zfs create > -V 50gb zroot/new-volume/' then use a 'zfs send ... | zfs receive ...' > Not that I've ever done it, but I'm under the impression you should be able to just make the existing disk device bigger by 'truncating' it to a larger size, or by using ZFS commands to increase the size of a ZVOL. There should be no need to create a brand new disk image and dd or send/recv the data over. Of course that's the easy bit. The more dangerous part is resizing the partitions inside the guest (if it's not whole disk ZFS), and then resizing the filesystems. If the disk is GPT partitioned in the guest you will probably have to recover the partition table first, as the secondary copy will no longer exist at the end of the disk. You'll then need to resize the partitions (hopefully the 'main' partition you want to resize is the last on the disk as that'll probably make it easier). Once done you then need to resize the filesystem. For ZFS you can usually just 'zpool online -e'. For UFS you'll need to grow the filesystem as shown in the handbook. Adding another disk image to the guest and just using that is obviously a much simpler solution, if you're happy to permanently have multiple disks attached to the guest and reorganise your data. That's what I did when one of my mail vm's got too full; I just created a second disk, formatted and mounted it on '/mail', and moved all the users there. Matt > - Are these FreeBSD guests (and which filesystem - UFS or ZFS) or Linux > guests? > - If these are FreeBSD guests running UFS, look at > https://www.freebsd.org/doc/handbook/disks-growing.html > - If these are FreeBSD guests running ZFS, you can probably create the > filesystems on your new disk and then use zfs send & receive > - If these are linux guests, you will have to use a livecd and > something like 'gpart'. > Thanks, > Manas ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: vm-bhyve port upgrade
>> I am now looking at actually implementing static macs for all interfaces, as >> I’d rather guests saw the same mac address every run just in case they tie >> configuration to the >> mac (important for vm-bhyve as simply starting >> guests in a different order will change what tap devices they get). Also >> tap/slot/func isn’t much of a uniqueness guarantee >> across multiple hosts. > Yes, and udev treats MAC as ethX = MAC. So linux guests using static ip's > will be quite broken unless some more fiddling is done. Static MAC's aren't > the only way to handle > this, but it's the best IMO. > Adam I have thought about this briefly in the past but never really put much into it as I just do what every other bhyve manager does - which is let bhyve handle it. Looking into it more, when you create a vm-bhyve guest, it really should *just work* every time you run it, wherever you run it. It's not ideal to have your statically assigned interface disappear just because you've booted the guest with a different tap interface. As such I think the best option is to assign a mac that's fixed (which is effectively what happens with real hardware). As such I've updated the latest version on GitHub (0.9.16) to automatically assign a fixed mac to all interfaces if one isn't already set. The fixed mac is written to the config file just as if you'd set one manually - so you can also modify it if needed. It's not 100% perfect as I'm using the assigned bhyve prefix of 0x589cfc00 which only gives the last 4 bytes to play with. I use 4 bytes from a hash of the guest name, interface number and time which isn't quite as collision-safe as making the whole mac random but I think using the bhyve prefix makes sense (even though I think bhyve itself still uses NetApp macs...) I did consider not using a random mac and just using some stored configuration to keep track of what's assigned/next available but that's pretty much guaranteed to cause collisions if you have multiple hosts on the same network running independent copies of vm-bhyve. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: vm-bhyve port upgrade
On Thu, Nov 12, 2015 at 4:37 AM, Matt Churchyard via freebsd-virtualization mailto:freebsd-virtualization@freebsd.org>> wrote: Hello, For anyone interested I have submitted a PR to update the version of vm-bhyve in the ports tree. Primarily this fixes the off-putting, but completely benign error printed when users run 'vm init' (the very first thing to run...) I've no idea if I've got the diff format right though. Also adds various small fixes, and the following changes since the last ports version: Command to rename a guest Configuration options for utctime, hostbridge, disk options, debug mode, custom grub commands, virtual random device Snapshot and rollback commands when using ZFS Allows use of custom bridges and/or tap devices in addition to the normal automated networking Ability to specify a custom path for disk devices Guests can now automatically attach correctly to virtual switches if the real interface(s) (and thus the bridge) are using jumbo frames Template options to specify zfs dataset/zvol properties to apply when creating a guest (most useful for zvol volblocksize) New 'info' commands showing detailed guest/switch details including disk & network usage No longer replaces dnsmasq.conf, just provides a sample config for the user to apply if they want dhcp on a nat-enabled virtual switch. > If I statically define MAC addresses, is there a possibility for collisions? > I think it would be better to default to an auto generated permanent MAC. > -- > Adam At the moment the default is to just let bhyve handle mac addresses as normal. I have considered assigning static addresses to guests, as the bhyve man page suggests their mac addresses are only unique based on slot/func & device name. This could mean guests on different hosts get the same mac at the same time, or guests on the same machine - if they are run at different times and use the same tap interface. The current bhyve code also means the mac could change if I assign a different slot/func or tap device for a guest. If a user manually specifies a mac address in the guest configuration then obviously it becomes the users responsibility to make sure it’s unique in their environment. I am now looking at actually implementing static macs for all interfaces, as I’d rather guests saw the same mac address every run just in case they tie configuration to the mac (important for vm-bhyve as simply starting guests in a different order will change what tap devices they get). Also tap/slot/func isn’t much of a uniqueness guarantee across multiple hosts. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
vm-bhyve port upgrade
Hello, For anyone interested I have submitted a PR to update the version of vm-bhyve in the ports tree. Primarily this fixes the off-putting, but completely benign error printed when users run 'vm init' (the very first thing to run...) I've no idea if I've got the diff format right though. Also adds various small fixes, and the following changes since the last ports version: Command to rename a guest Configuration options for utctime, hostbridge, disk options, debug mode, custom grub commands, virtual random device Snapshot and rollback commands when using ZFS Allows use of custom bridges and/or tap devices in addition to the normal automated networking Ability to specify a custom path for disk devices Guests can now automatically attach correctly to virtual switches if the real interface(s) (and thus the bridge) are using jumbo frames Template options to specify zfs dataset/zvol properties to apply when creating a guest (most useful for zvol volblocksize) New 'info' commands showing detailed guest/switch details including disk & network usage No longer replaces dnsmasq.conf, just provides a sample config for the user to apply if they want dhcp on a nat-enabled virtual switch. Matt (churchers) ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
bhyve uefi error please help me!
> Hello! > not workink bhyve and windows please help me: > bhyve > -c 2 > -s 0,hostbridge > -s 3,ahci-hd,/images/win.img > -s 4,ahci-cd,/images/win_repack.iso > -s 10,virtio-net,tap0 > -s 31,lpc > -l com1,/dev/nmdm0A > -l com2,/dev/nmdm1A > -l bootrom,/path/to/BHYVE_UEFI.fd > -m 2G -H -w > windows > bhyve: invalid lpc device configuration 'bootrom,BHYVE_UEFI_20151002.fd' > https://people.freebsd.org/~grehan/bhyve_uefi/windows_install.txt Is this on FreeBSD CURRENT (FreeBSD-11)? Other people on this list will know for certain but that error suggests to me that your version of bhyve might not have bootrom support. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: C bhyve administration tool
>> Hello! >> >> Couple months ago I started writing a bhyve management tool in C for >> our startup, in preparation for migration to FreeBSD for our servers. >> The goal was to be able to create, drop, and auto-start/stop/restart >> VMs, individually or all at once, and provide a plugin infrastructure >> to expose some metrics / errors for reporting and aggregation. More >> importantly it would detect errors/failures, report them and attempt >> to restart the VM per assigned policy. >> >> The effort stalled due to some high priority work that crept in. >> However, I am planning on restarting soon and wanted to check if >> there was a need for something like this. I ran across Michael >> Dexter's vmrc, and although its shell based, it does provide quite a >> bit of features that we were looking for. There's also iohyve. And I >> heard Peter and Neel are working on something as well. If there is >> interest, this would be open sourced from the start. Or if there is >> an existing effort that addresses these problems, I would love to >> contribute to that project. >> >> Thoughts, comments, concerns, please share. >> > >> vm-bhyve is most mature to this point IME, just don't use the ports >> version. > >> Adam > > Thanks Adam :) > > I could do with updating the port. Unfortunately this is my first > porting experience so not exactly sure of the 'correct' channel to go > through to get things updated. It took 2 months for my original port > request to be addressed and I'm not sure re-opening that bug is the > right way to do it. > > If anyone uses the port, the error that gets printed when running 'vm > init' can be completely ignored. It's just trying to load any existing > virtual switches from the switch configuration file, and complaining > that the switch configuration file doesn't exist. Apart from that and > a few newer features/tweaks, I don't think there's anything > particularly show-stopping about the current port/package if people > prefer installing it that way to downloading from GitHub. > > Of course it would still be interesting to see a "real" bhyve > management tool (possibly even an official one) written in C. > >to update an existing port, create a PR with the patch to update it, and then >wait. If a committer doesn't grab it within a reasonable time, post to >freebsd-ports to get their >attention. >Do run poudriere testport and provide the output. >-- >Larry Rosenman http://www.lerctr.org/~ler --- >Hi Matt, >if you need someone to help you updating the port and get it committed, drop >me a line and I'll help. >Lars Hi Lars, Larry Thanks both for your help. Once I'm sure I won't be making any changes 5 minutes after submitting it, and don't have any other silly bugs scaring new users I will submit a PR. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: C bhyve administration tool
> Hello! > > Couple months ago I started writing a bhyve management tool in C for > our startup, in preparation for migration to FreeBSD for our servers. > The goal was to be able to create, drop, and auto-start/stop/restart > VMs, individually or all at once, and provide a plugin infrastructure > to expose some metrics / errors for reporting and aggregation. More > importantly it would detect errors/failures, report them and attempt > to restart the VM per assigned policy. > > The effort stalled due to some high priority work that crept in. > However, I am planning on restarting soon and wanted to check if there > was a need for something like this. I ran across Michael Dexter's > vmrc, and although its shell based, it does provide quite a bit of > features that we were looking for. There's also iohyve. And I heard > Peter and Neel are working on something as well. If there is interest, > this would be open sourced from the start. Or if there is an existing > effort that addresses these problems, I would love to contribute to that > project. > > Thoughts, comments, concerns, please share. > >vm-bhyve is most mature to this point IME, just don't use the ports version. > Adam Thanks Adam :) I could do with updating the port. Unfortunately this is my first porting experience so not exactly sure of the 'correct' channel to go through to get things updated. It took 2 months for my original port request to be addressed and I'm not sure re-opening that bug is the right way to do it. If anyone uses the port, the error that gets printed when running 'vm init' can be completely ignored. It's just trying to load any existing virtual switches from the switch configuration file, and complaining that the switch configuration file doesn't exist. Apart from that and a few newer features/tweaks, I don't think there's anything particularly show-stopping about the current port/package if people prefer installing it that way to downloading from GitHub. Of course it would still be interesting to see a "real" bhyve management tool (possibly even an official one) written in C. Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Illumos boot
>> -s 0,hostbridge > Things should work if you leave out the hostbridge. The PCIe > capability that is tacked on to this will make Illumos use MSI/MSIx for > the virtio adapter which apparently hits a bug in the driver. Without > it, the virtio driver will fall back to legacy interrupts. This also > means that the virtio adapter will be confined to slots 3/4/5/6. > [root@smartos ~]# echo ::interrupts | mdb -k > IRQ Vect IPL BusTrg Type CPU Share APIC/INT# ISR(s) > 10x40 5 ISAEdg Fixed 0 1 0x0/0x1 i8042_intr > 30xb1 12 ISAEdg Fixed 0 1 0x0/0x3 asyintr > 40xb0 12 ISAEdg Fixed 0 1 0x0/0x4 asyintr > 90x81 9 PCILvl Fixed 1 1 0x0/0x9 acpi_wrapper_isr > 12 0x41 5 ISAEdg Fixed 1 1 0x0/0xc i8042_intr > 16 0x42 5 PCILvl Fixed 1 1 0x0/0x10 ahci_intr > 17 0x43 5 PCILvl Fixed 0 1 0x0/0x11 ahci_intr > 18 0x60 6 PCILvl Fixed 1 1 0x0/0x12 virtio_intx_dispatch > 160 0xa0 0 Edg IPIall 0 - poke_cpu > 208 0xd0 14 Edg IPIall 1 - kcpc_hw_overflow_intr > 209 0xd1 14 Edg IPIall 1 - cbe_fire > 210 0xd3 14 Edg IPIall 1 - cbe_fire > 240 0xe0 15 Edg IPIall 1 - xc_serv > 241 0xe1 15 Edg IPIall 1 - apic_error_intr > (I believe Andriy Gapon (cc'd) has a fix for this in Illumos) > later, > Peter. Thanks Peter. I can't believe how many times I checked you Illumos instructions making sure there were no obvious differences, and didn't notice the hostbridge was missing. I think I just had it in my mind that the hostbridge was a basic required device. Regards Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Illumos boot
> Hi, >> .. > /pci@0,0/pci8086,2821@4/disk@0,0 (sd1) online > NOTICE: vioif0: Got MAC address from host: e4:94:1:0:ff:ff > pseudo-device: tun0 > tun0 is /pseudo/tun@0 > pseudo-device: lx_systrace0 > lx_systrace0 is /pseudo/lx_systrace@0 > > panic[cpu0]/thread=ff0002566c40: BAD TRAP: type=d (#gp General > protection) rp=ff00025664c0 addr=20 > > sched: #gp General protection > addr=0x20 > pid=0, pc=0xf80d375a, sp=0xff00025665b0, eflags=0x10282 > cr0: 8005003b cr4: > 406b8 > cr2: fed3b5accr3: 1dc0cr8: c > >rdi: 7f1a90ff00c3 rsi: ff00c33321f8 rdx: ff00c37f5828 >rcx: ff00c3868603 r8: ff00ca7aa600 r9: 2ba6 >rax:0 rbx: ff00c30d0ef0 rbp: ff00025665c0 >r10: fb8554c4 r11:1 r12: 1f >r13: ff00c319f880 r14: 10 r15: 20 >fsb:0 gsb: fbc326a0 ds: 4b > es: 4b fs:0 gs: 1c3 >trp:d err:0 rip: f80d375a > cs: 30 rfl:10282 rsp: ff00025665b0 > ss: 38 > > This is the log of the bhyve options used (apart from 1 cpu, 1G ram) > > Oct 13 14:22:43: [bhyve devices: -s 0,hostbridge -s 31,lpc -s > 4:0,ahci-hd,/data/vm/smartos/disk0.img -s 7:0,virtio-net,tap1] Oct 13 > 14:22:43: [bhyve console: -l com1,/dev/nmdm1A -l com2,/dev/nmdm2A] > Oct 13 14:22:43: [bhyve options: -Hw -l > bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd] > Oct 13 14:22:43: [bhyve iso device: -s > 3:0,ahci-cd,/data/vm/.iso/smartos-latest.iso] > Ouch, even with the additional verbosity, the output isn't very insightful. > All I can glean is that you are reasonably far along in the boot > process. > You could try to run with KMDB (-k boot option) and do some disassembly > around that program counter to see if any specific module is > implicated. > Alternately, I'd omit the network device and see how far that gets you. It does appear that I can get as far as a login console without the virtio-net device Matt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Illumos boot
>Hi, >Please see inline. >On Oct 13, 2015, at 7:17 AM, Matt Churchyard via freebsd-virtualization > wrote: > In my quest to continue expanding guest support in my vm-bhyve utility (See > https://github.com/churchers/vm-bhyve :) ), I've found the Windows support > pretty solid once I got clear on the slot requirements. I'm now trying an OS > that requires CSM (Illumos) but unfortunately I'm currently struggling to get > it to boot up correctly. > > Here's an example of the command I'm generating at the moment (This is > running on an Intel Core-i3): > > bhyve -c 2 -m 2G -s 0,hostbridge -s 31,lpc \ > -s 3,ahci-cd,/data/vm/.iso/smartos-latest.iso \ > -s 4:0,ahci-hd,/data/vm/smartos/disk0.img \ > -s 5:0,virtio-net,tap0 \ > -l com1,stdio -l com2,/dev/nmdm2A \ > -H -l bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd \ > smartos > > I have com1 set to stdio so I can easily watch the output as it runs. > It tends to get as far as "Legacy INT19 Boot...", then fall over. > Depending on whether I put the network interface directly in the slot after > the HDD, I seem to get different errors - > > slot 3 - cd > slot 4 - hdd > slot 5 - virtio-net > > panic[cpu0]/thread=ff01457cdb40: BAD TRAP: type=e (#pf Page fault) > rp=ff0004a69a60 addr=40 occurred in module "genunix" due to a NULL > pointer dereference > > slot 3 - cd > slot 4 - hdd > slot 7 - virtio-net > > panic[cpu1]/thread=ff0004002c40: BAD TRAP: type=d (#gp General > protection) rp=ff0004002740 addr=0 > > On com2 I see the boot menu, then one and a half lines of dots. The second > line of dots stops about 2/3 of the way across. >Have you tried booting illumos in verbose mode - edit the grub command line >and provide '-v'. That may give you a better backtrace than a >program >counter. This is what I get from a verbose boot: Bhyve-HandleProtocol: Copying DevPath: PciRoot(0x0)/Pci(0x3,0x0)/Sata(0x0,0x0,0x0) [32] Legacy INT19 Boot... cpu0: x86 (chipid 0x0 GenuineIntel 206A7 family 6 model 42 step 7 clock 3109 MHz) cpu0: Intel(r) Core(tm) i3-2100 CPU @ 3.10GHz pseudo-device: stmf_sbd0 stmf_sbd0 is /pseudo/stmf_sbd@0 pseudo-device: lofi0 lofi0 is /pseudo/lofi@0 pseudo-device: devinfo0 devinfo0 is /pseudo/devinfo@0 acpinex0 at root acpinex0 is /fw iscsi0 at root iscsi0 is /iscsi xsvc0 at root: space 0 offset 0 xsvc0 is /xsvc@0,0 acpinex: sb@0, acpinex1 acpinex1 is /fw/sb@0 pseudo-device: pseudo1 pseudo1 is /pseudo/zconsnex@1 pseudo-device: pseudo2 pseudo2 is /pseudo/zfdnex@2 /pci@0,0/pci8086,2821@3 : SATA CD/DVD (ATAPI) device at port 0 model BHYVE SATA DVD ROM firmware 001 serial number BHYVE-EA14-A68A-54FA supported features: DMA SATA Gen3 signaling speed (6.0Gbps) pseudo-device: llc10 llc10 is /pseudo/llc1@0 pseudo-device: power0 power0 is /pseudo/power@0 pseudo-device: ramdisk1024 ramdisk1024 is /pseudo/ramdisk@1024 pseudo-device: ucode0 ucode0 is /pseudo/ucode@0 pseudo-device: zfs0 zfs0 is /pseudo/zfs@0 pseudo-device: srn0 srn0 is /pseudo/srn@0 pseudo-device: dtrace0 dtrace0 is /pseudo/dtrace@0 pseudo-device: dcpc0 dcpc0 is /pseudo/dcpc@0 pseudo-device: fasttrap0 fasttrap0 is /pseudo/fasttrap@0 pseudo-device: fbt0 fbt0 is /pseudo/fbt@0 pseudo-device: profile0 profile0 is /pseudo/profile@0 pseudo-device: lockstat0 lockstat0 is /pseudo/lockstat@0 pseudo-device: sdt0 sdt0 is /pseudo/sdt@0 pseudo-device: systrace0 systrace0 is /pseudo/systrace@0 pseudo-device: ipd0 ipd0 is /pseudo/ipd@0 pseudo-device: stmf0 stmf0 is /pseudo/stmf@0 sd0 at ahci0: target 0 lun 0 sd0 is /pci@0,0/pci8086,2821@3/cdrom@0,0 pseudo-device: fssnap0 fssnap0 is /pseudo/fssnap@0 /pci@0,0/pci8086,2821@3/cdrom@0,0 (sd0) online /pci@0,0/pci8086,2821@4 : SATA disk device at port 0 model BHYVE SATA DISK firmware 001 serial number BHYVE-3083-1AF1-1754 supported features: 48-bit LBA, DMA, Native Command Queueing SATA Gen3 signaling speed (6.0Gbps) Supported queue depth 32 capacity = 62914560 sectors WARNING: kvm: no hardware support pseudo-device: pool0 pool0 is /pseudo/pool@0 pseudo-device: bpf0 bpf0 is /pseudo/bpf@0 sd1 at ahci1: target 0 lun 0 sd1 is /pci@0,0/pci8086,2821@4/disk@0,0 pseudo-device: pm0 pm0 is /pseudo/pm@0 pseudo-device: nsmb0 nsmb0 is /pseudo/nsmb@0 pseudo-device: tap0 tap0 is /pseudo/tap@0 /pci@0,0/pci8086,2821@4/disk@0,0 (sd1) online NOTICE: vioif0: Got MAC address from host: e4:94:1:0:ff:ff pseudo-device: tun0 tun0 is /pseudo/tun@0 pseudo-device: lx_systrace0 lx_systrace0 is /pseudo/lx_systrace@0 panic[cpu0]/thread=ff0002566c40: BAD TRAP: type=d (#gp General protection) rp=ff00025664c0 addr=20 sched: #gp General protection addr=0x20 pid=0, pc=0xf80d375a, sp=0xfff
Illumos boot
Hello In my quest to continue expanding guest support in my vm-bhyve utility (See https://github.com/churchers/vm-bhyve :) ), I've found the Windows support pretty solid once I got clear on the slot requirements. I'm now trying an OS that requires CSM (Illumos) but unfortunately I'm currently struggling to get it to boot up correctly. Here's an example of the command I'm generating at the moment (This is running on an Intel Core-i3): bhyve -c 2 -m 2G -s 0,hostbridge -s 31,lpc \ -s 3,ahci-cd,/data/vm/.iso/smartos-latest.iso \ -s 4:0,ahci-hd,/data/vm/smartos/disk0.img \ -s 5:0,virtio-net,tap0 \ -l com1,stdio -l com2,/dev/nmdm2A \ -H -l bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd \ smartos I have com1 set to stdio so I can easily watch the output as it runs. It tends to get as far as "Legacy INT19 Boot...", then fall over. Depending on whether I put the network interface directly in the slot after the HDD, I seem to get different errors - slot 3 - cd slot 4 - hdd slot 5 - virtio-net panic[cpu0]/thread=ff01457cdb40: BAD TRAP: type=e (#pf Page fault) rp=ff0004a69a60 addr=40 occurred in module "genunix" due to a NULL pointer dereference slot 3 - cd slot 4 - hdd slot 7 - virtio-net panic[cpu1]/thread=ff0004002c40: BAD TRAP: type=d (#gp General protection) rp=ff0004002740 addr=0 On com2 I see the boot menu, then one and a half lines of dots. The second line of dots stops about 2/3 of the way across. Interestingly, my code normally puts the CD after the HDD, which Windows seems happy with as long as the slots are consecutive. In SmartOS this gives me a different error: slot 3 - hdd slot 4 - cd slot 5 - virtio-net PlatformBdsBootFail Boot Failed. Harddisk 1 Find PE image /home/grehan/proj/stock_edk2/Build/BhyveX64/DEBUG_GCC48/X64/UefiCpuPkg/CpuDxe/CpuDxe/DEBUG/CpuDxe.dll (ImageBase=7F8DC000, EntryPoint=7F8DC2AF) Regards, Matt Churchyard ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Options for zfs inside a VM backed by zfs on the host
> On Wed, Aug 26, 2015 at 11:10:44PM -0700, Marcus Reid wrote: > On Wed, Aug 26, 2015 at 05:25:52PM -0400, Vick Khera wrote: > > > Opinions? Preferably well-reasoned ones. :) > > > > > However, having the ARC eating up lots of memory twice seems pretty > > bletcherous. You can probably do some tuning to reduce that, but I > > never liked tuning the ARC much. > I just realized that you can turn primarycache off per-dataset. Does it make > more sense to turn primarycache=none on the zvol on the host, or > on the > datasets in the vm? I'm thinking on the host, but it might be worth > experimenting. I'd be very wary of disabling ARC on the main host, it can have pretty serious side effects. It could possibly be useful in the guest though, as data should be cached already by ARC on the host, you're just going through an extra step of reading through the virtual disk driver, and into host ARC, instead of directly from the guest memory. Would need testing to know what performance was like and if there are any side effects. I do agree that it doesn't seem unnecessary to have any redundancy in the guest if the host pool is redundant. Save for any glaring bugs in the virtual disk emulation, you wouldn't expect to get errors on the guest pool if the host pool is already checksumming the data. It's also worth testing with guest ARC enabled but just limited to a fairly small size, so you're not disabling it entirely, but doing at little double-caching as possible. ZFS features seems perfect for virtual hosts, although it's not ideal that you have to give up a big chunk of host RAM for ARC. You may also find that you need to limit host ARC, then only use "MAX_RAM - MY_ARC_LIMIT" for guests. Otherwise you'll have ZFS and VMs fighting for memory and enough of us have seen what shouldn't, but unfortunately does happen in that situation. Matt - > Marcus > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
