It would be interesting to test running it under Xen with FreeBSD as the dom0.
-Dustin On Sat, Oct 27, 2018 at 1:04 PM Harry Schmalzbauer <free...@omnilan.de> wrote: > Am 22.10.2018 um 13:26 schrieb Harry Schmalzbauer: > … > > Test-Runs: > > Each hypervisor had only the one bench-guest running, no other > > tasks/guests were running besides system's native standard processes. > > Since the time between powering up the guest and finishing logon > > differed notably (~5s vs. ~20s) from one host to the other, I did a > > quick synthetic IO-Test beforehand. > > I'm using IOmeter since heise.de published a great test pattern called > > IOmix – about 18 years ago I guess. This access pattern has always > > perfectly reflected the system performance for human computer usage > > with non-caculation-centric applications, and still is my favourite, > > despite throughput and latency changed by some orders of manitudes > > during the last decade (and I had defined something for "fio" which > > mimics IOmix and shows reasonable relational results; but I'm still > > prefering IOmeter for homogenous IO benchmarking). > > > > The results is about factor 7 :-( > > ~3800iops&69MB/s (CPU-guest-usage 42%IOmeter+12%irq) > > vs. > > ~29000iops&530MB/s (CPU-guest-usage 11%IOmeter+19%irq) > > > > > > [with debug kernel and debug-malloc, numbers are 3000iops&56MB/s, > > virtio-blk instead of ahci,hd: results in 5660iops&104MB/s with > > non-debug kernel > > – much better, but even higher CPU load and still factor 4 slower] > > > > What I don't understand is, why the IOmeter process differs that much > > in CPU utilization!?! It's the same binary on the same OS (guest) > > with the same OS-driver and the same underlying hardware – "just" the > > AHCI emulation and the vmm differ... > > > > Unfortunately, the picture for virtio-net vs. vmxnet3 is similar sad. > > Copying a single 5GB file from CIFS share to DB-ssd results in 100% > > guest-CPU usage, where 40% are irqs and the throughput max out at > > ~40MB/s. > > When copying the same file from the same source with the same guest on > > the same host but host booted ESXi, there's 20% guest-CPU usage while > > transfering 111MB/s – the uplink GbE limit. > > > > These synthetic benchmark very well explain the "feelable" difference > > when using a guest between the two hypervisors, but > … > > To add an additional and rather surprinsing result, at least for me: > > Virtualbox provides > 'VBoxManage internalcommands createrawvmdk -filename > "testbench_da0.vmdk" -rawdisk /dev/da0' > > So I could use the exactly same test setup as for ESXi and bhyve. > FreeBSD-Virtualbox (running on the same host installation like bhyve) > performed quiet well, although it doesn't survive IOmix benchmark run > when the "testbench_da0.vmdk" (the "raw" SSD-R0-array) is hooked up to > the SATA controller. > But connected to the emulated SAS controller(LSI1068), it runs without > problems and results in 9600iops@185MB/s with 1%IOmeter+7%irq CPU > utilization (yes, 1% vs. 42% for IOmeter load). > Still far away from what ESXi provides, but almost double performance of > virtio-blk with bhyve, and most important, much less load (host and > guest show exactly the same low values as opposed to the very high loads > which are shown on host and guest with bhyve:virtio-blk). > The HDtune random access benchmark also shows the factor 2, linear over > all block sizes. > > Virtualbox's virtio-net setup gives ~100MB/s with peaks at 111 and ~40% > CPU load. > Guest uses the same driver like with bhyve:virtio-blk, while backend of > virtualbox:virtio-net is vboxnetflt utilizing netgraph and vboxnetadp.ko > vs. tap(4). > So not only the IO efficiency (lower throughput but also much lower CPU > utilization) is remarbably better, but also the network performance. > Even low-bandwidth RDP sessions via GbE-LAN suffer from micro hangs > under bhyve and virtio-net. And 40MB/s transfers cause 100% CPU load on > bhyve – both runs had exactly the same WIndows virtio-net driver in use > (RedHat 141). > > Conclusion: Virtualbox vs. ESXi shows a 0.5% efficiency factor, while > bhyve vs. ESXi shows 0.25% overall efficiency factor. > I tried to provide a test environment with shortest hardware paths > possible. At least the benchmarks were run 100% reproducable with the > same binaries. > > So I'm really interested if > … > > Are these (emulation(only?) related, I guess) performace issues well > > known? I mean, does somebody know what needs to be done in what area, > > in order to catch up with the other results? So it's just a matter of > > time/resources? > > Or are these results surprising and extensive analysis must be done > > before anybody can tell how to fix the IO limitations? > > > > Is the root cause for the problematic low virtio-net throughput > > probably the same as for the disk IO limits? Both really hurt in my > > use case and the host is not idling in relation, but even showing > > higher load with lower results. So even if the lower > > user-experience-performance would be considered as toleratable, the > > guests/host ratio was only half dense. > > Thanks, > > -harry > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscr...@freebsd.org" > _______________________________________________ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"