Re: VMWare ESX and FBSD 7.2 AMD64 guest
On Thursday 23 July 2009 19:44:15 Steve Bertrand wrote: This message has a foot that has nearly touched down over the OT borderline. We received an HP Proliant DL360G5 collocation box yesterday that has two processors, and 8GB of memory. All the client wants to use this box for is a single instance of Windows web hosting. Knowing the sites the client wants to aggregate into IIS, I know that the box is far over-rated. Making a long story short, they have agreed to allow us to put their Windows server inside of a virtual-ized container, so we can use the unused horsepower for other vm's (test servers etc). My problem is performance. I'm only willing to make this box virtual if I can keep the abstraction performance loss to 25% (my ultimate goal would be 15%). The following is what I have, followed by my benchmark findings: # 7.2-RELEASE AMD64 FreeBSD 7.2-RELEASE #0: Fri May 1 07:18:07 UTC 2009 r...@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC Timecounter i8254 frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU5150 @ 2.66GHz (2666.78-MHz K8-class CPU) Origin = GenuineIntel Id = 0x6f6 Stepping = 6 usable memory = 8575160320 (8177 MB) avail memory = 8273620992 (7890 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7: Did you give the VM 4 virtual processors as well? How much RAM did it have? What type of storage does the server have? Did the VM just get a .vmdk on VMFS? What version of ESX? Benchmarks: # time make -j4 buildworld (under vmware) 5503.038u 3049.500s 1:15:46.25 188.1% 5877+1961k 3298+586716io 2407pf+0w # time make -j4 buildworld (native) 4777.568u 992.422s 33:02.12 291.1%6533+2099k 25722+586485io 3487pf+0w Note that the user time is within your 15% margin (if you round to the nearest percent). The system time is what's running away. My guess is that that is largely due to disk I/O and virtualization of same. What you can do to address this depends on what hardware you have. Giving the VM a raw slice/LUN/disk instead of a .vmdk file may improve matters somewhat. If you do use a disk file be sure that it lives on a stripe (or whatever unit is relevant) boundary of the underlying storage. Ways to do that (if any) depend on the storage. Improving the RAID performance, etc. of the storage will improve your benchmark overall, and may or may not narrow the divide. The (virtual) storage driver (mpt IIRC) might have some parameters you could tweak, but I don't know about that off the top of my head. ...both builds were from the exact same sources, and both runs were running with the exact same environment. I was extremely careful to ensure that the environments were exactly the same. I'd appreciate any feedback on tweaks that I can make (either to VMWare, or FreeBSD itself) to make the virtualized environment much more efficient. See above about storage. Similar questions come up periodically; searching the archives if you haven't already may prove fruitful. You may want to try running with different kernel HZ settings for instance. I would also try to isolate the performance of different components and evaluate their importance for your actual intended load. CPU and RAM probably perform like you expect out of the box. Disk and network I/O won't be as close to native speed, but the difference and the impact are variable depending on your hardware and load. A lightly-loaded Windows server is the poster child of virtualization candidates. If your decision is to dedicate the box to Winders or to virtualize and use the excess capacity for something else I would say it's a no-brainer if the cost of ESX isn't a factor (or if ESXi gives you similar performance). If that's already a given and your decision is between running a specific FreeBSD instance on the ESX host or on its own hardware then you're wise to spec out the performance differences. HTH, JN ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: VMWare ESX and FBSD 7.2 AMD64 guest
John Nielsen wrote: On Thursday 23 July 2009 19:44:15 Steve Bertrand wrote: My problem is performance. I'm only willing to make this box virtual if I can keep the abstraction performance loss to 25% (my ultimate goal would be 15%). usable memory = 8575160320 (8177 MB) avail memory = 8273620992 (7890 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7: Did you give the VM 4 virtual processors as well? How much RAM did it have? What type of storage does the server have? Did the VM just get a .vmdk on VMFS? What version of ESX? I gave it all four procs to use, and all available memory. See below about storage system. The system time is what's running away. My guess is that that is largely due to disk I/O and virtualization of same. What you can do to address this depends on what hardware you have. Giving the VM a raw slice/LUN/disk instead of a .vmdk file may improve matters somewhat. If you do use a disk file be sure that it lives on a stripe (or whatever unit is relevant) boundary of the underlying storage. Ways to do that (if any) depend on the storage. Improving the RAID performance, etc. of the storage will improve your benchmark overall, and may or may not narrow the divide. The storage system is the following, with 512MB cache. I'm trying to figure out if the cache has a battery backup installed, as I've read that disk performance could be affected without it. kernel: ciss0: HP Smart Array P400i With six Fujitsu MHW2120BS 120GB 5.4k SATA laptop drives. After performing multiple in-OS and outside-of-OS benchmark tests, the maximum read speed I can achieve is ~7MBps. Before I reconfigured the machine from the default RAID6 to RAID1+0, I was capped at ~5. This is certainly a huge bottleneck. I'm not impressed in any way with that type of performance, when a lesser system that I have running FBSD 7.2 and ZFS can achieve ~160MBps. I know the drives are only 5.4k, but ~7MB just isn't right. I'm off to see what we can do about that. Thanks John, Steve smime.p7s Description: S/MIME Cryptographic Signature
RE: VMWare ESX and FBSD 7.2 AMD64 guest
This message has a foot that has nearly touched down over the OT borderline. We received an HP Proliant DL360G5 collocation box yesterday that has two processors, and 8GB of memory. All the client wants to use this box for is a single instance of Windows web hosting. Knowing the sites the client wants to aggregate into IIS, I know that the box is far over-rated. Making a long story short, they have agreed to allow us to put their Windows server inside of a virtual-ized container, so we can use the unused horsepower for other vm's (test servers etc). My problem is performance. I'm only willing to make this box virtual if I can keep the abstraction performance loss to 25% (my ultimate goal would be 15%). The following is what I have, followed by my benchmark findings: # 7.2-RELEASE AMD64 FreeBSD 7.2-RELEASE #0: Fri May 1 07:18:07 UTC 2009 r...@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC Timecounter i8254 frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU5150 @ 2.66GHz (2666.78-MHz K8-class CPU) Origin = GenuineIntel Id = 0x6f6 Stepping = 6 usable memory = 8575160320 (8177 MB) avail memory = 8273620992 (7890 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7: Benchmarks: # time make -j4 buildworld (under vmware) 5503.038u 3049.500s 1:15:46.25 188.1% 5877+1961k 3298+586716io 2407pf+0w # time make -j4 buildworld (native) 4777.568u 992.422s 33:02.12 291.1%6533+2099k 25722+586485io 3487pf+0w ...both builds were from the exact same sources, and both runs were running with the exact same environment. I was extremely careful to ensure that the environments were exactly the same. I'd appreciate any feedback on tweaks that I can make (either to VMWare, or FreeBSD itself) to make the virtualized environment much more efficient. Off-list is fine. Cheers, Steve I haven't actually done any benchmarks to compare the performance, but I have been running production FreeBSD servers on VMware for a couple of years. I currently have two 6.2 systems running CUPS, one on VMware Server, and the other on ESX 3.5. I also have a 7.0 system and two 7.1 systems running Squid on ESX 3.5 as well. The thing that I noticed as the biggest bottle neck for any guest within VMware is the Disk I/O (with the exception of video which isn't an issue for a server). Compiling software does take longer, because of this, however if you tune your disks properly the performance under real application load doesn't seem to be an issue. Using soft updates on the file system seems to help out a lot, but be aware of the consequences. That being said, on the Systems I have running squid we average 9G of traffic a day on the busiest system with about 11% cache hit rate, These proxies sit close to idle after hours. Looking at the information from systat -vmstat, the system is almost idle during the day under the full load as well, you just can't touch FreeBSD with only 2 DSL lines for web traffic. Its faster than the old native system was, however there is an iSCSI SAN behind the ESX server for disk access, and we went from a Dell PowerEdge 850 to a Dell PowerEdge 2950. It does share that server with around 15 or more other servers (Mostly windows, some Linux) depending on the current load. Which brings us to another point, It seems to do just fine when VMware VMotion moves it between servers. Not sure if this information helps you out any, but my recommendation would be that if your application will be very disk intensive, avoid the Virtual machine. In my case with the Squid, gaining the redundancy of the VMware coupled with VMotion was worth the potential hit in performance. As we are soon implementing a second data center across town that will house additional VMware servers and thanks to a 10G fiber ring, will allow us to migrate servers while running between datacenters. Also keep in mind that as of vSphere 4 (We will be upgrading to this once the new data center is complete, just waiting on the shipment of the racks at this point), VMware does officially support FreeBSD 7.1, so you might want to go with that instead of 7.2, as there may be a performance issue with 7.2, but it's also just as likely that it was a timing issue on releases that 7.1 is supported and 7.2 isn't. As of ESXi 4.0 (released 5-21-2009), I believe it has the same code base as vSphere 4, so the same guests should be supported. Thanks, Dean Weimer Network Administrator Orscheln Management Co ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: VMWare ESX and FBSD 7.2 AMD64 guest
John Nielsen wrote: On Thursday 23 July 2009 19:44:15 Steve Bertrand wrote: I'd appreciate any feedback on tweaks that I can make (either to VMWare, or FreeBSD itself) to make the virtualized environment much more efficient. See above about storage. Similar questions come up periodically; searching the archives if you haven't already may prove fruitful. You may want to try running with different kernel HZ settings for instance. You should certainly try setting both kern.hz and vfs.read_max in the FreeBSD VM. I would recommend: In loader.conf: kern.hz=100 In /etc/sysctl.conf: vfs.read_max=32 You may also try increasing vfs.hirunningspace. I've had good results with setting it to 32MB on write-intensive systems. Tuning vfs.read_max can give some boosts to physical-hardware FreeBSD systems as well. -Steve Polyack ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: VMWare ESX and FBSD 7.2 AMD64 guest
From: John Nielsen li...@jnielsen.net Subject: Re: VMWare ESX and FBSD 7.2 AMD64 guest To: freebsd-questions@freebsd.org Cc: Steve Bertrand st...@ibctech.ca Date: Friday, July 24, 2009, 10:22 AM On Thursday 23 July 2009 19:44:15 Steve Bertrand wrote: This message has a foot that has nearly touched down over the OT borderline. We received an HP Proliant DL360G5 collocation box yesterday that has two processors, and 8GB of memory. All the client wants to use this box for is a single instance of Windows web hosting. Knowing the sites the client wants to aggregate into IIS, I know that the box is far over-rated. Making a long story short, they have agreed to allow us to put their Windows server inside of a virtual-ized container, so we can use the unused horsepower for other vm's (test servers etc). My problem is performance. I'm only willing to make this box virtual if I can keep the abstraction performance loss to 25% (my ultimate goal would be 15%). The following is what I have, followed by my benchmark findings: # 7.2-RELEASE AMD64 FreeBSD 7.2-RELEASE #0: Fri May 1 07:18:07 UTC 2009 r...@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC Timecounter i8254 frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU 5150 @ 2.66GHz (2666.78-MHz K8-class CPU) Origin = GenuineIntel Id = 0x6f6 Stepping = 6 usable memory = 8575160320 (8177 MB) avail memory = 8273620992 (7890 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7: Did you give the VM 4 virtual processors as well? How much RAM did it have? What type of storage does the server have? Did the VM just get a .vmdk on VMFS? What version of ESX? Benchmarks: # time make -j4 buildworld (under vmware) 5503.038u 3049.500s 1:15:46.25 188.1% 5877+1961k 3298+586716io 2407pf+0w # time make -j4 buildworld (native) 4777.568u 992.422s 33:02.12 291.1% 6533+2099k 25722+586485io 3487pf+0w Note that the user time is within your 15% margin (if you round to the nearest percent). The system time is what's running away. My guess is that that is largely due to disk I/O and virtualization of same. What you can do to address this depends on what hardware you have. Giving the VM a raw slice/LUN/disk instead of a .vmdk file may improve matters somewhat. If you do use a disk file be sure that it lives on a stripe (or whatever unit is relevant) boundary of the underlying storage. Ways to do that (if any) depend on the storage. Improving the RAID performance, etc. of the storage will improve your benchmark overall, and may or may not narrow the divide. The (virtual) storage driver (mpt IIRC) might have some parameters you could tweak, but I don't know about that off the top of my head. ...both builds were from the exact same sources, and both runs were running with the exact same environment. I was extremely careful to ensure that the environments were exactly the same. I'd appreciate any feedback on tweaks that I can make (either to VMWare, or FreeBSD itself) to make the virtualized environment much more efficient. See above about storage. Similar questions come up periodically; searching the archives if you haven't already may prove fruitful. You may want to try running with different kernel HZ settings for instance. I would also try to isolate the performance of different components and evaluate their importance for your actual intended load. CPU and RAM probably perform like you expect out of the box. Disk and network I/O won't be as close to native speed, but the difference and the impact are variable depending on your hardware and load. A lightly-loaded Windows server is the poster child of virtualization candidates. If your decision is to dedicate the box to Winders or to virtualize and use the excess capacity for something else I would say it's a no-brainer if the cost of ESX isn't a factor (or if ESXi gives you similar performance). If that's already a given and your decision is between running a specific FreeBSD instance on the ESX host or on its own hardware then you're wise to spec out the performance differences. HTH, JN If I recall correctly from ESX (well, VI) training*, there may be a minor scheduling issue affecting things here. If you set up the VM with 4 processors, ESX schedules time on the CPU only when there's 4 things to execute (well, there's another time period it also uses, so even a single thread will get run eventually, but anyway...). The physical instance will run one thread immediately even if there's nothing else waiting, whereas the VM will NOT execute a single thread necessarily immediately. I would retry using perhaps -j8 or even -j12 to make sure the 4 CPUs see plenty of work to do
Re: VMWare ESX and FBSD 7.2 AMD64 guest
Richard Mahlerwein wrote: If I recall correctly from ESX (well, VI) training*, there may be a minor scheduling issue affecting things here. If you set up the VM with 4 processors, ESX schedules time on the CPU only when there's 4 things to execute (well, there's another time period it also uses, so even a single thread will get run eventually, but anyway...). The physical instance will run one thread immediately even if there's nothing else waiting, whereas the VM will NOT execute a single thread necessarily immediately. I would retry using perhaps -j8 or even -j12 to make sure the 4 CPUs see plenty of work to do and see if the numbers don't slide closer to one another. For what it's worth, if there were a raw LUN available and made available to the VM, the disk performance of that LUN should very nearly match native performance, because it IS native performance. VMWare (if I understood right in the first place and remember correctly as well, I supposed I should * this as well. :) ) doesn't add anything to slow that down. Plugging in a USB drive to the Host and making it available to the guest would also be at native USB/drive speeds, assuming you can do that (I've never tried to use USB drives on our blade center!). I've isolated the problem to the SATA RAID system (or subsystem). Booting from CD/USB key and running a wide array of bench tests, I can not read from the RAID setup faster than 10MBps. Regardless of anything else, this is my priority. RAID 0 is the only config where I can read faster than ~7MBps. The board does not have any standard IDE interfaces, and I don't have any PCIe-IDE cards that aren't in use, so I can't really bypass the HP RAID card. I will however slap a 200GB USB drive against the box, and see if I can get faster performance from USB than I can the native SATA setup. FWIW, I do have the battery backed cache installed... Steve smime.p7s Description: S/MIME Cryptographic Signature
RE: VMWare ESX and FBSD 7.2 AMD64 guest
From: Dean Weimer dwei...@orscheln.com Subject: RE: VMWare ESX and FBSD 7.2 AMD64 guest To: freebsd-questions@freebsd.org Cc: st...@ibctech.ca Date: Friday, July 24, 2009, 10:49 AM [snip] servers while running between datacenters. Also keep in mind that as of vSphere 4 (We will be upgrading to this once the new data center is complete, just waiting on the shipment of the racks at this point), VMware does officially support FreeBSD 7.1, so you might want to go with that instead of 7.2, as there may be a performance issue with Awesome news! That's teach me to keep shuttling the nearly-spam I get from VMware into the trash can right away. I'd love to hear about your experience with the upgrade and how things go later. We're looking to do something very similar sometime in the next 6 to 9 months. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
VMWare ESX and FBSD 7.2 AMD64 guest
This message has a foot that has nearly touched down over the OT borderline. We received an HP Proliant DL360G5 collocation box yesterday that has two processors, and 8GB of memory. All the client wants to use this box for is a single instance of Windows web hosting. Knowing the sites the client wants to aggregate into IIS, I know that the box is far over-rated. Making a long story short, they have agreed to allow us to put their Windows server inside of a virtual-ized container, so we can use the unused horsepower for other vm's (test servers etc). My problem is performance. I'm only willing to make this box virtual if I can keep the abstraction performance loss to 25% (my ultimate goal would be 15%). The following is what I have, followed by my benchmark findings: # 7.2-RELEASE AMD64 FreeBSD 7.2-RELEASE #0: Fri May 1 07:18:07 UTC 2009 r...@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC Timecounter i8254 frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU5150 @ 2.66GHz (2666.78-MHz K8-class CPU) Origin = GenuineIntel Id = 0x6f6 Stepping = 6 usable memory = 8575160320 (8177 MB) avail memory = 8273620992 (7890 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7: Benchmarks: # time make -j4 buildworld (under vmware) 5503.038u 3049.500s 1:15:46.25 188.1% 5877+1961k 3298+586716io 2407pf+0w # time make -j4 buildworld (native) 4777.568u 992.422s 33:02.12 291.1% 6533+2099k 25722+586485io 3487pf+0w ...both builds were from the exact same sources, and both runs were running with the exact same environment. I was extremely careful to ensure that the environments were exactly the same. I'd appreciate any feedback on tweaks that I can make (either to VMWare, or FreeBSD itself) to make the virtualized environment much more efficient. Off-list is fine. Cheers, Steve smime.p7s Description: S/MIME Cryptographic Signature
