Hi Zak, we have prepared this test build for you:
http://www.virtualbox.org/download/testcase/virtualbox-4.0_4.0.7-71651~Ubuntu~maverick_amd64.deb Kind regards, Frank On Sunday 08 May 2011 13:41:01 Knut St. Osmundsen wrote: > Hi Zak, > > I've raised the max VM limit to 8191 in r36988). We can provide a test > build if you are unable to build from SVN, just let us know. > > Kind Regards, > Knut > > On May 4, 2011, at 12:41 AM, Zak Perschon wrote: > > I had to restart the machine running all of the VMs, so I decided to > > increase the amount of files per process like you suggested, relaunched, > > and we are now hitting the projected 1023 virtual machine cap! Here is > > the error that VboxManage pushes out if anyone is interested (A vbox.log > > file was also created for the VM): > > > > VBoxManage: error: VM creation failed (GVMM) (VERR_GVM_TOO_MANY_VMS) > > VBoxManage: error: Unknown error creating VM (VERR_GVM_TOO_MANY_VMS) > > VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component > > Console, interface IConsole, callee > > > > Now that we made it this far, I guess the next step would be attempting > > to remove the 1023 machine limit that virtualbox is imposing on us? > > Thanks again for the help, its been very uplifting to have something > > actually show progress (bashing our heads against a slew of different > > virtualization solutions has been exhausting). > > > > -Zak > > > > > Date: Tue, 3 May 2011 20:39:14 +0200 > > > From: [email protected] > > > To: [email protected] > > > Subject: Re: [vbox-dev] Linux Host VM Cap? > > > > > > On 03.05.2011 17:39, Zak Perschon wrote: > > > > After letting my mass-deploy script complete, we seem to have hit a > > > > cap at 1010 running VMs (A heck of a lot better than 128!). There > > > > doesn't seem to be a log created for the 1010th VM that failed, but > > > > here is the error message that I get when attempting to start it: > > > > > > > > VBoxManage: error: The Virtual Machine 'TinyCore-1011' has terminated > > > > unexpectidly during startup with exit code 1 > > > > VboxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), > > > > component Machine, interface IMachine, callee > > > > > > > > Any ideas as to where to go from here? > > > > > > Of course :) It looks like you're running into the maximum number of > > > open files _per process_ now, which defaults to 1024 on most Linux > > > distros. The critical component in this respect is the VBoxXPCOMIPCD > > > process (automatically started). This one is the central message broker > > > for API requests. Since it also has a couple of files open the limit of > > > API clients is a bit lower than 1024. > > > > > > Not sure if it's worth the effort to get 13 VMs more, but if you want > > > to get to the current hardcoded maximum you have to stop all VMs > > > again. Please double check with the "ps" command that VBoxXPCOMIPCD > > > has terminated (should happen approx. 5 seconds after the last API > > > client is gone). > > > > > > Then set a higher limit either straight with ulimit -n 2048, or > > > following the descriptions (the comments are interesting as well) in > > > http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open- > > > files/ > > > > > > > The current system resource usage is as follows: > > > > > > > > CPU Usage: 10~% average idling, 20~% when we are utilizing all of the > > > > 1010 machines. > > > > > > This sounds very good. Must be a really stripped down distribution > > > which doesn't do much user friendly (i.e. CPU wasting) background > > > activities. Another reason for the low utilization might actually be > > > lock > > > contention, as all those VMs need to coordinate their activities to > > > some degree, and that could cause delays. We'd have some ideas what to > > > change if this really happens. > > > > > > > Memory: 146.8GB of 2019.9GB (which seems low, we have 1010 machines > > > > using 150MB of ram and 8 MB of vram, but we are getting replies from > > > > all of the machines) > > > > Disk Space: 48.6GB of 1.3TB (The machine is supposed to have an > > > > attached SAS drive, but its still on order) > > > > > > The memory usage is suspiciously low. There could be a reason though - > > > VirtualBox allocates VM memory lazily. If a VM doesn't touch all its > > > memory then it will be below the expected value. > > > > > > > The original goal of 1800 was our "conservative" estimate, we are > > > > actually aiming to get around 4,000 if possible. So far, Virtualbox > > > > has been the only VM solution we've tried that has gotten us this > > > > close (most either have a cap at around 300~ VMs that can't > > > > increase, or they don't support our host machine's hardware and > > > > crash). If you'd like, once we are finished with this I can provide > > > > a bit more detail on the VM solutions we've tried and the > > > > limitations we've ran in to. > > > > > > For over 1023 it needs some code tweaking... we'll help you with that > > > of course. Don't know yet when our expert can spare a few minutes for > > > looking into this. > > > > > > Klaus > > > > > > > -Zak > > > > > > > > > Date: Tue, 3 May 2011 17:00:55 +0200 > > > > > From: [email protected] > > > > > To: [email protected] > > > > > CC: [email protected] > > > > > Subject: Re: [vbox-dev] Linux Host VM Cap? > > > > > > > > > > Hi Zak, > > > > > > > > > > On 03.05.2011 00:29, Zak Perschon wrote: > > > > > > Hello, > > > > > > > > > > > > Here is a quick update for you (thanks a ton for the help by the > > > > > > way!): > > > > > > > > > > > > I've been able to breach the 128 cap we had before w/ the fix you > > > > > > suggested, but have ran into some "user error issues" that have > > > > > > caused me to have to re-deploy/restart my VMs. The highest I've > > > > > > gotten so far is 675 running VMs, but I'm sure I'll be able to > > > > > > hit the 1023 mark. It takes a uprising amount of time to > > > > > > deploy/start 1000 VMS.... > > > > > > > > > > Keeping fingers crossed... with 675 VMs, 150MB RAM each (how much > > > > > VRAM, I guessed 16MB?), you're around 280GB of RAM used by all of > > > > > them, using the rough rule of thumb that RAM usage of a VM is > > > > > approx. > > > > > 1.02*(RAM+VRAM)+250MB. Varies greatly with the VM config details of > > > > > > > > course. > > > > > > > > > With your monster gear having 1.8TB of free RAM the theoretical > > > > > maximum would be around 4300 of the tiny VMs you outlined > > > > > (ignoring the disk and CPU capacity). > > > > > > > > > > I'm working on updating the manual. The information for Solaris is > > > > > already there (in chapter 9), and the Linux information should also > > > > > go there. > > > > > > > > > > Just out of curiosity - are other virtualizers significantly faster > > > > > with such insanely high VM counts? It's really a very exotic use > > > > > case. > > > > > > > > > > BTW, we'd appreciate keeping such threads on the mailing list - > > > > > they show that VirtualBox is a serious virtualization solution, > > > > > and also help others if they run into such extreme situations. > > > > > > > > > > Klaus > > > > > > > > > > > -Zak > > > > > > > > > > > > > Date: Mon, 2 May 2011 19:01:21 +0200 > > > > > > > From: [email protected] > > > > > > > To: [email protected] > > > > > > > Subject: Re: [vbox-dev] Linux Host VM Cap? > > > > > > > > > > > > > > On 01.05.2011 17:56, Knut St. Osmundsen wrote: > > > > > > > > Hi Zak! > > > > > > > > > > > > > > > > The VMM imposes a limit of 1023 VMs on 64-bit hosts and 127 > > > > > > > > VMs on > > > > > > > > > > > > 32-bit hosts. See: > > > > http://www.virtualbox.org/browser/trunk/src/VBox/VMM/VMMR0/GVMMR0.cpp > > > > #L121 > > > > > > > > > > > That means your plan to have 1800 VMs will not work without > > > > > > > > bumping the > > > > > > > > > > > hardcoded limits, and in this area it needs much more thinking > > > > > > > > than just > > > > > > > > > > > changing a #define. > > > > > > > > > > > > > > Anyway, let's get you to that limit first... > > > > > > > > > > > > > > If you run "ipcs -ls" you'll get a "max number of arrays = 128" > > > > > > > line, which is what you're most likely bumping into. > > > > > > > > > > > > > > You can read the kernel defaults with "sysctl kernel.sem" as > > > > > > > > well, and > > > > > > > > > > > the last number needs to be increased to a bit more than 1023, > > > > > > > > just to > > > > > > > > > > > be on the safe side if some other app also needs a few > > > > > > > semaphores. > > > > > > > > > > > > > > "sysctl -w kernel.sem=250 32000 32 1200" should eliminate this > > > > > > > limit. You can also put this in /etc/sysctl.conf, so that it > > > > > > > is set > > > > > > > > straight on > > > > > > > > > > > system boot, as otherwise these settings are lost on a reboot. > > > > > > > > > > > > > > So... how many VMs can you start now? > > > > > > > > > > > > > > Klaus > > > > > > > > > > > > > > > Seeing you have 2TB of memory and 128 cores, I would assume > > > > > > > > you're running 64-bit Ubuntu. So, it is probably some other > > > > > > > > limit that > > > > > > > > you're > > > > > > > > > > > > hitting. It could be SysV semaphores, it could be memory > > > > > > > > available > > > > > > > > > > > > below > > > > > > > > > > > > > > 4GB, ... Could you please provide some more details regarding > > > > > > > > what happens when you are unable to launch more VMs? For > > > > > > > > instance, does > > > > > > > > > > > > a new > > > > > > > > > > > > > > VBox.log file get created for the VM? If it does, please zip > > > > > > > > it > > > > > > > > up and > > > > > > > > > > > > send it to me (or Klaus) per private mail and we'll work it > > > > > > > > out. > > > > > > > > > > > > > > I think the 64bit host OS assumption is safe... I would > > > > > > > seriously ask people to have their brain inspected if they'd > > > > > > > use a 32bit OS for > > > > > > > > this. > > > > > > > > > > > > On Apr 29, 2011, at 7:43 PM, Zak Perschon wrote: > > > > > > > >> Ah, sorry forgot to include the original support thread. The > > > > > > > > limit I > > > > > > > > > > > >> hit was 128 active VMs at once. Once I hit that number, I > > > > > > > >> was > > > > > > > > unable > > > > > > > > > > > >> to launch any more unless I stopped one that was already > > > > > > > > running. My > > > > > > > > > > > >> original thread can be found here (though there is little > > > > > > > > information > > > > > > > > > > > >> contained in > > > > > > > >> it):http://forums.virtualbox.org/viewtopic.php?f=7&t=40955 > > > > > > > >> <http://forums.virtualbox.org/viewtopic.php?f=7&t=40955> > > > > > > > >> > > > > > > > >> -Zak > > > > > > > >> > > > > > > > >> > Date: Fri, 29 Apr 2011 16:25:44 +0200 > > > > > > > >> > From:[email protected] > > > > > > > > <mailto:[email protected]> > > > > > > > > > > > >> > To:[email protected] > > > > > > > >> > <mailto:[email protected]> Subject: Re: [vbox-dev] > > > > > > > >> > Linux Host VM Cap? > > > > > > > >> > > > > > > > > >> > On 25.04.2011 18:00, Zak Perschon wrote: > > > > > > > >> > > I've come across what is seemingly a hard cap for the > > > > > > > > amount of > > > > > > > > > > > >> > > concurrent VMs I can run on a Linux based machine (after > > > > > > > > posting > > > > > > > > > > > >> on the > > > > > > > >> > > > > > > > >> > > support forums, I was directed here). Apparently this > > > > > > > >> > > same > > > > > > > > cap was > > > > > > > > > > > >> > > encountered on a solaris host and a fix was deployed to > > > > > > > > > > > > increase the > > > > > > > > > > > > > >> > > amount of VMs that were allowed to run at the same time. > > > > > > > >> > > Was > > > > > > > > > > > > this fix > > > > > > > > > > > > > >> > > supposed to be included in other OS releases? Any > > > > > > > >> > > information > > > > > > > > > > > > on this > > > > > > > > > > > > > >> > > would be greatly appreciated. > > > > > > > >> > > > > > > > > >> > That's a rather vague statement, and if you could tell > > > > > > > >> > what > > > > > > > > > > > > number of > > > > > > > > > > > > > >> > concurrently running VMs you can achieve I might be able > > > > > > > >> > to > > > > > > > > > > > > infer what > > > > > > > > > > > > > >> > limit is causing trouble for you. > > > > > > > >> > > > > > > > > >> > It could be the SysV semaphore count, the number of open > > > > > > > > files per > > > > > > > > > > > >> > process, and so on. All these defaults depend on the > > > > > > > >> > kernel configuration and/or distribution, so it's hard to > > > > > > > >> > generalize. > > > > > > > >> > > > > > > > > >> > Solaris is easy compared to that, as it is very uniform > > > > > > > > compared to > > > > > > > > > > > >> Linux. > > > > > > > >> > > > > > > > >> > Klaus > > > > > > > >> > > > > > > > > >> > > Here is the information on the hardware/VM profiles I am > > > > > > > >> > > using > > > > > > > >> > > > > > > > > > >> > > Host Machine: > > > > > > > >> > > 128 Processor Cores (64 dual core Xenon processors > > > > > > > > > > > > [email protected] GHz) > > > > > > > > > > > > > >> > > 2.0 TB Memory (1.8~ Usable) > > > > > > > >> > > 1.8 TB RAID 0 (6 x 300GB HDD running at 10k RPM) > > > > > > > >> > > 16TB SAS > > > > > > > >> > > > > > > > > > >> > > VM Profile > > > > > > > >> > > Compiled TinyCore Linux 2.6 > > > > > > > >> > > Required Memory: 150~MB > > > > > > > >> > > Required HDD Space: 50MB > > > > > > > >> > > > > > > > > > >> > > -Zak > > > > > > > >> > > > > > > > > >> > _______________________________________________ > > > > > > > >> > vbox-dev mailing list > > > > > > > >> > > > > > > > > >> >[email protected] <mailto:[email protected]> > > > > > > > >> >http://vbox.innotek.de/mailman/listinfo/vbox-dev > > > > > > > >> > > > > > > > >> _______________________________________________ > > > > > > > >> vbox-dev mailing list > > > > > > > >> [email protected] <mailto:[email protected]> > > > > > > > >> http://vbox.innotek.de/mailman/listinfo/vbox-dev > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Kind regards / Mit freundlichen Gruessen / Vennlig hilsen, > > > > > > > > bird > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > ORACLE Deutschland B.V. & Co. KG Knut St. Osmundsen > > > > > > > > Werkstrasse 24 Senior Staff Engineer, VirtualBox > > > > > > > > 71384 Weinstadt, Germany mailto:[email protected] > > > > > > > > > > > > > > > > Hauptverwaltung: Riesstr. 25, D-80992 Muenchen > > > > > > > > Registergericht: Amtsgericht Muenchen, HRA 95603 > > > > > > > > > > > > > > > > Komplementaerin: ORACLE Deutschland Verwaltung B.V. > > > > > > > > Rijnzathe 6, 3454PV De Meern, Niederlande > > > > > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr. > > > > > > > > 30143697 Geschaeftsfuehrer: J. Kunz, M. van de Molen, A. van > > > > > > > > der Ven > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > vbox-dev mailing list > > > > > > > > [email protected] > > > > > > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev > > > > > > > > > > > > > > -- > > > > > > > Oracle <http://www.oracle.com> > > > > > > > Dr. Klaus Espenlaub | Snr. Manager Software Development Desktop > > > > > > > Virtualization > > > > > > > Phone: +49 7151 60405 205 <tel:+49715160405205> > > > > > > > Oracle VM VirtualBox > > > > > > > > > > > > > > ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 > > > > > > > Weinstadt > > > > > > > > > > > > > > ORACLE Deutschland B.V. & Co. KG > > > > > > > Hauptverwaltung: Riesstr. 25, D-80992 München > > > > > > > Registergericht: Amtsgericht München, HRA 95603 > > > > > > > > > > > > > > Komplementärin: ORACLE Deutschland Verwaltung B.V. > > > > > > > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande > > > > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr. > > > > > > > 30143697 Geschäftsführer: Jürgen Kunz, Marcel van de Molen, > > > > > > > Alexander van > > > > > > > > der Ven > > > > > > > > > > > Green Oracle <http://www.oracle.com/commitment> Oracle is > > > > > > > > committed to > > > > > > > > > > > developing practices and products that help protect the > > > > > > > environment > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > vbox-dev mailing list > > > > > > > [email protected] > > > > > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev > > > > > > > > > > -- > > > > > Oracle <http://www.oracle.com> > > > > > Dr. Klaus Espenlaub | Snr. Manager Software Development Desktop > > > > > Virtualization > > > > > Phone: +49 7151 60405 205 <tel:+49715160405205> > > > > > Oracle VM VirtualBox > > > > > > > > > > ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt > > > > > > > > > > ORACLE Deutschland B.V. & Co. KG > > > > > Hauptverwaltung: Riesstr. 25, D-80992 München > > > > > Registergericht: Amtsgericht München, HRA 95603 > > > > > > > > > > Komplementärin: ORACLE Deutschland Verwaltung B.V. > > > > > Hertogswetering 163/167, 3543 AS Utrecht, Niederlande > > > > > Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 > > > > > Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van > > > > > der Ven > > > > > > > > > > Green Oracle <http://www.oracle.com/commitment> Oracle is committed > > > > > to developing practices and products that help protect the > > > > > environment > > > > > > > > _______________________________________________ > > > > vbox-dev mailing list > > > > [email protected] > > > > http://vbox.innotek.de/mailman/listinfo/vbox-dev > > > > _______________________________________________ > > vbox-dev mailing list > > [email protected] > > http://vbox.innotek.de/mailman/listinfo/vbox-dev -- ORACLE Deutschland B.V. & Co. KG Dr.-Ing. Frank Mehnert Werkstrasse 24 Staff Engineer, VirtualBox 71384 Weinstadt, Germany mailto:[email protected] Hauptverwaltung: Riesstr. 25, D-80992 München Registergericht: Amtsgericht München, HRA 95603 Komplementärin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ vbox-dev mailing list [email protected] http://vbox.innotek.de/mailman/listinfo/vbox-dev
