I have personally run about 350 one-executor slaves (each slave represented a unique resource). The hardware was reasonably beefy, at least 128 GB RAM and 1TB storage, and the Jenkins server ran a bunch of other services as well. The limiting factor appeared to be disk I/O; remember that all logs and artifacts have to be saved to disk on the server end. Judicious use of “discard old builds” might be needed to keep disk usage in check.
My system was running builds 24/7. It can be difficult if you are doing the same, as there is no safe time to bounce the service. --Rob From: jenkinsci-users@googlegroups.com [mailto:jenkinsci-users@googlegroups.com] On Behalf Of Stephen Connolly Sent: Tuesday, July 29, 2014 4:21 AM To: jenkinsci-users@googlegroups.com Subject: Re: Any upper limit on slaves? With appropriate tuning of various parameters, we have seen customers get over 1000 executors on-line (with an average of 2-3 per slave) With a large number of slaves, you are usually better using one of the on-demand retention strategies. Getting all 1000 executors running builds can be problematic though. I am currently working on a scalability framework for testing how Jenkins scales. It will be here https://github.com/jenkinsci/scalability-test-framework once I finish some refactoring that I identified as required - I want people to have a mostly stable API when I publish this framework. Some other tooling I have built: * https://github.com/jenkinsci/mock-load-builder-plugin which will create build jobs that should load up the remoting channel with load representative of a chatty build * https://github.com/jenkinsci/random-job-builder-plugin which will build jobs selected at random at a specified rate. With that tooling you can set up a Jenkins instance with a load of jobs and have those jobs queue up in a semi-realistic - if stressed - way. Using all the above I can report that: A 1.553 Jenkins master on an m3.large can support: * Connecting 60 JNLP slaves and having them idle (but the system will be unresponsive for 2-3 minutes after startup); OR * Connecting 60 SSH slaves and having them idle (but the system will be unresponsive for 5-6 minutes after startup); OR * Connecting 60 CloudBees NIO SSH slaves and having them idle All with 2 executors per slave For both of the SSH slave options above, you need to tell the JVM to use /dev/./urandom as the entropy source. I suspect the newer NIO JNLP mode will have removed / reduced the unresponsivity of the JNLP master after startup... also that is a side-effect of me starting all the JNLP slaves at the same time. Real systems will likely have JNLP slaves connect in a more staggered way and not suffer the thread contention that locks up the Web UI. On each of those test systems I created 3000 mock jobs organized in folders. I then upped the rate of builds until the Web UI became unresponsive. * JNLP hits system load > 5 and web UI is unusable at somewhere between 50 and 55 concurrent builds on an m3.large * Traditional SSH hits system load > 5 and web UI is unusable at between 10 and 12 concurrent builds on an m3.large * CloudBees NIO SSH hits system load of 4 at 15 concurrent builds, but Web UI remains usable all the way up to 120 concurrent build. The build duration - however - is increased once you go past 15 concurrent builds. So the cause and effect here is that back-pressure is being forced on the build in order to allow the master to remain usable... I picked a m3.large as being a reasonably cost-effective machine type that would let me scale up to a size where you should start seeing problems of scalability but not so large that I need a massive army of machines to saturate it. HTH On 28 July 2014 23:51, Maureen Barger <mobar...@gmail.com<mailto:mobar...@gmail.com>> wrote: Hi - I am wondering if there is a limit to how many slaves can connect to one master. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com<mailto:jenkinsci-users+unsubscr...@googlegroups.com>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com<mailto:jenkinsci-users+unsubscr...@googlegroups.com>. For more options, visit https://groups.google.com/d/optout. Click here<https://www.mailcontrol.com/sr/!oag5VG71g!GX2PQPOmvUgfUspOYWKRbGsXAOJ84oQ4x7JVx4eYAnf8VFAIdfjuAZ!IcHOUH37CzR1Lppn2zgQ==> to report this email as spam. ________________________________ This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message. Thank you. Please consider the environment before printing this email. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
