|
In our Jenkins install, we currently have 98 build nodes attached, with one offline (97 online nodes attached). When adding more nodes, we are consistently seeing an OutOfMemory error when attempting to add the 101st node. This error leaves Jenkins in a strange state – the UI is still responsive, but no jobs can be submitted via the CLI. A restart allows jobs to once again be submitted.
We have done a fair amount of cleanup to our plugin installs and decreasing build history, but the behavior is the same. We have not yet done any in-depth memory profiling on the machine to see what happens, but it seems that if it were a real OOM error, the point at which it would occur would be random. We see it consistently occur when adding the 101st build node. This leads me to believe that there is a hard-coded implicit limit on the number of online build nodes which can be attached to a single Jenkins instance at a time.
If there is such a limit, it would be good to a) make that explicit and b) remove the limit. If no such limit exists, additional assistance on what could be contributing to these errors would be greatly appreciated.
For reference, we are providing java args:
-Xmx12G
-Xms12G
-XX:MaxNewSize=4G
-XX:NewSize=4G
-XX:MaxPermSize=512M
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:+CMSParallelRemarkEnabled
-XX:ParallelGCThreads=6
-XX:CMSInitiatingOccupancyFraction=45
|
