|
Some of our builds are getting "stuck" when running the Warnings plugin during the publish phase of the build. The builds affected are configured to use the "Javac (Eclipse)" parser and the "Javadoc" parser. The parsers are configured against the console log only.
When a build gets stuck running the Warnings plugin, any other build that also uses the Warnings plugin and is running at about the same time also gets stuck. Eventually, all of our slaves are consumed with builds stuck at the warnings plugin step and no further builds can be built.
We are able to kill all of the builds except the one build which triggered the problem. In order to kill the original build we end up restarting the entire Jenkins system.
We did some investigation and it appears that the Warnings plugin is getting stuck while using a regular _expression_ to parse the console log. The thread dump of Jenkins shows that the Warnings plugin has a deep stack running a regular _expression_ (see attached), and a quick peak at the system itself shows that the plugin is consuming 100% of one of our server's cpu cores.
It is clear from the thread stack and the build console log that the plugin is getting stuck when using the "Java (Eclipse)" plugin.
We know there have been problems in the past with parsing very large logs or files, however, the build console log is not very large (23kb).
We let the stuck build run for about 18 hours before we decided to kill it. We know the plugin was doing work (a) because the CPU core was pegged at 100%, and (b) the stack trace for the plugin worker thread would change periodically.
|
