We recently upgraded to Jenkins 1.490, where we encountered severe page load time issues after running stable for months. We traced these issues to the lazy page load features, and chose to downgrade until performance of those features improves.
Unfortuantely, upon downgrading, jenkins performance issues occured in other ways. After a fresh restart, Jenkins may remain snappy and performant for hours, but inevitably at some point, some pages take a long time to load. Once that starts, over the next hour, all pages will take a long time to load, eventually becomming all but unresponsive (up to 2 minute load times). We are trying to investigate the root cause. I've used a lot of java applications and it feels like what happens when there are no free threads able to process a request. I honestly don't know where to begin, we've been looking at thread dumps, heap dumps and statistics from javaMelody (the monitoring plugin) as well as perfmon on our master server looking for clues as to what could be causing the issues. The machine we're running is a beefy 8 core, 32GB of ram Win 2008R2 server. We've seen the same performance issues on versions 1.480, 1.481, 1.484, and 1.485. Looking at thread dumps I don't perceive any erroneous threads in wait, I see the executor threads (we run 7 build agents, with 3 executors each) either working when builds are in process or waiting when not, etc. I'm not exactly sure which threads are responsible for the website. We've never encountered these kinds of issues with Jenkins before after a year of successful use, but we're at the end of our rope, having to restart Jenkins at least twice a day causing massive disruption in our CI routines. I've tried finding "best practice" documentation for troubleshooting performance issues, ways in which to identify if a plugin is the issue or what might be causing problems but I'm coming up dry. Any help that can be provided would be great. It seems like this should be an easy problem to find root cause of, there has to be some kind of lock contention on some resource going on. -- View this message in context: http://jenkins.361315.n4.nabble.com/Jenkins-1-480-1-485-performance-Page-loads-progressively-get-slower-until-unuseable-2-minute-load-ti-tp4649103.html Sent from the Jenkins issues mailing list archive at Nabble.com.
