Re: Jenkins queue times steadily increasing for a few months now

2019-09-24 Thread Robert Bradshaw
Yeah, that's useful. I was asking about getting things at the jenkins job level. E.g. are our PostCommits taking up all the time, or our Precommits? On Tue, Sep 24, 2019 at 1:23 PM Lukasz Cwik wrote: > > We can get the per gradle task profile with the --profile flag: > https://jakewharton.com/st

Re: Jenkins queue times steadily increasing for a few months now

2019-09-24 Thread Lukasz Cwik
We can get the per gradle task profile with the --profile flag: https://jakewharton.com/static/files/trace/profile.html This information also appears within the build scans that are sent to Gradle. Integrating with either of these sources of information would allow us to figure out whether its new

Re: Jenkins queue times steadily increasing for a few months now

2019-09-24 Thread Robert Bradshaw
Does anyone know how to gather stats on where the time is being spent? Several times the idea of consolidating many of the (expensive) validates runner integration tests into a single pipeline, and then running things individually only if that fails, has come up. I think that'd be a big win if inde

Re: Jenkins queue times steadily increasing for a few months now

2019-09-24 Thread Daniel Oliveira
Those ideas all sound good. I especially agree with trying to reduce tests first and then if we've done all we can there and latency is still too high, it means we need more workers. Also in addition to reducing the amount of tests, there's also running less important tests less frequently, particu

Re: Jenkins queue times steadily increasing for a few months now

2019-09-19 Thread Mikhail Gryzykhin
Hi Daniel, Generally this looks feasible since jobs wait for new worker to be available to start. Over time we added more tests and did not deprecate enough, this increases load on workers. I wonder if we can add something like total runtime of all running jobs? This will be a safeguard metric th

Jenkins queue times steadily increasing for a few months now

2019-09-19 Thread Daniel Oliveira
Hi everyone, A little while ago I was taking a look at the Precommit Latency metrics on Grafana (link ) and saw that the monthly 90th percentile metric has been really increasing the past few months, from aroun