Re: Jenkins slowness

[Ismaël Mejía]

Not exactly related to this but we also found that one Spark test is
holding the Validates Runner tests for Spark for 10 extra minutes,
that could eventually add to reduce Jenkins use.

More investigation still required.
https://issues.apache.org/jira/browse/BEAM-6633

On Thu, Feb 7, 2019 at 11:23 PM Udi Meiri  wrote:
>
> No problem Michael - I wasn't aware of the state of the state of the Gradle 5 
> migration.
>
> On Thu, Feb 7, 2019 at 1:45 PM Michael Luckey  
> wrote:
>>
>> Migration to gradle 5 should not be an issue. Unless something unforeseen 
>> happens as we did not run all possible task yet.
>>
>> Should only be a merge of an pending + one additional PR upgrading the 
>> versions.
>>
>> On 7. Feb 2019, at 21:23, Udi Meiri  wrote:
>>
>> I suggest disabling Jacoco and re-enabling the build cache until we can 
>> migrate to Gradle 5. I imagine the migration to v5 is not a simple change.
>> Meanwhile, I can't run postcommits on PRs on Jenkins (run seed job + run 
>> postcommit).
>>
>> On Thu, Feb 7, 2019, 12:00 Chamikara Jayalath >>
>>> Seems like there was a spike for all build times yesterday probably added 
>>> up to give slow Jenkins scheduling times for triggers. Also, seems like we 
>>> had three spikes that are about a week apart recently.
>>>
>>>
>>> On Thu, Feb 7, 2019 at 11:46 AM Michael Luckey  wrote:
>>>>
>>>> What might have some influence is the implicit disabling of the build 
>>>> cache by activating Jacoco report. There seems to be a increase of 
>>>> beam_PreCommit_Java_Cron with 
>>>> https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/ and looking 
>>>> into cacheable task there seems to be lots of work done now which 
>>>> previously was cacheable.
>>>>
>>>> Not sure, whether this is the culprit- or part of it -, but I d suggest to 
>>>> upgrade to gradle 5 pretty fast.
>>>>
>>>> On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri  wrote:
>>>>>
>>>>> If anyone has done any investigation/is working on this please share.
>>>>>
>>>>> I'm investigating Jenkins slowness. I've noticed it happening since 
>>>>> yesterday: precommits taking 3 hours to start, phrase commands similarly 
>>>>> taking as much time to register.
>>>>>
>>>>> My current theory is that we have a job that's are taking much longer 
>>>>> than usual to run.
>>
>>

Re: Jenkins slowness

[Udi Meiri]

No problem Michael - I wasn't aware of the state of the state of the Gradle
5 migration.

On Thu, Feb 7, 2019 at 1:45 PM Michael Luckey 
wrote:

> Migration to gradle 5 should not be an issue. Unless something unforeseen
> happens as we did not run all possible task yet.
>
> Should only be a merge of an pending + one additional PR upgrading the
> versions.
>
> On 7. Feb 2019, at 21:23, Udi Meiri  wrote:
>
> I suggest disabling Jacoco and re-enabling the build cache until we can
> migrate to Gradle 5. I imagine the migration to v5 is not a simple change.
> Meanwhile, I can't run postcommits on PRs on Jenkins (run seed job + run
> postcommit).
>
> On Thu, Feb 7, 2019, 12:00 Chamikara Jayalath 
>> Seems like there was a spike for all build times yesterday probably added
>> up to give slow Jenkins scheduling times for triggers. Also, seems like we
>> had three spikes that are about a week apart recently.
>>
>>
>> On Thu, Feb 7, 2019 at 11:46 AM Michael Luckey 
>> wrote:
>>
>>> What might have some influence is the implicit disabling of the build
>>> cache by activating Jacoco report. There seems to be a increase of
>>> beam_PreCommit_Java_Cron with
>>> https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/ and looking
>>> into cacheable task there seems to be lots of work done now which
>>> previously was cacheable.
>>>
>>> Not sure, whether this is the culprit- or part of it -, but I d suggest
>>> to upgrade to gradle 5 pretty fast.
>>>
>>> On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri  wrote:
>>>
>>>> If anyone has done any investigation/is working on this please share.
>>>>
>>>> I'm investigating Jenkins slowness. I've noticed it happening since
>>>> yesterday: precommits taking 3 hours to start, phrase commands similarly
>>>> taking as much time to register.
>>>>
>>>> My current theory is that we have a job that's are taking much longer
>>>> than usual to run.
>>>>
>>>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Jenkins slowness

[Michael Luckey]

Migration to gradle 5 should not be an issue. Unless something unforeseen 
happens as we did not run all possible task yet. 

Should only be a merge of an pending + one additional PR upgrading the versions.

> On 7. Feb 2019, at 21:23, Udi Meiri  wrote:
> 
> I suggest disabling Jacoco and re-enabling the build cache until we can 
> migrate to Gradle 5. I imagine the migration to v5 is not a simple change.
> Meanwhile, I can't run postcommits on PRs on Jenkins (run seed job + run 
> postcommit).
> 
> On Thu, Feb 7, 2019, 12:00 Chamikara Jayalath  <mailto:chamik...@google.com> wrote:
> Seems like there was a spike for all build times yesterday probably added up 
> to give slow Jenkins scheduling times for triggers. Also, seems like we had 
> three spikes that are about a week apart recently.
> 
> 
> On Thu, Feb 7, 2019 at 11:46 AM Michael Luckey  <mailto:adude3...@gmail.com>> wrote:
> What might have some influence is the implicit disabling of the build cache 
> by activating Jacoco report. There seems to be a increase of 
> beam_PreCommit_Java_Cron with 
> https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/ 
> <https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/> and looking 
> into cacheable task there seems to be lots of work done now which previously 
> was cacheable.
> 
> Not sure, whether this is the culprit- or part of it -, but I d suggest to 
> upgrade to gradle 5 pretty fast.
> 
> On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri  <mailto:eh...@google.com>> wrote:
> If anyone has done any investigation/is working on this please share.
> 
> I'm investigating Jenkins slowness. I've noticed it happening since 
> yesterday: precommits taking 3 hours to start, phrase commands similarly 
> taking as much time to register.
> 
> My current theory is that we have a job that's are taking much longer than 
> usual to run.

Re: Jenkins slowness

[Yifan Zou]

I saw many non-beam jobs are running on our nodes. It probably be one of
the reasons which caused long waiting time.
e.g. https://builds.apache.org/computer/beam13/builds

On Thu, Feb 7, 2019 at 12:24 PM Udi Meiri  wrote:

> There is also excessive python test logging tracked here:
> https://issues.apache.org/jira/browse/BEAM-6603
>
> On Thu, Feb 7, 2019, 12:23 Udi Meiri 
>> I suggest disabling Jacoco and re-enabling the build cache until we can
>> migrate to Gradle 5. I imagine the migration to v5 is not a simple change.
>> Meanwhile, I can't run postcommits on PRs on Jenkins (run seed job + run
>> postcommit).
>>
>> On Thu, Feb 7, 2019, 12:00 Chamikara Jayalath > wrote:
>>
>>> Seems like there was a spike for all build times yesterday probably
>>> added up to give slow Jenkins scheduling times for triggers. Also, seems
>>> like we had three spikes that are about a week apart recently.
>>>
>>>
>>> On Thu, Feb 7, 2019 at 11:46 AM Michael Luckey 
>>> wrote:
>>>
>>>> What might have some influence is the implicit disabling of the build
>>>> cache by activating Jacoco report. There seems to be a increase of
>>>> beam_PreCommit_Java_Cron with
>>>> https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/ and
>>>> looking into cacheable task there seems to be lots of work done now which
>>>> previously was cacheable.
>>>>
>>>> Not sure, whether this is the culprit- or part of it -, but I d suggest
>>>> to upgrade to gradle 5 pretty fast.
>>>>
>>>> On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri  wrote:
>>>>
>>>>> If anyone has done any investigation/is working on this please share.
>>>>>
>>>>> I'm investigating Jenkins slowness. I've noticed it happening since
>>>>> yesterday: precommits taking 3 hours to start, phrase commands similarly
>>>>> taking as much time to register.
>>>>>
>>>>> My current theory is that we have a job that's are taking much longer
>>>>> than usual to run.
>>>>>
>>>>

Re: Jenkins slowness

[Udi Meiri]

I suggest disabling Jacoco and re-enabling the build cache until we can
migrate to Gradle 5. I imagine the migration to v5 is not a simple change.
Meanwhile, I can't run postcommits on PRs on Jenkins (run seed job + run
postcommit).

On Thu, Feb 7, 2019, 12:00 Chamikara Jayalath  Seems like there was a spike for all build times yesterday probably added
> up to give slow Jenkins scheduling times for triggers. Also, seems like we
> had three spikes that are about a week apart recently.
>
>
> On Thu, Feb 7, 2019 at 11:46 AM Michael Luckey 
> wrote:
>
>> What might have some influence is the implicit disabling of the build
>> cache by activating Jacoco report. There seems to be a increase of
>> beam_PreCommit_Java_Cron with
>> https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/ and looking
>> into cacheable task there seems to be lots of work done now which
>> previously was cacheable.
>>
>> Not sure, whether this is the culprit- or part of it -, but I d suggest
>> to upgrade to gradle 5 pretty fast.
>>
>> On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri  wrote:
>>
>>> If anyone has done any investigation/is working on this please share.
>>>
>>> I'm investigating Jenkins slowness. I've noticed it happening since
>>> yesterday: precommits taking 3 hours to start, phrase commands similarly
>>> taking as much time to register.
>>>
>>> My current theory is that we have a job that's are taking much longer
>>> than usual to run.
>>>
>>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Jenkins slowness

[Udi Meiri]

There is also excessive python test logging tracked here:
https://issues.apache.org/jira/browse/BEAM-6603

On Thu, Feb 7, 2019, 12:23 Udi Meiri  I suggest disabling Jacoco and re-enabling the build cache until we can
> migrate to Gradle 5. I imagine the migration to v5 is not a simple change.
> Meanwhile, I can't run postcommits on PRs on Jenkins (run seed job + run
> postcommit).
>
> On Thu, Feb 7, 2019, 12:00 Chamikara Jayalath 
>> Seems like there was a spike for all build times yesterday probably added
>> up to give slow Jenkins scheduling times for triggers. Also, seems like we
>> had three spikes that are about a week apart recently.
>>
>>
>> On Thu, Feb 7, 2019 at 11:46 AM Michael Luckey 
>> wrote:
>>
>>> What might have some influence is the implicit disabling of the build
>>> cache by activating Jacoco report. There seems to be a increase of
>>> beam_PreCommit_Java_Cron with
>>> https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/ and looking
>>> into cacheable task there seems to be lots of work done now which
>>> previously was cacheable.
>>>
>>> Not sure, whether this is the culprit- or part of it -, but I d suggest
>>> to upgrade to gradle 5 pretty fast.
>>>
>>> On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri  wrote:
>>>
>>>> If anyone has done any investigation/is working on this please share.
>>>>
>>>> I'm investigating Jenkins slowness. I've noticed it happening since
>>>> yesterday: precommits taking 3 hours to start, phrase commands similarly
>>>> taking as much time to register.
>>>>
>>>> My current theory is that we have a job that's are taking much longer
>>>> than usual to run.
>>>>
>>>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Jenkins slowness

[Chamikara Jayalath]

Seems like there was a spike for all build times yesterday probably added
up to give slow Jenkins scheduling times for triggers. Also, seems like we
had three spikes that are about a week apart recently.


On Thu, Feb 7, 2019 at 11:46 AM Michael Luckey  wrote:

> What might have some influence is the implicit disabling of the build
> cache by activating Jacoco report. There seems to be a increase of
> beam_PreCommit_Java_Cron with
> https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/ and looking
> into cacheable task there seems to be lots of work done now which
> previously was cacheable.
>
> Not sure, whether this is the culprit- or part of it -, but I d suggest to
> upgrade to gradle 5 pretty fast.
>
> On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri  wrote:
>
>> If anyone has done any investigation/is working on this please share.
>>
>> I'm investigating Jenkins slowness. I've noticed it happening since
>> yesterday: precommits taking 3 hours to start, phrase commands similarly
>> taking as much time to register.
>>
>> My current theory is that we have a job that's are taking much longer
>> than usual to run.
>>
>

Re: Jenkins slowness

[Michael Luckey]

What might have some influence is the implicit disabling of the build cache
by activating Jacoco report. There seems to be a increase of
beam_PreCommit_Java_Cron with
https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/ and looking
into cacheable task there seems to be lots of work done now which
previously was cacheable.

Not sure, whether this is the culprit- or part of it -, but I d suggest to
upgrade to gradle 5 pretty fast.

On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri  wrote:

> If anyone has done any investigation/is working on this please share.
>
> I'm investigating Jenkins slowness. I've noticed it happening since
> yesterday: precommits taking 3 hours to start, phrase commands similarly
> taking as much time to register.
>
> My current theory is that we have a job that's are taking much longer than
> usual to run.
>

Jenkins slowness

[Udi Meiri]

If anyone has done any investigation/is working on this please share.

I'm investigating Jenkins slowness. I've noticed it happening since
yesterday: precommits taking 3 hours to start, phrase commands similarly
taking as much time to register.

My current theory is that we have a job that's are taking much longer than
usual to run.


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Jenkins slowness

[Udi Meiri]

Precommits times for Python and Java have been slowly climbing:
http://104.154.241.245/d/_TNndF2iz/pre-commit-test-latency?orgId=1&from=1546974237153&to=1549566237153&theme=light

On Thu, Feb 7, 2019 at 10:54 AM Udi Meiri  wrote:

> If anyone has done any investigation/is working on this please share.
>
> I'm investigating Jenkins slowness. I've noticed it happening since
> yesterday: precommits taking 3 hours to start, phrase commands similarly
> taking as much time to register.
>
> My current theory is that we have a job that's are taking much longer than
> usual to run.
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Jenkins slowness

[Udi Meiri]

Another run on Jenkins was faster:
Ran 1535 tests in 267.273s

On Fri, Mar 9, 2018 at 10:52 AM Lukasz Cwik  wrote:

> The only statistics I get to see are the build load statistics:
> https://builds.apache.org/computer/beam1/load-statistics
>
> On Fri, Mar 9, 2018 at 10:43 AM, Udi Meiri  wrote:
>
>> Is there a way to see OS load statistics?
>>
>> On Fri, Mar 9, 2018 at 10:24 AM Jason Kuster 
>> wrote:
>>
>>> Jenkins executors are probably less beefy than your development machine
>>> and also have two slots, thus are likely running two builds at once,
>>> causing extra slowness.
>>>
>>>
>>> On Fri, Mar 9, 2018 at 10:18 AM Udi Meiri  wrote:
>>>
 Hi,

 Does anybody know why Jenkins hosts take so long to run? For example,
 beam1 was running beam_PostCommit_Python_Verify and I saw this time for
 running "tox -e py27":

 Ran 1535 tests in 403.860s

 on my workstation I got:

 Ran 1535 tests in 160.242s

 Is there any way to troubleshoot this? Each run takes about an hour to
 run, not including time waiting in the queue.

>>>
>>>
>>> --
>>> ---
>>> Jason Kuster
>>> Apache Beam / Google Cloud Dataflow
>>>
>>> See something? Say something. go/jasonkuster-feedback
>>> 
>>>
>>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Jenkins slowness

[Lukasz Cwik]

The only statistics I get to see are the build load statistics:
https://builds.apache.org/computer/beam1/load-statistics

On Fri, Mar 9, 2018 at 10:43 AM, Udi Meiri  wrote:

> Is there a way to see OS load statistics?
>
> On Fri, Mar 9, 2018 at 10:24 AM Jason Kuster 
> wrote:
>
>> Jenkins executors are probably less beefy than your development machine
>> and also have two slots, thus are likely running two builds at once,
>> causing extra slowness.
>>
>>
>> On Fri, Mar 9, 2018 at 10:18 AM Udi Meiri  wrote:
>>
>>> Hi,
>>>
>>> Does anybody know why Jenkins hosts take so long to run? For example,
>>> beam1 was running beam_PostCommit_Python_Verify and I saw this time for
>>> running "tox -e py27":
>>>
>>> Ran 1535 tests in 403.860s
>>>
>>> on my workstation I got:
>>>
>>> Ran 1535 tests in 160.242s
>>>
>>> Is there any way to troubleshoot this? Each run takes about an hour to
>>> run, not including time waiting in the queue.
>>>
>>
>>
>> --
>> ---
>> Jason Kuster
>> Apache Beam / Google Cloud Dataflow
>>
>> See something? Say something. go/jasonkuster-feedback
>> 
>>
>

Re: Jenkins slowness

[Udi Meiri]

Is there a way to see OS load statistics?

On Fri, Mar 9, 2018 at 10:24 AM Jason Kuster  wrote:

> Jenkins executors are probably less beefy than your development machine
> and also have two slots, thus are likely running two builds at once,
> causing extra slowness.
>
>
> On Fri, Mar 9, 2018 at 10:18 AM Udi Meiri  wrote:
>
>> Hi,
>>
>> Does anybody know why Jenkins hosts take so long to run? For example,
>> beam1 was running beam_PostCommit_Python_Verify and I saw this time for
>> running "tox -e py27":
>>
>> Ran 1535 tests in 403.860s
>>
>> on my workstation I got:
>>
>> Ran 1535 tests in 160.242s
>>
>> Is there any way to troubleshoot this? Each run takes about an hour to
>> run, not including time waiting in the queue.
>>
>
>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> 
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Jenkins slowness

[Jason Kuster]

Jenkins executors are probably less beefy than your development machine and
also have two slots, thus are likely running two builds at once, causing
extra slowness.


On Fri, Mar 9, 2018 at 10:18 AM Udi Meiri  wrote:

> Hi,
>
> Does anybody know why Jenkins hosts take so long to run? For example,
> beam1 was running beam_PostCommit_Python_Verify and I saw this time for
> running "tox -e py27":
>
> Ran 1535 tests in 403.860s
>
> on my workstation I got:
>
> Ran 1535 tests in 160.242s
>
> Is there any way to troubleshoot this? Each run takes about an hour to
> run, not including time waiting in the queue.
>


-- 
---
Jason Kuster
Apache Beam / Google Cloud Dataflow

See something? Say something. go/jasonkuster-feedback

Jenkins slowness

[Udi Meiri]

Hi,

Does anybody know why Jenkins hosts take so long to run? For example, beam1
was running beam_PostCommit_Python_Verify and I saw this time for running
"tox -e py27":

Ran 1535 tests in 403.860s

on my workstation I got:

Ran 1535 tests in 160.242s

Is there any way to troubleshoot this? Each run takes about an hour to run,
not including time waiting in the queue.


smime.p7s
Description: S/MIME Cryptographic Signature