Re: Builds on Jenkins
I'll take a look at Jenkins when back in the office. Worst case probably ask infra on hipchat On Thu, Jun 14, 2018, 7:42 AM Andriy Redko wrote: > Hey guys, > > So here is what I found. I was able to get the thread dump with Apache > Maven 3.5.2 and here is the victim: > > "DefaultMetadataResolver-1-0" #31 daemon prio=5 os_prio=0 > tid=0x1a9ae800 nid=0x18bc waiting on condition [0x2021e000] >java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.eclipse.aether.connector.basic.PartialFile$LockFile.lock(PartialFile.java:113) > > at > org.eclipse.aether.connector.basic.PartialFile$LockFile.(PartialFile.java:69) > > at > org.eclipse.aether.connector.basic.PartialFile$Factory.newInstance(PartialFile.java:278) > > at > org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:438) > > at > org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:360) > > at > org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75) > > at > org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:583) > > at > org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:232) > > at > org.eclipse.aether.internal.impl.DefaultMetadataResolver$ResolveTask.run(DefaultMetadataResolver.java:593) > > at > org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Maven stuck on obtaining file lock, in this case for: > > Downloading from apache.snapshots: > https://repository.apache.org/snapshots/org/apache/cxf/xjcplugins/cxf-xjc-dv/3.2.2-SNAPSHOT/maven-metadata.xml > Downloading from apache.snapshots: > https://repository.apache.org/snapshots/org/apache/cxf/xjcplugins/cxf-xjc-dv/3.2.2-SNAPSHOT/maven-metadata.xml > Downloaded from apache.snapshots: > https://repository.apache.org/snapshots/org/apache/cxf/xjcplugins/cxf-xjc-dv/3.2.2-SNAPSHOT/maven-metadata.xml > (785 B at 2.1 kB/s) > > It seems to be related to https://issues.apache.org/jira/browse/MNG-6323 > and fixed in Apache Maven 3.5.3. Indeed, I have tried Apache Maven 3.5.3 > and the issue is not there anymore, at least for me. It is reproducible > 100% (just switching from 3.5.3 back to 3.5.2 and below manifests the > problem right away). It is still not clear to me why Colm sees slow builds > with 3.5.3, could be another issue? > > Regarding Jenkins, I am not able to find what is behind "Maven 3 (latest)" > in the tooling, I don't have right permissions. Dennis, John any chance you > guys could help to figure it out? Thanks. > > Best Regards, > Andriy Redko > > > > Friday, June 8, 2018, 6:58:37 AM, you wrote: > > > The latest - 3.5.3. > > Colm. > > On Fri, Jun 8, 2018 at 11:55 AM, Andriy Redko wrote: > > Hi Colm, > > Thanks for confirming, what version of Apache Maven you are using? As a > temp workaround, > if you build with Maven 3.2.1 (or I think 3.2.x in general), the issue > shouldn't manifest. > > Best Regards, > Andriy Redko > > > Friday, June 8, 2018, 5:14:38 AM, you wrote: > > COh> I've also noticed Maven "pausing" recently when downloading SNAPSHOTs > on my local machine (ubuntu). > > > COh> Colm. > > > COh> On Fri, Jun 8, 2018 at 2:27 AM, Andriy Redko > wrote: > > COh> Hey guys, > > COh> Quick update on this. The issue is not resolved, BUT I accidentally > reproduced the it on my local box. So I used to > COh> build with `Apache Maven 3.2.1` and have never seen any issues, but > recently I run the CXF build with `Apache Maven 3.5.2`, > COh> by accident actually, and ... the same 15 mins delay on snapshot > downloads ... !!! 100% of the time, I don't know the cause > COh> yet but I believe this is not an infrastructure problem so I closed > COh> https://issues.apache.org/jira/browse/INFRA-16489 ticket. > COh> I will be looking into this, at least with a goal to pinpoint the > Maven version to blame, so we could figure out the root of > COh> it. > > COh> PS: FYI, running build on Windows box / Java 8. Will check on Linux > shortly (but gut feeling tells me the 15 mins delay > COh> will be there) > > COh> Best Regards, > COh> Andriy Redko > > COh> Sunday, May 6, 2018, 2:42:05 AM, you wrote: > > DK>> Hi Andriy, > > DK>> thanks for the analysis. > >>> John, guys, any hints who might help us here? Infra team? > > DK>> Could you please open an issue at: > DK>> https://issues.apache.org/jira/projects/INFRA/ > > DK>> Cheers > DK>> Dennis > > > > > > > > > > -- > Colm O
Re: Builds on Jenkins
Hey guys, So here is what I found. I was able to get the thread dump with Apache Maven 3.5.2 and here is the victim: "DefaultMetadataResolver-1-0" #31 daemon prio=5 os_prio=0 tid=0x1a9ae800 nid=0x18bc waiting on condition [0x2021e000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.eclipse.aether.connector.basic.PartialFile$LockFile.lock(PartialFile.java:113) at org.eclipse.aether.connector.basic.PartialFile$LockFile.(PartialFile.java:69) at org.eclipse.aether.connector.basic.PartialFile$Factory.newInstance(PartialFile.java:278) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:438) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:360) at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:583) at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:232) at org.eclipse.aether.internal.impl.DefaultMetadataResolver$ResolveTask.run(DefaultMetadataResolver.java:593) at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Maven stuck on obtaining file lock, in this case for: Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/cxf/xjcplugins/cxf-xjc-dv/3.2.2-SNAPSHOT/maven-metadata.xml Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/cxf/xjcplugins/cxf-xjc-dv/3.2.2-SNAPSHOT/maven-metadata.xml Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/cxf/xjcplugins/cxf-xjc-dv/3.2.2-SNAPSHOT/maven-metadata.xml (785 B at 2.1 kB/s) It seems to be related to https://issues.apache.org/jira/browse/MNG-6323 and fixed in Apache Maven 3.5.3. Indeed, I have tried Apache Maven 3.5.3 and the issue is not there anymore, at least for me. It is reproducible 100% (just switching from 3.5.3 back to 3.5.2 and below manifests the problem right away). It is still not clear to me why Colm sees slow builds with 3.5.3, could be another issue? Regarding Jenkins, I am not able to find what is behind "Maven 3 (latest)" in the tooling, I don't have right permissions. Dennis, John any chance you guys could help to figure it out? Thanks. Best Regards, Andriy Redko Friday, June 8, 2018, 6:58:37 AM, you wrote: The latest - 3.5.3. Colm. On Fri, Jun 8, 2018 at 11:55 AM, Andriy Redko wrote: Hi Colm, Thanks for confirming, what version of Apache Maven you are using? As a temp workaround, if you build with Maven 3.2.1 (or I think 3.2.x in general), the issue shouldn't manifest. Best Regards, Andriy Redko Friday, June 8, 2018, 5:14:38 AM, you wrote: COh> I've also noticed Maven "pausing" recently when downloading SNAPSHOTs on my local machine (ubuntu). COh> Colm. COh> On Fri, Jun 8, 2018 at 2:27 AM, Andriy Redko wrote: COh> Hey guys, COh> Quick update on this. The issue is not resolved, BUT I accidentally reproduced the it on my local box. So I used to COh> build with `Apache Maven 3.2.1` and have never seen any issues, but recently I run the CXF build with `Apache Maven 3.5.2`, COh> by accident actually, and ... the same 15 mins delay on snapshot downloads ... !!! 100% of the time, I don't know the cause COh> yet but I believe this is not an infrastructure problem so I closed COh> https://issues.apache.org/jira/browse/INFRA-16489 ticket. COh> I will be looking into this, at least with a goal to pinpoint the Maven version to blame, so we could figure out the root of COh> it. COh> PS: FYI, running build on Windows box / Java 8. Will check on Linux shortly (but gut feeling tells me the 15 mins delay COh> will be there) COh> Best Regards, COh> Andriy Redko COh> Sunday, May 6, 2018, 2:42:05 AM, you wrote: DK>> Hi Andriy, DK>> thanks for the analysis. >>> John, guys, any hints who might help us here? Infra team? DK>> Could you please open an issue at: DK>> https://issues.apache.org/jira/projects/INFRA/ DK>> Cheers DK>> Dennis -- Colm O hEigeartaigh Talend Community Coder http://coders.talend.com
Re: Builds on Jenkins
I will definitely do that, haven't had time yet, it was late night discovery. You may use -U option when running the mvn command, it should force the snapshots refresh. Best Regards, Andriy Redko DK> Have you tried to do a thread dump to see where it's hanging? So far I DK> couldn't reproduce this, maybe I have to try with a clean local repo. DK> Dennis DK> Am 08.06.2018 um 11:14 schrieb Colm O hEigeartaigh: >> I've also noticed Maven "pausing" recently when downloading SNAPSHOTs on my >> local machine (ubuntu). >> Colm. >> On Fri, Jun 8, 2018 at 2:27 AM, Andriy Redko wrote: >>> Hey guys, >>> Quick update on this. The issue is not resolved, BUT I accidentally >>> reproduced the it on my local box. So I used to >>> build with `Apache Maven 3.2.1` and have never seen any issues, but >>> recently I run the CXF build with `Apache Maven 3.5.2`, >>> by accident actually, and ... the same 15 mins delay on snapshot downloads >>> ... !!! 100% of the time, I don't know the cause >>> yet but I believe this is not an infrastructure problem so I closed >>> https://issues.apache.org/jira/browse/INFRA-16489 ticket. >>> I will be looking into this, at least with a goal to pinpoint the Maven >>> version to blame, so we could figure out the root of >>> it. >>> PS: FYI, running build on Windows box / Java 8. Will check on Linux >>> shortly (but gut feeling tells me the 15 mins delay >>> will be there) >>> Best Regards, >>> Andriy Redko >>> Sunday, May 6, 2018, 2:42:05 AM, you wrote: >>> DK> Hi Andriy, >>> DK> thanks for the analysis. > John, guys, any hints who might help us here? Infra team? >>> DK> Could you please open an issue at: >>> DK> https://issues.apache.org/jira/projects/INFRA/ >>> DK> Cheers >>> DK> Dennis
Re: Builds on Jenkins
The latest - 3.5.3. Colm. On Fri, Jun 8, 2018 at 11:55 AM, Andriy Redko wrote: > Hi Colm, > > Thanks for confirming, what version of Apache Maven you are using? As a > temp workaround, > if you build with Maven 3.2.1 (or I think 3.2.x in general), the issue > shouldn't manifest. > > Best Regards, > Andriy Redko > > > Friday, June 8, 2018, 5:14:38 AM, you wrote: > > COh> I've also noticed Maven "pausing" recently when downloading SNAPSHOTs > on my local machine (ubuntu). > > > COh> Colm. > > > COh> On Fri, Jun 8, 2018 at 2:27 AM, Andriy Redko > wrote: > > COh> Hey guys, > > COh> Quick update on this. The issue is not resolved, BUT I accidentally > reproduced the it on my local box. So I used to > COh> build with `Apache Maven 3.2.1` and have never seen any issues, but > recently I run the CXF build with `Apache Maven 3.5.2`, > COh> by accident actually, and ... the same 15 mins delay on snapshot > downloads ... !!! 100% of the time, I don't know the cause > COh> yet but I believe this is not an infrastructure problem so I closed > COh> https://issues.apache.org/jira/browse/INFRA-16489 ticket. > COh> I will be looking into this, at least with a goal to pinpoint the > Maven version to blame, so we could figure out the root of > COh> it. > > COh> PS: FYI, running build on Windows box / Java 8. Will check on Linux > shortly (but gut feeling tells me the 15 mins delay > COh> will be there) > > COh> Best Regards, > COh> Andriy Redko > > COh> Sunday, May 6, 2018, 2:42:05 AM, you wrote: > > DK>> Hi Andriy, > > DK>> thanks for the analysis. > >>> John, guys, any hints who might help us here? Infra team? > > DK>> Could you please open an issue at: > DK>> https://issues.apache.org/jira/projects/INFRA/ > > DK>> Cheers > DK>> Dennis > > > > > > > -- Colm O hEigeartaigh Talend Community Coder http://coders.talend.com
Re: Builds on Jenkins
I've also noticed Maven "pausing" recently when downloading SNAPSHOTs on my local machine (ubuntu). Colm. On Fri, Jun 8, 2018 at 2:27 AM, Andriy Redko wrote: > Hey guys, > > Quick update on this. The issue is not resolved, BUT I accidentally > reproduced the it on my local box. So I used to > build with `Apache Maven 3.2.1` and have never seen any issues, but > recently I run the CXF build with `Apache Maven 3.5.2`, > by accident actually, and ... the same 15 mins delay on snapshot downloads > ... !!! 100% of the time, I don't know the cause > yet but I believe this is not an infrastructure problem so I closed > https://issues.apache.org/jira/browse/INFRA-16489 ticket. > I will be looking into this, at least with a goal to pinpoint the Maven > version to blame, so we could figure out the root of > it. > > PS: FYI, running build on Windows box / Java 8. Will check on Linux > shortly (but gut feeling tells me the 15 mins delay > will be there) > > Best Regards, > Andriy Redko > > Sunday, May 6, 2018, 2:42:05 AM, you wrote: > > DK> Hi Andriy, > > DK> thanks for the analysis. > >> John, guys, any hints who might help us here? Infra team? > > DK> Could you please open an issue at: > DK> https://issues.apache.org/jira/projects/INFRA/ > > DK> Cheers > DK> Dennis > > -- Colm O hEigeartaigh Talend Community Coder http://coders.talend.com
Re: Builds on Jenkins
Hey guys, Quick update on this. The issue is not resolved, BUT I accidentally reproduced the it on my local box. So I used to build with `Apache Maven 3.2.1` and have never seen any issues, but recently I run the CXF build with `Apache Maven 3.5.2`, by accident actually, and ... the same 15 mins delay on snapshot downloads ... !!! 100% of the time, I don't know the cause yet but I believe this is not an infrastructure problem so I closed https://issues.apache.org/jira/browse/INFRA-16489 ticket. I will be looking into this, at least with a goal to pinpoint the Maven version to blame, so we could figure out the root of it. PS: FYI, running build on Windows box / Java 8. Will check on Linux shortly (but gut feeling tells me the 15 mins delay will be there) Best Regards, Andriy Redko Sunday, May 6, 2018, 2:42:05 AM, you wrote: DK> Hi Andriy, DK> thanks for the analysis. >> John, guys, any hints who might help us here? Infra team? DK> Could you please open an issue at: DK> https://issues.apache.org/jira/projects/INFRA/ DK> Cheers DK> Dennis
Re: Builds on Jenkins
Done, https://issues.apache.org/jira/browse/INFRA-16489, thanks Dennis. Sunday, May 6, 2018, 2:42:05 AM, you wrote: DK> Hi Andriy, DK> thanks for the analysis. >> John, guys, any hints who might help us here? Infra team? DK> Could you please open an issue at: DK> https://issues.apache.org/jira/projects/INFRA/ DK> Cheers DK> Dennis
Re: Builds on Jenkins
Hi Andriy, thanks for the analysis. > John, guys, any hints who might help us here? Infra team? > Could you please open an issue at: https://issues.apache.org/jira/projects/INFRA/ Cheers Dennis
Re: Builds on Jenkins
Ok, so the mystery of the long builds seems to be revealed, but not the cause. The downloading of the xjcplugins artifacts from http://repository.apache.org takes 15min, as per Maven log. And there are at least 4 times (may be more, but 4 for sure) it is done during the build (in total, +1h). Here is the "evidence": 17:58:03 [INFO] Downloaded from Nexus: http://repository.apache.org/snapshots/org/apache/cxf/xjcplugins/cxf-xjc-javadoc/3.2.2-SNAPSHOT/maven-metadata.xml (790 B at 2.7 kB/s) 18:13:03 [INFO] Downloaded from Nexus: http://repository.apache.org/snapshots/org/apache/cxf/xjcplugins/cxf-xjc-javadoc/3.2.2-SNAPSHOT/maven-metadata.xml (790 B at 0 B/s) 18:13:03 [INFO] Downloaded from Nexus: http://repository.apache.org/snapshots/org/apache/cxf/xjcplugins/cxf-xjc-dv/3.2.2-SNAPSHOT/maven-metadata.xml (785 B at 2.6 kB/s) 18:28:04 [INFO] Downloaded from Nexus: http://repository.apache.org/snapshots/org/apache/cxf/xjcplugins/cxf-xjc-dv/3.2.2-SNAPSHOT/maven-metadata.xml (785 B at 0 B/s) To keep builds going, I set the "No activity timeout" to 20min for now, will be reverted back once we solve the problem. John, guys, any hints who might help us here? Infra team? Thanks! Best Regards, Andriy Redko Saturday, May 5, 2018, 10:02:44 AM, you wrote: JDA> Andriy, JDA> You now have access to configure the jobs. JDA> I've reconfigured the JDK18 job to show timestamps. I also changed the JDA> time out strategy from absolute to no activity. Still needs tweaking. JDA> Though I agree a 3 hour build is not good. JDA> John JDA> On Sat, May 5, 2018 at 9:48 AM Andriy Redko <drr...@gmail.com> wrote: >> Hey guys, >> I think many of you noticed that builds on Jenkins ( >> https://builds.apache.org/job/CXF-Trunk-JDK18/, >> https://builds.apache.org/job/CXF-Trunk-PR/) >> started to take much longer (more than 3 hours) and eventually are aborted >> due to timeout (the last one we have is dated April 24th). Anyone has >> looked at it or has an idea why it started to happen? I don't have access >> to Jenkins configuration, but if someone could assist with adding the >> Timestamper (https://plugins.jenkins.io/timestamper), it would help to >> capture the timeline. The agents builds are run on seem to be the same. >> On another topic, since JDK9 is end of life, do we need to keep >> https://builds.apache.org/job/CXF-Master-JDK9/ active? >> Thanks! >> Best Regards, >> Andriy Redko
Re: Builds on Jenkins
Thanks a lot, John. On it. Best Regards, Andriy Redko On Sat, May 5, 2018, 10:02 AM John D. Ament <johndam...@apache.org> wrote: > Andriy, > > You now have access to configure the jobs. > > I've reconfigured the JDK18 job to show timestamps. I also changed the > time out strategy from absolute to no activity. Still needs tweaking. > Though I agree a 3 hour build is not good. > > John > > On Sat, May 5, 2018 at 9:48 AM Andriy Redko <drr...@gmail.com> wrote: > > > Hey guys, > > > > I think many of you noticed that builds on Jenkins ( > > https://builds.apache.org/job/CXF-Trunk-JDK18/, > > https://builds.apache.org/job/CXF-Trunk-PR/) > > started to take much longer (more than 3 hours) and eventually are > aborted > > due to timeout (the last one we have is dated April 24th). Anyone has > > looked at it or has an idea why it started to happen? I don't have access > > to Jenkins configuration, but if someone could assist with adding the > > Timestamper (https://plugins.jenkins.io/timestamper), it would help to > > capture the timeline. The agents builds are run on seem to be the same. > > > > On another topic, since JDK9 is end of life, do we need to keep > > https://builds.apache.org/job/CXF-Master-JDK9/ active? > > Thanks! > > > > Best Regards, > > Andriy Redko > > > > > > > > > > > > > > >
Re: Builds on Jenkins
Andriy, You now have access to configure the jobs. I've reconfigured the JDK18 job to show timestamps. I also changed the time out strategy from absolute to no activity. Still needs tweaking. Though I agree a 3 hour build is not good. John On Sat, May 5, 2018 at 9:48 AM Andriy Redko <drr...@gmail.com> wrote: > Hey guys, > > I think many of you noticed that builds on Jenkins ( > https://builds.apache.org/job/CXF-Trunk-JDK18/, > https://builds.apache.org/job/CXF-Trunk-PR/) > started to take much longer (more than 3 hours) and eventually are aborted > due to timeout (the last one we have is dated April 24th). Anyone has > looked at it or has an idea why it started to happen? I don't have access > to Jenkins configuration, but if someone could assist with adding the > Timestamper (https://plugins.jenkins.io/timestamper), it would help to > capture the timeline. The agents builds are run on seem to be the same. > > On another topic, since JDK9 is end of life, do we need to keep > https://builds.apache.org/job/CXF-Master-JDK9/ active? > Thanks! > > Best Regards, > Andriy Redko > > > > > > >
Builds on Jenkins
Hey guys, I think many of you noticed that builds on Jenkins (https://builds.apache.org/job/CXF-Trunk-JDK18/, https://builds.apache.org/job/CXF-Trunk-PR/) started to take much longer (more than 3 hours) and eventually are aborted due to timeout (the last one we have is dated April 24th). Anyone has looked at it or has an idea why it started to happen? I don't have access to Jenkins configuration, but if someone could assist with adding the Timestamper (https://plugins.jenkins.io/timestamper), it would help to capture the timeline. The agents builds are run on seem to be the same. On another topic, since JDK9 is end of life, do we need to keep https://builds.apache.org/job/CXF-Master-JDK9/ active? Thanks! Best Regards, Andriy Redko
