[jira] [Comment Edited] (MNG-6474) Maven intermittently hanging at end of multithreaded build
[ https://issues.apache.org/jira/browse/MNG-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861156#comment-16861156 ] Michael Osipov edited comment on MNG-6474 at 6/11/19 7:07 PM: -- If you have two jobs (VMs) accessing the same local repo than you are in trouble. The repo impl is not concurrent-safe. Use Takari's implementation for that or distinct repos. was (Author: michael-o): If you have two jobs (VM) accessing the same local repo than you are in trouble. The repo impl is not concurrent-safe. Use Takari's implementation for that or distinct repos. > Maven intermittently hanging at end of multithreaded build > -- > > Key: MNG-6474 > URL: https://issues.apache.org/jira/browse/MNG-6474 > Project: Maven > Issue Type: Bug >Affects Versions: 3.5.2 >Reporter: Beirtí Ó'Nunáin >Priority: Major > Attachments: Trunk_Build_after_every_commit_UNDEFINED-r15120.log, > thread dump.txt > > > I can't reproduce this with a small skeleton project. It has only started > happening since we upgraded from 3.3 to 3.5. > > I have a multithreaded multi-module build which builds a number of large > (300mb+) artifacts upon SVN commits. I'm running the following goals: > > mvn clean deploy -T 1C > > Reasonably often, perhaps 1 in 10 builds, the build hangs just before where > I'd expect the build to have finished. I'm guessing that maven is attempting > to install an artifact to the local repository (looking at the stack trace) > while the build is about to complete as when I compare the build log from the > hung build to a successful build, the following line is the "Reactor Summary" > This issue happens both on my local machine and on our TeamCity server (Both > multi-processor, multi-core 14 vs 32 cores) > The Thread dump isn't indicating any kind of DeadLock. Our sysadmins have > checked the box for any antivirus scanning which may be impacting the file > copy indicated in the log but can't see anything going on at that time. > Full thread dump from TeamCity attached > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (MNG-6474) Maven intermittently hanging at end of multithreaded build
[ https://issues.apache.org/jira/browse/MNG-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612061#comment-16612061 ] Michael Osipov edited comment on MNG-6474 at 9/12/18 12:37 PM: --- The error does happen if you build in a single thread? Can you please enable thread name logging with SLF4J Simple and rerun? If you can reproduce this locally, it'd be awesome if you could run also Wireshark. She should see the hang if this is socket-related. HAve you tried to enable socket timeout in Wagon? was (Author: michael-o): The error does happen if you build in a single thread? Can you please enable thread name logging with SLF4J Simple and rerun= > Maven intermittently hanging at end of multithreaded build > -- > > Key: MNG-6474 > URL: https://issues.apache.org/jira/browse/MNG-6474 > Project: Maven > Issue Type: Bug >Affects Versions: 3.5.2 >Reporter: Beirtí Ó'Nunáin >Priority: Major > Attachments: Trunk_Build_after_every_commit_UNDEFINED-r15120.log, > thread dump.txt > > > I can't reproduce this with a small skeleton project. It has only started > happening since we upgraded from 3.3 to 3.5. > > I have a multithreaded multi-module build which builds a number of large > (300mb+) artifacts upon SVN commits. I'm running the following goals: > > mvn clean deploy -T 1C > > Reasonably often, perhaps 1 in 10 builds, the build hangs just before where > I'd expect the build to have finished. I'm guessing that maven is attempting > to install an artifact to the local repository (looking at the stack trace) > while the build is about to complete as when I compare the build log from the > hung build to a successful build, the following line is the "Reactor Summary" > This issue happens both on my local machine and on our TeamCity server (Both > multi-processor, multi-core 14 vs 32 cores) > The Thread dump isn't indicating any kind of DeadLock. Our sysadmins have > checked the box for any antivirus scanning which may be impacting the file > copy indicated in the log but can't see anything going on at that time. > Full thread dump from TeamCity attached > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (MNG-6474) Maven intermittently hanging at end of multithreaded build
[
https://issues.apache.org/jira/browse/MNG-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611144#comment-16611144
]
Michael Osipov edited comment on MNG-6474 at 9/11/18 7:46 PM:
--
Please reformat the strace as it is outputted by Java. It is very hard to read.
A quick look says me that this is incomplete. There is a Resolver thread
waiting for something, likely an HTTP reponse. I miss what
{{0x6038e000}} and {{0x814b9900}} are. Can you provide more
details? Attach to the VM with VisualVM and check for this ref? Can you also
please provide a debug log from Maven 3.5.4 where this hang happens? I think
this thread is created by
{{org.eclipse.aether.internal.impl.DefaultMetadataResolver}} if I look at the
thread factory.
was (Author: michael-o):
Please reformat the strace as it is outputted by Java. It is very hard to read.
A quick look says me that this is incomplete. There is a Resolver thread
waiting for something, likely an HTTP reponse. I miss what
{{0x6038e000}} and {{0x814b9900}} are. Can you provide more
details? Attach to the VM with VisualVM and check for this ref? Can you also
please provide a debug log from Maven 3.5.4 where this hang happens?
> Maven intermittently hanging at end of multithreaded build
> --
>
> Key: MNG-6474
> URL: https://issues.apache.org/jira/browse/MNG-6474
> Project: Maven
> Issue Type: Bug
>Affects Versions: 3.5.2
>Reporter: Beirtí Ó'Nunáin
>Priority: Major
>
> I can't reproduce this with a small skeleton project. It has only started
> happening since we upgraded from 3.3 to 3.5.
>
> I have a multithreaded multi-module build which builds a number of large
> (300mb+) artifacts upon SVN commits. I'm running the following goals:
>
> mvn clean deploy -T 1C
>
> Reasonably often, perhaps 1 in 10 builds, the build hangs just before where
> I'd expect the build to have finished. I'm guessing that maven is attempting
> to install an artifact to the local repository (looking at the stack trace)
> while the build is about to complete as when I compare the build log from the
> hung build to a successful build, the following line is the "Reactor Summary"
> This issue happens both on my local machine and on our TeamCity server (Both
> multi-processor, multi-core 14 vs 32 cores)
> The Thread dump isn't indicating any kind of DeadLock. Our sysadmins have
> checked the box for any antivirus scanning which may be impacting the file
> copy indicated in the log but can't see anything going on at that time.
> Here's the thread dump from TeamCity when it times out:
>
> {quote}
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode):
> "resolver-5" #63 daemon prio=5 os_prio=0 tid=0x596ff800 nid=0x1e98
> waiting on condition [0x6038e000] java.lang.Thread.State: WAITING
> (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for
> <0x814b9900> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745) "resolver-4" #62 daemon prio=5
> os_prio=0 tid=0x596fe800 nid=0x134c waiting on condition
> [0x5f42e000] java.lang.Thread.State: WAITING (parking) at
> sun.misc.Unsafe.park(Native Method) - parking to wait for
> <0x814b9900> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745) "resolver-3" #61 daemon prio=5
> os_prio=0 tid=0x596fe000 nid=0x17c0 waiting on condition
> [0x5fa4e000] java.lang.Thread.State: WAITING (parking) at
> sun.misc.Unsafe.park(Native Method) - parking to wait for
>
[jira] [Comment Edited] (MNG-6474) Maven intermittently hanging at end of multithreaded build
[
https://issues.apache.org/jira/browse/MNG-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611144#comment-16611144
]
Michael Osipov edited comment on MNG-6474 at 9/11/18 7:41 PM:
--
Please reformat the strace as it is outputted by Java. It is very hard to read.
A quick look says me that this is incomplete. There is a Resolver thread
waiting for something, likely an HTTP reponse. I miss what
{{0x6038e000}} and {{0x814b9900}} are. Can you provide more
details? Attach to the VM with VisualVM and check for this ref? Can you also
please provide a debug log from Maven 3.5.4 where this hang happens?
was (Author: michael-o):
Please reformat the strace as it is outputted by Java. It is very hard to read.
A quick look says me that this is incomplete. There is a Resolver thread
waiting for something, likely an HTTP reponse. I miss what
{{0x6038e000}} and {{0x814b9900}} are. Can you provide more
details? Attach to the VM with VisualVM and check for this ref?
> Maven intermittently hanging at end of multithreaded build
> --
>
> Key: MNG-6474
> URL: https://issues.apache.org/jira/browse/MNG-6474
> Project: Maven
> Issue Type: Bug
>Affects Versions: 3.5.2
>Reporter: Beirtí Ó'Nunáin
>Priority: Major
>
> I can't reproduce this with a small skeleton project. It has only started
> happening since we upgraded from 3.3 to 3.5.
>
> I have a multithreaded multi-module build which builds a number of large
> (300mb+) artifacts upon SVN commits. I'm running the following goals:
>
> mvn clean deploy -T 1C
>
> Reasonably often, perhaps 1 in 10 builds, the build hangs just before where
> I'd expect the build to have finished. I'm guessing that maven is attempting
> to install an artifact to the local repository (looking at the stack trace)
> while the build is about to complete as when I compare the build log from the
> hung build to a successful build, the following line is the "Reactor Summary"
> This issue happens both on my local machine and on our TeamCity server (Both
> multi-processor, multi-core 14 vs 32 cores)
> The Thread dump isn't indicating any kind of DeadLock. Our sysadmins have
> checked the box for any antivirus scanning which may be impacting the file
> copy indicated in the log but can't see anything going on at that time.
> Here's the thread dump from TeamCity when it times out:
>
> {quote}
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode):
> "resolver-5" #63 daemon prio=5 os_prio=0 tid=0x596ff800 nid=0x1e98
> waiting on condition [0x6038e000] java.lang.Thread.State: WAITING
> (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for
> <0x814b9900> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745) "resolver-4" #62 daemon prio=5
> os_prio=0 tid=0x596fe800 nid=0x134c waiting on condition
> [0x5f42e000] java.lang.Thread.State: WAITING (parking) at
> sun.misc.Unsafe.park(Native Method) - parking to wait for
> <0x814b9900> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745) "resolver-3" #61 daemon prio=5
> os_prio=0 tid=0x596fe000 nid=0x17c0 waiting on condition
> [0x5fa4e000] java.lang.Thread.State: WAITING (parking) at
> sun.misc.Unsafe.park(Native Method) - parking to wait for
> <0x814b9900> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>
