[ https://issues.apache.org/jira/browse/SPARK-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14607577#comment-14607577 ]
Josh Rosen commented on SPARK-8415: ----------------------------------- It looks like this problem occurs when a huge number of retests get queued up and triggered all at once. In these cases, we can end up with 2+ builds proceeding in lockstep on the same Jenkins worker, leading to bad thrashing while trying to acquire the Ivy lock. When 4+ pull request builds fire off on the same machine within a minute or two of each other, things can get really bad. > Jenkins compilation spends lots of time re-resolving dependencies and waiting > to acquire Ivy cache lock > ------------------------------------------------------------------------------------------------------- > > Key: SPARK-8415 > URL: https://issues.apache.org/jira/browse/SPARK-8415 > Project: Spark > Issue Type: Bug > Components: Build, Project Infra > Reporter: Josh Rosen > > When watching a pull request build, I noticed that the compilation + > packaging + test compilation phases spent huge amounts of time waiting to > acquire the Ivy cache lock. We should see whether we can tell SBT to skip > the resolution steps for some of these commands, since this could speed up > the compilation process when Jenkins is heavily loaded. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org