[jira] [Commented] (SPARK-8415) Jenkins compilation spends lots of time re-resolving dependencies and waiting to acquire Ivy cache lock
[
https://issues.apache.org/jira/browse/SPARK-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14624061#comment-14624061
]
Josh Rosen commented on SPARK-8415:
---
Oh, and I also added a {{mkdir -p "$HOME"}} to the execute shell command.
> Jenkins compilation spends lots of time re-resolving dependencies and waiting
> to acquire Ivy cache lock
> ---
>
> Key: SPARK-8415
> URL: https://issues.apache.org/jira/browse/SPARK-8415
> Project: Spark
> Issue Type: Bug
> Components: Build, Project Infra
>Reporter: Josh Rosen
>Assignee: Josh Rosen
>
> When watching a pull request build, I noticed that the compilation +
> packaging + test compilation phases spent huge amounts of time waiting to
> acquire the Ivy cache lock. We should see whether we can tell SBT to skip
> the resolution steps for some of these commands, since this could speed up
> the compilation process when Jenkins is heavily loaded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8415) Jenkins compilation spends lots of time re-resolving dependencies and waiting to acquire Ivy cache lock
[
https://issues.apache.org/jira/browse/SPARK-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14624058#comment-14624058
]
Josh Rosen commented on SPARK-8415:
---
I figured out how to configure AMPLab Jenkins to use a separate ivy cache for
each pull request builder workspace. In the Jenkins environment / properties
injection, I adeded the following lines
{code}
HOME=/home/sparkivy/${JOB_NAME}_${EXECUTOR_NUMBER}
SBT_OPTS=-Duser.home=/home/sparkivy/${JOB_NAME}_${EXECUTOR_NUMBER}
-Dsbt.ivy.home=/home/sparkivy/${JOB_NAME}_${EXECUTOR_NUMBER}/.ivy2
{code}
Here, {{/home/sparkivy}} is a directory that's outside of the build workspace
so it won't be deleted by the {{git clean -fdx}} in our Jenkins build. The
substitutions ensure that each build gets its own independent directory. I'm
going to mark this issue as resolved since I'm switching the main
SparkPullRequestBuilder to use this configuration change.
> Jenkins compilation spends lots of time re-resolving dependencies and waiting
> to acquire Ivy cache lock
> ---
>
> Key: SPARK-8415
> URL: https://issues.apache.org/jira/browse/SPARK-8415
> Project: Spark
> Issue Type: Bug
> Components: Build, Project Infra
>Reporter: Josh Rosen
>
> When watching a pull request build, I noticed that the compilation +
> packaging + test compilation phases spent huge amounts of time waiting to
> acquire the Ivy cache lock. We should see whether we can tell SBT to skip
> the resolution steps for some of these commands, since this could speed up
> the compilation process when Jenkins is heavily loaded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8415) Jenkins compilation spends lots of time re-resolving dependencies and waiting to acquire Ivy cache lock
[ https://issues.apache.org/jira/browse/SPARK-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14607577#comment-14607577 ] Josh Rosen commented on SPARK-8415: --- It looks like this problem occurs when a huge number of retests get queued up and triggered all at once. In these cases, we can end up with 2+ builds proceeding in lockstep on the same Jenkins worker, leading to bad thrashing while trying to acquire the Ivy lock. When 4+ pull request builds fire off on the same machine within a minute or two of each other, things can get really bad. > Jenkins compilation spends lots of time re-resolving dependencies and waiting > to acquire Ivy cache lock > --- > > Key: SPARK-8415 > URL: https://issues.apache.org/jira/browse/SPARK-8415 > Project: Spark > Issue Type: Bug > Components: Build, Project Infra >Reporter: Josh Rosen > > When watching a pull request build, I noticed that the compilation + > packaging + test compilation phases spent huge amounts of time waiting to > acquire the Ivy cache lock. We should see whether we can tell SBT to skip > the resolution steps for some of these commands, since this could speed up > the compilation process when Jenkins is heavily loaded. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8415) Jenkins compilation spends lots of time re-resolving dependencies and waiting to acquire Ivy cache lock
[ https://issues.apache.org/jira/browse/SPARK-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14607205#comment-14607205 ] Josh Rosen commented on SPARK-8415: --- Here's an example of a build that spent lots of time contended on the lock: https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/36086/consoleFull > Jenkins compilation spends lots of time re-resolving dependencies and waiting > to acquire Ivy cache lock > --- > > Key: SPARK-8415 > URL: https://issues.apache.org/jira/browse/SPARK-8415 > Project: Spark > Issue Type: Bug > Components: Build, Project Infra >Reporter: Josh Rosen > > When watching a pull request build, I noticed that the compilation + > packaging + test compilation phases spent huge amounts of time waiting to > acquire the Ivy cache lock. We should see whether we can tell SBT to skip > the resolution steps for some of these commands, since this could speed up > the compilation process when Jenkins is heavily loaded. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
