[jira] [Commented] (SPARK-39522) Uses Docker image cache over a custom image
[ https://issues.apache.org/jira/browse/SPARK-39522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558067#comment-17558067 ] Yikun Jiang commented on SPARK-39522: - I have some questions here, the cache seems to just reduce the build time (docker build), and the current infra uses static sha images (docker run/pull), so in theory there won't be any improvement. So, what you mean is that we want to migrate dongjoon-hyun/ApacheSparkGitHubActionImage to apache/spark, and then in each pyspark workflow, do a full docker build in every pr triggered job, and speed it up by caching? > Uses Docker image cache over a custom image > --- > > Key: SPARK-39522 > URL: https://issues.apache.org/jira/browse/SPARK-39522 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > We should probably replace the base image > (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, > https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain > ubunto image w/ Docker image cache. See also > https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39522) Uses Docker image cache over a custom image
[ https://issues.apache.org/jira/browse/SPARK-39522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558260#comment-17558260 ] Hyukjin Kwon commented on SPARK-39522: -- Hey, I appreciate it for taking a look!. Yes! > Uses Docker image cache over a custom image > --- > > Key: SPARK-39522 > URL: https://issues.apache.org/jira/browse/SPARK-39522 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > We should probably replace the base image > (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, > https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain > ubunto image w/ Docker image cache. See also > https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39522) Uses Docker image cache over a custom image
[ https://issues.apache.org/jira/browse/SPARK-39522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558570#comment-17558570 ] Yikun Jiang commented on SPARK-39522: - Some investigation: - The build job are running in each user's downstream repo, so we have to use "Registry cache" as a bridge. - The complete flow would be: # (apache repo) Build the image cache in apache repo, this image will be refreshed if dockerfile changes merged. # (user repo) Build the latest infra image in each pr based on image cache and PR changes Dockerfile, and upload it to user gchr.io. # (user repo) Use the latest infra image of Step2 to running pyspark, sparkr, lint. > Uses Docker image cache over a custom image > --- > > Key: SPARK-39522 > URL: https://issues.apache.org/jira/browse/SPARK-39522 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > We should probably replace the base image > (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, > https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain > ubunto image w/ Docker image cache. See also > https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39522) Uses Docker image cache over a custom image
[ https://issues.apache.org/jira/browse/SPARK-39522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558585#comment-17558585 ] Apache Spark commented on SPARK-39522: -- User 'Yikun' has created a pull request for this issue: https://github.com/apache/spark/pull/36980 > Uses Docker image cache over a custom image > --- > > Key: SPARK-39522 > URL: https://issues.apache.org/jira/browse/SPARK-39522 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > We should probably replace the base image > (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, > https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain > ubunto image w/ Docker image cache. See also > https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39522) Uses Docker image cache over a custom image
[ https://issues.apache.org/jira/browse/SPARK-39522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17559156#comment-17559156 ] Apache Spark commented on SPARK-39522: -- User 'Yikun' has created a pull request for this issue: https://github.com/apache/spark/pull/37005 > Uses Docker image cache over a custom image > --- > > Key: SPARK-39522 > URL: https://issues.apache.org/jira/browse/SPARK-39522 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > We should probably replace the base image > (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, > https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain > ubunto image w/ Docker image cache. See also > https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39522) Uses Docker image cache over a custom image
[ https://issues.apache.org/jira/browse/SPARK-39522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17559157#comment-17559157 ] Apache Spark commented on SPARK-39522: -- User 'Yikun' has created a pull request for this issue: https://github.com/apache/spark/pull/37003 > Uses Docker image cache over a custom image > --- > > Key: SPARK-39522 > URL: https://issues.apache.org/jira/browse/SPARK-39522 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > We should probably replace the base image > (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, > https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain > ubunto image w/ Docker image cache. See also > https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39522) Uses Docker image cache over a custom image
[ https://issues.apache.org/jira/browse/SPARK-39522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17559196#comment-17559196 ] Apache Spark commented on SPARK-39522: -- User 'Yikun' has created a pull request for this issue: https://github.com/apache/spark/pull/37006 > Uses Docker image cache over a custom image > --- > > Key: SPARK-39522 > URL: https://issues.apache.org/jira/browse/SPARK-39522 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > We should probably replace the base image > (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, > https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain > ubunto image w/ Docker image cache. See also > https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
