Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-07-14 Thread Yikun Jiang
With the help from the community, the cache based job switch has been completed! * About the ghcr images: You might notice that two images are generated in apache ghcr: - Image cache: spark/apache-spark-github-action-image-cache

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-26 Thread Yikun Jiang
> There’s one last task to simply caching the Docker image ( https://issues.apache.org/jira/browse/SPARK-39522). I will have to be less active for this week and next week because of the Spark Summit. Would appreciate if somebody finds some time to take a stab. I did some investigations on spark

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-23 Thread Dongjoon Hyun
Yep, I'll be there too. Thank you for the adjustment. See you soon. :) Dongjoon. On Thu, Jun 23, 2022 at 4:59 PM Hyukjin Kwon wrote: > Alright, I'll be there after Holden's talk Thursday > https://databricks.com/dataaisummit/session/tools-assisted-apache-spark-version-migrations-21-32 > w/

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-23 Thread Hyukjin Kwon
Alright, I'll be there after Holden's talk Thursday https://databricks.com/dataaisummit/session/tools-assisted-apache-spark-version-migrations-21-32 w/ Dongjoon (since he manages OSS Jenkins too). Let's have a quickie chat :-). On Thu, 23 Jun 2022 at 06:16, Hyukjin Kwon wrote: > Oops, I was

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-22 Thread Hyukjin Kwon
Oops, I was confused about the time and distance in the US. I won't make it too. Let me find another time slot that works for more ppl. On Thu, 23 Jun 2022 at 00:19, Dongjoon Hyun wrote: > Thank you, Hyukjin! :) > > BTW, unfortunately, it seems that I cannot join that quick meeting. > I have

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-22 Thread Dongjoon Hyun
Thank you, Hyukjin! :) BTW, unfortunately, it seems that I cannot join that quick meeting. I have another schedule at South Bay around 7PM and need to leave San Francisco at least 5PM. Dongjoon. On Wed, Jun 22, 2022 at 3:39 AM Hyukjin Kwon wrote: > (cc @Yikun Jiang @Gengliang Wang > @Maxim

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-22 Thread Hyukjin Kwon
(cc @Yikun Jiang @Gengliang Wang @Maxim Gekk @Yang,Jie(INF) FYI) On Wed, 22 Jun 2022 at 19:34, Hyukjin Kwon wrote: > Couple of updates: > >- > >All builds passed now with all combinations we defined in the GitHub >Actions (e.g., branch-3.2, branch-3.3, JDK 11, >JDK 17 and

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-22 Thread Hyukjin Kwon
Couple of updates: - All builds passed now with all combinations we defined in the GitHub Actions (e.g., branch-3.2, branch-3.3, JDK 11, JDK 17 and Scala 2.13), see https://github.com/apache/spark/actions cc @Tom Graves @Dongjoon Hyun FYI - except one test that is

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-20 Thread Hyukjin Kwon
Just chatted offline - both I and Holden have multiple sessions :-). Probably let's meet up for a quick chat after your talk https://databricks.com/dataaisummit/session/what-do-when-your-job-goes-oom-night-flowcharts ? On Mon, 20 Jun 2022 at 22:23, Holden Karau wrote: > How about a hallway

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-20 Thread Holden Karau
How about a hallway meet up at Data AI summit to talk about build CI if folks are Interested? On Sun, Jun 19, 2022 at 7:50 PM Hyukjin Kwon wrote: > Increased the priority to a blocker - I don't think we can release with > these build failures and poor CI > > On Mon, 20 Jun 2022 at 10:39,

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-19 Thread Hyukjin Kwon
Increased the priority to a blocker - I don't think we can release with these build failures and poor CI On Mon, 20 Jun 2022 at 10:39, Hyukjin Kwon wrote: > There are too many test failures here. I pinged in some PRs I could > identify from a cursory look but would be great for you guys to take

Re: [SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-19 Thread Hyukjin Kwon
There are too many test failures here. I pinged in some PRs I could identify from a cursory look but would be great for you guys to take a look if you guys haven't tested your change against other environments like JDK 11, Scala 2.13. On Mon, 20 Jun 2022 at 10:04, Hyukjin Kwon wrote: > Hi all,

[SPARK-39515] Improve scheduled jobs in GitHub Actions

2022-06-19 Thread Hyukjin Kwon
Hi all, I am trying to rework GitHub Actions CI at https://issues.apache.org/jira/browse/SPARK-39515. Any help would be very appreciated.