Couple of updates: -
All builds passed now with all combinations we defined in the GitHub Actions (e.g., branch-3.2, branch-3.3, JDK 11, JDK 17 and Scala 2.13), see https://github.com/apache/spark/actions cc @Tom Graves <tgraves...@yahoo.com> @Dongjoon Hyun <dongjoon.h...@gmail.com> FYI - except one test that is being failed due to OOM. That’s being fixed at https://github.com/apache/spark/pull/36954, see also https://github.com/apache/spark/pull/36787#discussion_r901190636 - I am now adding PySpark, SparkR jobs to the scheduled builds at https://github.com/apache/spark/pull/36940 and see if they pass. We might need a couple of more fixes there. - There’s one last task to simply caching the Docker image ( https://issues.apache.org/jira/browse/SPARK-39522). I will have to be less active for this week and next week because of the Spark Summit. Would appreciate if somebody finds some time to take a stab. About a quick hallway meetup, I will be there after Holden’s talk at least to say hello to her :-). Let’s have a quick chat about our CI. We still have some general problems to cope with like the lack of resources in GitHub Actions. On Tue, 21 Jun 2022 at 11:49, Hyukjin Kwon <gurwls...@gmail.com> wrote: > Just chatted offline - both I and Holden have multiple sessions :-). > Probably let's meet up for a quick chat after your talk > https://databricks.com/dataaisummit/session/what-do-when-your-job-goes-oom-night-flowcharts > ? > > > On Mon, 20 Jun 2022 at 22:23, Holden Karau <hol...@pigscanfly.ca> wrote: > >> How about a hallway meet up at Data AI summit to talk about build CI if >> folks are >> Interested? >> >> On Sun, Jun 19, 2022 at 7:50 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: >> >>> Increased the priority to a blocker - I don't think we can release with >>> these build failures and poor CI >>> >>> On Mon, 20 Jun 2022 at 10:39, Hyukjin Kwon <gurwls...@gmail.com> wrote: >>> >>>> There are too many test failures here. I pinged in some PRs I could >>>> identify from a cursory look but would be great for you guys to take a look >>>> if you guys haven't tested your change against other environments like JDK >>>> 11, Scala 2.13. >>>> >>>> On Mon, 20 Jun 2022 at 10:04, Hyukjin Kwon <gurwls...@gmail.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I am trying to rework GitHub Actions CI at >>>>> https://issues.apache.org/jira/browse/SPARK-39515. Any help would be >>>>> very appreciated. >>>>> >>>>> >>>>> -- >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> >
