> There’s one last task to simply caching the Docker image ( https://issues.apache.org/jira/browse/SPARK-39522). I will have to be less active for this week and next week because of the Spark Summit. Would appreciate if somebody finds some time to take a stab.
I did some investigations on spark container jobs (pyspark/sparkr/lint) using cache, and draft a doc to help you guys understand #36980 <https://github.com/apache/spark/pull/36980>: https://docs.google.com/document/d/1_uiId-U1DODYyYZejAZeyz2OAjxcnA-xfwjynDF6vd0 > About a quick hallway meetup, I will be there after Holden’s talk at least to say hello to her :-). Something topic I was interesting about and related to build CI: - K8S integrations <https://github.com/apache/spark/pull/35830> test on GA: - To help various OS <https://github.com/apache/spark/pull/35142> and multi architecture/hardware (x86/arm64, gpu) integration support, what we can do to help improving. Please feel free to ping me if necessary. It's a little bit pity I couldn't have the opportunity to be there, I hope you guys have a fabulous meet on summit! Regards, Yikun On Fri, Jun 24, 2022 at 11:15 AM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > Yep, I'll be there too. Thank you for the adjustment. See you soon. :) > > Dongjoon. > > On Thu, Jun 23, 2022 at 4:59 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: > >> Alright, I'll be there after Holden's talk Thursday >> https://databricks.com/dataaisummit/session/tools-assisted-apache-spark-version-migrations-21-32 >> w/ Dongjoon (since he manages OSS Jenkins too). >> Let's have a quickie chat :-). >> >> On Thu, 23 Jun 2022 at 06:16, Hyukjin Kwon <gurwls...@gmail.com> wrote: >> >>> Oops, I was confused about the time and distance in the US. I won't make >>> it too. >>> Let me find another time slot that works for more ppl. >>> >>> On Thu, 23 Jun 2022 at 00:19, Dongjoon Hyun <dongjoon.h...@gmail.com> >>> wrote: >>> >>>> Thank you, Hyukjin! :) >>>> >>>> BTW, unfortunately, it seems that I cannot join that quick meeting. >>>> I have another schedule at South Bay around 7PM and need to leave San >>>> Francisco at least 5PM. >>>> >>>> Dongjoon. >>>> >>>> >>>> On Wed, Jun 22, 2022 at 3:39 AM Hyukjin Kwon <gurwls...@gmail.com> >>>> wrote: >>>> >>>>> (cc @Yikun Jiang <yikunk...@gmail.com> @Gengliang Wang >>>>> <gengliang.w...@databricks.com> @Maxim Gekk >>>>> <maxim.g...@databricks.com> @Yang,Jie(INF) <yangji...@baidu.com> FYI) >>>>> >>>>> On Wed, 22 Jun 2022 at 19:34, Hyukjin Kwon <gurwls...@gmail.com> >>>>> wrote: >>>>> >>>>>> Couple of updates: >>>>>> >>>>>> - >>>>>> >>>>>> All builds passed now with all combinations we defined in the >>>>>> GitHub Actions (e.g., branch-3.2, branch-3.3, JDK 11, >>>>>> JDK 17 and Scala 2.13), see >>>>>> https://github.com/apache/spark/actions cc @Tom Graves >>>>>> <tgraves...@yahoo.com> @Dongjoon Hyun <dongjoon.h...@gmail.com> >>>>>> FYI >>>>>> - >>>>>> >>>>>> except one test that is being failed due to OOM. That’s being >>>>>> fixed at https://github.com/apache/spark/pull/36954, see >>>>>> also >>>>>> https://github.com/apache/spark/pull/36787#discussion_r901190636 >>>>>> - >>>>>> >>>>>> I am now adding PySpark, SparkR jobs to the scheduled builds at >>>>>> https://github.com/apache/spark/pull/36940 >>>>>> and see if they pass. We might need a couple of more fixes there. >>>>>> - >>>>>> >>>>>> There’s one last task to simply caching the Docker image ( >>>>>> https://issues.apache.org/jira/browse/SPARK-39522). >>>>>> I will have to be less active for this week and next week because >>>>>> of the Spark Summit. Would appreciate if somebody >>>>>> finds some time to take a stab. >>>>>> >>>>>> About a quick hallway meetup, I will be there after Holden’s talk at >>>>>> least to say hello to her :-). >>>>>> Let’s have a quick chat about our CI. We still have some general >>>>>> problems to cope with like the lack of resources in >>>>>> GitHub Actions. >>>>>> >>>>>> >>>>>> >>>>>> On Tue, 21 Jun 2022 at 11:49, Hyukjin Kwon <gurwls...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Just chatted offline - both I and Holden have multiple sessions :-). >>>>>>> Probably let's meet up for a quick chat after your talk >>>>>>> https://databricks.com/dataaisummit/session/what-do-when-your-job-goes-oom-night-flowcharts >>>>>>> ? >>>>>>> >>>>>>> >>>>>>> On Mon, 20 Jun 2022 at 22:23, Holden Karau <hol...@pigscanfly.ca> >>>>>>> wrote: >>>>>>> >>>>>>>> How about a hallway meet up at Data AI summit to talk about build >>>>>>>> CI if folks are >>>>>>>> Interested? >>>>>>>> >>>>>>>> On Sun, Jun 19, 2022 at 7:50 PM Hyukjin Kwon <gurwls...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Increased the priority to a blocker - I don't think we can release >>>>>>>>> with these build failures and poor CI >>>>>>>>> >>>>>>>>> On Mon, 20 Jun 2022 at 10:39, Hyukjin Kwon <gurwls...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> There are too many test failures here. I pinged in some PRs I >>>>>>>>>> could identify from a cursory look but would be great for you guys >>>>>>>>>> to take >>>>>>>>>> a look if you guys haven't tested your change against other >>>>>>>>>> environments like JDK 11, Scala 2.13. >>>>>>>>>> >>>>>>>>>> On Mon, 20 Jun 2022 at 10:04, Hyukjin Kwon <gurwls...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I am trying to rework GitHub Actions CI at >>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-39515. Any help >>>>>>>>>>> would be very appreciated. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>> >>>>>>>
