- Integration with additional external data sources or systems, say Hive - Enhancements to the Spark UI for improved monitoring and debugging - Enhancements to machine learning (MLlib) algorithms and capabilities, like TensorFlow or PyTorch,( if any in the pipeline)
HTH Mich Talebzadeh, Technologist | Architect | Data Engineer | Generative AI | FinCrime London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Thu, 2 May 2024 at 17:02, Steve Loughran <ste...@cloudera.com.invalid> wrote: > There's a new parquet RC up this week which would be good to pull in. > > On Thu, 2 May 2024 at 03:20, Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> +1 love to see it! >> >> On Thu, May 2, 2024 at 10:08 AM Holden Karau <holden.ka...@gmail.com> >> wrote: >> >>> +1 :) yay previews >>> >>> On Wed, May 1, 2024 at 5:36 PM Chao Sun <sunc...@apache.org> wrote: >>> >>>> +1 >>>> >>>> On Wed, May 1, 2024 at 5:23 PM Xiao Li <gatorsm...@gmail.com> wrote: >>>> >>>>> +1 for next Monday. >>>>> >>>>> We can do more previews when the other features are ready for preview. >>>>> >>>>> Tathagata Das <tathagata.das1...@gmail.com> 于2024年5月1日周三 08:46写道： >>>>> >>>>>> Next week sounds great! Thank you Wenchen! >>>>>> >>>>>> On Wed, May 1, 2024 at 11:16 AM Wenchen Fan <cloud0...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Yea I think a preview release won't hurt (without a branch cut). We >>>>>>> don't need to wait for all the ongoing projects to be ready. How about >>>>>>> we >>>>>>> do a 4.0 preview release based on the current master branch next Monday? >>>>>>> >>>>>>> On Wed, May 1, 2024 at 11:06 PM Tathagata Das < >>>>>>> tathagata.das1...@gmail.com> wrote: >>>>>>> >>>>>>>> Hey all, >>>>>>>> >>>>>>>> Reviving this thread, but Spark master has already accumulated a >>>>>>>> huge amount of changes. As a downstream project maintainer, I want to >>>>>>>> really start testing the new features and other breaking changes, and >>>>>>>> it's >>>>>>>> hard to do that without a Preview release. So the sooner we make a >>>>>>>> Preview >>>>>>>> release, the faster we can start getting feedback for fixing things >>>>>>>> for a >>>>>>>> great Spark 4.0 final release. >>>>>>>> >>>>>>>> So I urge the community to produce a Spark 4.0 Preview soon even if >>>>>>>> certain features targeting the Delta 4.0 release are still incomplete. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Apr 17, 2024 at 8:35 AM Wenchen Fan <cloud0...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thank you all for the replies! >>>>>>>>> >>>>>>>>> To @Nicholas Chammas <nicholas.cham...@gmail.com> : Thanks for >>>>>>>>> cleaning up the error terminology and documentation! I've merged the >>>>>>>>> first >>>>>>>>> PR and let's finish others before the 4.0 release. >>>>>>>>> To @Dongjoon Hyun <dongjoon.h...@gmail.com> : Thanks for driving >>>>>>>>> the ANSI on by default effort! Now the vote has passed, let's flip the >>>>>>>>> config and finish the DataFrame error context feature before 4.0. >>>>>>>>> To @Jungtaek Lim <kabhwan.opensou...@gmail.com> : Ack. We can >>>>>>>>> treat the Streaming state store data source as completed for 4.0 then. >>>>>>>>> To @Cheng Pan <cheng...@apache.org> : Yea we definitely should >>>>>>>>> have a preview release. Let's collect more feedback on the ongoing >>>>>>>>> projects >>>>>>>>> and then we can propose a date for the preview release. >>>>>>>>> >>>>>>>>> On Wed, Apr 17, 2024 at 1:22 PM Cheng Pan <pan3...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> will we have preview release for 4.0.0 like we did for 2.0.0 and >>>>>>>>>> 3.0.0? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Cheng Pan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > On Apr 15, 2024, at 09:58, Jungtaek Lim < >>>>>>>>>> kabhwan.opensou...@gmail.com> wrote: >>>>>>>>>> > >>>>>>>>>> > W.r.t. state data source - reader (SPARK-45511), there are >>>>>>>>>> several follow-up tickets, but we don't plan to address them soon. >>>>>>>>>> The >>>>>>>>>> current implementation is the final shape for Spark 4.0.0, unless >>>>>>>>>> there are >>>>>>>>>> demands on the follow-up tickets. >>>>>>>>>> > >>>>>>>>>> > We may want to check the plan for transformWithState - my >>>>>>>>>> understanding is that we want to release the feature to 4.0.0, but >>>>>>>>>> there >>>>>>>>>> are several remaining works to be done. While the tentative timeline >>>>>>>>>> for >>>>>>>>>> releasing is June 2024, what would be the tentative timeline for the >>>>>>>>>> RC cut? >>>>>>>>>> > (cc. Anish to add more context on the plan for >>>>>>>>>> transformWithState) >>>>>>>>>> > >>>>>>>>>> > On Sat, Apr 13, 2024 at 3:15 AM Wenchen Fan < >>>>>>>>>> cloud0...@gmail.com> wrote: >>>>>>>>>> > Hi all, >>>>>>>>>> > >>>>>>>>>> > It's close to the previously proposed 4.0.0 release date (June >>>>>>>>>> 2024), and I think it's time to prepare for it and discuss the >>>>>>>>>> ongoing >>>>>>>>>> projects: >>>>>>>>>> > • >>>>>>>>>> > ANSI by default >>>>>>>>>> > • Spark Connect GA >>>>>>>>>> > • Structured Logging >>>>>>>>>> > • Streaming state store data source >>>>>>>>>> > • new data type VARIANT >>>>>>>>>> > • STRING collation support >>>>>>>>>> > • Spark k8s operator versioning >>>>>>>>>> > Please help to add more items to this list that are missed >>>>>>>>>> here. I would like to volunteer as the release manager for Apache >>>>>>>>>> Spark >>>>>>>>>> 4.0.0 if there is no objection. Thank you all for the great work >>>>>>>>>> that fills >>>>>>>>>> Spark 4.0! >>>>>>>>>> > >>>>>>>>>> > Wenchen Fan >>>>>>>>>> >>>>>>>>>> >>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >>
