There's a new parquet RC up this week which would be good to pull in. On Thu, 2 May 2024 at 03:20, Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote:
> +1 love to see it! > > On Thu, May 2, 2024 at 10:08 AM Holden Karau <holden.ka...@gmail.com> > wrote: > >> +1 :) yay previews >> >> On Wed, May 1, 2024 at 5:36 PM Chao Sun <sunc...@apache.org> wrote: >> >>> +1 >>> >>> On Wed, May 1, 2024 at 5:23 PM Xiao Li <gatorsm...@gmail.com> wrote: >>> >>>> +1 for next Monday. >>>> >>>> We can do more previews when the other features are ready for preview. >>>> >>>> Tathagata Das <tathagata.das1...@gmail.com> 于2024年5月1日周三 08:46写道： >>>> >>>>> Next week sounds great! Thank you Wenchen! >>>>> >>>>> On Wed, May 1, 2024 at 11:16 AM Wenchen Fan <cloud0...@gmail.com> >>>>> wrote: >>>>> >>>>>> Yea I think a preview release won't hurt (without a branch cut). We >>>>>> don't need to wait for all the ongoing projects to be ready. How about we >>>>>> do a 4.0 preview release based on the current master branch next Monday? >>>>>> >>>>>> On Wed, May 1, 2024 at 11:06 PM Tathagata Das < >>>>>> tathagata.das1...@gmail.com> wrote: >>>>>> >>>>>>> Hey all, >>>>>>> >>>>>>> Reviving this thread, but Spark master has already accumulated a >>>>>>> huge amount of changes. As a downstream project maintainer, I want to >>>>>>> really start testing the new features and other breaking changes, and >>>>>>> it's >>>>>>> hard to do that without a Preview release. So the sooner we make a >>>>>>> Preview >>>>>>> release, the faster we can start getting feedback for fixing things for >>>>>>> a >>>>>>> great Spark 4.0 final release. >>>>>>> >>>>>>> So I urge the community to produce a Spark 4.0 Preview soon even if >>>>>>> certain features targeting the Delta 4.0 release are still incomplete. >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>>> On Wed, Apr 17, 2024 at 8:35 AM Wenchen Fan <cloud0...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thank you all for the replies! >>>>>>>> >>>>>>>> To @Nicholas Chammas <nicholas.cham...@gmail.com> : Thanks for >>>>>>>> cleaning up the error terminology and documentation! I've merged the >>>>>>>> first >>>>>>>> PR and let's finish others before the 4.0 release. >>>>>>>> To @Dongjoon Hyun <dongjoon.h...@gmail.com> : Thanks for driving >>>>>>>> the ANSI on by default effort! Now the vote has passed, let's flip the >>>>>>>> config and finish the DataFrame error context feature before 4.0. >>>>>>>> To @Jungtaek Lim <kabhwan.opensou...@gmail.com> : Ack. We can >>>>>>>> treat the Streaming state store data source as completed for 4.0 then. >>>>>>>> To @Cheng Pan <cheng...@apache.org> : Yea we definitely should >>>>>>>> have a preview release. Let's collect more feedback on the ongoing >>>>>>>> projects >>>>>>>> and then we can propose a date for the preview release. >>>>>>>> >>>>>>>> On Wed, Apr 17, 2024 at 1:22 PM Cheng Pan <pan3...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> will we have preview release for 4.0.0 like we did for 2.0.0 and >>>>>>>>> 3.0.0? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Cheng Pan >>>>>>>>> >>>>>>>>> >>>>>>>>> > On Apr 15, 2024, at 09:58, Jungtaek Lim < >>>>>>>>> kabhwan.opensou...@gmail.com> wrote: >>>>>>>>> > >>>>>>>>> > W.r.t. state data source - reader (SPARK-45511), there are >>>>>>>>> several follow-up tickets, but we don't plan to address them soon. The >>>>>>>>> current implementation is the final shape for Spark 4.0.0, unless >>>>>>>>> there are >>>>>>>>> demands on the follow-up tickets. >>>>>>>>> > >>>>>>>>> > We may want to check the plan for transformWithState - my >>>>>>>>> understanding is that we want to release the feature to 4.0.0, but >>>>>>>>> there >>>>>>>>> are several remaining works to be done. While the tentative timeline >>>>>>>>> for >>>>>>>>> releasing is June 2024, what would be the tentative timeline for the >>>>>>>>> RC cut? >>>>>>>>> > (cc. Anish to add more context on the plan for >>>>>>>>> transformWithState) >>>>>>>>> > >>>>>>>>> > On Sat, Apr 13, 2024 at 3:15 AM Wenchen Fan <cloud0...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> > Hi all, >>>>>>>>> > >>>>>>>>> > It's close to the previously proposed 4.0.0 release date (June >>>>>>>>> 2024), and I think it's time to prepare for it and discuss the ongoing >>>>>>>>> projects: >>>>>>>>> > • >>>>>>>>> > ANSI by default >>>>>>>>> > • Spark Connect GA >>>>>>>>> > • Structured Logging >>>>>>>>> > • Streaming state store data source >>>>>>>>> > • new data type VARIANT >>>>>>>>> > • STRING collation support >>>>>>>>> > • Spark k8s operator versioning >>>>>>>>> > Please help to add more items to this list that are missed here. >>>>>>>>> I would like to volunteer as the release manager for Apache Spark >>>>>>>>> 4.0.0 if >>>>>>>>> there is no objection. Thank you all for the great work that fills >>>>>>>>> Spark >>>>>>>>> 4.0! >>>>>>>>> > >>>>>>>>> > Wenchen Fan >>>>>>>>> >>>>>>>>> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> >
