Re: Apache Spark 3.3 Release
@Maxim Thanks for driving the release! > Not sure about SPARK-36057 since the current state. @Igor Costa Thanks for your attention, as dongjoon said, basic framework abilities of custom scheduler have been supported, we are also planning to mark this as beta in 3.3.0. Of course, we will do more tests to make sure it is more stable and also welcome more input to make it better continuously. > I don't think that could be a blocker for Apache Spark 3.2.0. Yep, and v3.3.0, : ) Regards, Yikun
Re: Apache Spark 3.3 Release
I've reviewed most of the actual code in that area. That's pretty much an experimental feature still. I don't think that could be a blocker for Apache Spark 3.2.0. Dongjoon. On Fri, Mar 4, 2022 at 12:25 PM Igor Costa wrote: > Thanks Maxim, > > The code freeze by end of this month would be fine. Not sure about > SPARK-36057 since the current state. > > > > Thanks > > On Fri, 4 Mar 2022 at 19:27, Jungtaek Lim > wrote: > >> Thanks Maxim for volunteering to drive the release! I support the plan >> (March 15th) to perform a release branch cut. >> >> Btw, would we be open for modification of critical/blocker issues after >> the release branch cut? I have a blocker JIRA ticket and the PR is open for >> reviewing, but need some time to gain traction as well as going through >> actual reviews. My guess is yes but to confirm again. >> >> On Fri, Mar 4, 2022 at 4:20 AM Dongjoon Hyun >> wrote: >> >>> Thank you, Max, for volunteering for Apache Spark 3.3 release manager. >>> >>> Ya, I'm also +1 for the original plan. >>> >>> Dongjoon >>> >>> On Thu, Mar 3, 2022 at 10:52 AM Mridul Muralidharan >>> wrote: >>> Agree with Sean, code freeze by mid March sounds good. Regards, Mridul On Thu, Mar 3, 2022 at 12:47 PM Sean Owen wrote: > I think it's fine to pursue the existing plan - code freeze in two > weeks and try to close off key remaining issues. Final release pending on > how those go, and testing, but fine to get the ball rolling. > > On Thu, Mar 3, 2022 at 12:45 PM Maxim Gekk > wrote: > >> Hello All, >> >> I would like to bring on the table the theme about the new Spark >> release 3.3. According to the public schedule at >> https://spark.apache.org/versioning-policy.html, we planned to start >> the code freeze and release branch cut on March 15th, 2022. Since this >> date >> is coming soon, I would like to take your attention on the topic and >> gather >> objections that you might have. >> >> Bellow is the list of ongoing and active SPIPs: >> >> Spark SQL: >> - [SPARK-31357] DataSourceV2: Catalog API for view metadata >> - [SPARK-35801] Row-level operations in Data Source V2 >> - [SPARK-37166] Storage Partitioned Join >> >> Spark Core: >> - [SPARK-20624] Add better handling for node shutdown >> - [SPARK-25299] Use remote storage for persisting shuffle data >> >> PySpark: >> - [SPARK-26413] RDD Arrow Support in Spark Core and PySpark >> >> Kubernetes: >> - [SPARK-36057] Support Customized Kubernetes Schedulers >> >> Probably, we should finish if there are any remaining works for Spark >> 3.3, and switch to QA mode, cut a branch and keep everything on track. I >> would like to volunteer to help drive this process. >> >> Best regards, >> Max Gekk >> > -- > Sent from Gmail Mobile >
Re: Apache Spark 3.3 Release
Thanks Maxim, The code freeze by end of this month would be fine. Not sure about SPARK-36057 since the current state. Thanks On Fri, 4 Mar 2022 at 19:27, Jungtaek Lim wrote: > Thanks Maxim for volunteering to drive the release! I support the plan > (March 15th) to perform a release branch cut. > > Btw, would we be open for modification of critical/blocker issues after > the release branch cut? I have a blocker JIRA ticket and the PR is open for > reviewing, but need some time to gain traction as well as going through > actual reviews. My guess is yes but to confirm again. > > On Fri, Mar 4, 2022 at 4:20 AM Dongjoon Hyun > wrote: > >> Thank you, Max, for volunteering for Apache Spark 3.3 release manager. >> >> Ya, I'm also +1 for the original plan. >> >> Dongjoon >> >> On Thu, Mar 3, 2022 at 10:52 AM Mridul Muralidharan >> wrote: >> >>> >>> Agree with Sean, code freeze by mid March sounds good. >>> >>> Regards, >>> Mridul >>> >>> On Thu, Mar 3, 2022 at 12:47 PM Sean Owen wrote: >>> I think it's fine to pursue the existing plan - code freeze in two weeks and try to close off key remaining issues. Final release pending on how those go, and testing, but fine to get the ball rolling. On Thu, Mar 3, 2022 at 12:45 PM Maxim Gekk wrote: > Hello All, > > I would like to bring on the table the theme about the new Spark > release 3.3. According to the public schedule at > https://spark.apache.org/versioning-policy.html, we planned to start > the code freeze and release branch cut on March 15th, 2022. Since this > date > is coming soon, I would like to take your attention on the topic and > gather > objections that you might have. > > Bellow is the list of ongoing and active SPIPs: > > Spark SQL: > - [SPARK-31357] DataSourceV2: Catalog API for view metadata > - [SPARK-35801] Row-level operations in Data Source V2 > - [SPARK-37166] Storage Partitioned Join > > Spark Core: > - [SPARK-20624] Add better handling for node shutdown > - [SPARK-25299] Use remote storage for persisting shuffle data > > PySpark: > - [SPARK-26413] RDD Arrow Support in Spark Core and PySpark > > Kubernetes: > - [SPARK-36057] Support Customized Kubernetes Schedulers > > Probably, we should finish if there are any remaining works for Spark > 3.3, and switch to QA mode, cut a branch and keep everything on track. I > would like to volunteer to help drive this process. > > Best regards, > Max Gekk > -- Sent from Gmail Mobile
