Re: [VOTE] Spark 3.1.3 RC3

2022-02-02 Thread Mridul Muralidharan
Hi, Minor nit: the tag mentioned under [1] looks like a typo - I used "v3.1.3-rc3" for my vote (3.2.1 is mentioned in a couple of places, treat them as 3.1.3 instead) +1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul

Re: MetadataFetchFailedException due to decommission block migrations

2022-02-02 Thread Dongjoon Hyun
Thank you for sharing, Emil. > I willing to help up to develop a fix, but might need some guidance of > how this case could be handled better. Could you file an official Apache JIRA for your finding and propose a PR for that too with the test case? We can continue our discussion on your PR.

Re: [VOTE] Spark 3.1.3 RC3

2022-02-02 Thread Mridul Muralidharan
Thanks Tom ! I missed [1] (or probably forgot) the 3.1 part of the discussion given it centered around 3.2 ... Regards, Mridul [1] https://www.mail-archive.com/dev@spark.apache.org/msg28484.html On Wed, Feb 2, 2022 at 8:55 AM Thomas Graves wrote: > It was discussed doing all the maintenance

Re: [VOTE] Spark 3.1.3 RC3

2022-02-02 Thread Thomas Graves
It was discussed doing all the maintenance lines back at beginning of December (Dec 6) when we were talking about release 3.2.1. Tom On Wed, Feb 2, 2022 at 2:07 AM Mridul Muralidharan wrote: > > Hi Holden, > > Not that I am against releasing 3.1.3 (given the fixes that have already > gone

Re: [VOTE] Spark 3.1.3 RC3

2022-02-02 Thread Sean Owen
+1 from me, same result as the last release on my end. I think releasing 3.1.3 is fine, it's 7 months since 3.1.2. On Tue, Feb 1, 2022 at 7:12 PM Holden Karau wrote: > Please vote on releasing the following candidate as Apache Spark version > 3.1.3. > > The vote is open until Feb. 4th at 5 PM

MetadataFetchFailedException due to decommission block migrations

2022-02-02 Thread Emil Ejbyfeldt
As noted in SPARK-34939 there is race when using broadcast for map output status. Explanation from SPARK-34939 > After map statuses are broadcasted and the executors obtain serialized broadcasted map statuses. If any fetch failure happens after, Spark scheduler invalidates cached map statuses

Re: [VOTE] Spark 3.1.3 RC3

2022-02-02 Thread Mridul Muralidharan
Hi Holden, Not that I am against releasing 3.1.3 (given the fixes that have already gone in), but did we discuss releasing it ? I might have missed the thread ... Regards, Mridul On Tue, Feb 1, 2022 at 7:12 PM Holden Karau wrote: > Please vote on releasing the following candidate as Apache