[jira] [Resolved] (SPARK-30835) Add support for YARN decommissioning & pre-emption

[ https://issues.apache.org/jira/browse/SPARK-30835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-30835. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-30835) Add support for YARN decommissioning & pre-emption

[ https://issues.apache.org/jira/browse/SPARK-30835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-30835: --- Assignee: Abhishek Dixit > Add support for YARN decommissioning &a

Re: [VOTE] Release Spark 3.3.0 (RC6)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes The test "SPARK-33084: Add jar support Ivy URI in SQL" in sql.SQLQuerySuite fails; but other than that, rest looks good. Regards, Mridul On Mon, Jun 13, 2022 at 4:25 PM Tom Graves

[jira] [Commented] (SPARK-38987) Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true

[ https://issues.apache.org/jira/browse/SPARK-38987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17550681#comment-17550681 ] Mridul Muralidharan commented on SPARK-38987: - Manually marking as resolved - some env issue

[jira] [Resolved] (SPARK-38987) Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true

[ https://issues.apache.org/jira/browse/SPARK-38987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38987. - Fix Version/s: 3.4.0 Assignee: Aravind Patnam Resolution: Fixed

Re: Spark Push-Based Shuffle causing multiple stage failures

+CC zhouye...@gmail.com On Mon, May 23, 2022 at 7:11 AM Han Altae-Tran wrote: > Hi, > > First of all, I am very thankful for all of the amazing work that goes > into this project! It has opened up so many doors for me! I am a long > time Spark user, and was very excited to start working with

Re: [VOTE] Release Spark 3.3.0 (RC1)

I will also try to get a PR out to fix the first test failure that Sean reported. I will have a PR ready by EOD. Regards, Mridul On Fri, May 6, 2022 at 10:31 AM Gengliang Wang wrote: > Hi Maxim, > > Thanks for the work! > There is a bug fix from Bruce merged on branch-3.3 right after the RC1

[jira] [Assigned] (SPARK-38973) When push-based shuffle is enabled, a stage may not complete when retried

[ https://issues.apache.org/jira/browse/SPARK-38973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38973: --- Assignee: Chandni Singh > When push-based shuffle is enabled, a stage

[jira] [Resolved] (SPARK-38973) When push-based shuffle is enabled, a stage may not complete when retried

[ https://issues.apache.org/jira/browse/SPARK-38973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38973. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-38856) Fix a rejectedExecutionException error when push-based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-38856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38856. - Target Version/s: 3.4.0 Assignee: weixiuli Resolution

[jira] [Resolved] (SPARK-38555) Avoid contention and get or create clientPools quickly in the TransportClientFactory

[ https://issues.apache.org/jira/browse/SPARK-38555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38555. - Assignee: weixiuli Resolution: Fixed > Avoid contention and

[jira] [Updated] (SPARK-38555) Avoid contention and get or create clientPools quickly in the TransportClientFactory

[ https://issues.apache.org/jira/browse/SPARK-38555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-38555: Fix Version/s: 3.4.0 > Avoid contention and get or create clientPools quic

[jira] [Commented] (SPARK-38388) Repartition + Stage retries could lead to incorrect data

[ https://issues.apache.org/jira/browse/SPARK-38388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17507257#comment-17507257 ] Mridul Muralidharan commented on SPARK-38388: - Agree with [~jiangxb1987], either

[jira] [Resolved] (SPARK-38428) Check the FetchShuffleBlocks message only once to improve iteration in external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-38428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38428. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-38309) SHS has incorrect percentiles for shuffle read bytes and shuffle total blocks metrics

[ https://issues.apache.org/jira/browse/SPARK-38309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38309. - Fix Version/s: 3.3.0 3.2.2 3.1.3

[jira] [Assigned] (SPARK-38309) SHS has incorrect percentiles for shuffle read bytes and shuffle total blocks metrics

[ https://issues.apache.org/jira/browse/SPARK-38309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38309: --- Assignee: Rob Reeves > SHS has incorrect percentiles for shuffle r

Re: Apache Spark 3.3 Release

Agree with Sean, code freeze by mid March sounds good. Regards, Mridul On Thu, Mar 3, 2022 at 12:47 PM Sean Owen wrote: > I think it's fine to pursue the existing plan - code freeze in two weeks > and try to close off key remaining issues. Final release pending on how > those go, and testing,

[jira] [Resolved] (SPARK-38344) Avoid to submit task when there are no requests to push up in push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-38344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38344. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-38344) Avoid to submit task when there are no requests to push up in push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-38344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38344: --- Assignee: weixiuli > Avoid to submit task when there are no reque

[jira] [Resolved] (SPARK-37688) ExecutorMonitor should ignore SparkListenerBlockUpdated event if executor was not active

[ https://issues.apache.org/jira/browse/SPARK-37688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37688. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37688) ExecutorMonitor should ignore SparkListenerBlockUpdated event if executor was not active

[ https://issues.apache.org/jira/browse/SPARK-37688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37688: --- Assignee: hujiahua > ExecutorMonitor should ign

Re: [VOTE] Spark 3.1.3 RC4

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Feb 16, 2022 at 8:32 AM Thomas graves wrote: > +1 > > Tom > > On Mon, Feb 14, 2022 at 2:55 PM Holden Karau wrote: > > > > Please vote on releasing the

[jira] [Resolved] (SPARK-34826) Adaptive fetch of shuffle mergers for Push based shuffle

[ https://issues.apache.org/jira/browse/SPARK-34826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34826. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-34826) Adaptive fetch of shuffle mergers for Push based shuffle

[ https://issues.apache.org/jira/browse/SPARK-34826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34826: --- Assignee: Venkata krishnan Sowrirajan > Adaptive fetch of shuffle merg

Re: [VOTE] Spark 3.1.3 RC3

ds, Mridul [1] "The tag to be voted on is v3.2.1-rc1" - the commit hash and git url are correct. On Wed, Feb 2, 2022 at 9:30 AM Mridul Muralidharan wrote: > > Thanks Tom ! > I missed [1] (or probably forgot) the 3.1 part of the discussion given it > centered around 3.2 ..

Re: [VOTE] Spark 3.1.3 RC3

nce lines back at beginning of > December (Dec 6) when we were talking about release 3.2.1. > > Tom > > On Wed, Feb 2, 2022 at 2:07 AM Mridul Muralidharan > wrote: > > > > Hi Holden, > > > > Not that I am against releasing 3.1.3 (given the fixes tha

Re: [VOTE] Spark 3.1.3 RC3

Hi Holden, Not that I am against releasing 3.1.3 (given the fixes that have already gone in), but did we discuss releasing it ? I might have missed the thread ... Regards, Mridul On Tue, Feb 1, 2022 at 7:12 PM Holden Karau wrote: > Please vote on releasing the following candidate as Apache

[jira] [Assigned] (SPARK-37793) Invalid LocalMergedBlockData cause task hang

[ https://issues.apache.org/jira/browse/SPARK-37793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37793: --- Assignee: Chandni Singh > Invalid LocalMergedBlockData cause task h

[jira] [Resolved] (SPARK-37793) Invalid LocalMergedBlockData cause task hang

[ https://issues.apache.org/jira/browse/SPARK-37793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37793. - Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed

[jira] [Resolved] (SPARK-37675) Push-based merge finalization bugs in the RemoteBlockPushResolver

[ https://issues.apache.org/jira/browse/SPARK-37675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37675. - Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue

[jira] [Assigned] (SPARK-37675) Push-based merge finalization bugs in the RemoteBlockPushResolver

[ https://issues.apache.org/jira/browse/SPARK-37675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37675: --- Assignee: Chandni Singh > Push-based merge finalization b

Re: [VOTE] Release Spark 3.2.1 (RC2)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Fri, Jan 21, 2022 at 9:01 PM Sean Owen wrote: > +1 with same result as last time. > > On Thu, Jan 20, 2022 at 9:59 PM huaxin gao wrote: > >> Please vote on

[jira] [Assigned] (SPARK-37712) Spark request yarn cluster metrics slow and cause unnecessary delay

[ https://issues.apache.org/jira/browse/SPARK-37712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37712: --- Assignee: angerszhu > Spark request yarn cluster metrics slow and ca

[jira] [Resolved] (SPARK-37712) Spark request yarn cluster metrics slow and cause unnecessary delay

[ https://issues.apache.org/jira/browse/SPARK-37712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37712. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

+1 (binding) This should be a great improvement ! Regards, Mridul On Wed, Jan 12, 2022 at 4:04 AM Kent Yao wrote: > +1 (non-binding) > > Thomas Graves 于2022年1月12日周三 11:52写道： > >> +1 (binding). >> >> One minor note since I haven't had time to look at the implementation >> details is please

[jira] [Assigned] (SPARK-37847) PushBlockStreamCallback should check isTooLate first to avoid NPE

[ https://issues.apache.org/jira/browse/SPARK-37847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37847: --- Assignee: Cheng Pan > PushBlockStreamCallback should check isTooLate fi

[jira] [Resolved] (SPARK-37847) PushBlockStreamCallback should check isTooLate first to avoid NPE

[ https://issues.apache.org/jira/browse/SPARK-37847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37847. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37710) Add detailed log message for java.io.IOException occurring on Kryo flow

[ https://issues.apache.org/jira/browse/SPARK-37710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37710: --- Assignee: Eren Avsarogullari > Add detailed log mess

[jira] [Resolved] (SPARK-37710) Add detailed log message for java.io.IOException occurring on Kryo flow

[ https://issues.apache.org/jira/browse/SPARK-37710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37710. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-33701) Adaptive shuffle merge finalization for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-33701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33701. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-33701) Adaptive shuffle merge finalization for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-33701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33701: --- Assignee: Venkata krishnan Sowrirajan > Adaptive shuffle me

Re: Time for Spark 3.2.1?

+1 for maintenance release, and also +1 for doing this in Jan ! Thanks, Mridul On Tue, Dec 7, 2021 at 11:41 PM Gengliang Wang wrote: > +1 for new maintenance releases for all 3.x branches as well. > > On Wed, Dec 8, 2021 at 8:19 AM Hyukjin Kwon wrote: > >> SGTM! >> >> On Wed, 8 Dec 2021 at

Re: [FYI] Build and run tests on Java 17 for Apache Spark 3.3

Nice job ! There are some nice API's which should be interesting to explore with JDK 17 :-) Regards. Mridul On Fri, Nov 12, 2021 at 7:08 PM Yuming Wang wrote: > Cool, thank you Dongjoon. > > On Sat, Nov 13, 2021 at 4:09 AM shane knapp ☠ wrote: > >> woot! nice work everyone! :) >> >> On Fri,

[jira] [Resolved] (SPARK-37023) Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry

[ https://issues.apache.org/jira/browse/SPARK-37023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37023. - Fix Version/s: 3.3.0 3.2.1 Resolution: Fixed Issue

[jira] [Assigned] (SPARK-37023) Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry

[ https://issues.apache.org/jira/browse/SPARK-37023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37023: --- Assignee: Minchu Yang > Avoid fetching merge status w

Re: Update Spark 3.3 release window?

+1 to EOL 2.x Mid march sounds like a good placeholder for 3.3. Regards, Mridul On Wed, Oct 27, 2021 at 10:38 PM Sean Owen wrote: > Seems fine to me - as good a placeholder as anything. > Would that be about time to call 2.x end-of-life? > > On Wed, Oct 27, 2021 at 9:36 PM Hyukjin Kwon wrote:

Re: [ANNOUNCE] Apache Spark 3.2.0

Congratulations everyone ! And thanks Gengliang for sheparding the release out :-) Regards, Mridul On Tue, Oct 19, 2021 at 9:25 AM Yuming Wang wrote: > Congrats and thanks! > > On Tue, Oct 19, 2021 at 10:17 PM Gengliang Wang wrote: > >> Hi all, >> >> Apache Spark 3.2.0 is the third release

Re: [ANNOUNCE] Apache Spark 3.2.0

Congratulations everyone ! And thanks Gengliang for sheparding the release out :-) Regards, Mridul On Tue, Oct 19, 2021 at 9:25 AM Yuming Wang wrote: > Congrats and thanks! > > On Tue, Oct 19, 2021 at 10:17 PM Gengliang Wang wrote: > >> Hi all, >> >> Apache Spark 3.2.0 is the third release

[jira] [Resolved] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36834. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36834: --- Assignee: Thejdeep Gudivada > Namespace log lines in External Shuf

[jira] [Assigned] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36834: --- Assignee: (was: Thejdeep Gudivada) > Namespace log lines in Exter

[jira] [Assigned] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36834: --- Assignee: Thejdeep Gudivada > Namespace log lines in External Shuf

[jira] [Assigned] (SPARK-36798) When SparkContext is stopped, metrics system should be flushed after listeners have finished processing

[ https://issues.apache.org/jira/browse/SPARK-36798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36798: --- Assignee: Harsh Panchal > When SparkContext is stopped, metrics sys

[jira] [Resolved] (SPARK-36798) When SparkContext is stopped, metrics system should be flushed after listeners have finished processing

[ https://issues.apache.org/jira/browse/SPARK-36798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36798. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phadoop-2.7 -Pyarn -Pmesos -Pkubernetes. Regards, Mridul On Wed, Oct 6, 2021 at 12:55 PM Michael Heuer wrote: > +1 (non-binding) > >michael > > > On Oct 6, 2021, at 11:49 AM, Gengliang Wang wrote: > >

[jira] [Commented] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425016#comment-17425016 ] Mridul Muralidharan commented on SPARK-36892: - Sounds good [~Gengliang.Wang], I am not aware

[jira] [Commented] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424097#comment-17424097 ] Mridul Muralidharan commented on SPARK-36926: - +CC [~gengliang], [~cloud_fan] > Discrepa

[jira] [Comment Edited] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424052#comment-17424052 ] Mridul Muralidharan edited comment on SPARK-36892 at 10/4/21, 5:26 PM

[jira] [Commented] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424052#comment-17424052 ] Mridul Muralidharan commented on SPARK-36892: - [~Gengliang.Wang] Thanks for holding

Re: [VOTE] Release Spark 3.2.0 (RC6)

Yi Wu helped identify an issue which causes correctness (duplication) and hangs - waiting for validation to complete before submitting a patch. Regards, Mridul On Wed, Sep 29, 2021 at 11:34 AM Holden Karau wrote: > PySpark smoke tests pass,

[jira] [Created] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

Mridul Muralidharan created SPARK-36892: --- Summary: Disable batch fetch for a shuffle when push based shuffle is enabled Key: SPARK-36892 URL: https://issues.apache.org/jira/browse/SPARK-36892

Re: [VOTE] Release Spark 3.2.0 (RC3)

ue, Sep 21, 2021 at 2:05 PM Chao Sun wrote: >>> >>>> Mridul, is the LZ4 failure about Parquet? I think Parquet currently >>>> uses Hadoop compression codec while Hadoop 2.7 still depends on native lib >>>> for the LZ4. Maybe we should run the test only f

[jira] [Resolved] (SPARK-36615) SparkContext should register shutdown hook earlier

[ https://issues.apache.org/jira/browse/SPARK-36615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36615. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-36615) SparkContext should register shutdown hook earlier

[ https://issues.apache.org/jira/browse/SPARK-36615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36615: --- Assignee: angerszhu > SparkContext should register shutdown hook earl

Re: [VOTE] Release Spark 3.2.0 (RC3)

Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes, this worked fine. I found that including "-Phadoop-2.7" failed on lz4 tests ("native lz4 library not available"). Regards, Mridul On Tue, Sep 21, 2021 at 10:18 AM Gengliang Wang wrote:

[jira] [Commented] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415891#comment-17415891 ] Mridul Muralidharan commented on SPARK-36772: - Thanks [~Gengliang.Wang] ! And thanks

[jira] [Comment Edited] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415803#comment-17415803 ] Mridul Muralidharan edited comment on SPARK-36772 at 9/15/21, 11:55 PM

[jira] [Commented] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415803#comment-17415803 ] Mridul Muralidharan commented on SPARK-36772: - +CC [~Gengliang.Wang] > FinalizeShuffleMe

[jira] [Created] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

Mridul Muralidharan created SPARK-36772: --- Summary: FinalizeShuffleMerge fails with an exception due to attempt id not matching Key: SPARK-36772 URL: https://issues.apache.org/jira/browse/SPARK-36772

[jira] [Assigned] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36705: --- Assignee: Minchu Yang > Disable push based shuffle when IO encrypt

[jira] [Resolved] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36705. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Commented] (SPARK-36700) BlockManager re-registration is broken due to deferred removal of BlockManager

[ https://issues.apache.org/jira/browse/SPARK-36700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412732#comment-17412732 ] Mridul Muralidharan commented on SPARK-36700: - Thanks for chasing this down [~Ngone51

Re: [VOTE] Release Spark 3.2.0 (RC2)

I have filed a blocker, SPARK-36705 which will need to be addressed. Regards, Mridul On Sun, Sep 5, 2021 at 8:47 AM Gengliang Wang wrote: > Hi all, > > the voting fails. > Liang-Chi reported a new block SPARK-36669 >

[jira] [Created] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

Mridul Muralidharan created SPARK-36705: --- Summary: Disable push based shuffle when IO encryption is enabled or serializer is not relocatable Key: SPARK-36705 URL: https://issues.apache.org/jira/browse/SPARK

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi, Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Pmesos -Pkubernetes I am seeing test failures which are addressed by #33790 - this is in branch-3.2, but after the RC tag. After updating to the

Re: -1s on committed but not released code?

Hi Holden, In the past, I have seen discussions on the merged pr to thrash out the details. Usually it would be clear whether to revert and reformulate the change or concerns get addressed and possibly result in follow up work. This is usually helped by the fact that we typically are

[jira] [Assigned] (SPARK-35548) Handling new attempt has started error message in BlockPushErrorHandler in client

[ https://issues.apache.org/jira/browse/SPARK-35548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35548: --- Assignee: Qi Zhu (was: zhuqi) > Handling new attempt has started er

[jira] [Assigned] (SPARK-36344) Fix some typos in ShuffleBlockPusher class.

[ https://issues.apache.org/jira/browse/SPARK-36344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36344: --- Assignee: Qi Zhu (was: zhuqi) > Fix some typos in ShuffleBlockPus

[jira] [Assigned] (SPARK-35548) Handling new attempt has started error message in BlockPushErrorHandler in client

[ https://issues.apache.org/jira/browse/SPARK-35548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35548: --- Assignee: zhuqi > Handling new attempt has started error mess

[jira] [Resolved] (SPARK-35548) Handling new attempt has started error message in BlockPushErrorHandler in client

[ https://issues.apache.org/jira/browse/SPARK-35548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35548. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-36374) Push-based shuffle documentation

[ https://issues.apache.org/jira/browse/SPARK-36374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36374. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-36374) Push-based shuffle documentation

[ https://issues.apache.org/jira/browse/SPARK-36374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36374: --- Assignee: Venkata krishnan Sowrirajan > Push-based shuffle documentat

[jira] [Resolved] (SPARK-36483) Fix intermittent test failure due to netty dependency version bump

[ https://issues.apache.org/jira/browse/SPARK-36483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36483. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-36483) Fix intermittent test failure due to netty dependency version bump

[ https://issues.apache.org/jira/browse/SPARK-36483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36483: --- Assignee: Min Shen > Fix intermittent test failure due to netty depende

[jira] [Commented] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397725#comment-17397725 ] Mridul Muralidharan commented on SPARK-36378: - Thanks for marking this resolved [~Ngone51

Re: ASF board report draft for August

Hi Matei, 3.2 will also include support for pushed based shuffle (spip SPARK-30602). Regards, Mridul On Mon, Aug 9, 2021 at 9:26 PM Hyukjin Kwon wrote: > > Are you referring to what version of Koala project? 1.8.1? > > Yes, the latest version 1.8.1. > > 2021년 8월 10일 (화) 오전 11:07, Igor Costa

[jira] [Resolved] (SPARK-36423) Randomize blocks within a push request before pushing to improve block merge ratio

[ https://issues.apache.org/jira/browse/SPARK-36423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36423. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-36423) Randomize blocks within a push request before pushing to improve block merge ratio

[ https://issues.apache.org/jira/browse/SPARK-36423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36423: --- Assignee: Min Shen > Randomize blocks within a push request before push

[jira] [Assigned] (SPARK-36173) [CORE] Support getting CPU number in TaskContext

[ https://issues.apache.org/jira/browse/SPARK-36173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36173: --- Assignee: Xiaochang Wu > [CORE] Support getting CPU number in TaskCont

[jira] [Resolved] (SPARK-36173) [CORE] Support getting CPU number in TaskContext

[ https://issues.apache.org/jira/browse/SPARK-36173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36173. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-30602: --- Assignee: Min Shen > SPIP: Support push-based shuffle to improve shuf

[jira] [Commented] (SPARK-35275) Add checksum for shuffle blocks

[ https://issues.apache.org/jira/browse/SPARK-35275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391640#comment-17391640 ] Mridul Muralidharan commented on SPARK-35275: - Do we want to add a documentation task

[jira] [Assigned] (SPARK-36206) Diagnose shuffle data corruption by checksum

[ https://issues.apache.org/jira/browse/SPARK-36206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36206: --- Assignee: wuyi > Diagnose shuffle data corruption by check

[jira] [Resolved] (SPARK-36206) Diagnose shuffle data corruption by checksum

[ https://issues.apache.org/jira/browse/SPARK-36206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36206. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Updated] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-35917: Fix Version/s: (was: 3.2.0) > Disable push-based shuffle until the feat

[jira] [Updated] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-35917: Fix Version/s: 3.2.0 > Disable push-based shuffle until the feature is compl

[jira] [Assigned] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35917: --- Assignee: (was: Mridul Muralidharan) > Disable push-based shuf

[jira] [Assigned] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36378: --- Assignee: (was: Mridul Muralidharan) > Minor changes to address a

[jira] [Assigned] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-30602: --- Shepherd: Mridul Muralidharan Assignee: (was: Mridul Muralidharan

[jira] [Updated] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-30602: Fix Version/s: 3.2.0 > SPIP: Support push-based shuffle to improve shuf

<    1   2   3   4   5   6   7   8   9   10   >