[jira] [Comment Edited] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391318#comment-17391318 ] Mridul Muralidharan edited comment on SPARK-30602 at 8/2/21, 5:30 AM

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391324#comment-17391324 ] Mridul Muralidharan commented on SPARK-30602: - Thanks for all the work in getting

[jira] [Assigned] (SPARK-36266) Rename classes in shuffle RPC used for block push operations

[ https://issues.apache.org/jira/browse/SPARK-36266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36266: --- Assignee: Min Shen > Rename classes in shuffle RPC used for block p

[jira] [Resolved] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-30602. - Resolution: Fixed The only pending task here is documentation. [~vsowrirajan

[jira] [Assigned] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-30602: --- Assignee: Mridul Muralidharan > SPIP: Support push-based shuf

[jira] [Updated] (SPARK-32923) Add support to properly handle different type of stage retries

[ https://issues.apache.org/jira/browse/SPARK-32923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32923: Fix Version/s: 3.2.0 > Add support to properly handle different type of st

[jira] [Assigned] (SPARK-32923) Add support to properly handle different type of stage retries

[ https://issues.apache.org/jira/browse/SPARK-32923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32923: --- Assignee: Venkata krishnan Sowrirajan > Add support to properly han

[jira] [Issue Comment Deleted] (SPARK-32923) Add support to properly handle different type of stage retries

[ https://issues.apache.org/jira/browse/SPARK-32923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32923: Comment: was deleted (was: This has been handled by SPARK-32923) > Add supp

[jira] [Resolved] (SPARK-32923) Add support to properly handle different type of stage retries

[ https://issues.apache.org/jira/browse/SPARK-32923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32923. - Resolution: Fixed This has been handled by SPARK-32923 > Add supp

[jira] [Assigned] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36378: --- Assignee: Mridul Muralidharan > Minor changes to address a few identif

[jira] [Resolved] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36378. - Resolution: Won't Fix Let us move this outside of the SPIP and into individual

[jira] [Assigned] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35917: --- Assignee: Mridul Muralidharan > Disable push-based shuffle un

[jira] [Resolved] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35917. - Resolution: Won't Fix > Disable push-based shuffle until the feat

[jira] [Commented] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391313#comment-17391313 ] Mridul Muralidharan commented on SPARK-35917: - Closing this Jira - as push based shuffle has

[jira] [Resolved] (SPARK-36266) Rename classes in shuffle RPC used for block push operations

[ https://issues.apache.org/jira/browse/SPARK-36266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36266. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-35546) Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way

[ https://issues.apache.org/jira/browse/SPARK-35546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35546: --- Assignee: Ye Zhou > Enable push-based shuffle when multiple app attem

[jira] [Assigned] (SPARK-35276) Write checksum files for shuffle

[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35276: --- Assignee: wuyi > Write checksum files for shuf

[jira] [Resolved] (SPARK-35276) Write checksum files for shuffle

[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35276. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures

[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32922. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures

[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32922: --- Assignee: Chandni Singh > Add support for ShuffleBlockFetcherItera

[jira] [Resolved] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

[ https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35258. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

[ https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35258: --- Assignee: Erik Krogen > Enhance ESS ExternalBlockHandler with additio

[jira] [Resolved] (SPARK-35836) Remove reference to spark.shuffle.push.based.enabled in ShuffleBlockPusherSuite

[ https://issues.apache.org/jira/browse/SPARK-35836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35836. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-35836) Remove reference to spark.shuffle.push.based.enabled in ShuffleBlockPusherSuite

[ https://issues.apache.org/jira/browse/SPARK-35836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35836: --- Assignee: Chandni Singh > Remove refere

[jira] [Assigned] (SPARK-35671) Add Support in the ESS to serve merged shuffle block meta and data to executors

[ https://issues.apache.org/jira/browse/SPARK-35671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35671: --- Assignee: Chandni Singh > Add Support in the ESS to serve merged shuf

[jira] [Resolved] (SPARK-35671) Add Support in the ESS to serve merged shuffle block meta and data to executors

[ https://issues.apache.org/jira/browse/SPARK-35671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35671. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

Re: [VOTE] Release Spark 3.0.3 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Pmesos -Pkubernetes Regards, Mridul PS: Might be related to some quirk of my local env - the first test run (after clean + package) usually fails for me (typically for hive tests) - with a

[jira] [Resolved] (SPARK-34898) Send ExecutorMetricsUpdate EventLog appropriately

[ https://issues.apache.org/jira/browse/SPARK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34898. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-35613) Cache commonly occurring strings from SQLMetrics, JsonProtocol and AccumulatorV2

[ https://issues.apache.org/jira/browse/SPARK-35613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35613. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-35613) Cache commonly occurring strings from SQLMetrics, JsonProtocol and AccumulatorV2

[ https://issues.apache.org/jira/browse/SPARK-35613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35613: --- Assignee: Venkata krishnan Sowrirajan > Cache commonly occurring stri

[jira] [Assigned] (SPARK-33350) Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

[ https://issues.apache.org/jira/browse/SPARK-33350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33350: --- Assignee: Ye Zhou > Add support to DiskBlockManager to create me

[jira] [Resolved] (SPARK-33350) Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

[ https://issues.apache.org/jira/browse/SPARK-33350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33350. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32920. - Resolution: Fixed > Add support in Spark driver to coordinate the finalizat

[jira] [Updated] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32920: Fix Version/s: 3.2.0 > Add support in Spark driver to coordinate the finalizat

[jira] [Assigned] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32920: --- Assignee: Venkata krishnan Sowrirajan > Add support in Spark dri

Re: Apache Spark 3.0.3 Release?

+1 Regards, Mridul On Tue, Jun 8, 2021 at 10:11 PM Hyukjin Kwon wrote: > Yeah, +1 > > 2021년 6월 9일 (수) 오후 12:06, Yi Wu 님이 작성: > >> Hi, All. >> >> Since Apache Spark 3.0.2 tag creation (Feb 16), >> new 119 patches (92 issues >> >>

Re: Resolves too old JIRAs as incomplete

+1, thanks Takeshi ! Regards, Mridul On Wed, May 19, 2021 at 8:48 PM Takeshi Yamamuro wrote: > Hi, dev, > > As you know, we have too many open JIRAs now: > # of open JIRAs=2698: JQL='project = SPARK AND status in (Open, "In > Progress", Reopened)' > > We've recently released v2.4.8(EOL), so

[jira] [Resolved] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

[ https://issues.apache.org/jira/browse/SPARK-35263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35263. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

[ https://issues.apache.org/jira/browse/SPARK-35263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35263: --- Assignee: Erik Krogen > Refactor ShuffleBlockFetcherIteratorSu

Re: [VOTE] Release Spark 2.4.8 (RC4)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested. Regards, Mridul On Sun, May 9, 2021 at 4:22 PM Liang-Chi Hsieh wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.4.8. > > The vote is open until May 14th at 9AM PST and passes if

[jira] [Resolved] (SPARK-32921) Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario

[ https://issues.apache.org/jira/browse/SPARK-32921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32921. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-32921) Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario

[ https://issues.apache.org/jira/browse/SPARK-32921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32921: --- Assignee: Venkata krishnan Sowrirajan > Extend MapOutputTracker to supp

[jira] [Assigned] (SPARK-35049) Remove unused MapOutputTracker in BlockStoreShuffleReader

[ https://issues.apache.org/jira/browse/SPARK-35049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35049: --- Assignee: angerszhu > Remove unused MapOutputTrac

[jira] [Resolved] (SPARK-35049) Remove unused MapOutputTracker in BlockStoreShuffleReader

[ https://issues.apache.org/jira/browse/SPARK-35049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35049. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

Re: [VOTE] Release Spark 2.4.8 (RC1)

Do we have a fix for this in 3.x/master which can be backported without too much surrounding change ? Given we are expecting 2.4.7 to probably be the last release for 2.4, if we can fix it, that would be great. Regards, Mridul On Wed, Apr 7, 2021 at 9:31 PM Liang-Chi Hsieh wrote: > Thanks for

Re: Mesos + Spark users going forward?

Unfortunate about Mesos, +1 on deprecation of mesos integration. Regards, Mridul On Wed, Apr 7, 2021 at 7:12 AM Sean Owen wrote: > I noted that Apache Mesos is moving to the attic, so won't be actively > developed soon: > >

[jira] [Assigned] (SPARK-34949) Executor.reportHeartBeat reregisters blockManager even when Executor is shutting down

[ https://issues.apache.org/jira/browse/SPARK-34949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34949: --- Assignee: Sumeet > Executor.reportHeartBeat reregisters blockManager e

[jira] [Resolved] (SPARK-34949) Executor.reportHeartBeat reregisters blockManager even when Executor is shutting down

[ https://issues.apache.org/jira/browse/SPARK-34949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34949. - Fix Version/s: 3.1.2 3.2.0 Resolution: Fixed Issue

Re: [VOTE] SPIP: Support pandas API layer on PySpark

+1 Regards, Mridul On Sat, Mar 27, 2021 at 6:09 PM Xiao Li wrote: > +1 > > Xiao > > Takeshi Yamamuro 于2021年3月26日周五 下午4:14写道： > >> +1 (non-binding) >> >> On Sat, Mar 27, 2021 at 4:53 AM Liang-Chi Hsieh wrote: >> >>> +1 (non-binding) >>> >>> >>> rxin wrote >>> > +1. Would open up a huge

Re: Welcoming six new Apache Spark committers

Congratulations, looking forward to more exciting contributions ! Regards, Mridul On Fri, Mar 26, 2021 at 8:21 PM Dongjoon Hyun wrote: > > Congratulations! :) > > Bests, > Dongjoon. > > On Fri, Mar 26, 2021 at 5:55 PM angers zhu wrote: > >> Congratulations >> >> Prashant Sharma

[jira] [Resolved] (SPARK-34840) Fix cases of corruption in merged shuffle blocks that are pushed

[ https://issues.apache.org/jira/browse/SPARK-34840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34840. - Fix Version/s: 3.1.2 3.2.0 Resolution: Fixed Issue

[jira] [Assigned] (SPARK-34840) Fix cases of corruption in merged shuffle blocks that are pushed

[ https://issues.apache.org/jira/browse/SPARK-34840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34840: --- Assignee: Chandni Singh > Fix cases of corruption in merged shuffle blo

Re: [ANNOUNCE] Announcing Apache Spark 3.1.1

Thanks Hyukjin and congratulations everyone on the release ! Regards, Mridul On Tue, Mar 2, 2021 at 8:54 PM Yuming Wang wrote: > Great work, Hyukjin! > > On Wed, Mar 3, 2021 at 9:50 AM Hyukjin Kwon wrote: > >> We are excited to announce Spark 3.1.1 today. >> >> Apache Spark 3.1.1 is the

Re: [ANNOUNCE] Announcing Apache Spark 3.1.1

Thanks Hyukjin and congratulations everyone on the release ! Regards, Mridul On Tue, Mar 2, 2021 at 8:54 PM Yuming Wang wrote: > Great work, Hyukjin! > > On Wed, Mar 3, 2021 at 9:50 AM Hyukjin Kwon wrote: > >> We are excited to announce Spark 3.1.1 today. >> >> Apache Spark 3.1.1 is the

Re: Apache Spark 3.2 Expectation

Nit: Java 17 -> should be available by Sept 2021 :-) Adoption would also depend on some of our nontrivial dependencies supporting it - it might be a stretch to get it in for Apache Spark 3.2 ? Features: Push based shuffle and disaggregated shuffle should also be in 3.2 Regards, Mridul On

Re: [VOTE] Release Spark 3.1.1 (RC3)

different > results between Spark 3.0 and Spark 3.1. We need a few more days to > understand whether these changes are expected. > > Xiao > > > Mridul Muralidharan 于2021年2月24日周三 上午10:41写道： > >> >> Sounds good, thanks for clarifying Hyukjin ! >> +1 on release. >

Re: [VOTE] Release Spark 3.1.1 (RC3)

rk/commit/0d5d248bdc4cdc71627162a3d20c42ad19f24ef4 > and .. KafkaDelegationTokenSuite is flaky ( > https://issues.apache.org/jira/browse/SPARK-31250). > > 2021년 2월 24일 (수) 오후 5:19, Mridul Muralidharan 님이 작성: > >> >> Signatures, digests, etc check out fine. >> Checked out tag and build/tested

Re: [VOTE] Release Spark 3.1.1 (RC3)

Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes I keep getting test failures with * org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *

[jira] [Assigned] (SPARK-24818) Ensure all the barrier tasks in the same stage are launched together

[ https://issues.apache.org/jira/browse/SPARK-24818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-24818: --- Assignee: wuyi > Ensure all the barrier tasks in the same st

[jira] [Resolved] (SPARK-24818) Ensure all the barrier tasks in the same stage are launched together

[ https://issues.apache.org/jira/browse/SPARK-24818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-24818. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

Re: [DISCUSS] assignee practice on committers+ (possible issue on preemption)

I agree, Assignee has been used primarily to give recognition to the contributor who ended up submitting the patch which got merged. Typically jira's remain unassigned - even if it were to be assigned, it conveys no meaning or ownership or ongoing work : IMO it is equivalent to an unassigned

Re: [VOTE] Release Spark 3.1.1 (RC2)

Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes I keep getting test failures with org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite: removing this suite gets the build through though - does

Re: [VOTE] Release Spark 3.1.1 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes The sha512 signature for spark-3.1.1.tgz tripped up my scripts :-) Regards, Mridul On Wed, Jan 20, 2021 at 8:17 PM 郑瑞峰 wrote: > +1

[jira] [Assigned] (SPARK-34069) Kill barrier tasks should respect SPARK_JOB_INTERRUPT_ON_CANCEL

[ https://issues.apache.org/jira/browse/SPARK-34069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34069: --- Assignee: ulysses you > Kill barrier tasks should resp

[jira] [Resolved] (SPARK-34069) Kill barrier tasks should respect SPARK_JOB_INTERRUPT_ON_CANCEL

[ https://issues.apache.org/jira/browse/SPARK-34069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34069. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-32917) Add support for executors to push shuffle blocks after successful map task completion

[ https://issues.apache.org/jira/browse/SPARK-32917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32917: --- Assignee: Chandni Singh > Add support for executors to push shuffle blo

[jira] [Resolved] (SPARK-32917) Add support for executors to push shuffle blocks after successful map task completion

[ https://issues.apache.org/jira/browse/SPARK-32917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32917. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

Re: Recovering SparkR on CRAN?

I agree, is there something we can do to ensure CRAN publish goes through consistently and predictably ? If possible, it would be good to continue supporting it. Regards, Mridul On Tue, Dec 22, 2020 at 7:48 PM Felix Cheung wrote: > Ok - it took many years to get it first published, so it was

[jira] [Assigned] (SPARK-33669) Wrong error message from YARN application state monitor when sc.stop in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-33669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33669: --- Assignee: Su Qilong > Wrong error message from YARN application st

[jira] [Resolved] (SPARK-33669) Wrong error message from YARN application state monitor when sc.stop in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-33669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33669. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33185. - Resolution: Fixed Issue resolved by pull request 30450 [https://github.com

[jira] [Assigned] (SPARK-32918) RPC implementation to support control plane coordination for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-32918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32918: --- Assignee: Ye Zhou > RPC implementation to support control pl

[jira] [Resolved] (SPARK-32918) RPC implementation to support control plane coordination for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-32918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32918. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions

[ https://issues.apache.org/jira/browse/SPARK-32919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32919: --- Assignee: Venkata krishnan Sowrirajan > Add support in Spark dri

[jira] [Resolved] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions

[ https://issues.apache.org/jira/browse/SPARK-32919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32919. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-31069) high cpu caused by chunksBeingTransferred in external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-31069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-31069: --- Assignee: angerszhu > high cpu caused by chunksBeingTransfer

[jira] [Assigned] (SPARK-31069) high cpu caused by chunksBeingTransferred in external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-31069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-31069: --- Assignee: angerszhu (was: angerszhu) > high cpu cau

[jira] [Resolved] (SPARK-31069) high cpu caused by chunksBeingTransferred in external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-31069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-31069. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request

Re: [DISCUSS] Review/merge phase, and post-review

I try to follow the second option. In general, when multiple reviewers are looking at the code, sometimes addressing review comments might open up other avenues of discussion/optimization/design discussions : atleast in core, I have seen this happen often. A day or so delay is worth the increased

[jira] [Assigned] (SPARK-32915) RPC implementation to support pushing and merging shuffle blocks

[ https://issues.apache.org/jira/browse/SPARK-32915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32915: --- Assignee: Min Shen > RPC implementation to support pushing and merg

[jira] [Resolved] (SPARK-32916) Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-32916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32916. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-32916) Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-32916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32916: --- Assignee: Chandni Singh > Add support for external shuffle service in Y

[jira] [Assigned] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33185: --- Assignee: Erik Krogen > YARN: Print direct links to driver logs alongs

[jira] [Resolved] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33185. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request

Re: [VOTE] Standardize Spark Exception Messages SPIP

+1 Regards, Mridul On Wed, Nov 4, 2020 at 12:41 PM Xinyi Yu wrote: > Hi all, > > We had the discussion of SPIP: Standardize Spark Exception Messages at > > http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-SPIP-Standardize-Spark-Exception-Messages-td30341.html > < >

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

I like the idea of consistent messages; it makes understanding errors easier. Having said that, Exception messages themselves are not part of the exposed contract to users; and are subject to change. We should leave that flexibility open to spark developers ... I am currently viewing this proposal

[jira] [Resolved] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events

[ https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33088. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-32915) RPC implementation to support pushing and merging shuffle blocks

[ https://issues.apache.org/jira/browse/SPARK-32915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32915. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request

Re: Apache Spark 3.1 Preparation Status (Oct. 2020)

+1 on pushing the branch cut for increased dev time to match previous releases. Regards, Mridul On Sat, Oct 3, 2020 at 10:22 PM Xiao Li wrote: > Thank you for your updates. > > Spark 3.0 got released on Jun 18, 2020. If Nov 1st is the target date of > the 3.1 branch cut, the feature

Re: Apache Spark 3.1 Preparation Status (Oct. 2020)

+1 on pushing the branch cut for increased dev time to match previous releases. Regards, Mridul On Sat, Oct 3, 2020 at 10:22 PM Xiao Li wrote: > Thank you for your updates. > > Spark 3.0 got released on Jun 18, 2020. If Nov 1st is the target date of > the 3.1 branch cut, the feature

[jira] [Updated] (SPARK-32738) thread safe endpoints may hang due to fatal error

[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32738: Fix Version/s: 2.4.8 > thread safe endpoints may hang due to fatal er

[RESULT] [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

Hi, The vote passed with 16 +1's (6 binding) and no -1's +1s (* = binding): Xingbo Jiang Venkatakrishnan Sowrirajan Tom Graves (*) Chandni Singh DB Tsai (*) Xiao Li (*) Angers Zhu Joseph Torres Kalyan Dongjoon Hyun (*) Wenchen Fan (*) Yi Wu 叶先进 郑瑞峰 Takeshi Yamamuro Mridul Muralidharan

Re: [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

Adding my +1 as well, before closing the vote. Regards, Mridul On Sun, Sep 13, 2020 at 9:59 PM Mridul Muralidharan wrote: > Hi, > > I'd like to call for a vote on SPARK-30602 - SPIP: Support push-based > shuffle to improve shuffle efficiency. > Please take a look at: > >

[jira] [Updated] (SPARK-32738) thread safe endpoints may hang due to fatal error

[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32738: Fix Version/s: 3.0.2 > thread safe endpoints may hang due to fatal er

[VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

Hi, I'd like to call for a vote on SPARK-30602 - SPIP: Support push-based shuffle to improve shuffle efficiency. Please take a look at: - SPIP jira: https://issues.apache.org/jira/browse/SPARK-30602 - SPIP doc:

Re: [VOTE] Release Spark 2.4.7 (RC3)

t; On Wed, Sep 9, 2020 at 6:12 AM Mridul Muralidharan > wrote: > >> >> +1 >> >> Signatures, digests, etc check out fine. >> Checked out tag and built/tested with -Pyarn -Phadoop-2.7 -Phive >> -Phive-thriftserver -Pmesos -Pkubernetes >> >&

Re: [VOTE] Release Spark 2.4.7 (RC3)

+1 Signatures, digests, etc check out fine. Checked out tag and built/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes Thanks, Mridul On Tue, Sep 8, 2020 at 8:55 AM Prashant Sharma wrote: > Please vote on releasing the following candidate as Apache Spark >

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183622#comment-17183622 ] Mridul Muralidharan commented on SPARK-30602: - SPIP proposal document: [https

Re: Push-based shuffle SPIP

Hi, Thanks for sending out the proposal Min ! For the SPIP requirements, I am willing to act as the shepherd for this proposal. The jira + paper + proposal provides the high level design and implementation details. The vldb paper discusses the performance gains in detail for the inhouse

[jira] [Assigned] (SPARK-32663) TransportClient getting closed when there are outstanding requests to the server

[ https://issues.apache.org/jira/browse/SPARK-32663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32663: --- Assignee: Attila Zsolt Piros > TransportClient getting closed w

<    1   2   3   4   5   6   7   8   9   10   >