Re: Question on Celeborn workers,

With push based shuffle in Apache Spark (magnet), we have both the map output and reducer orientated merged output preserved - with reducer oriented view chosen by default for reads and fallback to mapper output when reducer output is missing/failures. That mitigates this specific issue for

[jira] [Resolved] (SPARK-45429) [CORE] Add helper classes for RPC SSL communication

[ https://issues.apache.org/jira/browse/SPARK-45429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45429. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-45429) [CORE] Add helper classes for RPC SSL communication

[ https://issues.apache.org/jira/browse/SPARK-45429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45429: --- Assignee: Hasnain Lakhani > [CORE] Add helper classes for RPC

Re: [PROPOSAL] Spark stage resubmission for shuffle fetch failure

On Sat, Oct 14, 2023 at 3:49 AM Mridul Muralidharan wrote: > > A reducer oriented view of shuffle, especially without replication, could > indeed be susceptible to this issue you described (a single fetch failure > would require all mappers to need to be recomputed) - note, not neces

Re: [PROPOSAL] Spark stage resubmission for shuffle fetch failure

utation could be (partially or totally) reused. > > Regards, > > --- Sungwoo > > On Sat, Oct 14, 2023 at 5:24 PM Mridul Muralidharan > wrote: > >> >> Hi, >> >> Spark will try to minimize the recomputation cost as much as possible. >> For example, if

Re: [PROPOSAL] Spark stage resubmission for shuffle fetch failure

Hi, Spark will try to minimize the recomputation cost as much as possible. For example, if parent stage was DETERMINATE, it simply needs to recompute the missing (mapper) partitions (which resulted in fetch failure). Note, this by itself could require further recomputation in the DAG if the

[jira] [Resolved] (SPARK-45427) [CORE] Add RPC SSL settings to SSLOptions and SparkTransportConf

[ https://issues.apache.org/jira/browse/SPARK-45427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45427. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-45427) [CORE] Add RPC SSL settings to SSLOptions and SparkTransportConf

[ https://issues.apache.org/jira/browse/SPARK-45427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45427: --- Assignee: Hasnain Lakhani > [CORE] Add RPC SSL settings to SSLOpti

[jira] [Assigned] (SPARK-45426) [CORE] Add support for a ReloadingTrustManager

[ https://issues.apache.org/jira/browse/SPARK-45426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45426: --- Assignee: Hasnain Lakhani > [CORE] Add support for a ReloadingTrustMana

[jira] [Resolved] (SPARK-45426) [CORE] Add support for a ReloadingTrustManager

[ https://issues.apache.org/jira/browse/SPARK-45426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45426. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

Re: [PROPOSAL] Spark stage resubmission for shuffle fetch failure

Hi, So there are a couple of things here based on whether the stages are DETERMINATE or INDETERMINATE. The exit I added to my example was to trigger some of these cases, and you can come up with more involved scenarios where this would apply :-) At a high level, we have the following: a) If

[jira] [Assigned] (SPARK-45408) [CORE] Add RPC SSL settings to TransportConf

[ https://issues.apache.org/jira/browse/SPARK-45408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45408: --- Assignee: Hasnain Lakhani > [CORE] Add RPC SSL settings to TransportC

[jira] [Resolved] (SPARK-45408) [CORE] Add RPC SSL settings to TransportConf

[ https://issues.apache.org/jira/browse/SPARK-45408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45408. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-45283) Make StatusTrackerSuite less fragile

[ https://issues.apache.org/jira/browse/SPARK-45283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45283. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-45283) Make StatusTrackerSuite less fragile

[ https://issues.apache.org/jira/browse/SPARK-45283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45283: --- Assignee: Bo Xiong > Make StatusTrackerSuite less frag

[jira] [Assigned] (SPARK-45376) [CORE] Add netty-tcnative-boringssl-static dependency

[ https://issues.apache.org/jira/browse/SPARK-45376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45376: --- Assignee: Hasnain Lakhani > [CORE] Add netty-tcnative-boringssl-sta

[jira] [Resolved] (SPARK-45376) [CORE] Add netty-tcnative-boringssl-static dependency

[ https://issues.apache.org/jira/browse/SPARK-45376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45376. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations ! Looking forward to more exciting contributions :-) Regards, Mridul On Tue, Oct 3, 2023 at 2:51 AM Hussein Awala wrote: > Congrats to all of you! > > On Tue 3 Oct 2023 at 08:15, Rui Wang wrote: > >> Congratulations! Well deserved! >> >> -Rui >> >> >> On Mon, Oct 2, 2023 at

[jira] [Assigned] (SPARK-45250) Support stage level task resource profile for yarn cluster when dynamic allocation disabled

[ https://issues.apache.org/jira/browse/SPARK-45250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45250: --- Assignee: Bobby Wang > Support stage level task resource profile for y

[jira] [Resolved] (SPARK-45250) Support stage level task resource profile for yarn cluster when dynamic allocation disabled

[ https://issues.apache.org/jira/browse/SPARK-45250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45250. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-45378) [CORE] Add convertToNettyForSsl to ManagedBuffer

[ https://issues.apache.org/jira/browse/SPARK-45378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45378. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-45378) [CORE] Add convertToNettyForSsl to ManagedBuffer

[ https://issues.apache.org/jira/browse/SPARK-45378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45378: --- Assignee: Hasnain Lakhani > [CORE] Add convertToNettyFor

[jira] [Updated] (SPARK-45227) Fix a subtle thread-safety issue with CoarseGrainedExecutorBackend where an executor process randomly gets stuck

[ https://issues.apache.org/jira/browse/SPARK-45227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-45227: Fix Version/s: 3.3.4 > Fix a subtle thread-safety is

[jira] [Closed] (SPARK-45227) Fix a subtle thread-safety issue with CoarseGrainedExecutorBackend where an executor process randomly gets stuck

[ https://issues.apache.org/jira/browse/SPARK-45227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan closed SPARK-45227. --- > Fix a subtle thread-safety issue with CoarseGrainedExecutorBackend wh

[jira] [Assigned] (SPARK-45227) Fix a subtle thread-safety issue with CoarseGrainedExecutorBackend where an executor process randomly gets stuck

[ https://issues.apache.org/jira/browse/SPARK-45227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45227: --- Assignee: Bo Xiong > Fix a subtle thread-safety is

[jira] [Resolved] (SPARK-45227) Fix a subtle thread-safety issue with CoarseGrainedExecutorBackend where an executor process randomly gets stuck

[ https://issues.apache.org/jira/browse/SPARK-45227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45227. - Resolution: Fixed > Fix a subtle thread-safety is

[jira] [Updated] (SPARK-45227) Fix a subtle thread-safety issue with CoarseGrainedExecutorBackend where an executor process randomly gets stuck

[ https://issues.apache.org/jira/browse/SPARK-45227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-45227: Fix Version/s: 3.4.2 4.0.0 3.5.1 >

[jira] [Assigned] (SPARK-45057) Deadlock caused by rdd replication level of 2

[ https://issues.apache.org/jira/browse/SPARK-45057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-45057: --- Assignee: Zhongwei Zhu > Deadlock caused by rdd replication level o

[jira] [Resolved] (SPARK-45057) Deadlock caused by rdd replication level of 2

[ https://issues.apache.org/jira/browse/SPARK-45057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-45057. - Fix Version/s: 3.3.4 3.5.1 4.0.0

[jira] [Resolved] (SPARK-44937) Add SSL/TLS support for RPC and Shuffle communications

[ https://issues.apache.org/jira/browse/SPARK-44937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-44937. - Fix Version/s: 3.3.4 3.5.1 4.0.0

[jira] [Assigned] (SPARK-44937) Add SSL/TLS support for RPC and Shuffle communications

[ https://issues.apache.org/jira/browse/SPARK-44937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-44937: --- Assignee: Hasnain Lakhani > Add SSL/TLS support for RPC and Shuf

Re: Migrating the Junit framework used in Apache Spark 4.0 from 4.x to 5.x

+1 for moving to a newer version. Thanks for driving this Jie Yang ! Regards, Mridul On Mon, Sep 25, 2023 at 10:15 AM 杨杰 wrote: > Hi all, > > In SPARK-44170 (apache/spark#43074 [1]), I’m trying to migrate the Junit > test framework used in Spark 4.0 from Junit4 to Junit5. > > > Although this

[jira] [Resolved] (SPARK-44756) Executor hangs when RetryingBlockTransferor fails to initiate retry

[ https://issues.apache.org/jira/browse/SPARK-44756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-44756. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-44756) Executor hangs when RetryingBlockTransferor fails to initiate retry

[ https://issues.apache.org/jira/browse/SPARK-44756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-44756: --- Assignee: Harunobu Daikoku > Executor hangs when RetryingBlockTransfe

Re: [PROPOSAL] Spark stage resubmission for shuffle fetch failure

Hi, I am not yet very familiar with Celeborn, so will restrict my notes on the proposal in context to Apache Spark: a) For Option 1, there is SPARK-25299 - which was started a few years back. Unfortunately, the work there has stalled: but if there is interest in pushing that forward, I can

Re: [DISCUSSION] Support memory file storage.

n move existing shuffle files to different storage tires. > > c) As mentioned above, the enhancement is intended to act as a storage > tier that's why I explained the details about how it is handled > internally. > > Thanks again for your email. Please let me know if you have any >

Re: [DISCUSSION] Support memory file storage.

Hi, This should be a nontrivial improvement to Celeborn imo, thanks Ethan ! I had a few queries: a) Are we viewing this enhancement as a cache or as a tiered storage layer ? When going over it, I felt the proposal might be doing both - though leaning more as a cache, but wanted to get

[jira] [Resolved] (SPARK-44306) Group FileStatus with few RPC calls within Yarn Client

[ https://issues.apache.org/jira/browse/SPARK-44306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-44306. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-44306) Group FileStatus with few RPC calls within Yarn Client

[ https://issues.apache.org/jira/browse/SPARK-44306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-44306: --- Assignee: SHU WANG > Group FileStatus with few RPC calls within Y

Re: [DISCUSS] Support authentication in Celeborn

To add to what Chandni mentioned, using self-signed certificates and trusting them is another (though less secure) practice some deployments leverage. This ensures encryption over the wire, but does not allow for clients to validate identity of the Celeborn server components (so potentially

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Sat, Sep 9, 2023 at 10:02 AM Yuanjian Li wrote: > Please vote on releasing the following candidate(RC5) as Apache Spark > version 3.5.0. > > The vote is open

[jira] [Resolved] (SPARK-44845) spark job copies jars repeatedly if fs.defaultFS and application jar are same url

[ https://issues.apache.org/jira/browse/SPARK-44845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-44845. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-44845) spark job copies jars repeatedly if fs.defaultFS and application jar are same url

[ https://issues.apache.org/jira/browse/SPARK-44845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-44845: --- Assignee: zheju_he > spark job copies jars repeatedly if fs.defaul

Re: [VOTE] Release Apache Celeborn(Incubating) 0.3.1-incubating-rc0

+1 Signatures, digests, license, etc check out fine. Checked out tag and build/tested with -Pspark-3.1 Regards, Mridul On Thu, Aug 31, 2023 at 11:35 AM Cheng Pan wrote: > Hi Celeborn community, > > This is a call for a vote to release Apache Celeborn (Incubating) > 0.3.1-incubating-rc0 > >

[jira] [Assigned] (SPARK-44238) Introduce a new readFrom method with byte array input for BloomFilter

[ https://issues.apache.org/jira/browse/SPARK-44238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-44238: --- Assignee: Yang Jie > Introduce a new readFrom method with byte array in

[jira] [Resolved] (SPARK-44238) Introduce a new readFrom method with byte array input for BloomFilter

[ https://issues.apache.org/jira/browse/SPARK-44238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-44238. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-44162) Support G1GC in `spark.eventLog.gcMetrics.*` without warning

[ https://issues.apache.org/jira/browse/SPARK-44162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-44162: --- Assignee: Jia Fan > Support G1GC in `spark.eventLog.gcMetrics.*` with

[jira] [Resolved] (SPARK-44162) Support G1GC in `spark.eventLog.gcMetrics.*` without warning

[ https://issues.apache.org/jira/browse/SPARK-44162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-44162. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Aug 30, 2023 at 6:10 AM yangjie01 wrote: > Hi, Sean > > > > I have performed testing with Java 17 and Scala 2.13 using maven (`mvn > clean install` and

[jira] [Resolved] (SPARK-44242) Spark job submission failed because Xmx string is available on one parameter provided into spark.driver.extraJavaOptions

[ https://issues.apache.org/jira/browse/SPARK-44242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-44242. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-44242) Spark job submission failed because Xmx string is available on one parameter provided into spark.driver.extraJavaOptions

[ https://issues.apache.org/jira/browse/SPARK-44242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-44242: --- Assignee: Nicolas Fraison > Spark job submission failed because Xmx str

[jira] [Resolved] (SPARK-43987) Separate finalizeShuffleMerge Processing to Dedicated Thread Pools

[ https://issues.apache.org/jira/browse/SPARK-43987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-43987. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-43987) Separate finalizeShuffleMerge Processing to Dedicated Thread Pools

[ https://issues.apache.org/jira/browse/SPARK-43987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-43987: --- Assignee: SHU WANG > Separate finalizeShuffleMerge Processing to Dedica

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Fri, Aug 11, 2023 at 2:00 AM Cheng Pan wrote: > +1 (non-binding) > > Passed integration test with Apache Kyuubi. > > Thanks for driving this release. > >

[jira] [Updated] (SPARK-44272) Path Inconsistency when Operating statCache within Yarn Client

[ https://issues.apache.org/jira/browse/SPARK-44272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-44272: Affects Version/s: (was: 0.9.1) (was: 2.3.0

[jira] [Resolved] (SPARK-44272) Path Inconsistency when Operating statCache within Yarn Client

[ https://issues.apache.org/jira/browse/SPARK-44272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-44272. - Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue

[jira] [Assigned] (SPARK-44272) Path Inconsistency when Operating statCache within Yarn Client

[ https://issues.apache.org/jira/browse/SPARK-44272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-44272: --- Assignee: SHU WANG > Path Inconsistency when Operating statCache wit

[jira] [Commented] (SPARK-44215) Client receives zero number of chunks in merge meta response which doesn't trigger fallback to unmerged blocks

[ https://issues.apache.org/jira/browse/SPARK-44215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17740368#comment-17740368 ] Mridul Muralidharan commented on SPARK-44215: - Issue resolved by pull request 41762 https

[jira] [Updated] (SPARK-44215) Client receives zero number of chunks in merge meta response which doesn't trigger fallback to unmerged blocks

[ https://issues.apache.org/jira/browse/SPARK-44215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-44215: Fix Version/s: 3.3.3 > Client receives zero number of chunks in merge m

[jira] [Resolved] (SPARK-44215) Client receives zero number of chunks in merge meta response which doesn't trigger fallback to unmerged blocks

[ https://issues.apache.org/jira/browse/SPARK-44215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-44215. - Assignee: Chandni Singh Resolution: Fixed > Client receives zero num

[jira] [Updated] (SPARK-44215) Client receives zero number of chunks in merge meta response which doesn't trigger fallback to unmerged blocks

[ https://issues.apache.org/jira/browse/SPARK-44215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-44215: Fix Version/s: 3.5.0 3.4.2 > Client receives zero num

[jira] [Resolved] (SPARK-42784) Fix the problem of incomplete creation of subdirectories in push merged localDir

[ https://issues.apache.org/jira/browse/SPARK-42784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-42784. - Resolution: Fixed > Fix the problem of incomplete creation of subdirector

[jira] [Updated] (SPARK-42784) Fix the problem of incomplete creation of subdirectories in push merged localDir

[ https://issues.apache.org/jira/browse/SPARK-42784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-42784: Fix Version/s: 3.3.3 3.5.0 3.4.2 >

[jira] [Assigned] (SPARK-42784) Fix the problem of incomplete creation of subdirectories in push merged localDir

[ https://issues.apache.org/jira/browse/SPARK-42784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-42784: --- Assignee: Fencheng Mei > Fix the problem of incomplete creat

Re: [ANNOUNCE] Apache Spark 3.4.1 released

Thanks Dongjoon ! Regards, Mridul On Fri, Jun 23, 2023 at 6:58 PM Dongjoon Hyun wrote: > We are happy to announce the availability of Apache Spark 3.4.1! > > Spark 3.4.1 is a maintenance release containing stability fixes. This > release is based on the branch-3.4 maintenance branch of Spark.

Re: [ANNOUNCE] Apache Spark 3.4.1 released

Thanks Dongjoon ! Regards, Mridul On Fri, Jun 23, 2023 at 6:58 PM Dongjoon Hyun wrote: > We are happy to announce the availability of Apache Spark 3.4.1! > > Spark 3.4.1 is a maintenance release containing stability fixes. This > release is based on the branch-3.4 maintenance branch of Spark.

Re: [VOTE][RESULT] Release Spark 3.4.1 (RC1)

A late +1 from me too … forgot to send this yesterday :-) Regards, Mridul On Fri, Jun 23, 2023 at 3:20 AM Dongjoon Hyun wrote: > The vote passes with 15 +1s (10 binding +1s). > Thanks to all who helped with the release! > > (* = binding) > +1: > - Jia Fan > - Dongjoon Hyun * > - Liang-Chi

Re: [DISCUSS] Allow external contributors to run CI without approval

Agree, +1 Regards, Mridul On Fri, Jun 16, 2023 at 9:16 AM Cheng Pan wrote: > +1 for "only requires approval first time" > > Keyong Zhou 于 2023年6月16日周五 下午5:48写道： > > > +1 > > > > Thanks, > > Keyong Zhou > > > > Ethan Feng 于2023年6月16日周五 16:27写道： > > > > > Recent moves by Apache Infra have

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

I agree with Holden, we should have some understanding of what we are targeting for 4.0, given it is a major ver bump - and work from there on the release date. Regards, Mridul On Mon, Jun 12, 2023 at 8:53 PM Jia Fan wrote: > By the way, like Holden said, what's big feature for 4.0.0? I think

Re: Apache Spark 3.4.1 Release?

+1, thanks Dongjoon ! Regards, Mridul On Thu, Jun 8, 2023 at 7:16 PM Jia Fan wrote: > +1 > > > > > Jia Fan > > > > 2023年6月9日 08:00，Yuming Wang 写道： > > +1. > > On Fri, Jun 9, 2023 at 7:14 AM Chao Sun wrote: > >> +1 too >> >> On Thu, Jun 8, 2023 at 2:34 PM kazuyuki

[jira] [Resolved] (SPARK-43237) Handle null exception message in event log

[ https://issues.apache.org/jira/browse/SPARK-43237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-43237. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-43237) Handle null exception message in event log

[ https://issues.apache.org/jira/browse/SPARK-43237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-43237: --- Assignee: Zhongwei Zhu > Handle null exception message in event

[jira] [Resolved] (SPARK-43052) Handle stacktrace with null file name in event log

[ https://issues.apache.org/jira/browse/SPARK-43052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-43052. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-43052) Handle stacktrace with null file name in event log

[ https://issues.apache.org/jira/browse/SPARK-43052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-43052: --- Assignee: Zhongwei Zhu > Handle stacktrace with null file name in ev

[jira] [Assigned] (SPARK-43179) Add option for applications to control saving of metadata in the External Shuffle Service LevelDB

[ https://issues.apache.org/jira/browse/SPARK-43179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-43179: --- Assignee: Chandni Singh > Add option for applications to control sav

[jira] [Resolved] (SPARK-43179) Add option for applications to control saving of metadata in the External Shuffle Service LevelDB

[ https://issues.apache.org/jira/browse/SPARK-43179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-43179. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Mon, Apr 10, 2023 at 10:34 AM huaxin gao wrote: > +1 > > On Mon, Apr 10, 2023 at 8:17 AM Chao Sun wrote: > >> +1 (non-binding) >> >> On Mon, Apr 10, 2023

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Sat, Apr 8, 2023 at 12:13 PM L. C. Hsieh wrote: > +1 > > Thanks Xinrong. > > On Sat, Apr 8, 2023 at 8:23 AM yangjie01 wrote: > > > > +1 > > > > > > > >

Re: Apache Spark 3.2.4 EOL Release?

+1 Sounds good to me. Thanks, Mridul On Tue, Apr 4, 2023 at 1:39 PM huaxin gao wrote: > +1 > > On Tue, Apr 4, 2023 at 11:17 AM Chao Sun wrote: > >> +1 >> >> On Tue, Apr 4, 2023 at 11:12 AM Holden Karau >> wrote: >> >>> +1 >>> >>> On Tue, Apr 4, 2023 at 11:04 AM L. C. Hsieh wrote: >>>

Re: Slack for PySpark users

Thanks for flagging the concern Dongjoon, I was not aware of the discussion - but I can understand the concern. Would be great if you or Matei could update the thread on the result of deliberations, once it reaches a logical consensus: before we set up official policy around it. Regards, Mridul

Re: Slack for PySpark users

Thanks for flagging the concern Dongjoon, I was not aware of the discussion - but I can understand the concern. Would be great if you or Matei could update the thread on the result of deliberations, once it reaches a logical consensus: before we set up official policy around it. Regards, Mridul

[jira] [Created] (SPARK-42922) Use SecureRandom, instead of Random in security sensitive contexts

Mridul Muralidharan created SPARK-42922: --- Summary: Use SecureRandom, instead of Random in security sensitive contexts Key: SPARK-42922 URL: https://issues.apache.org/jira/browse/SPARK-42922

Re: Ammonite as REPL for Spark Connect

ng started > with connect, and/or doing debugging. > > On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan > wrote: > >> >> What is unclear to me is why we are introducing this integration, how >> users will leverage it. >> >> * Are we replacing spark-shell

Re: Ammonite as REPL for Spark Connect

che Spark. > > On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan > wrote: > >> >> Will this be maintained externally or included into Apache Spark ? >> >> Regards , >> Mridul >> >> >> >> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell >

Re: Ammonite as REPL for Spark Connect

Will this be maintained externally or included into Apache Spark ? Regards , Mridul On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell wrote: > Hi All, > > For Spark Connect Scala Client we are working on making the REPL > experience a bit nicer .

[jira] [Resolved] (SPARK-40082) DAGScheduler may not schduler new stage in condition of push-based shuffle enabled

[ https://issues.apache.org/jira/browse/SPARK-40082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40082. - Fix Version/s: 3.5.0 Assignee: Fencheng Mei Resolution: Fixed

[jira] [Resolved] (SPARK-42577) A large stage could run indefinitely due to executor lost

[ https://issues.apache.org/jira/browse/SPARK-42577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-42577. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-42577) A large stage could run indefinitely due to executor lost

[ https://issues.apache.org/jira/browse/SPARK-42577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-42577: --- Assignee: Tengfei Huang > A large stage could run indefinitely

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

Other than the tag issue, the sigs/artifacts/build/etc worked for me. So the next RC candidate looks promising ! Regards, Mridul On Thu, Mar 9, 2023 at 5:07 PM Xinrong Meng wrote: > Thank you Hyukjin! :) > > I would prefer to cut v3.4.0-rc4 now if there are no objections. > > On Fri, Mar 10,

[jira] [Resolved] (SPARK-42719) `MapOutputTracker#getMapLocation` should respect `spark.shuffle.reduceLocality.enabled`

[ https://issues.apache.org/jira/browse/SPARK-42719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-42719. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-42719) `MapOutputTracker#getMapLocation` should respect `spark.shuffle.reduceLocality.enabled`

[ https://issues.apache.org/jira/browse/SPARK-42719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-42719: --- Assignee: He Qi > `MapOutputTracker#getMapLocation` should resp

[jira] [Assigned] (SPARK-42689) Allow ShuffleDriverComponent to declare if shuffle data is reliably stored

[ https://issues.apache.org/jira/browse/SPARK-42689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-42689: --- Assignee: Mridul Muralidharan > Allow ShuffleDriverComponent to decl

[jira] [Resolved] (SPARK-42689) Allow ShuffleDriverComponent to declare if shuffle data is reliably stored

[ https://issues.apache.org/jira/browse/SPARK-42689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-42689. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request

[jira] [Updated] (SPARK-42689) Allow ShuffleDriverComponent to declare if shuffle data is reliably stored

[ https://issues.apache.org/jira/browse/SPARK-42689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-42689: Description: Currently, if there is an executor node loss, we assume the shuffle

[jira] [Created] (SPARK-42689) Allow ShuffleDriverComponent to declare if shuffle data is reliably stored

Mridul Muralidharan created SPARK-42689: --- Summary: Allow ShuffleDriverComponent to declare if shuffle data is reliably stored Key: SPARK-42689 URL: https://issues.apache.org/jira/browse/SPARK-42689

[jira] [Resolved] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-41497. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-41497: --- Assignee: Tengfei Huang > Accumulator undercounting in the case of re

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ... On Wed, Feb 22, 2023 at 2:07 AM Mridul Muralidharan wrote: > > Thanks Xinrong ! > The signature verifications are fine now ... will continue with testing > the release. > > > Regards, > Mridul > &

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

Thanks Xinrong ! The signature verifications are fine now ... will continue with testing the release. Regards, Mridul On Wed, Feb 22, 2023 at 1:27 AM Xinrong Meng wrote: > Hi Mridul, > > Would you please try that again? It should work now. > > On Wed, Feb 22, 2023 at

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

Hi Xinrong, Was it signed with the same key as present in KEYS [1] ? I am seeing errors with gpg when validating. For example: $ gpg --verify pyspark-3.4.0.tar.gz.asc gpg: assuming signed data in 'pyspark-3.4.0.tar.gz' gpg: Signature made Tue 21 Feb 2023 05:56:05 AM CST gpg:

<    1   2   3   4   5   6   7   8   9   10   >