Re: [VOTE] Release Spark 3.3.2 (RC1)

2023-02-11 Thread Mridul Muralidharan
ct in > > https://repository.apache.org/content/repositories/orgapachespark-1433/org/apache/spark/spark-mllib-local_2.13/3.3.2/ > . > Did I miss something? > > Liang-Chi > > On Sat, Feb 11, 2023 at 10:08 AM Mridul Muralidharan > wrote: > > > > > > Hi, >

Re: [VOTE] Release Spark 3.3.2 (RC1)

2023-02-11 Thread Mridul Muralidharan
Hi, The following file is missing in the staging repository - there is a corresponding asc sig file, without the artifact. * org/apache/spark/spark-mllib-local_2.13/3.3.2/spark-mllib-local_2.13-3.3.2-test-sources.jar Can we have this fixed please ? Rest of the signatures, digests, etc check out

[jira] [Resolved] (SPARK-42366) Log shuffle data corruption diagnose cause

2023-02-09 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-42366. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-42366) Log shuffle data corruption diagnose cause

2023-02-09 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-42366: --- Assignee: dzcxzl > Log shuffle data corruption diagnose ca

[jira] [Updated] (SPARK-42090) Introduce sasl retry count in RetryingBlockTransferor

2023-01-24 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-42090: Fix Version/s: 3.2.4 3.3.2 > Introduce sasl retry co

[jira] [Resolved] (SPARK-42149) Remove the env `SPARK_USE_CONC_INCR_GC` used to enable CMS GC for Yarn AM

2023-01-21 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-42149. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-42149) Remove the env `SPARK_USE_CONC_INCR_GC` used to enable CMS GC for Yarn AM

2023-01-21 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-42149: --- Assignee: Yang Jie > Remove the env `SPARK_USE_CONC_INCR_GC` u

[jira] [Updated] (SPARK-41415) SASL Request Retries

2023-01-20 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-41415: Fix Version/s: 3.2.4 3.3.2 > SASL Request Retr

[jira] [Resolved] (SPARK-42090) Introduce sasl retry count in RetryingBlockTransferor

2023-01-16 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-42090. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-42090) Introduce sasl retry count in RetryingBlockTransferor

2023-01-16 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-42090: --- Assignee: Ted Yu > Introduce sasl retry count in RetryingBlockTransfe

[jira] [Assigned] (SPARK-41415) SASL Request Retries

2023-01-14 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-41415: --- Assignee: Aravind Patnam (was: Aravind Patnam) > SASL Request Retr

[jira] [Resolved] (SPARK-41415) SASL Request Retries

2023-01-14 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-41415. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-41415) SASL Request Retries

2023-01-14 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-41415: --- Assignee: Aravind Patnam > SASL Request Retr

[jira] [Assigned] (SPARK-40480) Remove push-based shuffle data after query finished

2023-01-14 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40480: --- Assignee: Wan Kun > Remove push-based shuffle data after query finis

[jira] [Resolved] (SPARK-40480) Remove push-based shuffle data after query finished

2023-01-14 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40480. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-33573) Server side metrics related to push-based shuffle

2023-01-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33573. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-33573) Server side metrics related to push-based shuffle

2023-01-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33573: --- Assignee: Minchu Yang > Server side metrics related to push-based shuf

[jira] [Comment Edited] (SPARK-41953) Shuffle output location refetch during shuffle migration in decommission

2023-01-09 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17656480#comment-17656480 ] Mridul Muralidharan edited comment on SPARK-41953 at 1/10/23 7:48 AM

[jira] [Commented] (SPARK-41953) Shuffle output location refetch during shuffle migration in decommission

2023-01-09 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17656480#comment-17656480 ] Mridul Muralidharan commented on SPARK-41953: - A few things: * Looking at SPARK-27637, we

[jira] [Assigned] (SPARK-36620) Client side related push-based shuffle metrics

2023-01-06 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36620: --- Assignee: Thejdeep Gudivada > Client side related push-based shuf

[jira] [Resolved] (SPARK-36620) Client side related push-based shuffle metrics

2023-01-06 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36620. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Commented] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

2023-01-04 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654590#comment-17654590 ] Mridul Muralidharan commented on SPARK-41497: - Sounds good [~Ngone51], thanks

Re: Time for Spark 3.4.0 release?

2023-01-04 Thread Mridul Muralidharan
+1, Thanks ! Regards, Mridul On Wed, Jan 4, 2023 at 2:20 AM Gengliang Wang wrote: > +1, thanks for driving the release! > > > Gengliang > > On Tue, Jan 3, 2023 at 10:55 PM Dongjoon Hyun > wrote: > >> +1 >> >> Thank you! >> >> Dongjoon >> >> On Tue, Jan 3, 2023 at 9:44 PM Rui Wang wrote: >>

[jira] [Resolved] (SPARK-41792) Shuffle merge finalization removes the wrong finalization state from the DB

2023-01-01 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-41792. - Fix Version/s: 3.4.0 Assignee: Mridul Muralidharan Resolution

[jira] [Updated] (SPARK-41792) Shuffle merge finalization removes the wrong finalization state from the DB

2023-01-01 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-41792: Affects Version/s: (was: 3.3.0) > Shuffle merge finalization remo

[jira] [Created] (SPARK-41792) Shuffle merge finalization removes the wrong finalization state from the DB

2022-12-30 Thread Mridul Muralidharan (Jira)
Mridul Muralidharan created SPARK-41792: --- Summary: Shuffle merge finalization removes the wrong finalization state from the DB Key: SPARK-41792 URL: https://issues.apache.org/jira/browse/SPARK-41792

[jira] [Resolved] (SPARK-41469) Task rerun on decommissioned executor can be avoided if shuffle data has migrated

2022-12-27 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-41469. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-41469) Task rerun on decommissioned executor can be avoided if shuffle data has migrated

2022-12-27 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-41469: --- Assignee: wuyi > Task rerun on decommissioned executor can be avoi

[jira] [Assigned] (SPARK-41192) Task finished before speculative task scheduled leads to holding idle executors

2022-12-20 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-41192: --- Assignee: Yazhi Wang > Task finished before speculative task schedu

[jira] [Resolved] (SPARK-41192) Task finished before speculative task scheduled leads to holding idle executors

2022-12-20 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-41192. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Comment Edited] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

2022-12-14 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17646972#comment-17646972 ] Mridul Muralidharan edited comment on SPARK-41497 at 12/14/22 8:03 AM

[jira] [Commented] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

2022-12-13 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17646972#comment-17646972 ] Mridul Muralidharan commented on SPARK-41497: - [~Ngone51] Agree, that is what I was not sure

[jira] [Commented] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

2022-12-13 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17646962#comment-17646962 ] Mridul Muralidharan commented on SPARK-41497: - Agree, if we can determine that - do we have

[jira] [Comment Edited] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

2022-12-13 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17646825#comment-17646825 ] Mridul Muralidharan edited comment on SPARK-41497 at 12/13/22 9:20 PM

[jira] [Comment Edited] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

2022-12-13 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17646825#comment-17646825 ] Mridul Muralidharan edited comment on SPARK-41497 at 12/13/22 9:20 PM

[jira] [Commented] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

2022-12-13 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17646825#comment-17646825 ] Mridul Muralidharan commented on SPARK-41497: - > For example, a task is constructed by `r

[jira] [Comment Edited] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

2022-12-12 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17646477#comment-17646477 ] Mridul Muralidharan edited comment on SPARK-41497 at 12/13/22 7:41 AM

[jira] [Commented] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

2022-12-12 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17646477#comment-17646477 ] Mridul Muralidharan commented on SPARK-41497: - Agree - there appears to be a bunch

[jira] [Resolved] (SPARK-41360) Avoid BlockManager re-registration if the executor has been lost

2022-12-12 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-41360. - Fix Version/s: 3.2.4 3.3.2 3.4.0

[jira] [Assigned] (SPARK-41360) Avoid BlockManager re-registration if the executor has been lost

2022-12-12 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-41360: --- Assignee: wuyi > Avoid BlockManager re-registration if the executor

[jira] [Resolved] (SPARK-41187) [Core] LiveExecutor MemoryLeak in AppStatusListener when ExecutorLost happen

2022-12-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-41187. - Fix Version/s: 3.3.2 3.4.0 Resolution: Fixed > [C

[jira] [Assigned] (SPARK-41187) [Core] LiveExecutor MemoryLeak in AppStatusListener when ExecutorLost happen

2022-12-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-41187: --- Assignee: wineternity > [Core] LiveExecutor MemoryL

Re: [VOTE][SPIP] Asynchronous Offset Management in Structured Streaming

2022-11-30 Thread Mridul Muralidharan
+1 Regards, Mridul On Wed, Nov 30, 2022 at 8:55 PM Xingbo Jiang wrote: > +1 > > On Wed, Nov 30, 2022 at 5:59 PM Jungtaek Lim > wrote: > >> Starting with +1 from me. >> >> On Thu, Dec 1, 2022 at 10:54 AM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Hi all, >>> >>> I'd like to

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

2022-11-30 Thread Mridul Muralidharan
goal of this project. If that >> happens eventually, that would be a side-effect. Someone may have concerns >> that we have two different projects aiming for similar thing, but I'd >> rather see both projects having competition. If anyone willing to improve >> continuous m

[jira] [Assigned] (SPARK-40987) Avoid creating a directory when deleting a block, causing DAGScheduler to not work

2022-11-29 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40987: --- Assignee: dzcxzl > Avoid creating a directory when deleting a bl

[jira] [Resolved] (SPARK-40987) Avoid creating a directory when deleting a block, causing DAGScheduler to not work

2022-11-29 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40987. - Fix Version/s: 3.3.2 3.2.3 3.4.0

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

2022-11-23 Thread Mridul Muralidharan
Hi Jungtaek, Given the goal of the SPIP is reducing latency for stateless apps, and should reasonably fit continuous mode design goals, it feels odd to not support it fin the proposal. I know you have raised concerns about continuous mode in past as well in dev@ list, and we are further

[jira] [Resolved] (SPARK-40872) Fallback to original shuffle block when a push-merged shuffle chunk is zero-size

2022-11-20 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40872. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-40872) Fallback to original shuffle block when a push-merged shuffle chunk is zero-size

2022-11-20 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40872: --- Assignee: gaoyajun02 > Fallback to original shuffle block when a p

Re: [VOTE][RESULT] Release Spark 3.2.3, RC1

2022-11-18 Thread Mridul Muralidharan
eh (*) > - Huaxin Gao (*) > - Kazuyuki Tanimura > - Mridul Muralidharan (*) > - Yuming Wang > - Chris Nauroth > - Yang Jie > - Wenche Fan (*) > - Ruifeng Zheng > - Chao Sun > > +0: None > > -1: None > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >

[jira] [Resolved] (SPARK-40979) Keep removed executor info in decommission state

2022-11-16 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40979. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-40979) Keep removed executor info in decommission state

2022-11-16 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40979: --- Assignee: Zhongwei Zhu > Keep removed executor info in decommission st

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

2022-11-16 Thread Mridul Muralidharan
+1 Would be great to see history server performance improvements and lower resource utilization at driver ! Regards, Mridul On Wed, Nov 16, 2022 at 2:38 AM Kent Yao wrote: > +1, non-binding > > Gengliang Wang 于2022年11月16日周三 16:36写道: > > > > Hi all, > > > > I’d like to start a vote for SPIP:

[jira] [Resolved] (SPARK-40622) Result of a single task in collect() must fit in 2GB

2022-11-15 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40622. - Fix Version/s: 3.4.0 Resolution: Fixed > Result of a single t

[jira] [Assigned] (SPARK-40622) Result of a single task in collect() must fit in 2GB

2022-11-15 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40622: --- Assignee: Ziqi Liu > Result of a single task in collect() must fit in

Re: [VOTE] Release Spark 3.2.3 (RC1)

2022-11-15 Thread Mridul Muralidharan
+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Tue, Nov 15, 2022 at 1:00 PM kazuyuki tanimura wrote: > +1 (non-binding) > > Thank you Chao > > Kazu > > >  | Kazuyuki Tanimura | ktanim...@apple.com |

[jira] [Resolved] (SPARK-40901) Unable to store Spark Driver logs with Absolute Hadoop based URI FS Path

2022-11-10 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40901. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-40901) Unable to store Spark Driver logs with Absolute Hadoop based URI FS Path

2022-11-10 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40901: --- Assignee: Swaminathan Balachandran > Unable to store Spark Driver l

[jira] [Resolved] (SPARK-41051) Optimize ProcfsMetrics file acquisition

2022-11-08 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-41051. - Fix Version/s: 3.4.0 (was: 3.3.2) Resolution

[jira] [Assigned] (SPARK-40950) isRemoteAddressMaxedOut performance overhead on scala 2.13

2022-11-04 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40950: --- Assignee: Emil Ejbyfeldt > isRemoteAddressMaxedOut performance overh

[jira] [Resolved] (SPARK-40950) isRemoteAddressMaxedOut performance overhead on scala 2.13

2022-11-04 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40950. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-40968) Fix some wrong/misleading comments in DAGSchedulerSuite

2022-11-02 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40968: --- Assignee: Jiexing Li > Fix some wrong/misleading comme

[jira] [Resolved] (SPARK-40968) Fix some wrong/misleading comments in DAGSchedulerSuite

2022-11-02 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40968. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-40320) When the Executor plugin fails to initialize, the Executor shows active but does not accept tasks forever, just like being hung

2022-10-26 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40320: --- Assignee: miracle > When the Executor plugin fails to initial

[jira] [Updated] (SPARK-40320) When the Executor plugin fails to initialize, the Executor shows active but does not accept tasks forever, just like being hung

2022-10-26 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-40320: Fix Version/s: 3.4.0 > When the Executor plugin fails to initialize, the Execu

[jira] [Resolved] (SPARK-40320) When the Executor plugin fails to initialize, the Executor shows active but does not accept tasks forever, just like being hung

2022-10-26 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40320. - Resolution: Fixed > When the Executor plugin fails to initialize, the Execu

[jira] [Updated] (SPARK-40902) Quick submission of drivers in tests to mesos scheduler results in dropping drivers

2022-10-24 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-40902: Priority: Minor (was: Major) > Quick submission of drivers in tests to me

[jira] [Commented] (SPARK-40902) Quick submission of drivers in tests to mesos scheduler results in dropping drivers

2022-10-24 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623270#comment-17623270 ] Mridul Muralidharan commented on SPARK-40902: - +CC [~dongjoon], [~yumwang] > Qu

[jira] [Created] (SPARK-40902) Quick submission of drivers in tests to mesos scheduler results in dropping drivers

2022-10-24 Thread Mridul Muralidharan (Jira)
Mridul Muralidharan created SPARK-40902: --- Summary: Quick submission of drivers in tests to mesos scheduler results in dropping drivers Key: SPARK-40902 URL: https://issues.apache.org/jira/browse/SPARK-40902

Re: [VOTE] Release Spark 3.3.1 (RC4)

2022-10-21 Thread Mridul Muralidharan
ion: > https://github.com/apache/spark/actions?query=branch%3Abranch-3.3 > Apple Silicon Jenkins Farm: > https://apache-spark.s3.fr-par.scw.cloud/BRANCH-3.3.html > > Dongjoon. > > > On Fri, Oct 21, 2022 at 8:48 AM Mridul Muralidharan > wrote: > >> Hi, >> &g

Re: [VOTE] Release Spark 3.3.1 (RC4)

2022-10-21 Thread Mridul Muralidharan
Hi, I saw a couple of test failures I have not observed before: a) FsHistoryProviderSuite - "SPARK-33146: don't let one bad rolling log folder prevent loading other applications" b) MesosClusterSchedulerSuite - "accept/decline offers with driver constraints" I ended up 'ignore''ing them to

Re: Welcome Yikun Jiang as a Spark committer

2022-10-08 Thread Mridul Muralidharan
Congratulations ! Regards, Mridul On Sat, Oct 8, 2022 at 12:19 AM Yuming Wang wrote: > Congratulations Yikun! > > On Sat, Oct 8, 2022 at 12:40 PM Hyukjin Kwon wrote: > >> Hi all, >> >> The Spark PMC recently added Yikun Jiang as a committer on the project. >> Yikun is the major contributor of

Re: [VOTE] Release Spark 3.3.1 (RC2)

2022-10-03 Thread Mridul Muralidharan
+1 from me, with a few comments. I saw the following failures, are these known issues/flakey tests ? * PersistenceEngineSuite.ZooKeeperPersistenceEngine Looks like a port conflict issue from a quick look into logs (conflict with starting admin port at 8080) - is this expected behavior for the

[jira] [Assigned] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

2022-09-22 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40096: --- Assignee: Wan Kun > Finalize shuffle merge slow due to connection creat

[jira] [Resolved] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

2022-09-22 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40096. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-38888) Add `RocksDBProvider` similar to `LevelDBProvider`

2022-09-08 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-3. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-38888) Add `RocksDBProvider` similar to `LevelDBProvider`

2022-09-08 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-3: --- Assignee: Yang Jie > Add `RocksDBProvider` similar to `LevelDBProvi

[jira] [Resolved] (SPARK-40186) mergedShuffleCleaner should have been shutdown before db closed

2022-09-07 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40186. - Target Version/s: 3.4.0 Assignee: Yang Jie Resolution

[jira] [Resolved] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

2022-08-24 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40094. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

2022-08-24 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40094: --- Assignee: wangshengjie > Send TaskEnd event when task fai

[jira] [Assigned] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

2022-08-18 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38909: --- Assignee: Yang Jie > Encapsulate LevelDB u

[jira] [Resolved] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

2022-08-18 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38909. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

2022-08-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Kai-Hsun Chen (was: Kai-Hsun Chen) > Improve LaunchTask proc

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

2022-08-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Kai-Hsun Chen (was: Mridul Muralidharan) > Improve LaunchT

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

2022-08-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Mridul Muralidharan > Improve LaunchTask process to avoid St

[jira] [Resolved] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

2022-08-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-39955. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

Re: How to set platform-level defaults for array-like configs?

2022-08-11 Thread Mridul Muralidharan
Hi, Wenchen, would be great if you could chime in with your thoughts - given the feedback you originally had on the PR. It would be great to hear feedback from others on this, particularly folks managing spark deployments - how this is mitigated/avoided in your case, any other pain points with

Re: Welcoming three new PMC members

2022-08-09 Thread Mridul Muralidharan
Congratulations ! Great to have you join the PMC !! Regards, Mridul On Tue, Aug 9, 2022 at 11:57 AM vaquar khan wrote: > Congratulations > > On Tue, Aug 9, 2022, 11:40 AM Xiao Li wrote: > >> Hi all, >> >> The Spark PMC recently voted to add three new PMC members. Join me in >> welcoming them

Re: Welcome Xinrong Meng as a Spark committer

2022-08-09 Thread Mridul Muralidharan
Congratulations Xinrong ! Regards, Mridul On Tue, Aug 9, 2022 at 3:13 AM Hyukjin Kwon wrote: > Hi all, > > The Spark PMC recently added Xinrong Meng as a committer on the project. > Xinrong is the major contributor of PySpark especially Pandas API on Spark. > She has guided a lot of new

[jira] [Resolved] (SPARK-40004) Redundant `LevelDB.get` in `RemoteBlockPushResolver`

2022-08-08 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40004. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-40004) Redundant `LevelDB.get` in `RemoteBlockPushResolver`

2022-08-08 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40004: --- Assignee: Yang Jie > Redundant `LevelDB.get` in `RemoteBlockPushResol

Re: [VOTE] Apache Bahir Flink 1.1.0 (RC2)

2022-08-02 Thread Mridul Muralidharan
Boto wrote: > Hi Mridul, > > Thanks for checking.. > Do you have installed on your computer a docker environment? > This is needed for testcontainers that we use on testing.. > > Le me know if is that > > On 2022/07/29 15:11:21 Mridul Muralidharan wrote: > > The t

Re: [VOTE] Apache Bahir Flink 1.1.0 (RC2)

2022-07-29 Thread Mridul Muralidharan
The tests failed for me with [1] I also saw this at the top of build [2], but dont think it was related. Any pointers ? Thanks, Mridul [1] [INFO] Running org.apache.flink.streaming.connectors.flume.FlumeSinkTest SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J:

[jira] [Commented] (SPARK-39696) Uncaught exception in thread executor-heartbeater java.util.ConcurrentModificationException: mutation occurred during iteration

2022-07-25 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571169#comment-17571169 ] Mridul Muralidharan commented on SPARK-39696: - Can you give more details on how this can

[jira] [Resolved] (SPARK-33236) Enable Push-based shuffle service to store state in NM level DB for work preserving restart

2022-07-25 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33236. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-33236) Enable Push-based shuffle service to store state in NM level DB for work preserving restart

2022-07-25 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33236: --- Assignee: Ye Zhou > Enable Push-based shuffle service to store state in

Re: [VOTE] Release Spark 3.2.2 (RC1)

2022-07-12 Thread Mridul Muralidharan
+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with "-Pyarn -Pmesos -Pkubernetes" As always, the test "SPARK-33084: Add jar support Ivy URI in SQL" in sql.SQLQuerySuite fails in my env; but other than that, the rest looks good. Regards, Mridul On Tue, Jul 12,

[jira] [Resolved] (SPARK-39647) Block push fails with java.lang.IllegalArgumentException: Active local dirs list has not been updated by any executor registration even when the NodeManager hasn't been

2022-07-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-39647. - Fix Version/s: 3.3.1 3.4.0 Resolution: Fixed Issue

[jira] [Assigned] (SPARK-39647) Block push fails with java.lang.IllegalArgumentException: Active local dirs list has not been updated by any executor registration even when the NodeManager hasn't been

2022-07-11 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39647: --- Assignee: Chandni Singh > Block push fa

Re: Apache Spark 3.2.2 Release?

2022-07-06 Thread Mridul Muralidharan
+1 Thanks for driving this Dongjoon ! Regards, Mridul On Thu, Jul 7, 2022 at 12:36 AM Gengliang Wang wrote: > +1. > Thank you, Dongjoon. > > On Wed, Jul 6, 2022 at 10:21 PM Wenchen Fan wrote: > >> +1 >> >> On Thu, Jul 7, 2022 at 10:41 AM Xinrong Meng >> wrote: >> >>> +1 >>> >>> Thanks! >>>

<    1   2   3   4   5   6   7   8   9   10   >