[jira] [Created] (SPARK-38093) Set shuffleMergeAllowed to false for a determinate stage after the stage is finalized

2022-02-02 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-38093: --- Summary: Set shuffleMergeAllowed to false for a determinate stage after the stage is finalized Key: SPARK-38093 URL: https://issues.apache.org/jira/b

[jira] [Created] (SPARK-38092) Check if shuffleMergeId is the same as the current stage's shuffleMergeId before registering MergeStatus

2022-02-02 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-38092: --- Summary: Check if shuffleMergeId is the same as the current stage's shuffleMergeId before registering MergeStatus Key: SPARK-38092 URL: https://issue

[jira] [Created] (SPARK-38064) Push-based shuffle's internal implementation details should not be exposed as API

2022-01-28 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-38064: --- Summary: Push-based shuffle's internal implementation details should not be exposed as API Key: SPARK-38064 URL: https://issues.apache.org/jira/brows

[jira] [Created] (SPARK-37964) Replace usages of slaveTracker to workerTracker in MapOutputTrackerSuite

2022-01-19 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-37964: --- Summary: Replace usages of slaveTracker to workerTracker in MapOutputTrackerSuite Key: SPARK-37964 URL: https://issues.apache.org/jira/browse/SPARK-3

[jira] [Commented] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

2021-10-04 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424265#comment-17424265 ] Venkata krishnan Sowrirajan commented on SPARK-36926: - This looks li

[jira] [Comment Edited] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

2021-10-04 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424112#comment-17424112 ] Venkata krishnan Sowrirajan edited comment on SPARK-36926 at 10/4/21, 7:44 PM: ---

[jira] [Commented] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

2021-10-04 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424112#comment-17424112 ] Venkata krishnan Sowrirajan commented on SPARK-36926: - I tried to ge

[jira] [Comment Edited] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

2021-10-04 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424080#comment-17424080 ] Venkata krishnan Sowrirajan edited comment on SPARK-36926 at 10/4/21, 6:53 PM: ---

[jira] [Comment Edited] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

2021-10-04 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424080#comment-17424080 ] Venkata krishnan Sowrirajan edited comment on SPARK-36926 at 10/4/21, 6:05 PM: ---

[jira] [Commented] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

2021-10-04 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424080#comment-17424080 ] Venkata krishnan Sowrirajan commented on SPARK-36926: - The issue see

[jira] [Comment Edited] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

2021-10-04 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424080#comment-17424080 ] Venkata krishnan Sowrirajan edited comment on SPARK-36926 at 10/4/21, 6:04 PM: ---

[jira] [Updated] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

2021-10-04 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata krishnan Sowrirajan updated SPARK-36926: Description: When running TPCH scale 100 against 3.2, Query 22 has

[jira] [Created] (SPARK-36908) Document speculation metrics added as part of SPARK-36038

2021-10-01 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-36908: --- Summary: Document speculation metrics added as part of SPARK-36038 Key: SPARK-36908 URL: https://issues.apache.org/jira/browse/SPARK-36908

[jira] [Commented] (SPARK-36810) Handle HDSF read inconsistencies on Spark when observer Namenode is used

2021-09-20 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417769#comment-17417769 ] Venkata krishnan Sowrirajan commented on SPARK-36810: - I am currentl

[jira] [Created] (SPARK-36810) Handle HDSF read inconsistencies on Spark when observer Namenode is used

2021-09-20 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-36810: --- Summary: Handle HDSF read inconsistencies on Spark when observer Namenode is used Key: SPARK-36810 URL: https://issues.apache.org/jira/browse/SPARK-3

[jira] [Commented] (SPARK-36558) Stage has all tasks finished but with ongoing finalization can cause job hang

2021-08-23 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403508#comment-17403508 ] Venkata krishnan Sowrirajan commented on SPARK-36558: - One thing I m

[jira] [Commented] (SPARK-36558) Stage has all tasks finished but with ongoing finalization can cause job hang

2021-08-23 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403507#comment-17403507 ] Venkata krishnan Sowrirajan commented on SPARK-36558: - [~Ngone51] I

[jira] [Commented] (SPARK-36558) Stage has all tasks finished but with ongoing finalization can cause job hang

2021-08-23 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403404#comment-17403404 ] Venkata krishnan Sowrirajan commented on SPARK-36558: - [~Ngone51] I

[jira] [Resolved] (SPARK-36484) Handle Stale block fetch failure on the client side by not retrying the requests

2021-08-16 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata krishnan Sowrirajan resolved SPARK-36484. - Resolution: Won't Fix Thinking more about it, it doesn't seem to

[jira] [Created] (SPARK-36484) Handle Stale block fetch failure on the client side by not retrying the requests

2021-08-11 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-36484: --- Summary: Handle Stale block fetch failure on the client side by not retrying the requests Key: SPARK-36484 URL: https://issues.apache.org/jira/browse

[jira] [Created] (SPARK-36460) Pull out NoOpMergedShuffleFileManager inner class outside

2021-08-09 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-36460: --- Summary: Pull out NoOpMergedShuffleFileManager inner class outside Key: SPARK-36460 URL: https://issues.apache.org/jira/browse/SPARK-36460

[jira] [Created] (SPARK-36374) Push-based shuffle documentation

2021-08-01 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-36374: --- Summary: Push-based shuffle documentation Key: SPARK-36374 URL: https://issues.apache.org/jira/browse/SPARK-36374 Project: Spark

[jira] [Created] (SPARK-36332) Cleanup RemoteBlockPushResolver log messages

2021-07-28 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-36332: --- Summary: Cleanup RemoteBlockPushResolver log messages Key: SPARK-36332 URL: https://issues.apache.org/jira/browse/SPARK-36332 Project: Sp

[jira] [Created] (SPARK-36317) PruneFileSourcePartitionsSuite tests are failing after the fix to SPARK-36136

2021-07-27 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-36317: --- Summary: PruneFileSourcePartitionsSuite tests are failing after the fix to SPARK-36136 Key: SPARK-36317 URL: https://issues.apache.org/jira/browse/SP

[jira] [Created] (SPARK-36038) Basic speculation metrics at stage level

2021-07-07 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-36038: --- Summary: Basic speculation metrics at stage level Key: SPARK-36038 URL: https://issues.apache.org/jira/browse/SPARK-36038 Project: Spark

[jira] [Updated] (SPARK-35613) Cache commonly occurring strings from SQLMetrics, JsonProtocol and AccumulatorV2

2021-06-02 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata krishnan Sowrirajan updated SPARK-35613: Summary: Cache commonly occurring strings from SQLMetrics, JsonProt

[jira] [Created] (SPARK-35613) Cache commonly occurring strings from SQLMetrics and in JsonProtocol

2021-06-02 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-35613: --- Summary: Cache commonly occurring strings from SQLMetrics and in JsonProtocol Key: SPARK-35613 URL: https://issues.apache.org/jira/browse/SPARK-35613

[jira] [Created] (SPARK-35549) Register merge status even after shuffle dependency is merge finalized

2021-05-27 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-35549: --- Summary: Register merge status even after shuffle dependency is merge finalized Key: SPARK-35549 URL: https://issues.apache.org/jira/browse/SPARK-355

[jira] [Created] (SPARK-35547) Support push based shuffle when barrier scheduling is enabled

2021-05-27 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-35547: --- Summary: Support push based shuffle when barrier scheduling is enabled Key: SPARK-35547 URL: https://issues.apache.org/jira/browse/SPARK-35547

[jira] [Created] (SPARK-35536) Cancel finalizing the shuffle merge if the stage is cancelled while waiting until shuffle merge finalize wait time.

2021-05-26 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-35536: --- Summary: Cancel finalizing the shuffle merge if the stage is cancelled while waiting until shuffle merge finalize wait time. Key: SPARK-35536 URL: ht

[jira] [Created] (SPARK-35036) Improve push based shuffle to work with AQE by fetching partial map indexes for a reduce partition

2021-04-12 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-35036: --- Summary: Improve push based shuffle to work with AQE by fetching partial map indexes for a reduce partition Key: SPARK-35036 URL: https://issues.apac

[jira] [Created] (SPARK-34826) Adaptive fetch of shuffle mergers for Push based shuffle

2021-03-22 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-34826: --- Summary: Adaptive fetch of shuffle mergers for Push based shuffle Key: SPARK-34826 URL: https://issues.apache.org/jira/browse/SPARK-34826

[jira] [Commented] (SPARK-33778) Allow typesafe join for LeftSemi and LeftAnti

2020-12-15 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249873#comment-17249873 ] Venkata krishnan Sowrirajan commented on SPARK-33778: - Thanks [~ange

[jira] [Created] (SPARK-33778) Allow typesafe join for LeftSemi and LeftAnti

2020-12-14 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-33778: --- Summary: Allow typesafe join for LeftSemi and LeftAnti Key: SPARK-33778 URL: https://issues.apache.org/jira/browse/SPARK-33778 Project: S

[jira] [Created] (SPARK-33481) Better heuristics to compute number of shuffle mergers required for a ShuffleMapStage

2020-11-18 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-33481: --- Summary: Better heuristics to compute number of shuffle mergers required for a ShuffleMapStage Key: SPARK-33481 URL: https://issues.apache.org/jira/b

[jira] [Created] (SPARK-33329) Pluggable API to fetch shuffle merger locations with Push based shuffle

2020-11-03 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-33329: --- Summary: Pluggable API to fetch shuffle merger locations with Push based shuffle Key: SPARK-33329 URL: https://issues.apache.org/jira/browse/SPARK-33

[jira] [Created] (SPARK-32596) Clear Ivy resolution files as part of the finally block

2020-08-11 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-32596: --- Summary: Clear Ivy resolution files as part of the finally block Key: SPARK-32596 URL: https://issues.apache.org/jira/browse/SPARK-32596

[jira] [Commented] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

2020-04-13 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082731#comment-17082731 ] Venkata krishnan Sowrirajan commented on SPARK-31418: - [~tgraves] Cu

[jira] [Commented] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

2020-04-10 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17080942#comment-17080942 ] Venkata krishnan Sowrirajan commented on SPARK-31418: - Currently, I'

[jira] [Updated] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

2020-04-10 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata krishnan Sowrirajan updated SPARK-31418: Description: With Spark blacklisting, if a task fails on an execut

[jira] [Created] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

2020-04-10 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created SPARK-31418: --- Summary: Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation Key: SPARK-31418 URL: https://issue

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-08 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078520#comment-17078520 ] Venkata krishnan Sowrirajan commented on SPARK-22148: - Thanks for yo

[jira] [Comment Edited] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-07 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077674#comment-17077674 ] Venkata krishnan Sowrirajan edited comment on SPARK-22148 at 4/8/20, 12:04 AM: ---

[jira] [Comment Edited] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-07 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077674#comment-17077674 ] Venkata krishnan Sowrirajan edited comment on SPARK-22148 at 4/8/20, 12:04 AM: ---

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-07 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077674#comment-17077674 ] Venkata krishnan Sowrirajan commented on SPARK-22148: - Thanks for re

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-07 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077603#comment-17077603 ] Venkata krishnan Sowrirajan commented on SPARK-22148: - [~irashid][~D

[jira] [Created] (SPARK-27338) Deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator

2019-04-01 Thread Venkata krishnan Sowrirajan (JIRA)
Venkata krishnan Sowrirajan created SPARK-27338: --- Summary: Deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator Key: SPARK-27338 URL: https://issues.apache.org/jira/browse/SP

[jira] [Commented] (SPARK-27338) Deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator

2019-04-01 Thread Venkata krishnan Sowrirajan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806946#comment-16806946 ] Venkata krishnan Sowrirajan commented on SPARK-27338: - We found this

[jira] [Commented] (SPARK-26894) Fix Alias handling in AggregateEstimation

2019-03-21 Thread Venkata krishnan Sowrirajan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798154#comment-16798154 ] Venkata krishnan Sowrirajan commented on SPARK-26894: - Thanks for me

[jira] [Commented] (SPARK-26894) Fix Alias handling in AggregateEstimation

2019-02-15 Thread Venkata krishnan Sowrirajan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769773#comment-16769773 ] Venkata krishnan Sowrirajan commented on SPARK-26894: - Need to assig

[jira] [Created] (SPARK-26894) Fix Alias handling in AggregateEstimation

2019-02-15 Thread Venkata krishnan Sowrirajan (JIRA)
Venkata krishnan Sowrirajan created SPARK-26894: --- Summary: Fix Alias handling in AggregateEstimation Key: SPARK-26894 URL: https://issues.apache.org/jira/browse/SPARK-26894 Project: Spark

[jira] [Commented] (SPARK-14307) Alluxio Epic JIRA

2016-03-31 Thread Venkata krishnan Sowrirajan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220423#comment-15220423 ] Venkata krishnan Sowrirajan commented on SPARK-14307: - Invalid JIRA.

[jira] [Created] (SPARK-14307) Alluxio Epic JIRA

2016-03-31 Thread Venkata krishnan Sowrirajan (JIRA)
Venkata krishnan Sowrirajan created SPARK-14307: --- Summary: Alluxio Epic JIRA Key: SPARK-14307 URL: https://issues.apache.org/jira/browse/SPARK-14307 Project: Spark Issue Type:

[jira] [Closed] (SPARK-14307) Alluxio Epic JIRA

2016-03-31 Thread Venkata krishnan Sowrirajan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata krishnan Sowrirajan closed SPARK-14307. --- Resolution: Fixed > Alluxio Epic JIRA > -- > >