[GitHub] [spark] mridulm commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks

mridulm commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1478905577 Is your jira id `StoveM` @Stove-hust ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] mridulm commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks

mridulm commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1478833979 I could not cherry pick this into 3.4 and 3.3 - we should fix for those branches as well IMO. Can you create a PR against those two branches as well @Stove-hust ? Thanks -- This is a

[GitHub] [spark] mridulm commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks

mridulm commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1478833621 Merged to master. Thanks for working on this @Stove-hust ! Thanks for the review @otterc :-) -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] mridulm commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks

mridulm commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1476585572 The test failure is unrelated to this PR - once the changes above are made, the reexecution should pass -- This is an automated message from the Apache Git Service. To respond to the me

[GitHub] [spark] mridulm commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks

mridulm commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1475538274 Technically, 3 :-) The UT that I added will generate 2 tests - one for push based shuffle and one without. And we have the initial test you added. -- This is an automated message

[GitHub] [spark] mridulm commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks

mridulm commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1474986959 @Stove-hust To clarify - I meant add this as well. We should keep the UT you had added - and it is important to test the specific expectation as it stands today. -- This is an automa

[GitHub] [spark] mridulm commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks

mridulm commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1474782673 Instead of testing specifically for the flag - which is subject to change as the implementation evolves, we should test for behavior here. This is the reproducible test I was using

[GitHub] [spark] mridulm commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks

mridulm commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1473171295 So this is an interesting coincidence, I literally encountered a production job which seems to be hitting this exact same issue :-) I was in the process of creating a test case, but my