[GitHub] [spark] wankunde commented on pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

wankunde commented on PR #37533: URL: https://github.com/apache/spark/pull/37533#issuecomment-1242234794 I'm sorry for the late reply, I have updated the code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [spark] wankunde commented on pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

wankunde commented on PR #37533: URL: https://github.com/apache/spark/pull/37533#issuecomment-1229847247 Hi, @mridulm @otterc New processing flow: * Send all the finalize RPC tasks and wait for merged status in a new thread pool. * Wait for `shuffleMergeResultsTimeoutSec` in main t

[GitHub] [spark] wankunde commented on pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

wankunde commented on PR #37533: URL: https://github.com/apache/spark/pull/37533#issuecomment-1226729326 Hi, @otterc @mridulm , I updated the code, could you help to review the new code? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] wankunde commented on pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

wankunde commented on PR #37533: URL: https://github.com/apache/spark/pull/37533#issuecomment-1225359818 > So the issue is that the wait period timer doesn't take into account the time for connection creation which is a bug. However, in this PR you are adding another major change of excludi

[GitHub] [spark] wankunde commented on pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

wankunde commented on PR #37533: URL: https://github.com/apache/spark/pull/37533#issuecomment-1224250892 Hi, @otterc could you help me to review this PR? Or should I provide more information on this issue ? -- This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [spark] wankunde commented on pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

wankunde commented on PR #37533: URL: https://github.com/apache/spark/pull/37533#issuecomment-1220198154 > @wankunde > > > Send finalize RPCs will block the main thread due to creating connection to some unreachable nodes. > > Which main thread are you referring to here? Could

[GitHub] [spark] wankunde commented on pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

wankunde commented on PR #37533: URL: https://github.com/apache/spark/pull/37533#issuecomment-1218157844 Hi, @otterc Could you help to review this PR ? thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo