[GitHub] [spark] Ngone51 commented on pull request #32385: [WIP][SPARK-35275][CORE] Add checksum for shuffle blocks and diagnose corruption

Ngone51 commented on pull request #32385: URL: https://github.com/apache/spark/pull/32385#issuecomment-856938701 Hi @tgravescs @mridulm @otterc , I have resolved the regression issue and made the checksum as a built-in feature of Spark. And I have updated PR #32401 (which adds c

[GitHub] [spark] Ngone51 commented on pull request #32385: [WIP][SPARK-35275][CORE] Add checksum for shuffle blocks and diagnose corruption

Ngone51 commented on pull request #32385: URL: https://github.com/apache/spark/pull/32385#issuecomment-857796097 Sure, take your time :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [spark] Ngone51 commented on pull request #32385: [WIP][SPARK-35275][CORE] Add checksum for shuffle blocks and diagnose corruption

Ngone51 commented on pull request #32385: URL: https://github.com/apache/spark/pull/32385#issuecomment-861274556 oh..@mridulm Sorry if I confused you here. I have planed to split this PR into two separate PRs to ease the review: * write checksum file (Ready to review https://github.

[GitHub] [spark] Ngone51 commented on pull request #32385: [WIP][SPARK-35275][CORE] Add checksum for shuffle blocks and diagnose corruption

Ngone51 commented on pull request #32385: URL: https://github.com/apache/spark/pull/32385#issuecomment-843722683 @tgravescs Thanks for the good points! I did find some perf regression by benchmarking with the change. I'll double-check it for sure and try to get rid of it if possible.