[GitHub] [spark] Ngone51 commented on pull request #32385: [WIP][SPARK-18188][CORE] Add checksum for shuffle blocks

2021-04-28 Thread GitBox
Ngone51 commented on pull request #32385: URL: https://github.com/apache/spark/pull/32385#issuecomment-828568544 I marked PR as `WIP` because I want to hear the community's feedback before working further, e.g., adding more unit tests. And there will be two following PRs (if the communit

[GitHub] [spark] Ngone51 commented on pull request #32385: [WIP][SPARK-18188][CORE] Add checksum for shuffle blocks

2021-04-28 Thread GitBox
Ngone51 commented on pull request #32385: URL: https://github.com/apache/spark/pull/32385#issuecomment-828569773 cc @mridulm @otterc @attilapiros @tgravescs @cloud-fan Please take a look, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [spark] Ngone51 commented on pull request #32385: [WIP][SPARK-18188][CORE] Add checksum for shuffle blocks and diagnose corruption

2021-04-29 Thread GitBox
Ngone51 commented on pull request #32385: URL: https://github.com/apache/spark/pull/32385#issuecomment-829384725 Hi all, to ease the review for everyone, I have planned to split these PRs into 2 smaller PR first: 1) calculate checksums and save them into a checksum file (I hav