[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-891117883 Thank you, everybody! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-890675274 @mridulm It's only about AppVeyor. Other tests can pass in GA. BTW, AFAIK, some tests today only run in GA, e.g, `pyspark-pandas`. -- This is an automated message from the Apa

[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-890671843 Ok I find some efforts to fix it in the community: https://github.com/apache/spark/pull/32961, https://github.com/apache/spark/pull/33180 -- This is an automated message from

[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-890670704 Not sure if the `continuous-integration/appveyor/pr` failure is relevant but it's been failed two times. @mridulm Do you have any idea? -- This is an automated message

[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-889569485 I'm working on the test failures. It's relevant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-888031775 Sure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-887652732 > There doesn't seem to by any UTs added to ShuffleBlockFetchIteratorSuite. Sure. I'll add there. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-886839493 FYI, there's a major change after addressing https://github.com/apache/spark/pull/33451#discussion_r676354691: Previously, we'd diagnose corruption when the first corrupti

[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-885707378 @otterc @mridulm Thanks for the review. I'll try to address them soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-885707103 @otterc > Though it avoids re-fetch of a corrupted block for which the cause of corruption is disk_issue, the act of finding the cause of corruption, which is by sending

[GitHub] [spark] Ngone51 commented on pull request #33451: [SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum

Ngone51 commented on pull request #33451: URL: https://github.com/apache/spark/pull/33451#issuecomment-883876789 cc @mridulm @tgravescs @otterc @cloud-fan Please help review, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G