[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-858861907 Updated to the latest master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-856517863 Added two unit tests for IndexShuffleBlockResolver getMergedBlockData and getMergedBlockMeta -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-855593936 Address most of the review comments. The unit test for getLocalMergedBlockData and getLocalMergedBlockMeta are yet to be added. I think we should add the unit tests for

[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-853615497 Updated with a slim version, which excludes the handling for multiple attempts case. @Ngone51 Would like to share a little bit more context. We had multiple round of

[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-853615497 Updated with a slim version, which excludes the handling for multiple attempts case. @Ngone51 Would like to share a little bit more context. We had multiple round of

[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-850036241 Created ticket for later improvement https://issues.apache.org/jira/browse/SPARK-35546 -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-849395958 @mridulm @Victsm @otterc @Ngone51 Updated the PR. Please help review. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-846476480 @otterc @mridulm @Ngone51 Addressed the comments. Please review. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-846476449 > 1. `RemoteBlockPushResolver` needs to ignore any `PushBlock` message that is from previous attempts otherwise it will still merge a block of previous attempt to files of

[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-823463844 @tgravescs @Ngone51 @attilapiros Can you help review this PR? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] zhouyejoe commented on pull request #32007: [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

zhouyejoe commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-820954486 Thanks for reviewing the PR. I am addressing the comments, will update the PR soon. -- This is an automated message from the Apache Git Service. To respond to the message,