[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-736067925 Thank you so much, @holdenk and @viirya . Merged to master for Apache Spark 3.1.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-735912350 Hi, @holdenk and @viirya and @dbtsai . Could you review this, please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-735552719 Got it, @zsxwing . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-735473004 Hi, @zsxwing and @viirya . I addressed your comments. Could you review once more please if you have a chance? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-734527796 Thank you so much for your review, @viirya ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733906102 BTW, cc @dbtsai , too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733905003 Thank you, @holdenk . Sorry for pinging you on the holiday season. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733866284 Could you review this PR once more please, @viirya ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733837333 Thank you for review, @Tagar . Yes, they are separate options: - `spark.storage.decommission.shuffleBlocks.enabled` - `spark.storage.decommission.rddBlocks.enabled` This PR is still under review. If this is accepted, I believe we can move on to that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733492363 It's a little weird because it passed locally. ``` $ build/sbt "core/testOnly *.FallbackStorageSuite" ... [info] FallbackStorageSuite: [info] - fallback storage APIs - copy/exists (957 milliseconds) [info] - migrate shuffle data to fallback storage (224 milliseconds) [info] - Upload from all decommissioned executors (5 seconds, 232 milliseconds) [info] - Upload multi stages (3 seconds, 67 milliseconds) [info] - Newly added executors should access old data from remote storage (8 seconds, 415 milliseconds) [info] ScalaTest [info] Run completed in 18 seconds, 975 milliseconds. [info] Total number of tests run: 5 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 5, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733260121 Could you review this, @holdenk , @viirya , @mridulm ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
