[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r655050720 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -661,16 +744,29 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r655047442 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r655046357 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r655046020 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -871,6 +1047,81 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r655045474 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -871,6 +1047,81 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r655045176 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r655045134 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r655044846 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r655034874 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648845832 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1392,298 @@ object

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648845832 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1392,298 @@ object

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649603473 ## File path: core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala ## @@ -22,31 +22,40 @@ import java.nio.ByteBuffer

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649548771 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1394,54 @@ object ShuffleBlockFetcherIterator

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649514612 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -436,24 +485,48 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649505327 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -347,77 +355,118 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649482176 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649479840 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649479384 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649472914 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649457152 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649449168 ## File path: core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala ## @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649435207 ## File path: core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala ## @@ -22,31 +22,40 @@ import java.nio.ByteBuffer

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649419829 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -347,20 +361,48 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r649419059 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -347,77 +355,118 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648845832 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1392,298 @@ object

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648845832 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1392,298 @@ object

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648847338 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -767,6 +908,43 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648845832 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1392,298 @@ object

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648705513 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -436,24 +487,51 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648537705 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -347,20 +361,48 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648520952 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -767,6 +908,43 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648685526 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1392,298 @@ object

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648683194 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1392,298 @@ object

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648683013 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1392,298 @@ object

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648682648 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -767,6 +908,43 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648594657 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1392,298 @@ object

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648595988 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1074,8 +1337,13 @@ object ShuffleBlockFetcherIterator

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648594657 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1124,4 +1392,298 @@ object

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648567584 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -376,48 +418,62 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648537705 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -347,20 +361,48 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648520952 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -767,6 +908,43 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648496347 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -1074,8 +1337,13 @@ object ShuffleBlockFetcherIterator

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r648493846 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -347,20 +361,48 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r646965608 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java ## @@ -413,6 +466,47 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r646965372 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java ## @@ -333,14 +382,18 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r646807905 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r646807905 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r646807905 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645907997 ## File path: common/network-common/src/main/java/org/apache/spark/network/client/BaseResponseCallback.java ## @@ -0,0 +1,31 @@ +/* + * Licensed to the

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645904465 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645901855 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -276,9 +342,13 @@ public void

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645902345 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645902038 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645901943 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645901796 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -246,6 +304,14 @@ private void

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645870074 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -246,6 +304,14 @@ private void

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645901032 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645900987 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645900747 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645900543 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java ## @@ -413,6 +466,47 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645895089 ## File path: core/src/main/scala/org/apache/spark/storage/BlockId.scala ## @@ -124,11 +134,12 @@ class UnrecognizedBlockId(name: String) @DeveloperApi

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645880908 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/FetchShuffleBlockChunks.java ## @@ -0,0 +1,126 @@ +/* + *

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645872258 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -276,9 +342,13 @@ public void

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645870074 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -246,6 +304,14 @@ private void

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645727804 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645718534 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645717463 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645709830 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645679706 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java ## @@ -88,82 +93,124 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645674391 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java ## @@ -413,6 +466,47 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r645664320 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java ## @@ -333,14 +382,18 @@ public

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r640210756 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -712,38 +824,66 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r640210756 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -712,38 +824,66 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r640322571 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -742,10 +742,10 @@ private[spark] class MapOutputTrackerMaster( *

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r640321817 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -720,7 +720,7 @@ private[spark] class MapOutputTrackerMaster( } }

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r640312469 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -108,13 +115,6 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r640209621 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -347,20 +360,48 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r640210756 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -712,38 +824,66 @@ final class

[GitHub] [spark] otterc commented on a change in pull request #32140: [WIP][SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data

otterc commented on a change in pull request #32140: URL: https://github.com/apache/spark/pull/32140#discussion_r640209621 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -347,20 +360,48 @@ final class