[ https://issues.apache.org/jira/browse/SPARK-34534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
haiyangyu updated SPARK-34534: ------------------------------ Attachment: image-2021-02-25-11-31-59-110.png > New protocol FetchShuffleBlocks in OneForOneBlockFetcher lead to data loss or > correctness > ----------------------------------------------------------------------------------------- > > Key: SPARK-34534 > URL: https://issues.apache.org/jira/browse/SPARK-34534 > Project: Spark > Issue Type: Bug > Components: Shuffle > Affects Versions: 3.0.0, 3.0.1, 3.0.2 > Reporter: haiyangyu > Priority: Major > Labels: Correctness, data-loss > Attachments: image-2021-02-25-11-17-12-714.png, > image-2021-02-25-11-27-34-429.png, image-2021-02-25-11-28-31-255.png, > image-2021-02-25-11-30-03-834.png, image-2021-02-25-11-31-59-110.png > > > We will build a new rpc message `FetchShuffleBlocks` when > `OneForOneBlockFetcher` init in replace of > `OpenBlocks` to use adaptive feature, this introduce additional problems as > follows. > `OneForOneBlockFetcher` will init a `blockIds` String array to catch chunk > fetch success, it will use index in `blockIds` to fetch blocks and match > blockId in `blockIds` when chunk data return. So the `blockIds` 's order must > be consistent with fetchChunk index. > !image-2021-02-25-11-17-12-714.png|width=875,height=502! -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org