[
https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424052#comment-17424052
]
Mridul Muralidharan edited comment on SPARK-36892 at 10/4/21, 5:26 PM:
-----------------------------------------------------------------------
[~Gengliang.Wang] Thanks for holding the release !
The team worked on testing the RC and fixed two issues:
* SPARK-36705 (https://github.com/apache/spark/pull/34158) and
* SPARK-36892 (https://github.com/apache/spark/pull/34156).
With these two fixes on top of RC6, in addition to internal tests, we had all
queries for TPCDS (scale 100) completing successfully and correctly.
There was an issue identified with mismatch for TPCH query 22 - [~apatnam] has
filed a jira (SPARK-36926) for it.
was (Author: mridulm80):
[~Gengliang.Wang] Thanks for holding the release !
The team worked on testing the RC and fixed two issues:
* SPARK-36705 (https://github.com/apache/spark/pull/34158) and
* SPARK-36892 (https://github.com/apache/spark/pull/34156).
With these two fixes on top of RC6, in addition to internal tests, we had all
queries for TPCDS (scale 100) completing successfully and correctly.
There was an issue identified with mismatch for TPCH query foo - [~apatnam] has
filed a jira (SPARK-36926) for it.
> Disable batch fetch for a shuffle when push based shuffle is enabled
> --------------------------------------------------------------------
>
> Key: SPARK-36892
> URL: https://issues.apache.org/jira/browse/SPARK-36892
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 3.2.0
> Reporter: Mridul Muralidharan
> Priority: Blocker
>
> When push based shuffle is enabled, efficient fetch of merged mapper shuffle
> output happens.
> Unfortunately, this currently interacts badly with
> spark.sql.adaptive.fetchShuffleBlocksInBatch, potentially causing shuffle
> fetch to hang and/or duplicate data to be fetched, causing correctness issues.
> Given batch fetch does not benefit spark stages reading merged blocks when
> push based shuffle is enabled, ShuffleBlockFetcherIterator.doBatchFetch can
> be disabled when push based shuffle is enabled.
> Thx to [~Ngone51] for surfacing this issue.
> +CC [~Gengliang.Wang]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
