Michael Munday created SPARK-32952:
--------------------------------------
Summary: Test failure on IBM Z: CoalesceShufflePartitionsSuite: -
determining the number of reducers: complex query 1
Key: SPARK-32952
URL: https://issues.apache.org/jira/browse/SPARK-32952
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.1
Environment: *{color:#FF0000}{color}*Linux on IBM Z (s390x).
Reporter: Michael Munday
I'm seeing the test 'CoalesceShufflePartitionsSuite: - determining the number
of reducers: complex query 1' fail on IBM Z with the wrong number of partitions
(1 instead of 2). It's strange because none of the other tests fail.
I'd be grateful for any hints as to how the number of partitions is calculated.
Could that calculation be affected by incorrect unsafe code? Is there a way to
trace the calculation?
{noformat}
CoalesceShufflePartitionsSuite:
- determining the number of reducers: aggregate
operator(minNumPostShufflePartitions: 5)
- determining the number of reducers: join
operator(minNumPostShufflePartitions: 5)
- determining the number of reducers: complex query
1(minNumPostShufflePartitions: 5)
- determining the number of reducers: complex query
2(minNumPostShufflePartitions: 5)
- determining the number of reducers: plan already
partitioned(minNumPostShufflePartitions: 5)
- determining the number of reducers: aggregate operator
- determining the number of reducers: join operator
- determining the number of reducers: complex query 1 *** FAILED ***
1 did not equal 2 (CoalesceShufflePartitionsSuite.scala:221)
- determining the number of reducers: complex query 2
- determining the number of reducers: plan already partitioned
- SPARK-24705 adaptive query execution works correctly when exchange reuse
enabled
- Do not reduce the number of shuffle partition for repartition
- Union two datasets with different pre-shuffle partition number{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
