[
https://issues.apache.org/jira/browse/SPARK-31220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-31220:
----------------------------------
Fix Version/s: (was: 3.0.0)
3.1.0
3.0.1
> repartition obeys spark.sql.adaptive.coalescePartitions.initialPartitionNum
> when spark.sql.adaptive.enabled
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-31220
> URL: https://issues.apache.org/jira/browse/SPARK-31220
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Yuming Wang
> Assignee: Yuming Wang
> Priority: Major
> Fix For: 3.0.1, 3.1.0
>
>
> {code:scala}
> spark.sql("CREATE TABLE spark_31220(id int)")
> spark.sql("set
> spark.sql.adaptive.coalescePartitions.initialPartitionNum=1000")
> spark.sql("set spark.sql.adaptive.enabled=true")
> {code}
> {noformat}
> scala> spark.sql("SELECT id from spark_31220 GROUP BY id").explain
> == Physical Plan ==
> AdaptiveSparkPlan(isFinalPlan=false)
> +- HashAggregate(keys=[id#5], functions=[])
> +- Exchange hashpartitioning(id#5, 1000), true, [id=#171]
> +- HashAggregate(keys=[id#5], functions=[])
> +- FileScan parquet default.spark_31220[id#5] Batched: true,
> DataFilters: [], Format: Parquet, Location:
> InMemoryFileIndex[file:/root/opensource/apache-spark/spark-warehouse/spark_31220],
> PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>
> scala> spark.sql("SELECT id from spark_31220 DISTRIBUTE BY id").explain
> == Physical Plan ==
> AdaptiveSparkPlan(isFinalPlan=false)
> +- Exchange hashpartitioning(id#5, 200), false, [id=#179]
> +- FileScan parquet default.spark_31220[id#5] Batched: true, DataFilters:
> [], Format: Parquet, Location:
> InMemoryFileIndex[file:/root/opensource/apache-spark/spark-warehouse/spark_31220],
> PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
