[ https://issues.apache.org/jira/browse/SPARK-31220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuming Wang updated SPARK-31220: -------------------------------- Summary: repartition obeys spark.sql.adaptive.coalescePartitions.initialPartitionNum when spark.sql.adaptive.enabled (was: distribute by obeys spark.sql.adaptive.coalescePartitions.initialPartitionNum when spark.sql.adaptive.enabled) > repartition obeys spark.sql.adaptive.coalescePartitions.initialPartitionNum > when spark.sql.adaptive.enabled > ----------------------------------------------------------------------------------------------------------- > > Key: SPARK-31220 > URL: https://issues.apache.org/jira/browse/SPARK-31220 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.1.0 > Reporter: Yuming Wang > Priority: Major > > {code:scala} > spark.sql("CREATE TABLE spark_31220(id int)") > spark.sql("set > spark.sql.adaptive.coalescePartitions.initialPartitionNum=1000") > spark.sql("set spark.sql.adaptive.enabled=true") > {code} > {noformat} > scala> spark.sql("SELECT id from spark_31220 GROUP BY id").explain > == Physical Plan == > AdaptiveSparkPlan(isFinalPlan=false) > +- HashAggregate(keys=[id#5], functions=[]) > +- Exchange hashpartitioning(id#5, 1000), true, [id=#171] > +- HashAggregate(keys=[id#5], functions=[]) > +- FileScan parquet default.spark_31220[id#5] Batched: true, > DataFilters: [], Format: Parquet, Location: > InMemoryFileIndex[file:/root/opensource/apache-spark/spark-warehouse/spark_31220], > PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int> > scala> spark.sql("SELECT id from spark_31220 DISTRIBUTE BY id").explain > == Physical Plan == > AdaptiveSparkPlan(isFinalPlan=false) > +- Exchange hashpartitioning(id#5, 200), false, [id=#179] > +- FileScan parquet default.spark_31220[id#5] Batched: true, DataFilters: > [], Format: Parquet, Location: > InMemoryFileIndex[file:/root/opensource/apache-spark/spark-warehouse/spark_31220], > PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int> > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org