[jira] [Created] (SPARK-42728) when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config

[kevinshin (Jira)]

kevinshin created SPARK-42728:
---------------------------------

             Summary: when enable AQE update or drop a small iceberg table 
should not generate as much task as 
`spark.sql.adaptive.coalescePartitions.initialPartitionNum` config
                 Key: SPARK-42728
                 URL: https://issues.apache.org/jira/browse/SPARK-42728
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.2.3, 3.2.1
         Environment: spark3.2.1 or 3.2.3; iceberg 0.14.1 or 1.1.0

 
            Reporter: kevinshin


spark config : 
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions 
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
 
Sql script: 
 
*create* *table* spark_catalog.test.iceberg5(id *bigint*, *data* string) 
*USING* iceberg;

*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 
'c');

*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;

*drop* *table* spark_catalog.test.iceberg5 PURGE;

spark web :  

!image-2023-03-09-17-33-14-940.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org