kevinshin created SPARK-42728: --------------------------------- Summary: when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config Key: SPARK-42728 URL: https://issues.apache.org/jira/browse/SPARK-42728 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.2.3, 3.2.1 Environment: spark3.2.1 or 3.2.3; iceberg 0.14.1 or 1.1.0
Reporter: kevinshin spark config : spark.sql.adaptive.enabled true spark.sql.adaptive.advisoryPartitionSizeInBytes 256m spark.sql.adaptive.coalescePartitions.enabled true spark.sql.adaptive.coalescePartitions.minPartitionNum 1 spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000 spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog spark.sql.catalog.spark_catalog.type hive Sql script: *create* *table* spark_catalog.test.iceberg5(id *bigint*, *data* string) *USING* iceberg; *INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 'c'); *update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3; *drop* *table* spark_catalog.test.iceberg5 PURGE; spark web : !image-2023-03-09-17-33-14-940.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org