[
https://issues.apache.org/jira/browse/SPARK-42728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kevinshin updated SPARK-42728:
------------------------------
Description:
spark config :
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
Sql script:
*create* *table* spark_catalog.test.iceberg5(id {*}bigint{*}, *data* string)
*USING* iceberg;
*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3,
'c');
*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;
*drop* *table* spark_catalog.test.iceberg5 PURGE;
spark web :
!image-2023-03-09-17-34-26-428.png!
was:
spark config :
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
Sql script:
*create* *table* spark_catalog.test.iceberg5(id *bigint*, *data* string)
*USING* iceberg;
*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3,
'c');
*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;
*drop* *table* spark_catalog.test.iceberg5 PURGE;
spark web :
!image-2023-03-09-17-33-14-940.png!
> when enable AQE update or drop a small iceberg table should not generate as
> much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum`
> config
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-42728
> URL: https://issues.apache.org/jira/browse/SPARK-42728
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.1, 3.2.3
> Environment: spark3.2.1 or 3.2.3; iceberg 0.14.1 or 1.1.0
>
> Reporter: kevinshin
> Priority: Minor
> Attachments: image-2023-03-09-17-34-26-428.png
>
>
> spark config :
> spark.sql.adaptive.enabled true
> spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
> spark.sql.adaptive.coalescePartitions.enabled true
> spark.sql.adaptive.coalescePartitions.minPartitionNum 1
> spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
> spark.sql.extensions
> org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
> spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
> spark.sql.catalog.spark_catalog.type hive
>
> Sql script:
>
> *create* *table* spark_catalog.test.iceberg5(id {*}bigint{*}, *data* string)
> *USING* iceberg;
> *INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3,
> 'c');
> *update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;
> *drop* *table* spark_catalog.test.iceberg5 PURGE;
> spark web :
> !image-2023-03-09-17-34-26-428.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
