[
https://issues.apache.org/jira/browse/SPARK-41386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Dong updated SPARK-41386:
-----------------------------
Description:
{*}Problem (/*+ REBALANCE(bot_mid) */){*}:
sparksession config:
{noformat}
config("spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled", "true")
config("spark.sql.adaptive.advisoryPartitionSizeInBytes", "20m")
config("spark.sql.adaptive.rebalancePartitionsSmallPartitionFactor",
"0.5"){noformat}
was:TODO:
> There are some small files when using rebalance(column)
> -------------------------------------------------------
>
> Key: SPARK-41386
> URL: https://issues.apache.org/jira/browse/SPARK-41386
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Zhe Dong
> Priority: Minor
>
> {*}Problem (/*+ REBALANCE(bot_mid) */){*}:
> sparksession config:
> {noformat}
> config("spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled",
> "true")
> config("spark.sql.adaptive.advisoryPartitionSizeInBytes", "20m")
> config("spark.sql.adaptive.rebalancePartitionsSmallPartitionFactor",
> "0.5"){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
