[
https://issues.apache.org/jira/browse/SPARK-29831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh resolved SPARK-29831.
---------------------------------
Resolution: Won't Fix
> Scan Hive partitioned table should not dramatically increase data parallelism
> -----------------------------------------------------------------------------
>
> Key: SPARK-29831
> URL: https://issues.apache.org/jira/browse/SPARK-29831
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
>
> Hive table scan operator reads each Hive partition as a HadoopRDD and unions
> all RDDs. The data parallelism of the result RDD can be dramatically
> increased, when reading a lot of partitions with a lot of files.
> Although users can also do coalesce by themselves, this ticket proposes to
> add a config to limit the maximum of the data parallelism. Because:
> 1. end-users might not understand details and get confused by big partition
> number. end-users might not know why/when/where to add coalesce.
> 2. users need to add coalesce to each time Hive table scan. It is annoying.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
