[ https://issues.apache.org/jira/browse/SPARK-40150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17582564#comment-17582564 ]
Apache Spark commented on SPARK-40150: -------------------------------------- User 'jackylee-ch' has created a pull request for this issue: https://github.com/apache/spark/pull/37601 > Dynamically merge File Splits > ----------------------------- > > Key: SPARK-40150 > URL: https://issues.apache.org/jira/browse/SPARK-40150 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.4.0 > Reporter: Jackey Lee > Priority: Major > > We currently use maxPartitionBytes and minPartitionNum to split files and use > openCostInBytes to merge file splits. But these are static configurations, > and the same configuration does not work in all scenarios. > This PR attempts to dynamically merge file splits, taking into the > concurrency while processing more data in one task. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org