[
https://issues.apache.org/jira/browse/FLINK-25010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454339#comment-17454339
]
Liu commented on FLINK-25010:
-----------------------------
We have 5000 partitions for testing. When the thread is 1, it takes about 341
seconds. When the thread is 3, it takes 122 seconds. More threads means less
time to split.
> Speed up hive's createMRSplits by multi thread
> ----------------------------------------------
>
> Key: FLINK-25010
> URL: https://issues.apache.org/jira/browse/FLINK-25010
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Hive
> Reporter: Liu
> Priority: Major
> Labels: pull-request-available
>
> We have thousands of hive partitions and the method createMRSplits will take
> much time, for example, ten minutes. We can speed up the process by multi
> thread for different partitions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
