[
https://issues.apache.org/jira/browse/SPARK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279947#comment-17279947
]
Cheng Su commented on SPARK-33207:
--
[~yumwang] - just an update, after
[
https://issues.apache.org/jira/browse/SPARK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219497#comment-17219497
]
Yuming Wang commented on SPARK-33207:
-
Thank you [~chengsu].
> Reduce the number of tasks launched
[
https://issues.apache.org/jira/browse/SPARK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219458#comment-17219458
]
Cheng Su commented on SPARK-33207:
--
[~yumwang] - I recently added the functionality to disable bucketed
[
https://issues.apache.org/jira/browse/SPARK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218827#comment-17218827
]
Cheng Su commented on SPARK-33207:
--
If you filter out empty `FilePartition` here, then `FileScanRDD`
[
https://issues.apache.org/jira/browse/SPARK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218823#comment-17218823
]
Yuming Wang commented on SPARK-33207:
-
But it still launch #-of-buckets tasks. May be we can change
[
https://issues.apache.org/jira/browse/SPARK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218545#comment-17218545
]
Cheng Su commented on SPARK-33207:
--
Thank [~yumwang] for bringing up the issue. We don't need to launch
[
https://issues.apache.org/jira/browse/SPARK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218290#comment-17218290
]
Yuming Wang commented on SPARK-33207:
-
cc [~chengsu]
> Reduce the number of tasks launched after
