codope commented on PR #11043: URL: https://github.com/apache/hudi/pull/11043#issuecomment-2125051546
> Do we need to use parallel stream to improve efficiency in `allFiles.map`? I tried but it seems like the usual `scala.collection.parallel` does not work with Scala 2.13 (which is used with Spark 3.5). To make it work for all Scala versions, i'll have to introduce a [new dependency](https://github.com/scala/scala-parallel-collections?tab=readme-ov-file#cross-building-dependency). I can do it in a followup PR, if necessary. However, I don't other index types using parallel collection, plus I have also optimized the code as per your other suggestions (suing pruned files instead of all files). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
