tom-s-powell opened a new pull request, #17790: URL: https://github.com/apache/iceberg/pull/17790
Updates the `SupportsPrefixOperations` with a new `listPrefix` method that takes a `delimiter`. This will list files in the prefix up to the delimiter and rather common sub-prefixes after the delimiter. A `supportsPrefixListingWithDelimiter` method lets implementation of `SupportsPrefixOperations` indicate if they support listing with a particular delimiter. The main motivation (also included in this PR) is to improve the performance of `remove_orphan_files`. Currently, when `prefix_listing=true` all listing of files in a table happens driver-side in Spark on a single thread. Hadoop differs in that it will list directories recursively on the driver up to some maximum-depth and afterwards parallelise listing in Spark. When you have tables with a large number of files and suitable number of subdirectories this improves performance because you can list subdirectories in parallel. It also alleviates memory pressure on the driver. This PR intends to bring that benefit when `prefix_listing=true`. Alongside this you can specify `prefix_listing_max_seed_depth` which will result in similar behaviour to Hadoop where common subprefixes up to that depth will be discovered on the driver. These prefixes will then be parallelized in Spark and complete listing will happen on executor-side. Default behaviour is to treat `prefix_listing_max_seed_depth=0` in which case we have a single partition when we list all files beneath the table location. This does move it executor-side rather than driver-side which should still reduce chance of driver OOMs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
