Hi JB, Dan, Russell, Thanks for the steer. I have reworked #17862 to the shape agreed here: when table.io() is HadoopConfigurable, the listing uses its Hadoop configuration; otherwise it uses the session configuration as today. No catalogName() or hadoopConf(), no new API. Since the change is small, the PR covers the 4.2, 4.1, 4.0 and 3.5 modules together rather than as separate backports. Let me know if you would rather have them split.
The docs now state the rule and point S3FileIO users to prefix_listing until the parallel FileIO listing exists. I updated #17860 with the same guidance, including the io-impl change to ResolvingFileIO that makes the fix apply to a Glue catalog. Thanks, Sebastian On Mon, Nov 9, 2026 at 9:17 PM Jean-Baptiste Onofré <[email protected]> wrote: > Hi, > > Thanks Dan, fair point. I missed your earlier comment on #11906 (as I > said, I'm back on Iceberg and back on pace :)) about enumerating the key > space (using the LocationProvider as a hint) and distributing the prefixes > to the executors. I agree it's a better direction than a delimiter walk: > with the object store layout the prefixes are predictable, so we can > compute them instead of discovering with LIST calls on the driver. And the > listing stays on table.io(), which is what we need for #17860 and for > REST catalogs with vended crendentials. > > One question: orphan files are not always where the LocationProvider would > put them (that's kind of the point :)). In #11906 you mentioned separate > tasks for the ranges above and below the enumerated prefixes. listPrefix() > can't express a range today, so would you add a start-after bound to > SupportsPrefixOperations (S3 and GCS support it natively), or do you see > another way? > > Sebastian, to answer your questions: > - I don't think we should make prefix_listing the default or remove Hadoop > from the action now. The Hadoop walk is still needed for FileIOs that don't > implement SupportsPrefixOperations. We can discuss the default once the > parallel FileIO listing exists and works on large tables. > - I would not close #17862, but rework it to the shape Dan and Russell > suggested: when table.io() is HadoopConfigurable, the listing uses its > confiiguration. No catalogName() or hadoopConf(), so no new API. It's a > small fix on its own: today the listing ignores the catalog hadoop.* > overrides even for HadoopFileIO and ResolvingFileIO users. > - It won't fix #17860 with S3FileIO, that will come form the parallel > FileIO listing. Until then, prefix_listing => true already uses the catalog > credentials, but it lists serially on the driver, so it's only OK for small > and medium tables. > > Before reviewing #17790, I think we should agree on the approach here. > > Regards > JB > > On Mon, Nov 9, 2026 at 4:49 PM Daniel Weeks <[email protected]> wrote: > >> Trying to catch up and follow where this discussion is going, but I >> generally agree with JB's or Russels's comment that we should use the >> table's FileIO to perform these operations. >> >> Falling back to the spark/hadoop config is a poor workaround for being >> able to perform a directory based traversal. >> >> On the other hand, I'm not in favor of adding a delimiter to the prefix >> listing as that hides a larger problem with how we perform the listing >> operations and it won't scale. WIth layouts like object store layout, you >> will end up performing huge numbers of list operations (which is already >> done with with the file system approach). >> >> I've discussed this in the past, and the right way to address this is to >> break up the listing namespace and use the prefix based listing to identify >> all objects under the table location. Recursive listing doesn't scale with >> object store layout. >> >> -Dan >> >> On Mon, Nov 9, 2026 at 5:17 AM Sebastian Daberdaku via dev < >> [email protected]> wrote: >> >>> Thank you, JB, for your input. >>> >>> So with this suggestion, if I understand correctly, prefix_listing would >>> become true by default (and maybe be dropped completely as an argument from >>> the remove_orphan_files procedure), and Hadoop possibly be removed >>> completely from the action, right? Or do you see applications where Hadoop >>> will still be required by this action, as in using a custom io-impl that >>> does not support prefix_listing? In this case, should I rewrite my PR to >>> derive the HadoopConfig from the io-impl if present, as Dan suggested? Or >>> is it not worth it and I should just close the PR and wait for #17790 to be >>> merged? >>> >>> Thanks again, >>> >>> Sebastian >>> >>> On Mon, Nov 9, 2026 at 11:49 AM Jean-Baptiste Onofré <[email protected]> >>> wrote: >>> >>>> Hi Sebastian and all, >>>> >>>> First, I'm happy to be back on Iceberg (I was busy with other stuff and >>>> didn't have much time for Iceberg, I'm trying to optimize my time to >>>> balance between projects and the foundation :)). >>>> >>>> Back on this thread, the cross account scenario is clear, thanks >>>> Sebastian. I agree it's a real problem for multi-tenant deployments. >>>> >>>> That said, I agree with Russell and Dan: we should not fix it by giving >>>> the listing a Hadoop configuration when the table FileIO is S3FileIO. Your >>>> workaround needs the same role declared twice, once for the AWS SDK and >>>> once for s3a. That shows the real issue imho: the Hadoop walk is a second >>>> IO path that doesn't use the table FileIO. Adding catalogName() or >>>> hadoopConf() to the action would make that second path official instead of >>>> removing it. >>>> >>>> For me, the right fix is the option 2: make the FileIO listing path >>>> scale, so the listing goes through table.io() and uses the same >>>> credentials as the reads and deletes. I saw that PR #17790 adds >>>> delimiter-based listing to FileIO (S3, GCS, ADLS and Hadoop), which is the >>>> missing piece. With it, the action can walk the first levels on the driver >>>> and hand the sub-prefixes to the executors with the table FileIO, the same >>>> way the Hadoop walk does today. I think we should focus on #17790 (I will >>>> do the review :)), then, with that, we can build the parallel FileIO walk >>>> on top of it. Once done, we can also discuss using FileIO listing by >>>> default when table.io() supports it. >>>> >>>> I'm not in favor of option 1. Making ResolvingFileIO the Glue default >>>> would change the default for every Glue user and pull Hadoop onto the >>>> classpath, just to fix one Spark maintenance action. >>>> >>>> Regards >>>> JB >>>> >>> -- *This communication, including any attachments, is intended solely for the recipient(s) addressed. * *If you have received this email in error, please notify the sender immediately and delete it from your system. * *Any unauthorized use, disclosure, or distribution, of this communication is strictly prohibited.*
