Thank you, JB, for your input. So with this suggestion, if I understand correctly, prefix_listing would become true by default (and maybe be dropped completely as an argument from the remove_orphan_files procedure), and Hadoop possibly be removed completely from the action, right? Or do you see applications where Hadoop will still be required by this action, as in using a custom io-impl that does not support prefix_listing? In this case, should I rewrite my PR to derive the HadoopConfig from the io-impl if present, as Dan suggested? Or is it not worth it and I should just close the PR and wait for #17790 to be merged?
Thanks again, Sebastian On Mon, Nov 9, 2026 at 11:49 AM Jean-Baptiste Onofré <[email protected]> wrote: > Hi Sebastian and all, > > First, I'm happy to be back on Iceberg (I was busy with other stuff and > didn't have much time for Iceberg, I'm trying to optimize my time to > balance between projects and the foundation :)). > > Back on this thread, the cross account scenario is clear, thanks > Sebastian. I agree it's a real problem for multi-tenant deployments. > > That said, I agree with Russell and Dan: we should not fix it by giving > the listing a Hadoop configuration when the table FileIO is S3FileIO. Your > workaround needs the same role declared twice, once for the AWS SDK and > once for s3a. That shows the real issue imho: the Hadoop walk is a second > IO path that doesn't use the table FileIO. Adding catalogName() or > hadoopConf() to the action would make that second path official instead of > removing it. > > For me, the right fix is the option 2: make the FileIO listing path scale, > so the listing goes through table.io() and uses the same credentials as > the reads and deletes. I saw that PR #17790 adds delimiter-based listing to > FileIO (S3, GCS, ADLS and Hadoop), which is the missing piece. With it, the > action can walk the first levels on the driver and hand the sub-prefixes to > the executors with the table FileIO, the same way the Hadoop walk does > today. I think we should focus on #17790 (I will do the review :)), then, > with that, we can build the parallel FileIO walk on top of it. Once done, > we can also discuss using FileIO listing by default when table.io() > supports it. > > I'm not in favor of option 1. Making ResolvingFileIO the Glue default > would change the default for every Glue user and pull Hadoop onto the > classpath, just to fix one Spark maintenance action. > > Regards > JB > -- *This communication, including any attachments, is intended solely for the recipient(s) addressed. * *If you have received this email in error, please notify the sender immediately and delete it from your system. * *Any unauthorized use, disclosure, or distribution, of this communication is strictly prohibited.*
