I hadn't seen this thread before we discussed it yesterday, but since then I've taken a look and have some reservations.
I was under the impression that the AuthManager implementation was relatively small (based on the recent work for the GCP AuthManager <https://github.com/apache/iceberg/pull/13212>), but after taking a look at the repo, this is far from a small contribution. I strongly support more robust security support (especially for OAuth2/OIDC), but I don't feel this is going to be a small effort to introduce. The broader community wasn't involved in decisions made about the implementation and I see elements that give me pause (like "impersonation flow" which I'm not familiar with and implementation details like extensions to immutables that aren't consistent with the broader codebase). If we decide that we want to take this on, I feel like we need to break it into smaller contributions and figure out the appropriate way to review and assimilate the functionality in a way that's consistent with the rest of the project. Due to this being security related, we should take extra precautions around what this introduces and how well the community understands the behaviors. However, looking at the complexity here relative to the approach with the GCP, I have to question whether this is the right path overall. The main competency of this project isn't to implement security protocols, so it's a lot to say we want a full and complete (possibly with extensions) native implementation of the OAuth2 specification (there are whole projects built around that alone). Was there any exploration of leveraging other standard implementations like Apache Oltu, Nimbus, etc. to build the implementation off of? -Dan On Thu, Jun 19, 2025 at 5:33 AM Alex Dutra <alex.du...@dremio.com.invalid> wrote: > Hi Ryan & JB, hi all, > > I think it would be easier to introduce this new manager as an > alternative manager. This would make the migration smoother as it > would give users time to migrate at their convenience. Besides, the > new manager has the notion of "dialects", and can be configured to > behave exactly like the current one (honoring the same config > options), making the migration even easier. > > > Why not contribute the functionality directly to the AuthManager already > in Iceberg? Is this incompatible or is there a reason the current one can't > be extended through contributions? > > There are a few reasons why I believe it's not possible to extend the > current manager indefinitely: > > 1. The current auth manager lives in iceberg-core; as we introduce > more features, it will become impractical to keep it there, especially > since some of the features will require third-party dependencies. As a > data point: the new manager contains almost 100 Java production > classes (not counting test classes and build scripts). > 2. The current auth manager has some well known shortcomings, notably > around token refreshes. It's not possible to fix that without > introducing regressions and potentially breaking many catalog clients > already in production. > 3. As we introduce features like Authorization Code grant support, > interactions with the IDP will become more complex than just a > request-response cycle. Since most of the current logic resides in the > OAuth2Util class, which is entirely public, it won't be an easy task > to introduce support for such complex flows while avoiding binary > incompatibilities. > > Thanks, > Alex > > > On Wed, Jun 18, 2025 at 11:35 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > > Hi > > > > I think it makes sense to directly add in AuthManager. I don't see > > blockers (with some adaptations). Alex ? > > > > From a donation process standpoint (if accepted), I'm happy to help > > with the SGA and IP Clearance. > > > > Regards > > JB > > > > On Wed, Jun 18, 2025 at 9:15 PM Ryan Blue <rdb...@gmail.com> wrote: > > > > > > I think it would be great to bring this functionality into Iceberg. > I'm curious about your plan for getting it in. It sounds like you're > suggesting adding the Dremio project to the Iceberg repo and making it > optional. Why not contribute the functionality directly to the AuthManager > already in Iceberg? Is this incompatible or is there a reason the current > one can't be extended through contributions? > > > > > > On Tue, Jun 17, 2025 at 11:23 AM Christian Thiel < > christian.t.b...@gmail.com> wrote: > > >> > > >> Hey Alex, > > >> > > >> Thanks for the Initiative — I really appreciate the effort here! > > >> > > >> Having good auth compatibility in the Catalog ecosystem is key to > establish secure standards by making them easy to use. While Iceberg should > stay open to other means of Authentication, OAuth2 is the most widely > adopted interoperable auth standard, and its role in Iceberg REST reflects > that. But with human-centric flows like Auth Code (with PKCE 😉) and Device > Code missing from most standard clients, users often default to handing out > personal Client ID/secret pairs—which is really bad from a security > perspective. > > >> > > >> While I can’t speak to the Java details, I fully support bringing the > functionality into Iceberg. I have tested the proposed code successfully > with Spark and different IdPs, including Auth & Device Code flows with > token refresh, as well as token refresh for Client Credential flows. > > >> > > >> Thanks! > > >> > > >> Christian > > >> > > >> > > >> > > >> On Mon, 16 Jun 2025 at 20:33, Alex Dutra > <alex.du...@dremio.com.invalid> wrote: > > >>> > > >>> Hi all, > > >>> > > >>> Dremio recently open-sourced a new implementation of the Auth Manager > > >>> API for OAuth2: > > >>> > > >>> https://github.com/dremio/iceberg-auth-manager > > >>> > > >>> I wrote a blog post about it a while ago [1]. > > >>> > > >>> Built on top of the Auth Manager API introduced in Iceberg 1.9.0, > this > > >>> project provides a more flexible and extensible OAuth2 manager > > >>> compared to the built-in equivalent in Iceberg Core. It follows > OAuth2 > > >>> standards strictly, but also provides compatibility with any existing > > >>> Apache Iceberg REST catalog, and contains no Dremio-specific > > >>> functionality. To date, this is the only OAuth2 manager fully > > >>> compliant with external identity providers. > > >>> > > >>> Dremio would like to contribute this code to the Apache Iceberg > > >>> project. I am therefore initiating this discussion to determine the > > >>> community's interest in accepting this donation. > > >>> > > >>> This project is beneficial to the community because it addresses > > >>> well-known limitations, such as token refresh problems [2][3][4], and > > >>> also because it introduces highly anticipated features like the > > >>> Authorization Code grant support [5]. Fixing these limitations or > > >>> adding support for such large features in the built-in manager, while > > >>> avoiding any risk of regressions, would have been a lot harder. > > >>> > > >>> Also worth mentioning: this project adheres to the "Iceberg OAuth2 > > >>> Client Authentication Guide", proposed by Christian Thiel [6]. > > >>> > > >>> This project could initially serve as a runtime-selectable > alternative > > >>> to the current built-in implementation. Upon reaching sufficient > > >>> maturity however, it could potentially replace the existing manager. > > >>> > > >>> Please share your thoughts by replying to this email. Alternatively, > > >>> we can discuss this topic at the Catalog Sync meeting this Wednesday, > > >>> June 18th, if that is a more comfortable option to everyone. > > >>> > > >>> Thanks, > > >>> > > >>> Alex > > >>> > > >>> [1] > https://medium.com/data-engineering-with-dremio/introducing-dremio-auth-manager-for-apache-iceberg-223827342d19 > > >>> [2]: https://github.com/apache/iceberg/issues/12196 > > >>> [3]: https://github.com/apache/iceberg/issues/12363 > > >>> [4]: https://github.com/apache/iceberg/issues/13030 > > >>> [5]: https://github.com/apache/iceberg/issues/10677 > > >>> [6]: > https://docs.google.com/document/d/1buW9PCNoHPeP7Br5_vZRTU-_3TExwLx6bs075gi94xc/edit?tab=t.0#heading=h.hufqidg1ij89 >
