rdsarvar commented on PR #2322: URL: https://github.com/apache/polaris/pull/2322#issuecomment-3178989292
> We have been (more precisely: are) discussing support for Hive, and implicitly HDFS, in Polaris. There are a couple of concerns about adding (more) Hadoop stuff to Polaris. > > Hadoop types like `org.apache.hadoop.security.UserGroupInformation` et al rely on functionality that has already been deprecated since Java 17 ([JEP-411](https://openjdk.org/jeps/411)) and is already _removed_ in Java since version 24 ([JEP-486](https://openjdk.org/jeps/486))). Although we technically do not need 24 (or the upcoming Java 25) yet, we might have to. > > As we want to support these use cases, the approach would be to have a Polaris instance that _federates_ to an instance, which can then have have hard "max Java version" restrictions but also do other "restrictive" things like Kerberos. > > This all falls into the "catalog federation" bucket. We'd appreciate your input on the dev-mailing-list and in-person discussions in our biweekly community sync! Hi @snazy thanks for the response -- Interesting solution, so just to recap to make sure I'm on the same page the thought is to run a separate sidecar / service which itself can be pegged to a lower Java version and support any HDFS related operations there? So in essence the workflow would now look like: 1. User makes request to Polaris catalog (ex: new table) 2. Polaris determines the storage type is HDFS and proxies the request to the HDFS catalog service (REST API I assume) 3. The HDFS catalog service would perform required actions on behalf of Polaris, returning what happened through structured JSON response 4. Polaris would respond to the user and make any required persistence changes Thus decoupling the potentially unsafe (or lesser version) Hadoop usage to users that are explicitly requiring it. I'll join the Slack workspace, do you have reference to the channel (or thread) that this is being discussed? Or is it mostly in the syncs / mailing list -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@polaris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
