A few questions/thoughts: 1. When you say having both catalogs side by side, are you planning to have the same tables in both catalogs? This could be tricky consistency wise as writes would need only update one catalog.
2. The main reason to have a Nessie catalog would be for its catalog versioning features. If you do want these features why not just use Nessie in Spark with Hadoop as the store vs using Hive? 3. Nessie has OAuth/token and AWS authentication available, you can find more details in the docs at projectnessie.org. How would you want authentication to be handled? Are there any other tools you plan on accessing your tables with? Then catalog access of these tools should be considered as well. *Alex Merced <https://bio.alexmerced.com/data> * *Senior Tech Evangeslit, Dremio **Dremio.com* <https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature>*/ **Follow Us on LinkedIn!* <https://www.linkedin.com/company/dremio> *Resources for Getting Hands-on with Apache Iceberg/Dremio* <https://medium.com/data-engineering-with-dremio/a-deep-intro-to-apache-iceberg-and-resources-for-learning-more-be51535cff74> On Mon, May 6, 2024 at 6:40 PM Pani Dhakshnamurthy <dpa...@gmail.com> wrote: > Hello Everyone, > I'm currently planning to build a Lakehouse solution, leveraging Apache > Spark and Hive Metastore in my Hadoop workflows. I'm also exploring the > potential benefits of integrating Nessie as my catalog solution. > > With this setup, we'll have two catalogs, and we anticipate working with > non-Iceberg tables alongside Iceberg tables. I'm keen to understand if > Nessie offers any advantages over Hive Metastore for Iceberg tables and if > anyone has experience using Nessie alongside Hive Metastore in production > environments. > > Additionally, I'd like to know if there are any known issues or challenges > associated with having both solutions in place, especially regarding > Iceberg table maintenance. Also, could you provide insights into the > authentication and authorization mechanisms available for Nessie? > > Your insights and guidance on these points would be highly valuable. > > Thank you for your time and assistance! > > Best regards, > Pani >
