Hello, It is a pleasure to hear from you. Thank you for sharing your insights. We have adopted a similar approach to address component stack upgrade and evolution challenges, and we are delighted to see that the community is actively advancing this work as well. Our general strategy focuses on two key principles: 1. Decoupling the Engine-Data Relationship This appears to be a consensus among industry peers. Given that our data processing remains predominantly structured, the data lake technology stack naturally became our preferred choice, as it inherently preserves data schema. In our production environment, we utilize HadoopCatalog to migrate data previously managed by legacy HMS versions into Iceberg. While alternative RestCatalog implementations are certainly worth considering, our preference for minimizing component stack complexity led us to favor a FileSystemCatalog-like solution. We are grateful that the community has incorporated similar experiences—HIVE now supports both HadoopCatalog and LocationBasedIcebergTable. Leveraging these features, we have achieved seamless data interoperability between Spark and HIVE4 in our production environment. This liberates us from concerns about engine upgrades potentially corrupting data or compromising accessibility. For Trino and other MPP databases, we now implementing a compatibility layer adhering to RestCatalog specifications, enabling these engines to access FileSystemCatalog-based tables. Our current production architecture maintains separate deployments: Spark 3.x operates with legacy HMS 3.x, while HIVE 4.x runs independently. This arrangement allows us to upgrade Spark and gracefully phase out the legacy HMS(3.x,2.x,...) at our discretion,and use hms 4.x. Should we need to reconfigure extensively—even clearing all metadata and redeploying engines—the data remains secure within Iceberg. We simply re-establish the connection between Iceberg and the engine. 2. Minimizing Hadoop Cluster Dependencies Similar to Spark-on-Kubernetes deployments, our approach involves bundling essential runtime dependencies within the engine's self-contained libraries, ensuring the engine operates exclusively with its own libraries during execution. This method has effectively decoupled Hadoop versioning from our engines. Provided base APIs remain stable, we can successfully run engines dependent on newer Hadoop versions atop older Hadoop/YARN infrastructures. We were honored to contribute this approach to the official documentation: https://hive.apache.org/docs/latest/admin/manual-installation/#installing-with-old-version-hadoopgreater-than-or-equal-310 Employing these techniques, we currently operate three or more distinct HIVE+HMS version combinations in production. In our customer engagements, we have similarly enabled HIVE4 (dependent on Hadoop 3.4+) to run within Hadoop 2.x environments. The above reflects our humble experience and observations. We would be delighted to exchange ideas should you have alternative approaches or insights to share. Please kindly point out any misconceptions or areas for improvement. Warm regards, Lisoda
