[
https://issues.apache.org/jira/browse/HIVE-29052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17987311#comment-17987311
]
Butao Zhang commented on HIVE-29052:
------------------------------------
Paste my reply in the email here:
https://lists.apache.org/thread/916pjwz96l866sy97f48g4bxqzrpy5w6
{code:java}
First of all, thank you very much to Stamatis for pointing out the work [1] the
Hive community has done in decoupling HMS from Hive, such as ticket HIVE-17159
[2]. Personally, I believe that maintaining the decoupling of HMS and Hive code
is absolutely the right approach. Therefore, without considering other
community factors, directly bundling hive-exec.jar and hive-iceberg-handler.jar
into the HMS package would seem entirely unreasonable—I might even give it a -1.
The main reason for my +1 is due to the current state of community development.
As we all know, next-generation table formats like Iceberg have brought
significant challenges to traditional Hive table formats. Meanwhile, HMS, as
the de facto standard catalog component, is also facing major competition from
new systems like Unity Catalog and various Iceberg REST catalogs. Fortunately,
the Hive community has recognized this. We have begun fully embracing the
Iceberg table format, reconsidering how HMS can interface with different
metadata services compatible with the HMS API (HIVE-27473)[3], integrating the
Iceberg REST catalog into HMS (HIVE-28059)[4], and even planning to implement
cool features like Federated Catalog in HMS (HIVE-28879)[5]. Personally, I
believe there are many valuable things we can do around HMS in the future.
Since many current and future efforts will focus on HMS, now is an excellent
time for the community to release Standalone HMS. This would send a positive
signal to both the Hive community and beyond: the Hive community is committed
to strengthening HMS Catalog as the de facto standard for metadata management.
I believe this will attract more community users to participate and contribute
to the growth of the Hive community, creating a win-win situation for both
users and the Hive ecosystem.
If we wait until the code decoupling work is thoroughly completed before
releasing Standalone HMS, it would be difficult for me to estimate the
timeline. I’m also unsure whether the community has sufficient resources in the
short term to undertake such significant decoupling efforts. If this
refactoring takes a year or even two, I fear we might miss the best window for
the release of Standalone HMS.
In summary, I believe now is the right time to release Standalone HMS. When
weighing the future development of the Hive community against code-level
decoupling, I would prioritize the former. But I also think we can continue the
code decoupling work in parallel after releasing Standalone HMS—the two efforts
can, to some extent, proceed simultaneously.
[1]
https://issues.apache.org/jira/browse/HIVE-29052?focusedCommentId=17987181&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17987181
[2] https://issues.apache.org/jira/browse/HIVE-17159
[3] https://issues.apache.org/jira/browse/HIVE-27473
[4] https://issues.apache.org/jira/browse/HIVE-28059
[5] https://issues.apache.org/jira/browse/HIVE-28879
{code}
> Error starting standalone HMS: "IcebergHouseKeeperService class not found"
> --------------------------------------------------------------------------
>
> Key: HIVE-29052
> URL: https://issues.apache.org/jira/browse/HIVE-29052
> Project: Hive
> Issue Type: Task
> Components: Standalone Metastore
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Butao Zhang
> Priority: Major
> Labels: pull-request-available
>
> During local standalone HMS deployment testing, the following exception log
> was observed:
>
> {code:java}
> 2025-06-28T17:19:28,840 ERROR [Metastore threads starter thread]
> leader.LeaseLeaderElection: Error notifying the listener:
> org.apache.hadoop.hive.metastore.leader.CompactorTasks@61e0c33d, leader: true
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.ql.txn.compactor.Initiator
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
> ~[?:?]
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
> ~[?:?]
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
> ~[?:?]
> at java.base/java.lang.Class.forName0(Native Method) ~[?:?]
> at java.base/java.lang.Class.forName(Class.java:375) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.leader.CompactorTasks.instantiateThread(CompactorTasks.java:53)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.CompactorTasks.getCompactorThreads(CompactorTasks.java:67)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.CompactorTasks.takeLeadership(CompactorTasks.java:136)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection.lambda$notifyListener$0(LeaseLeaderElection.java:141)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection.notifyListener(LeaseLeaderElection.java:138)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection.doWork(LeaseLeaderElection.java:120)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection.tryBeLeader(LeaseLeaderElection.java:181)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection.tryBeLeader(LeaseLeaderElection.java:63)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.LeaderElectionContext.lambda$start$2(LeaderElectionContext.java:125)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at java.base/java.lang.Thread.run(Thread.java:833) [?:?]
> at
> org.apache.hadoop.hive.metastore.leader.LeaderElectionContext.start(LeaderElectionContext.java:136)
> [hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$8.run(HiveMetaStore.java:855)
> [hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> 2025-06-28T17:19:28,847 ERROR [Metastore threads starter thread]
> leader.LeaseLeaderElection: Error notifying the listener:
> org.apache.hadoop.hive.metastore.leader.HouseKeepingTasks@33d3660, leader:
> true
> org.apache.hadoop.hive.metastore.api.MetaException:
> org.apache.iceberg.mr.hive.metastore.task.IcebergHouseKeeperService class not
> found
> at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.getClass(JavaUtils.java:56)
> ~[hive-standalone-metastore-common-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.HouseKeepingTasks.getRemoteOnlyTasks(HouseKeepingTasks.java:72)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.HouseKeepingTasks.takeLeadership(HouseKeepingTasks.java:112)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection.lambda$notifyListener$0(LeaseLeaderElection.java:141)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection.notifyListener(LeaseLeaderElection.java:138)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection.doWork(LeaseLeaderElection.java:120)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection.tryBeLeader(LeaseLeaderElection.java:181)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.LeaseLeaderElection.tryBeLeader(LeaseLeaderElection.java:63)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.leader.LeaderElectionContext.lambda$start$2(LeaderElectionContext.java:125)
> ~[hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at java.base/java.lang.Thread.run(Thread.java:833) [?:?]
> at
> org.apache.hadoop.hive.metastore.leader.LeaderElectionContext.start(LeaderElectionContext.java:136)
> [hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$8.run(HiveMetaStore.java:855)
> [hive-standalone-metastore-server-4.1.0.jar:4.1.0]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
