Hey Zsolt, I would divide this discussion into three topics:
1. What are the benefits of using the StartMiniHS2Cluster? 2. What other alternatives are there for testing HS2 with different metastore DBMS? 3. How can we make StartMiniHS2Cluster work as before? Regarding the point 1, I don't have an answer because I never used StartMiniHS2Cluster myself. Obviously, other people here are using it so I would be curious to know in which cases this is useful. There are various alternatives for testing HS2 with different metastore DBMS. The first and in my opinion the easiest to use is the classic qtest framework with the various CLI drivers. Basically, with the work that Laszlo started in HIVE-21954 it is pretty easy to run any kind of test over any metastore just by setting the respective system property. mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres The second is to use our brand new and shiny docker images contributed by Zhihua in HIVE-26400 and get the real feel of HS2 and HMS in a prod-like setup. I haven't played much with this in apache/master but I did use some similar images in our internal forks and it's pretty easy to get it up and running. The third is the old and classic hive-dev-box [1] started by Zoltan. With two/three commands you have a Hive cluster like environment and of course you can choose which DBMS you want for the metastore. Regarding point 3, I assume that it is pretty easy to fix by adding the postgresql (or other JDBC driver) dependency inside the hive-it-unit module. <dependency> <groupId>org.postgresql</groupId> <artifactId>postgresql</artifactId> <optional>true</optional> </dependency> Given that we have other alternatives do we really need to go into this direction? In fact, do we really need the StartMiniHS2Cluster class? Best, Stamatis [1] https://github.com/kgyrtkirk/hive-dev-box On Tue, Aug 15, 2023, 6:08 PM Zsolt Miskolczi <zsolt.miskol...@gmail.com> wrote: > Hey there!Do you know how it is possible to use minihs2 with embedded > metastore service but postgresql as the metastore database? > > Since I'm pretty sure it broke with that change > https://github.com/apache/hive/pull/4317, handling drivers is changed: > > We don't bundle postgresql drivers with the hive build.Firstly, when I copy > the postgresql driver jar into the > > packaging/target/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/lib > directory, I can run the schematool as it sees the driver. > The real issue is with miniHS2Cluster: it is basically a maven test and I > have really no idea how I can pass the driver to the test. If it would be a > simple java command, I would try to pass the -cp argument but I see no > other way to add extra jar file to the test execution.I want to run minihs2 > with this command: mvn test -Dtest=StartMiniHS2Cluster -DminiHS2.run=true > -DminiHS2.usePortsFromConf=true -DminiHS2.clusterType=LLAP > -DminiHS2.conf="../../data/conf/llap/hive-site.xml" > -Dpackaging.minimizeJar=false -DskipShade -Dremoteresources.skip=true > -Dmaven.javadoc.skip=true -Denforcer.skip=trueAnd get that error (it seems > the metastore client cannot find the proper driver): > > java.lang.RuntimeException: Error applying authorization policy on hive > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to > instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > at org.apache.hive.service.cli.CLIService.init(CLIService.java:122) > at org.apache.hive.service.CompositeService.init(CompositeService.java:59) > at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:243) > at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:394) > at > > org.apache.hive.jdbc.miniHS2.StartMiniHS2Cluster.testRunCluster(StartMiniHS2Cluster.java:78) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at > > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > at > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > at > > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > at > > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > at > > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > at > > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > at > > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377) > at > > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138) > at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451) > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to instantiate > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > at > > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:1001) > at > > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:2003) > at > > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:135) > at org.apache.hive.service.cli.CLIService.init(CLIService.java:119) > ... 32 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to > instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > at > > org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1033) > at > > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:993) > ... 35 more > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > at > > org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:88) > at > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:96) > at > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:149) > at > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:120) > at > > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:5769) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5847) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5827) > at > > org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1029) > ... 36 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > > org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86) > ... 43 more > Caused by: MetaException(message:Got exception: > org.apache.hadoop.hive.metastore.api.MetaException java.sql.SQLException: > No suitable driver) > at > > org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.throwMetaException(MetaStoreUtils.java:150) > at > > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.callEmbeddedMetastore(HiveMetaStoreClient.java:309) > at > > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:220) > at > > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.<init>(HiveMetaStoreClientWithLocalCache.java:120) > at > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:154) > ... 48 more > > > Any help is appreciated, > Thank you >
