Hi Anup,
As Sorabh mentioned, you seem to be using hive 2.1.1 jars in Drill classpath based on the stack trace. Did you build drill package by customizing hive version to 2.1.1 or added hive 2.1.1 jars in Drill classpath manually? I could see that Drill 1.12 (latest released), 1.11 and 1.10 are bundled with hive 1.2.1 client. You could try with drill 1.11 tarball that includes hive 1.2.1 client. Also, you may check the version of hive client jars in $DRILL_HOME/jars/3rdparty/ directory from the Drill 1.10 cluster where it is working fine. http://archive.apache.org/dist/drill/drill-1.11.0/ # List of hive client jars from Drill 1.11 tarball extracted. $ ls -1 apache-drill-1.11.0/jars/3rdparty/hive-* apache-drill-1.11.0/jars/3rdparty/hive-contrib-1.2.1.jar apache-drill-1.11.0/jars/3rdparty/hive-hbase-handler-1.2.1.jar apache-drill-1.11.0/jars/3rdparty/hive-metastore-1.2.1.jar $ Thanks, Arjun ________________________________ From: Anup Tiwari <[email protected]> Sent: Tuesday, February 13, 2018 12:01 PM To: [email protected] Subject: Re: Unable to setup hive plugin in Drill 1.11.0 Also forgot to mention that we are using Drill 1.10 with 2.1 on our one of clusters and it is working properly. Just the change is in Hadoop Version and we are using MySQL community version 5.7.16. Please find below details :- Apache Drill Version : 1.10.0MySQL Community :5.7.16Apache Hive : 2.1.1Apache Hadoop : 2.7.3 So does this mean the issue is with hadoop version ? As i can seehadoop 2.7.1 related jars in 3rdparty jar of drill. On Tue, Feb 13, 2018 11:33 AM, Anup Tiwari [email protected] wrote: Hi Sorabh, Thanks for reply. We are using below combination :- Apache Drill Version : 1.11.0MySQL Percona : 5.6.35-81.0Apache Hive : 2.1.1 Apache Hadoop : 2.8.0 As you have mentioned that Drill 1.11 with hive 2.1 combination is not supported so do we have any link from where i can get an idea of which combination is supported? Also does not supported means we can't use this combination(i mean don't we have any workaround)? On Tue, Feb 13, 2018 12:57 AM, Sorabh Hamirwasia [email protected] wrote: Hi Anup, >From the stack trace which you have shared it looks like you are trying to use Hive2.1 with drill-1.11.0. This s not a supported combination[1]. There is a PR[2] open for it in 1.13 though. Also from the stack trace it's pointing towards an invalid field in HiveConf METASTORE_BATCH_RETRIEVE_OBJECTS_MAX which got introduced in hive-2.1 only. java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]Caused by: java.lang.NoSuchFieldError: METASTORE_BATCH_RETRIEVE_OBJECTS_MAX at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:222) ~[hive-metastore-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:210) ~[hive-metastore-2.1.1.jar:2.1.1] at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient.<init>(DrillHiveMetaStoreClient.java:140) Usually any error/exxception should get logged into drillbit.out as well. May be you can try seeing there. [1]: https://github.com/apache/drill/blob/1.11.0/pom.xml#L51 [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/drill/blob/1.11.0/pom.xml#L51> apache/drill<https://github.com/apache/drill/blob/1.11.0/pom.xml#L51> github.com drill - Mirror of Apache Drill [2]: https://github.com/apache/drill/pull/1111 Thanks, Sorabh ________________________________ From: Anup Tiwari <[email protected]> Sent: Monday, February 12, 2018 9:21 AM To: [email protected] Subject: Re: Unable to setup hive plugin in Drill 1.11.0 I don't know where to look for logs as i have tried multiple time creating hive plugin. But one time i can see below stack is printed in drillbit.log(i don't know if it came due to plugin creation or not as i have tried multiple time but nothing got printed in drillbit.log except below which is present in logs only once) :- Error org.apache.drill.common.exceptions.ExecutionSetupException: Failure setting up new storage plugin configuration for config org.apache.drill.exec. store.hive.HiveStoragePluginConfig@f149f108 at org.apache.drill.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:354) ~[drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.StoragePluginRegistryImpl.createOrUpdate(StoragePluginRegistryImpl.java:238) ~[drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.StoragePluginRegistryImpl.getPlugin(StoragePluginRegistryImpl.java:289) ~[drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.server.rest.StorageResources.getStoragePluginJSON(StorageResources.java:101) [drill-java-exec-1.11.0.jar:1.11.0] at org.apache.drill.exec.server.rest.StorageResources.getStoragePlugin(StorageResources.java:115) [drill-java-exec-1.11.0.jar:1.11.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_72] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_72] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_72] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_72] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java :81) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcher Provider.java:195) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104 ) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) [jersey-server-2.8.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.8.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.8.jar:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) [jersey-common-2.8.jar:na] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252) [jersey-server-2.8.jar:na] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1023) [jersey-server-2.8.jar:na] at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372) [jersey-container-servlet-core-2.8.jar:na] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382) [jersey-container-servlet-core-2.8.jar:na] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:345) [jersey-container-servlet-core-2.8.jar:na] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:220) [jersey-container-servlet-core-2.8.jar:na] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:219) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.server.Server.handle(Server.java:462) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]Caused by: java.lang.NoSuchFieldError: METASTORE_BATCH_RETRIEVE_OBJECTS_MAX at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:222) ~[hive-metastore-2.1.1.jar:2.1.1] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:210) ~[hive-metastore-2.1.1.jar:2.1.1] at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient.<init>(DrillHiveMetaStoreClient.java:140) ~[drill-storage-hive-core-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient.<init>(DrillHiveMetaStoreClient.java:57) ~[drill-storage-hive-core-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.<init>(DrillHiveMetaStoreClient.java:471) ~[drill-storage-hive-core-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.<init>(DrillHiveMetaStoreClient.java:469) ~[drill-storage-hive-core-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient.createCloseableClientWithCaching(DrillHiveMetaStoreClient.java:136) ~[drill-storage-hive-core-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.<init>(HiveSchemaFactory.java:75) ~[drill-storage-hive-core-1.11.0.jar:1.11.0] at org.apache.drill.exec.store.hive.HiveStoragePlugin.<init>(HiveStoragePlugin.java:68) ~[drill-storage-hive-core-1.11.0.jar:1.11.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_72] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_72] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_72] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_72] at org.apache.drill.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:345) ~[drill-java-exec-1.11.0.jar:1.11.0] ... 45 common frames omitted On Mon, Feb 12, 2018 9:23 PM, Anup Tiwari [email protected] wrote: Hi All, Please find below information :- Apache Drill Version : 1.11.0MySQL Percona : 5.6.35-81.0Apache Hive : 2.1.1 mysql-connector-java-5.1.44-bin.jar is present in 3rdparty jar Hive Plugin info :- { "type" : "hive", "enabled" : true, "configProps" : { "hive.metastore.uris" : "thrift://prod-hadoop-xxx:9083", "hive.metastore.sasl.enabled" : "false", "fs.default.name" : "hdfs://prod-hadoop-xxx:9000" } } Error : "result" : "error (unable to create/ update storage)" Regards, Anup Tiwari Sent with Mixmax Regards, Anup Tiwari Regards, Anup Tiwari Regards, Anup Tiwari
