[jira] [Commented] (HIVE-13825) Using JOIN in 2 tables that has same path locations, but different colum names fail wtih an error exception
[ https://issues.apache.org/jira/browse/HIVE-13825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286829#comment-17286829 ] Dhirendra Pandit commented on HIVE-13825: - I am also having the same issue with below query. MERGE INTO default.s_satellite AS O using default.s_satellite AS N ON o.load_end_date=n.load_end_date AND n.l_link_key = o.l_link_key and n.load_date > o.load_date WHEN matched then update set load_end_date= n.load_date; I am trying to update the same table by comparing the value of the exiting entries. > Using JOIN in 2 tables that has same path locations, but different colum > names fail wtih an error exception > --- > > Key: HIVE-13825 > URL: https://issues.apache.org/jira/browse/HIVE-13825 > Project: Hive > Issue Type: Improvement >Reporter: Sergio Peña >Assignee: Vihang Karajgaonkar >Priority: Major > > The following scenario of 2 tables with same locations cannot be used on a > JOIN query: > {noformat} > hive> create table t1 (a string, b string) location > '/user/hive/warehouse/test1'; > OK > hive> create table t2 (c string, d string) location > '/user/hive/warehouse/test1'; > OK > hive> select t1.a from t1 join t2 on t1.a = t2.c; > ... > 2016-05-23 16:39:57 Starting to launch local task to process map join; > maximum memory = 477102080 > Execution failed with exit status: 2 > Obtaining error information > Task failed! > Task ID: > Stage-4 > Logs: > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask > {noformat} > The logs contain this error exception: > {noformat} > 2016-05-23T16:39:58,163 ERROR [main]: mr.MapredLocalTask (:()) - Hive Runtime > Error: Map local work failed > java.lang.RuntimeException: cannot find field a from [0:c, 1:d] > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485) > at > org.apache.hadoop.hive.serde2.BaseStructObjectInspector.getStructFieldRef(BaseStructObjectInspector.java:133) > at > org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55) > at > org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:973) > at > org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:999) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:75) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504) > at > org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504) > at > org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365) > at > org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:499) > at > org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:403) > at > org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:383) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:751) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-14631) HiveServer2 regularly fails to connect to metastore
[ https://issues.apache.org/jira/browse/HIVE-14631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283056#comment-17283056 ] Dhirendra Pandit edited comment on HIVE-14631 at 2/11/21, 2:51 PM: --- I am also having the same problem, any workaround? I tried with both Tez and Mr engine with Hadoop version 3.2.1 and hive 3.1.2. But no help so far :( was (Author: pandit_dhiren...@yahoo.com): I am also having the same problem, any workaround? > HiveServer2 regularly fails to connect to metastore > --- > > Key: HIVE-14631 > URL: https://issues.apache.org/jira/browse/HIVE-14631 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.1, 2.0.0, 2.1.0 > Environment: Hive 2.1.0, Hue 3.10.0, Hadoop 2.7.2, Tez 0.8.3 >Reporter: Alexandre Linte >Priority: Major > > I have a cluster secured with Kerberos and Hive is configured to work with > Tez by default. Everything works well through hive-cli and beeline; however, > I'm facing a strange behavior through Hue. > I can have a lot of client connections (these can reach 600) and after a day, > the client connections fail. But this is not the case for all clients > connection attempts. > When it fails, I have the following logs on the HiveServer2: > {noformat} > Aug 3 09:28:04 hiveserver2.bigdata.fr Executing > command(queryId=hiveserver2_20160803092803_a216edf1-bb51-43a7-81a6-f40f1574b112): > INSERT INTO TABLE shfs3453.camille_test VALUES ('coucou') > Aug 3 09:28:04 hiveserver2.bigdata.fr Query ID = > hiveserver2_20160803092803_a216edf1-bb51-43a7-81a6-f40f1574b112 > Aug 3 09:28:04 hiveserver2.bigdata.fr Total jobs = 1 > Aug 3 09:28:04 hiveserver2.bigdata.fr Launching Job 1 out of 1 > Aug 3 09:28:04 hiveserver2.bigdata.fr Starting task [Stage-1:MAPRED] in > parallel > Aug 3 09:28:04 hiveserver2.bigdata.fr Trying to connect to metastore with > URI thrift://metastore01.bigdata.fr:9083 > Aug 3 09:28:04 hiveserver2.bigdata.fr Failed to connect to the MetaStore > Server... > Aug 3 09:28:04 hiveserver2.bigdata.fr Waiting 1 seconds before next > connection attempt. > Aug 3 09:28:05 hiveserver2.bigdata.fr Trying to connect to metastore with > URI thrift://metastore01.bigdata.fr:9083 > Aug 3 09:28:05 hiveserver2.bigdata.fr Failed to connect to the MetaStore > Server... > Aug 3 09:28:05 hiveserver2.bigdata.fr Waiting 1 seconds before next > connection attempt. > Aug 3 09:28:06 hiveserver2.bigdata.fr Trying to connect to metastore with > URI thrift://metastore01.bigdata.fr:9083 > Aug 3 09:28:06 hiveserver2.bigdata.fr Failed to connect to the MetaStore > Server... > Aug 3 09:28:06 hiveserver2.bigdata.fr Waiting 1 seconds before next > connection attempt. > Aug 3 09:28:08 hiveserver2.bigdata.fr FAILED: Execution Error, return code > -1 from org.apache.hadoop.hive.ql.exec.tez.TezTask > Aug 3 09:28:08 hiveserver2.bigdata.fr Completed executing > command(queryId=hiveserver2_20160803092803_a216edf1-bb51-43a7-81a6-f40f1574b112); > Time taken: 4.002 seconds > {noformat} > At the same time I have the following logs on the Metastore are: > {noformat} > Aug 3 09:28:03 metastore01.bigdata.fr 180: get_table : db=shfs3453 > tbl=camille_test > Aug 3 09:28:03 metastore01.bigdata.fr > ugi=shfs3453#011ip=10.77.64.228#011cmd=get_table : db=shfs3453 > tbl=camille_test#011 > Aug 3 09:28:04 metastore01.bigdata.fr 180: get_table : db=shfs3453 > tbl=camille_test > Aug 3 09:28:04 metastore01.bigdata.fr > ugi=shfs3453#011ip=10.77.64.228#011cmd=get_table : db=shfs3453 > tbl=camille_test#011 > Aug 3 09:28:04 metastore01.bigdata.fr 180: get_table : db=shfs3453 > tbl=camille_test > Aug 3 09:28:04 metastore01.bigdata.fr > ugi=shfs3453#011ip=10.77.64.228#011cmd=get_table : db=shfs3453 > tbl=camille_test#011 > Aug 3 09:28:04 metastore01.bigdata.fr SASL negotiation failure > Aug 3 09:28:04 metastore01.bigdata.fr Error occurred during processing of > message. > Aug 3 09:28:05 metastore01.bigdata.fr SASL negotiation failure > Aug 3 09:28:05 metastore01.bigdata.fr Error occurred during processing of > message. > Aug 3 09:28:06 metastore01.bigdata.fr SASL negotiation failure > Aug 3 09:28:06 metastore01.bigdata.fr Error occurred during processing of > message. > {noformat} > To solve the connections issue, I have to restart the HiveServer2. > Note: I also created a JIRA for Hue: > https://issues.cloudera.org/browse/HUE-4748 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-14631) HiveServer2 regularly fails to connect to metastore
[ https://issues.apache.org/jira/browse/HIVE-14631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283056#comment-17283056 ] Dhirendra Pandit commented on HIVE-14631: - I am also having the same problem, any workaround? > HiveServer2 regularly fails to connect to metastore > --- > > Key: HIVE-14631 > URL: https://issues.apache.org/jira/browse/HIVE-14631 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.1, 2.0.0, 2.1.0 > Environment: Hive 2.1.0, Hue 3.10.0, Hadoop 2.7.2, Tez 0.8.3 >Reporter: Alexandre Linte >Priority: Major > > I have a cluster secured with Kerberos and Hive is configured to work with > Tez by default. Everything works well through hive-cli and beeline; however, > I'm facing a strange behavior through Hue. > I can have a lot of client connections (these can reach 600) and after a day, > the client connections fail. But this is not the case for all clients > connection attempts. > When it fails, I have the following logs on the HiveServer2: > {noformat} > Aug 3 09:28:04 hiveserver2.bigdata.fr Executing > command(queryId=hiveserver2_20160803092803_a216edf1-bb51-43a7-81a6-f40f1574b112): > INSERT INTO TABLE shfs3453.camille_test VALUES ('coucou') > Aug 3 09:28:04 hiveserver2.bigdata.fr Query ID = > hiveserver2_20160803092803_a216edf1-bb51-43a7-81a6-f40f1574b112 > Aug 3 09:28:04 hiveserver2.bigdata.fr Total jobs = 1 > Aug 3 09:28:04 hiveserver2.bigdata.fr Launching Job 1 out of 1 > Aug 3 09:28:04 hiveserver2.bigdata.fr Starting task [Stage-1:MAPRED] in > parallel > Aug 3 09:28:04 hiveserver2.bigdata.fr Trying to connect to metastore with > URI thrift://metastore01.bigdata.fr:9083 > Aug 3 09:28:04 hiveserver2.bigdata.fr Failed to connect to the MetaStore > Server... > Aug 3 09:28:04 hiveserver2.bigdata.fr Waiting 1 seconds before next > connection attempt. > Aug 3 09:28:05 hiveserver2.bigdata.fr Trying to connect to metastore with > URI thrift://metastore01.bigdata.fr:9083 > Aug 3 09:28:05 hiveserver2.bigdata.fr Failed to connect to the MetaStore > Server... > Aug 3 09:28:05 hiveserver2.bigdata.fr Waiting 1 seconds before next > connection attempt. > Aug 3 09:28:06 hiveserver2.bigdata.fr Trying to connect to metastore with > URI thrift://metastore01.bigdata.fr:9083 > Aug 3 09:28:06 hiveserver2.bigdata.fr Failed to connect to the MetaStore > Server... > Aug 3 09:28:06 hiveserver2.bigdata.fr Waiting 1 seconds before next > connection attempt. > Aug 3 09:28:08 hiveserver2.bigdata.fr FAILED: Execution Error, return code > -1 from org.apache.hadoop.hive.ql.exec.tez.TezTask > Aug 3 09:28:08 hiveserver2.bigdata.fr Completed executing > command(queryId=hiveserver2_20160803092803_a216edf1-bb51-43a7-81a6-f40f1574b112); > Time taken: 4.002 seconds > {noformat} > At the same time I have the following logs on the Metastore are: > {noformat} > Aug 3 09:28:03 metastore01.bigdata.fr 180: get_table : db=shfs3453 > tbl=camille_test > Aug 3 09:28:03 metastore01.bigdata.fr > ugi=shfs3453#011ip=10.77.64.228#011cmd=get_table : db=shfs3453 > tbl=camille_test#011 > Aug 3 09:28:04 metastore01.bigdata.fr 180: get_table : db=shfs3453 > tbl=camille_test > Aug 3 09:28:04 metastore01.bigdata.fr > ugi=shfs3453#011ip=10.77.64.228#011cmd=get_table : db=shfs3453 > tbl=camille_test#011 > Aug 3 09:28:04 metastore01.bigdata.fr 180: get_table : db=shfs3453 > tbl=camille_test > Aug 3 09:28:04 metastore01.bigdata.fr > ugi=shfs3453#011ip=10.77.64.228#011cmd=get_table : db=shfs3453 > tbl=camille_test#011 > Aug 3 09:28:04 metastore01.bigdata.fr SASL negotiation failure > Aug 3 09:28:04 metastore01.bigdata.fr Error occurred during processing of > message. > Aug 3 09:28:05 metastore01.bigdata.fr SASL negotiation failure > Aug 3 09:28:05 metastore01.bigdata.fr Error occurred during processing of > message. > Aug 3 09:28:06 metastore01.bigdata.fr SASL negotiation failure > Aug 3 09:28:06 metastore01.bigdata.fr Error occurred during processing of > message. > {noformat} > To solve the connections issue, I have to restart the HiveServer2. > Note: I also created a JIRA for Hue: > https://issues.cloudera.org/browse/HUE-4748 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22717) Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
[ https://issues.apache.org/jira/browse/HIVE-22717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282849#comment-17282849 ] Dhirendra Pandit commented on HIVE-22717: - Is there any workaround available for this solution. I am using Hadoop 3.2.1, Hive 3.1.2 and Tez 0.10.1 version and facing the same problem during tez job submission. You can find the logs below. 2021-02-11 04:01:33,573 [INFO] [main] |common.TezClassLoader|: Created TezClassLoader with parent classloader: sun.misc.Launcher$AppClassLoader@4b67cf4d, thread: 1, system classloader: sun.misc.Launcher$AppClassLoader@4b67cf4d 2021-02-11 04:01:34,097 [INFO] [main] |app.DAGAppMaster|: Creating DAGAppMaster for applicationId=application_1613010819541_0003, attemptNum=2, AMContainerId=container_1613010819541_0003_02_01, jvmPid=12590, userFromEnv=init, cliSessionOption=true, pwd=/tmp/hadoop-init/nm-local-dir/usercache/init/appcache/application_1613010819541_0003/container_1613010819541_0003_02_01, localDirs=/tmp/hadoop-init/nm-local-dir/usercache/init/appcache/application_1613010819541_0003, logDirs=/home/init/bigdata/hadoop-3.2.1/logs/userlogs/application_1613010819541_0003/container_1613010819541_0003_02_01 2021-02-11 04:01:34,164 [ERROR] [main] |app.DAGAppMaster|: Error starting DAGAppMaster java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) at org.apache.tez.common.TezUtilsInternal.addUserSpecifiedTezConfiguration(TezUtilsInternal.java:84) at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2377) > Exception in thread "main" java.lang.NoSuchMethodError: > com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V > -- > > Key: HIVE-22717 > URL: https://issues.apache.org/jira/browse/HIVE-22717 > Project: Hive > Issue Type: Bug >Reporter: ZHangTianLong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281467#comment-17281467 ] Dhirendra Pandit commented on HIVE-22126: - I am also having a similar problem with hive 3.1.1, Tez job is failing with error message "com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V" > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch, > HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch, > HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)