Re: Hive throwing exception when using Sqoop
As an update for others if you have this problem, I have setup a vm with the
same packages, and things are running fine. What I've traced it down to is
how the shims are loaded. The shim 2.0S (the correct shim for 0.20.2) is
used if the class "org.apache.hadoop.security.UnixUserInformation" doesn't
exist (or if it's a secure Hadoop). For some reason, my install is using
the 2.0 shim instead of 2.0S like it should. This makes me think that the
class "org.apache.hadoop.security.UnixUserInformation" actually exists in my
install for some reason, even though I'm using 0.20.2 (and I've searched
through the hadoop code, and can't find anything relating to it).
My assumption is that there's a 0.20.1 jar being included somewhere, so I
need to track it down and remove it.
Thanks,
-Josh Braegger
On Mon, Oct 24, 2011 at 8:49 PM, Joshua Braegger wrote:
> Hello,
>
> I am trying to use Sqoop to import data into Hive, and am running the
> following command:
>
> sqoop import --connect 'jdbc:mysql://dbhost/dbname' --username xxuser
> --table xxtable --password xxpassword --hive-import --direct
>
> and am receiving the following exception from Hive:
>
> 11/10/24 20:36:31 INFO hive.HiveImport: Exception in thread "main"
> java.lang.NoSuchMethodError:
> org.apache.hadoop.security.UserGroupInformation.login(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/security/UserGroupInformation;
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> org.apache.hadoop.hive.shims.Hadoop20Shims.getUGIForConf(Hadoop20Shims.java:448)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:51)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:222)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:241)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:466)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> java.lang.reflect.Method.invoke(Method.java:616)
> 11/10/24 20:36:31 INFO hive.HiveImport: at
> org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
> I've seen others with the same issue, but no mention of a resolution. I'm
> fairly certain this is an issue with Hive, but haven't been able to
> reproduce the issue directly with Hive... only using Sqoop. I'm using the
> Cloudera-distributed release of all 3 applications (downloaded from
> https://ccp.cloudera.com/display/SUPPORT/CDH3+Downloadable+Tarballs). The
> specific versions are:
>
> * hadoop-0.20.2-cdh3u2
> * hive-0.7.1-cdh3u2
> * sqoop-1.3.0-cdh3u2
>
> I have seen that others are using hive 0.7.1 with hadoop 0.20.2, but can't
> figure out why it's working at all for them. In the file
> "shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java",
> getUGIForConf is defined as:
>
> public UserGroupInformation getUGIForConf(Configuration conf) throws
> LoginException {
> UserGroupInformation ugi =
> UnixUserGroupInformation.readFromConf(conf,
> UnixUserGroupInformation.UGI_PROPERTY_NAME);
> if(ugi == null) {
> ugi = UserGroupInformation.login(conf);
> }
> return ugi;
> }
>
> and the offending line is "ugi = UserGroupInformation.login(conf)".
> Looking at the source for hadoop's UserGroupInformation.java, there is no
> "login()" method defined -- the only ones I could see working are
> "getLoginUser" and "getCurrentUser". Attempting to change the "login" to
> one of these fails immediately, because 0.20.1 (which Hive is built with)
> doesn't contain these methods, but 0.20.2 does. Looking at the shems,
> there's no support for sub-versions, only major version (.19, .20, etc), so
> I'm not sure how trivial it would be to break it apart into minor releases.
>
> Given that I know these two versions should be pretty compatible with each
> other (since Cloudera is distributing them together), they are stable
> releases, and I haven't seen others mentioning a problem, does anyone know
> 1) if this is actually a problem and 2) Is there a workaround or simple fix
> that can be applied.
>
> Thanks,
> -Josh Braegger
>
Problem With HDFS USERS using JDBC
Hello, any body could tell me how can I do to set the hdfs user and group in a LOAD sentence from the JDBC driver? I have tried with many diferents way but ever that I tried the user used by the hive server was HIVE. Thanks Gabriel.
Hive runtime error while run TestCliDriver auto_join21.q
Hi, Guys
I met an error which is similar like described in HIVE-1478, but not exactly
the same when run auto_join21.q in TestCliDriver.
Do you have some ideas on this?
== Re-Produce ==
Hive: 0.7.1
ANT: 1.8.2
Hadoop: 0.20.2
command: ant test -Dtestcase=TestCliDriver -Dqfile=auto_join21.q
[junit] 11/10/25 10:03:57 INFO persistence.HashMapWrapper: maximum memory:
1048576000
[junit] 11/10/25 10:03:57 INFO persistence.HashMapWrapper: maximum memory:
1048576000
[junit] 11/10/25 10:03:57 INFO exec.MapJoinOperator: Initialization Done 1
MAPJOIN
[junit] 11/10/25 10:03:57 INFO exec.HashTableDummyOperator: Initialization
Done 5 HASHTABLEDUMMY
[junit] 11/10/25 10:03:57 INFO exec.MapOperator: Processing path
pfile:/root/libing/N20111024_1337/hive-0.7.1-ibm/src/build/ql/test/data/warehouse/src/kv1.txt
[junit] 11/10/25 10:03:57 INFO exec.MapOperator: Processing alias src1 for
file
pfile:/root/libing/N20111024_1337/hive-0.7.1-ibm/src/build/ql/test/data/warehouse/src
[junit] 11/10/25 10:03:57 INFO exec.MapJoinOperator: *** Load from
HashTable File: input :
pfile:/root/libing/N20111024_1337/hive-0.7.1-ibm/src/build/ql/test/data/warehouse/src/kv1.txt
[junit] 11/10/25 10:03:57 INFO exec.MapJoinOperator: Load back 1
hashtable file from tmp file
uri:file:/tmp/root/hive_2011-10-25_10-03-53_617_6340495404173422880/-local-10003/HashTable-Stage-5/MapJoin-1--.hashtable
[junit] 11/10/25 10:03:57 WARN lazybinary.LazyBinaryStruct: Extra bytes
detected at the end of the row! Ignoring similar problems.
[junit] 11/10/25 10:03:57 INFO exec.MapJoinOperator: Load back 1
hashtable file from tmp file
uri:file:/tmp/root/hive_2011-10-25_10-03-53_617_6340495404173422880/-local-10003/HashTable-Stage-5/MapJoin-2--.hashtable
[junit] 11/10/25 10:03:57 WARN lazybinary.LazyBinaryStruct: Extra bytes
detected at the end of the row! Ignoring similar problems.
[junit] 11/10/25 10:03:57 INFO exec.MapOperator: 12 forwarding 1 rows
[junit] 11/10/25 10:03:57 INFO exec.TableScanOperator: 0 forwarding 1 rows
[junit] 11/10/25 10:03:57 FATAL ExecMapper:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"key":"238","value":"val_238"}
[junit] at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
[junit] at
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
[junit] at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
[junit] at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
[junit] at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
[junit] at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
[junit] Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text
incompatible with org.apache.hadoop.io.BooleanWritable
[junit] at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.joinObjectsRightOuterJoin(CommonJoinOperator.java:489)
[junit] at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.joinObjects(CommonJoinOperator.java:639)
[junit] at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:681)
[junit] at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:685)
[junit] at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:685)
[junit] at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:845)
[junit] at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:264)
[junit] at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
[junit] at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
[junit] at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:78)
[junit] at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
[junit] at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
[junit] at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:527)
[junit] ... 5 more
[junit]
[junit] 11/10/25 10:03:57 INFO exec.MapOperator: 12 finished. closing...
[junit] 11/10/25 10:03:57 INFO exec.MapOperator: 12 forwarded 1 rows
[junit] 11/10/25 10:03:57 INFO exec.MapOperator: DESERIALIZE_ERRORS:0
[junit] 11/10/25 10:03:57 INFO exec.TableScanOperator: 0 finished.
closing...
[junit] 11/10/25 10:03:57 INFO exec.TableScanOperator: 0 forwarded 1 rows
[junit] 11/10/25 10:03:57 INFO exec.TableScanOperator: 0 Close done
[junit] 11/10/25 10:03:57 INFO exec.MapOperator: 12 Close done
[junit] 11/10/25 10:03:57 INFO exec.HashTableDummyOperator: 4 finished.
closing...
[junit] 11/10/25 10:03:57 INFO exec.MapOperator: 12 Close done
[junit] 11/10/25 10:03:57 INFO ex
