Hi Bejoy, My normal hive query works but I am receiving an error while inserting data into hbase table..
see attached log.. hive> select * from pokes where foo=9; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201207260911_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207260911_0005 Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker= 0.0.0.0:8021 -kill job_201207260911_0005 2012-07-26 15:07:29,827 Stage-1 map = 0%, reduce = 0% 2012-07-26 15:07:34,942 Stage-1 map = 100%, reduce = 0% 2012-07-26 15:07:39,292 Stage-1 map = 100%, reduce = 100% Ended Job = job_201207260911_0005 OK 9 val_9 Time taken: 19.359 seconds hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where foo=9; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201207260911_0006, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207260911_0006 Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker= 0.0.0.0:8021 -kill job_201207260911_0006 2012-07-26 15:08:14,784 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:09:14,977 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:10:15,947 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:11:16,814 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:12:17,692 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:13:18,615 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:14:19,448 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:15:20,353 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:16:21,197 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:17:21,972 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:18:22,148 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:19:23,061 Stage-0 map = 0%, reduce = 0% 2012-07-26 15:19:24,079 Stage-0 map = 100%, reduce = 100% Ended Job = job_201207260911_0006 with errors FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask On Thu, Jul 26, 2012 at 3:39 PM, Bejoy Ks <bejoy...@yahoo.com> wrote: > Hi Vijay > > Your current error looks like some issue with the Select query. Is the > select query working as desired? > hive> SELECT * FROM pokes where foo=98; > > Regards > Bejoy KS > > ------------------------------ > *From:* vijay shinde <vijaysanj...@gmail.com> > *To:* user@hive.apache.org; Bejoy Ks <bejoy...@yahoo.com> > *Sent:* Friday, July 27, 2012 12:24 AM > > *Subject:* Re: HBASE and HIVE Integration > > Hi Bejoy, > > Thanks for your reply. My Hbase is working fine independently. > I was able reolve the zookeeper error. I had issues with hive-site.xml > configuration.. > > Now my job run little bit longer and this is the lastest error > > I am on Cloudera CDH3U4 demo VM > > Number of reduce tasks is set to 0 since there's no reduce operator > Starting Job = job_201207260911_0003, Tracking URL = > http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207260911_0003 > Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker= > 0.0.0.0:8021 -kill job_201207260911_0003 > 2012-07-26 13:34:55,900 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:35:56,207 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:36:56,364 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:37:57,342 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:38:57,398 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:39:57,467 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:40:57,852 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:41:58,724 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:42:59,700 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:44:00,582 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:45:00,977 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:46:01,110 Stage-0 map = 0%, reduce = 0% > 2012-07-26 13:46:05,443 Stage-0 map = 100%, reduce = 100% > Ended Job = job_201207260911_0003 with errors > > hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where > foo=98; > > java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing row {"foo":98,"bar":"val_98"} > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) > at org.apache.hadoop.mapred.Child.main(Child.java:264) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row {"foo":98,"bar":"val_98"} > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546) > at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) > ... 8 more > Caused by: or > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.MapRedTask > hive> > > > > On Thu, Jul 26, 2012 at 9:55 AM, Bejoy Ks <bejoy...@yahoo.com> wrote: > > Hi Vijay > > Is your hbase working independently without any issues. I mean, are you > able to insert data into hbase tables without using hive integration? > > Was the same error message thrown when you directly provided hbase.master > instead of zookeeper quorum? > > Regards > Bejoy KS > > > ________________________________ > From: Hezhiqiang (Ransom) <ransom.hezhiqi...@huawei.com> > To: "user@hive.apache.org" <user@hive.apache.org> > Cc: Zhaojun (Terry) <terry.zhao...@huawei.com> > Sent: Thursday, July 26, 2012 9:06 AM > Subject: RE: HBASE and HIVE Integration > > > > Hi，Vijay > You need to add zookeeper.jar to the hive-site.xml > <name>hive.aux.jars.path</name> > <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar, > file:///usr/lib/hive/lib/ zookeeper-3.4.3.jar</value> > </property> > <name>hbase.zookeeper.quorum</name> > <value>localhost</value> > </property> > > > Regards > Ransom > > From:vijay shinde [mailto:vijaysanj...@gmail.com] > Sent: Thursday, July 26, 2012 8:59 AM > To: user@hive.apache.org; Bejoy Ks > Subject: Re: HBASE and HIVE Integration > > Hi Bejoy, > > I made some changes as per your suggetion. > > Here is the error from the > http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 Job: > > Error: java.lang.ClassNotFoundException: > org.apache.zookeeper.KeeperException > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at sun.misc.Launcher$ > > I went ahead and updated the hadoop-env.sh file and set the class path for > hbase and zookeeper as follows: > # Extra Java CLASSPATH elements. Optional. > export > HADOOP_CLASSPATH="/usr/lib/hive/lib/hbase-0.92.jar:/usr/lib/hive/lib/zookeeper-3.4.3.jar:$HADOOP_CLASSPATH" > > Here is snippet of hive-site.xml file > > <name>hive.aux.jars.path</name> > > <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar</value> > </property> > <name>hbase.zookeeper.quorum</name> > <value>localhost</value> > </property> > > Error message while executing hive query > > [root@localhost hive]# ./bin/hive > Hive history file=/tmp/root/hive_job_log_root_201207252044_1993919630.txt > hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where > foo=98; > Total MapReduce jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks is set to 0 since there's no reduce operator > Starting Job = job_201207251858_0004, Tracking URL = > http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 > Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker= > 0.0.0.0:8021 -kill job_201207251858_0004 > 2012-07-25 20:46:38,207 Stage-0 map = 0%, reduce = 0% > 2012-07-25 20:47:35,920 Stage-0 map = 100%, reduce = 100% > Ended Job = job_201207251858_0004 with errors > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.MapRedTask > hive> > > I am lost need help badly!!!!!!!!!!!!! > > Vijay > > > > On Wed, Jul 25, 2012 at 9:47 AM, Bejoy Ks <bejoy...@yahoo.com> wrote: > Hi Vijay > > You have provided the hbase master directly. (It is fine for single node > hbase installation). But still can you try providing the zookeeper quorum > instead. > > If that doesn't work as well , please post the error log from the > mapreduce tasks? > > Just go the jobtracker page and drill down on the corresponding job to get > the failed tasks. From each failed tasks you can get the error logs. > http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005 > > > Regards > Bejoy KS > > > ________________________________ > > From:vijay shinde <vijaysanj...@gmail.com> > To:user@hive.apache.org; bejoy...@yahoo.com > Sent: Wednesday, July 25, 2012 6:58 PM > Subject: Re: HBASE and HIVE Integration > > Hi Bejoy, > > Thanks for quick reply. Here are some additional details > > Cloudera Version - CDH3U4 > > hive-site.xml > > > <property> > <name>hive.aux.jars.path</name> > > <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value> > </property> > Execution Log > > 1. start zookeeper > [root@localhost zookeeper]# ./bin/zkServer.sh start > > 2. start hbase > > 3. start hive. I am setting hive jars in hive-site.xml > > ./bin/hive -hiveconf hbase.master=127.0.1.1:60010 > > 4. Create new HBase table which is to be managed by Hive > > CREATE TABLE hive_hbasetable_k(key int, value string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") > TBLPROPERTIES ("hbase.table.name" = "hivehbasek"); > > 5. Create a logical table pokes in Hive > CREATE TABLE pokes (foo INT, bar STRING); > 6. HIve error while inserting the data from Hive Poke table to HBASE table > hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE > foo=98; > Total MapReduce jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks is set to 0 since there's no reduce operator > Starting Job = job_201207250246_0005, Tracking URL = > http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005 > Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker= > 0.0.0.0:8021 -kill job_201207250246_0005 > 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0% > 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0% > 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100% > Ended Job = job_201207250246_0005 with errors > > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.MapRedTask > Let me know if you need any additonal information. > > Thanks, > Vijay > On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <bejoy...@yahoo.com> wrote: > Hi Vijay > > Can you share more details like > > The CDH Version/Hive version you are using > > Steps you followed for hive hbase integration with the values you set > > The DDL used for hive hbase integration > > The actual error from failed map reduce task > Regards > Bejoy KS > > Sent from handheld, please excuse typos. > > ________________________________ > > From: vijay shinde <vijaysanj...@gmail.com> > Date: Wed, 25 Jul 2012 04:45:41 -0400 > To: <user@hive.apache.org> > ReplyTo: user@hive.apache.org > Subject: HBASE and HIVE Integration > > I am facing issue while executing HIVE queries on HBASE-HIVE integration. > I followed the wiki hbase-hive integration > https://cwiki.apache.org/Hive/hbaseintegration.html > > I have already passed all the required jars for auxpath in hive-site.xml > file. > I am using Cloudera CDH demo VM.. Any help would be highly appreciated > > hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE > foo=98; > Total MapReduce jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks is set to 0 since there's no reduce operator > Starting Job = job_201207250246_0005, Tracking URL = > http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005 > Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker= > 0.0.0.0:8021 -kill job_201207250246_0005 > 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0% > 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0% > 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100% > Ended Job = job_201207250246_0005 with errors > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.MapRedTask > hive> > > > > >
