Re: Hive-Hbase integration problem, ask for help
I've added this on as extra validation which ought to be added in HIVE-1222. JVS On Jun 15, 2010, at 3:59 PM, Basab Maulik wrote: I was not able to reproduce this problem on trunk (can't remember the label). The funny thing was both the create table and the insert overwrite worked even though the create table contained the invalid row format spec. Basab On Fri, Jun 11, 2010 at 1:33 PM, John Sichi jsi...@facebook.com wrote: You should not be specifying any ROW FORMAT for an HBase table. From the log in your earlier post, I couldn't tell what was going wrong; I don' think it contained the full exception stacks. You might be able to dig around in the actual log files to find more. JVS From: Zhou Shuaifeng [zhoushuaif...@huawei.com] Sent: Thursday, June 10, 2010 7:26 PM To: hive-dev@hadoop.apache.org Cc: 'zhaozhifeng 00129982' Subject: Hive-Hbase integration problem, ask for help Hi Guys, I download the hive source from SVN server, build it and try to run the hive-hbase integration. I works well on all file-based hive tables, but on the hbase-based tables, the 'insert' command cann't run successful. The 'select' command can run well. error info is below: hive INSERT OVERWRITE TABLE hive_zsf SELECT * FROM zsf WHERE id=3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201006081948_0021, Tracking URL = http://linux-01:50030/jobdetails.jsp?jobid=job_201006081948_0021 Kill Command = /opt/hadoop/hdfs/bin/../bin/hadoop job -Dmapred.job.tracker=linux-01:9001 -kill job_201006081948_0021 2010-06-09 16:05:43,898 Stage-0 map = 0%, reduce = 0% 2010-06-09 16:06:12,131 Stage-0 map = 100%, reduce = 100% Ended Job = job_201006081948_0021 with errors Task with the most failures(4): - Task ID: task_201006081948_0021_m_00 URL: http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021 http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021tipid=tas k_201006081948_0021_m_00http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021tipid=tas%0Ak_201006081948_0021_m_00 tipid=task_201006081948_0021_m_00 - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver I create a hbase-based table with hive, put some data into the hbase table through the hbase shell, and can select data from it through hive: CREATE TABLE hive_zsf1(id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf1:val) TBLPROPERTIES (hbase.table.name = hive_zsf1); hbase(main):001:0 scan 'hive_zsf1' ROW COLUMN+CELL 1 column=cf1:val, timestamp=1276157509028, value=zsf 2 column=cf1:val, timestamp=1276157539051, value=zzf 3 column=cf1:val, timestamp=1276157548247, value=zw 4 column=cf1:val, timestamp=1276157557115, value=cjl 4 row(s) in 0.0470 seconds hbase(main):002:0 hive select * from hive_zsf1 where id=3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201006081948_0038, Tracking URL = http://linux-01:50030/jobdetails.jsp?jobid=job_201006081948_0038 Kill Command = /opt/hadoop/hdfs/bin/../bin/hadoop job -Dmapred.job.tracker=linux-01:9001 -kill job_201006081948_0038 2010-06-11 10:25:42,049 Stage-1 map = 0%, reduce = 0% 2010-06-11 10:25:45,090 Stage-1 map = 100%, reduce = 0% 2010-06-11 10:25:48,133 Stage-1 map = 100%, reduce = 100% Ended Job = job_201006081948_0038 OK 3 zw Time taken: 13.526 seconds hive - This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
RE: Hive-Hbase integration problem, ask for help
I solved this problem by deleting all the jars previously existed and rebuilding the hive source. But still don't know what's the reason of the previous problem. Zhou -Original Message- From: John Sichi [mailto:jsi...@facebook.com] Sent: Friday, June 18, 2010 4:12 AM To: hive-dev@hadoop.apache.org Cc: zhaozhifeng 00129982; zhoushuaif...@huawei.com Subject: Re: Hive-Hbase integration problem, ask for help I've added this on as extra validation which ought to be added in HIVE-1222. JVS On Jun 15, 2010, at 3:59 PM, Basab Maulik wrote: I was not able to reproduce this problem on trunk (can't remember the label). The funny thing was both the create table and the insert overwrite worked even though the create table contained the invalid row format spec. Basab On Fri, Jun 11, 2010 at 1:33 PM, John Sichi jsi...@facebook.com wrote: You should not be specifying any ROW FORMAT for an HBase table. From the log in your earlier post, I couldn't tell what was going wrong; I don' think it contained the full exception stacks. You might be able to dig around in the actual log files to find more. JVS From: Zhou Shuaifeng [zhoushuaif...@huawei.com] Sent: Thursday, June 10, 2010 7:26 PM To: hive-dev@hadoop.apache.org Cc: 'zhaozhifeng 00129982' Subject: Hive-Hbase integration problem, ask for help Hi Guys, I download the hive source from SVN server, build it and try to run the hive-hbase integration. I works well on all file-based hive tables, but on the hbase-based tables, the 'insert' command cann't run successful. The 'select' command can run well. error info is below: hive INSERT OVERWRITE TABLE hive_zsf SELECT * FROM zsf WHERE id=3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201006081948_0021, Tracking URL = http://linux-01:50030/jobdetails.jsp?jobid=job_201006081948_0021 Kill Command = /opt/hadoop/hdfs/bin/../bin/hadoop job -Dmapred.job.tracker=linux-01:9001 -kill job_201006081948_0021 2010-06-09 16:05:43,898 Stage-0 map = 0%, reduce = 0% 2010-06-09 16:06:12,131 Stage-0 map = 100%, reduce = 100% Ended Job = job_201006081948_0021 with errors Task with the most failures(4): - Task ID: task_201006081948_0021_m_00 URL: http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021 http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021tip id=tas k_201006081948_0021_m_00http://linux-01:50030/taskdetails.jsp?jo bid=job_201006081948_0021tipid=tas%0Ak_201006081948_0021_m_00 tipid=task_201006081948_0021_m_00 - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver I create a hbase-based table with hive, put some data into the hbase table through the hbase shell, and can select data from it through hive: CREATE TABLE hive_zsf1(id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf1:val) TBLPROPERTIES (hbase.table.name = hive_zsf1); hbase(main):001:0 scan 'hive_zsf1' ROW COLUMN+CELL 1 column=cf1:val, timestamp=1276157509028, value=zsf 2 column=cf1:val, timestamp=1276157539051, value=zzf 3 column=cf1:val, timestamp=1276157548247, value=zw 4 column=cf1:val, timestamp=1276157557115, value=cjl 4 row(s) in 0.0470 seconds hbase(main):002:0 hive select * from hive_zsf1 where id=3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201006081948_0038, Tracking URL = http://linux-01:50030/jobdetails.jsp?jobid=job_201006081948_0038 Kill Command = /opt/hadoop/hdfs/bin/../bin/hadoop job -Dmapred.job.tracker=linux-01:9001 -kill job_201006081948_0038 2010-06-11 10:25:42,049 Stage-1 map = 0%, reduce = 0% 2010-06-11 10:25:45,090 Stage-1 map = 100%, reduce = 0% 2010-06-11 10:25:48,133 Stage-1 map = 100%, reduce = 100% Ended Job = job_201006081948_0038 OK 3 zw Time taken: 13.526 seconds hive - --- - This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
Re: Hive-Hbase integration problem, ask for help
I was not able to reproduce this problem on trunk (can't remember the label). The funny thing was both the create table and the insert overwrite worked even though the create table contained the invalid row format spec. Basab On Fri, Jun 11, 2010 at 1:33 PM, John Sichi jsi...@facebook.com wrote: You should not be specifying any ROW FORMAT for an HBase table. From the log in your earlier post, I couldn't tell what was going wrong; I don' think it contained the full exception stacks. You might be able to dig around in the actual log files to find more. JVS From: Zhou Shuaifeng [zhoushuaif...@huawei.com] Sent: Thursday, June 10, 2010 7:26 PM To: hive-dev@hadoop.apache.org Cc: 'zhaozhifeng 00129982' Subject: Hive-Hbase integration problem, ask for help Hi Guys, I download the hive source from SVN server, build it and try to run the hive-hbase integration. I works well on all file-based hive tables, but on the hbase-based tables, the 'insert' command cann't run successful. The 'select' command can run well. error info is below: hive INSERT OVERWRITE TABLE hive_zsf SELECT * FROM zsf WHERE id=3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201006081948_0021, Tracking URL = http://linux-01:50030/jobdetails.jsp?jobid=job_201006081948_0021 Kill Command = /opt/hadoop/hdfs/bin/../bin/hadoop job -Dmapred.job.tracker=linux-01:9001 -kill job_201006081948_0021 2010-06-09 16:05:43,898 Stage-0 map = 0%, reduce = 0% 2010-06-09 16:06:12,131 Stage-0 map = 100%, reduce = 100% Ended Job = job_201006081948_0021 with errors Task with the most failures(4): - Task ID: task_201006081948_0021_m_00 URL: http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021 http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021tipid=tas k_201006081948_0021_m_00http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021tipid=tas%0Ak_201006081948_0021_m_00 tipid=task_201006081948_0021_m_00 - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver I create a hbase-based table with hive, put some data into the hbase table through the hbase shell, and can select data from it through hive: CREATE TABLE hive_zsf1(id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf1:val) TBLPROPERTIES (hbase.table.name = hive_zsf1); hbase(main):001:0 scan 'hive_zsf1' ROW COLUMN+CELL 1 column=cf1:val, timestamp=1276157509028, value=zsf 2 column=cf1:val, timestamp=1276157539051, value=zzf 3 column=cf1:val, timestamp=1276157548247, value=zw 4 column=cf1:val, timestamp=1276157557115, value=cjl 4 row(s) in 0.0470 seconds hbase(main):002:0 hive select * from hive_zsf1 where id=3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201006081948_0038, Tracking URL = http://linux-01:50030/jobdetails.jsp?jobid=job_201006081948_0038 Kill Command = /opt/hadoop/hdfs/bin/../bin/hadoop job -Dmapred.job.tracker=linux-01:9001 -kill job_201006081948_0038 2010-06-11 10:25:42,049 Stage-1 map = 0%, reduce = 0% 2010-06-11 10:25:45,090 Stage-1 map = 100%, reduce = 0% 2010-06-11 10:25:48,133 Stage-1 map = 100%, reduce = 100% Ended Job = job_201006081948_0038 OK 3 zw Time taken: 13.526 seconds hive - This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
RE: Hive-Hbase integration problem, ask for help
You should not be specifying any ROW FORMAT for an HBase table. From the log in your earlier post, I couldn't tell what was going wrong; I don' think it contained the full exception stacks. You might be able to dig around in the actual log files to find more. JVS From: Zhou Shuaifeng [zhoushuaif...@huawei.com] Sent: Thursday, June 10, 2010 7:26 PM To: hive-dev@hadoop.apache.org Cc: 'zhaozhifeng 00129982' Subject: Hive-Hbase integration problem, ask for help Hi Guys, I download the hive source from SVN server, build it and try to run the hive-hbase integration. I works well on all file-based hive tables, but on the hbase-based tables, the 'insert' command cann't run successful. The 'select' command can run well. error info is below: hive INSERT OVERWRITE TABLE hive_zsf SELECT * FROM zsf WHERE id=3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201006081948_0021, Tracking URL = http://linux-01:50030/jobdetails.jsp?jobid=job_201006081948_0021 Kill Command = /opt/hadoop/hdfs/bin/../bin/hadoop job -Dmapred.job.tracker=linux-01:9001 -kill job_201006081948_0021 2010-06-09 16:05:43,898 Stage-0 map = 0%, reduce = 0% 2010-06-09 16:06:12,131 Stage-0 map = 100%, reduce = 100% Ended Job = job_201006081948_0021 with errors Task with the most failures(4): - Task ID: task_201006081948_0021_m_00 URL: http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021 http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021tipid=tas k_201006081948_0021_m_00 tipid=task_201006081948_0021_m_00 - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver I create a hbase-based table with hive, put some data into the hbase table through the hbase shell, and can select data from it through hive: CREATE TABLE hive_zsf1(id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf1:val) TBLPROPERTIES (hbase.table.name = hive_zsf1); hbase(main):001:0 scan 'hive_zsf1' ROW COLUMN+CELL 1 column=cf1:val, timestamp=1276157509028, value=zsf 2 column=cf1:val, timestamp=1276157539051, value=zzf 3 column=cf1:val, timestamp=1276157548247, value=zw 4 column=cf1:val, timestamp=1276157557115, value=cjl 4 row(s) in 0.0470 seconds hbase(main):002:0 hive select * from hive_zsf1 where id=3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201006081948_0038, Tracking URL = http://linux-01:50030/jobdetails.jsp?jobid=job_201006081948_0038 Kill Command = /opt/hadoop/hdfs/bin/../bin/hadoop job -Dmapred.job.tracker=linux-01:9001 -kill job_201006081948_0038 2010-06-11 10:25:42,049 Stage-1 map = 0%, reduce = 0% 2010-06-11 10:25:45,090 Stage-1 map = 100%, reduce = 0% 2010-06-11 10:25:48,133 Stage-1 map = 100%, reduce = 100% Ended Job = job_201006081948_0038 OK 3 zw Time taken: 13.526 seconds hive - This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
Hive-Hbase integration problem, ask for help
Hi Guys, I download the hive source from SVN server, build it and try to run the hive-hbase integration. I works well on all file-based hive tables, but on the hbase-based tables, the 'insert' command cann't run successful. The 'select' command can run well. error info is below: hive INSERT OVERWRITE TABLE hive_zsf SELECT * FROM zsf WHERE id=3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201006081948_0021, Tracking URL = http://linux-01:50030/jobdetails.jsp?jobid=job_201006081948_0021 Kill Command = /opt/hadoop/hdfs/bin/../bin/hadoop job -Dmapred.job.tracker=linux-01:9001 -kill job_201006081948_0021 2010-06-09 16:05:43,898 Stage-0 map = 0%, reduce = 0% 2010-06-09 16:06:12,131 Stage-0 map = 100%, reduce = 100% Ended Job = job_201006081948_0021 with errors Task with the most failures(4): - Task ID: task_201006081948_0021_m_00 URL: http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021 http://linux-01:50030/taskdetails.jsp?jobid=job_201006081948_0021tipid=tas k_201006081948_0021_m_00 tipid=task_201006081948_0021_m_00 - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver I create a hbase-based table with hive, put some data into the hbase table through the hbase shell, and can select data from it through hive: CREATE TABLE hive_zsf1(id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf1:val) TBLPROPERTIES (hbase.table.name = hive_zsf1); hbase(main):001:0 scan 'hive_zsf1' ROW COLUMN+CELL 1 column=cf1:val, timestamp=1276157509028, value=zsf 2 column=cf1:val, timestamp=1276157539051, value=zzf 3 column=cf1:val, timestamp=1276157548247, value=zw 4 column=cf1:val, timestamp=1276157557115, value=cjl 4 row(s) in 0.0470 seconds hbase(main):002:0 hive select * from hive_zsf1 where id=3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201006081948_0038, Tracking URL = http://linux-01:50030/jobdetails.jsp?jobid=job_201006081948_0038 Kill Command = /opt/hadoop/hdfs/bin/../bin/hadoop job -Dmapred.job.tracker=linux-01:9001 -kill job_201006081948_0038 2010-06-11 10:25:42,049 Stage-1 map = 0%, reduce = 0% 2010-06-11 10:25:45,090 Stage-1 map = 100%, reduce = 0% 2010-06-11 10:25:48,133 Stage-1 map = 100%, reduce = 100% Ended Job = job_201006081948_0038 OK 3 zw Time taken: 13.526 seconds hive - This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
