Re: Mapping HBase table in Hive
Thanks Bejoy, what do you mean by: If you need to map a full CF to a hive column, the data type of the hive column should be a Map. suppose I used sqoop to move data from mysql to hbase and used id as a column family, all the other columns will be QF then, right? The integration document is not clear, I think it needs more clarification or maybe I am still missing something. -- Ibrahim On Tue, Jan 8, 2013 at 9:35 PM, bejoy...@yahoo.com wrote: data type of
Re: Mapping HBase table in Hive
Hi Ibrahim. SQOOP is used to import data from rdbms to hbase in your case. Please get the schema from hbase for your corresponding table and post it here. We can point out how your mapping could be. Regards Bejoy KS Sent from remote device, Please excuse typos -Original Message- From: Ibrahim Yakti iya...@souq.com Date: Sun, 13 Jan 2013 11:22:51 To: useruser@hive.apache.org Reply-To: user@hive.apache.org Subject: Re: Mapping HBase table in Hive Thanks Bejoy, what do you mean by: If you need to map a full CF to a hive column, the data type of the hive column should be a Map. suppose I used sqoop to move data from mysql to hbase and used id as a column family, all the other columns will be QF then, right? The integration document is not clear, I think it needs more clarification or maybe I am still missing something. -- Ibrahim On Tue, Jan 8, 2013 at 9:35 PM, bejoy...@yahoo.com wrote: data type of
Re: Mapping HBase table in Hive
Thanks Bejoy, Seems it worked, in the mapping of the column family I used :key and that's it, in addition as per some articles there should be no spaces in mapping, below is the create table sample: CREATE EXTERNAL TABLE hbase_orders(id bigint, value bigint, date_lastchange string, date_inserted string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,date_inserted:bigint,date_lastchange:string,value:string) TBLPROPERTIES (hbase.table.name = orders); Thanks. -- Ibrahim On Sun, Jan 13, 2013 at 12:23 PM, bejoy...@yahoo.com wrote: ** Hi Ibrahim. SQOOP is used to import data from rdbms to hbase in your case. Please get the schema from hbase for your corresponding table and post it here. We can point out how your mapping could be. Regards Bejoy KS Sent from remote device, Please excuse typos -- *From: * Ibrahim Yakti iya...@souq.com *Date: *Sun, 13 Jan 2013 11:22:51 +0300 *To: *useruser@hive.apache.org *ReplyTo: * user@hive.apache.org *Subject: *Re: Mapping HBase table in Hive Thanks Bejoy, what do you mean by: If you need to map a full CF to a hive column, the data type of the hive column should be a Map. suppose I used sqoop to move data from mysql to hbase and used id as a column family, all the other columns will be QF then, right? The integration document is not clear, I think it needs more clarification or maybe I am still missing something. -- Ibrahim On Tue, Jan 8, 2013 at 9:35 PM, bejoy...@yahoo.com wrote: data type of
Got a hadoop server IPC version mismatch ERROR in TestCliDriver avro_joins.q
Hi, guys I applied the patches for HIVE-895 ( add SerDe for Avro serialized data ) and HIVE-3273 (Add avro jars into hive execution classpath ) on Hive-0.9.0. And then I ran the following command with hadoop-1.0.3 and avro-1.6.3 ant test -Dtestcase=TestCliDriver -Dqfile=avro_joins.q -Dtest.silent=false But I got an ERROR from hadoop in unit test. ( I can ran avro_joins.q successfully in a real hadoop-1.0.3 cluster). I found that IPC version 7 is from hadoop 2.x and version 4 is from hadoop-1.x, but I didn't set hadoop 2.x in any properties files. Do you know how this happened in unit test? Thanks, - Bing ERROR [junit] Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4 [junit] at org.apache.hadoop.ipc.Client.call(Client.java:740) [junit] at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) [junit] at $Proxy1.getProtocolVersion(Unknown Source) [junit] at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) [junit] at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) [junit] at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:207) [junit] at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:170) [junit] at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82) [junit] at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) [junit] at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) [junit] at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) [junit] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) [junit] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) [junit] at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:367) [junit] ... 10 more [junit] Job Submission failed with exception 'java.lang.RuntimeException(org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4)'
