[ 
https://issues.apache.org/jira/browse/HIVE-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741672#action_12741672
 ] 

Samuel Guo commented on HIVE-705:
---------------------------------

@schubert,

Thank you for you comment.

>> In you patch, we found many java files are modified, it is really a big 
>> effort. I don't know if there is any way to avoid such a big modification.
A HBase Table is quite different with a file in HDFS. The original Hive code is 
based on files. For example, when outputting the reduce results to the target 
table, Hive uses a FileSinkOperator to output the results to the temp file in 
the HDFS, and uses a MoveTask to rename the temp files in the HDFS to the 
target table dir. But when the the target table is based on a HBase Table, we 
do not need to deal with these file operations, and just output to the target 
HBase Table.

The modification of the original java files is to tell hive to deal with a 
hbase table in a differnt way. 

I will try to look into the code and find a way to avoid the modification.

>> 2. The performance is not good when we maped SQL columns to HBase columns in 
>> our past experience. For example, we have a table with 20 columns, then, 
>> each read or write of a row will comprise 20 key-value operations. It is 
>> ineffective.

A good point. The schema mapping does not effect the peformance during creating 
a hive table. The performance is effected if we get all the mapping columns out 
of hbase table in an actual query operation. Some code will be added to do the 
column-prune during hbase table scanning.

For example, an hbase table (cf1:(co1, col2, col3), cf2:(col4,col5,col6), ... , 
cfn:(colk,colj,coll)) is mapping to a hive table (column1, column2, column3, 
column4, ... ,column n).
If a query "select column3, column4 from hbasedhivetable" is invoked, we should 
not let hbase scan out all the columns. We know all the hive columns used in 
the query, map back to the hbase column, and get the scanning list "cf1:col3 
cf2:col4". We set the scanning list "cf1:col3 cf2:col4" in the HBaseInputFormat 
to let HBase just scan out the useful columns.

The code will be added in the new patch.

>> cf2: => {(col3, col5, col6), Default SerDe} 

Cool. Let different SerDe work on different hbase column.  I will try it in the 
new patch.

>> Look forward to have more communication with you in Chinese, by your 
>> convenience.
My Gtalk is : [email protected]


> Let Hive can analyse hbase's tables
> -----------------------------------
>
>                 Key: HIVE-705
>                 URL: https://issues.apache.org/jira/browse/HIVE-705
>             Project: Hadoop Hive
>          Issue Type: New Feature
>            Reporter: Samuel Guo
>         Attachments: hbase-0.19.3-test.jar, hbase-0.19.3.jar, 
> HIVE-705_draft.patch
>
>
> Add a serde over the hbase's tables, so that hive can analyse the data stored 
> in hbase easily.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to