[
https://issues.apache.org/jira/browse/HIVE-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741672#action_12741672
]
Samuel Guo commented on HIVE-705:
---------------------------------
@schubert,
Thank you for you comment.
>> In you patch, we found many java files are modified, it is really a big
>> effort. I don't know if there is any way to avoid such a big modification.
A HBase Table is quite different with a file in HDFS. The original Hive code is
based on files. For example, when outputting the reduce results to the target
table, Hive uses a FileSinkOperator to output the results to the temp file in
the HDFS, and uses a MoveTask to rename the temp files in the HDFS to the
target table dir. But when the the target table is based on a HBase Table, we
do not need to deal with these file operations, and just output to the target
HBase Table.
The modification of the original java files is to tell hive to deal with a
hbase table in a differnt way.
I will try to look into the code and find a way to avoid the modification.
>> 2. The performance is not good when we maped SQL columns to HBase columns in
>> our past experience. For example, we have a table with 20 columns, then,
>> each read or write of a row will comprise 20 key-value operations. It is
>> ineffective.
A good point. The schema mapping does not effect the peformance during creating
a hive table. The performance is effected if we get all the mapping columns out
of hbase table in an actual query operation. Some code will be added to do the
column-prune during hbase table scanning.
For example, an hbase table (cf1:(co1, col2, col3), cf2:(col4,col5,col6), ... ,
cfn:(colk,colj,coll)) is mapping to a hive table (column1, column2, column3,
column4, ... ,column n).
If a query "select column3, column4 from hbasedhivetable" is invoked, we should
not let hbase scan out all the columns. We know all the hive columns used in
the query, map back to the hbase column, and get the scanning list "cf1:col3
cf2:col4". We set the scanning list "cf1:col3 cf2:col4" in the HBaseInputFormat
to let HBase just scan out the useful columns.
The code will be added in the new patch.
>> cf2: => {(col3, col5, col6), Default SerDe}
Cool. Let different SerDe work on different hbase column. I will try it in the
new patch.
>> Look forward to have more communication with you in Chinese, by your
>> convenience.
My Gtalk is : [email protected]
> Let Hive can analyse hbase's tables
> -----------------------------------
>
> Key: HIVE-705
> URL: https://issues.apache.org/jira/browse/HIVE-705
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Samuel Guo
> Attachments: hbase-0.19.3-test.jar, hbase-0.19.3.jar,
> HIVE-705_draft.patch
>
>
> Add a serde over the hbase's tables, so that hive can analyse the data stored
> in hbase easily.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.