[
https://issues.apache.org/jira/browse/HIVE-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746592#action_12746592
]
Samuel Guo commented on HIVE-705:
---------------------------------
Attach a new patch.
1) move the related hbase code to the contrib package, as hbase just an
optional storage for hive, not neccessary.
I have tried to avoid modifying the hive original code and just add a hbase
serde to connect hive with hbase. But the hbase storage model is quite
different with file storage model. For example, a loadwork is used to
rename/copy files from temp dir to the target table's dir if a query's target
is a hive table. But in a hbased hive table, we can't rename a table now. So
it's hard to let a hbased hive table to follow the logic of a normal file-based
hive table. So I add some code(HiveFormatUtils) to distinguish a file-based
table from a not-file-based table.
2) fix some bugs in the draft patch, such as "select *" return nothing.
----------------------------------------------------------------------------------------------
How to use the hbase as hive's storage?
1) remember to add the contrib jar and the hbase jar in the hive's auxPath, so
m/r can populate the neccessary hbase-related jars to the whole hadoop m/r
cluster.
> $HIVE_HOME/bin/hive -auxPath ${contrib_jar},${hbase_jar}
2) modify the configuration to add the following configuration parameters.
"hbase.master" : pointer to the hbase's master.
"hive.othermetadata.handlers" :
"org.apache.hadoop.hive.contrib.hbase.HiveHBaseTableInputFormat:org.apache.hadoop.hive.contrib.hbase.HBaseMetadataHandler"
"hive.othermetadata.handlers" collects the metadata handlers to handle the
other metadata operations in the not-file-based hive tables. Take hbase as an
example. HBaseMetadataHandler will create the neccessary hbase table and its
family columns when we create a hbased hive table from hive's client. It also
drop the hbase table when we drop the hive table.
The metastore read the registered handlers map from the configuration file
during initialization. The registered handlers map is formated as
"table_format_classname:table_metadata_handler_classname,table_format_classname:table_metadata_handler_classname,...".
3) enjoy "hive over hbase"!
------------------------------------------------------------------------
Other problems.
1) Altering a hased-hive table is not supported now. :(
renaming a table in hbase is not supported now, so I just do not support rename
operation. ( maybe if we rename a hive table, we do not need to rename the base
hbase table.)
adding/replacing cloumns.
Now we need to specify the schema mapping in the SerDe properties explicitly.
If we want to adding columns, we need to call 'alter' twice to adding columns:
change the serde properties and the hive columns. Either change the serde
properties first or change the hive columns first will fail now, because we
validate the schema mapping during SerDe initialization. One of the hbase serde
validation is to check the counts of hive columns and hbase mapping columns. If
we first change the hive columns, the number of hive columns will be more than
hbase mapping columns, the HBase Serde initialization will fail this alter
operation. (maybe we need to remove the validation code from HBaseSerDe
initialization and do it in other place?)
2) more flexible schema mapping?
As Schubert metioned before, more flexible schema mapping will be useful for
user. This feature will be added later.
welcome for comments~
> Let Hive can analyse hbase's tables
> -----------------------------------
>
> Key: HIVE-705
> URL: https://issues.apache.org/jira/browse/HIVE-705
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Samuel Guo
> Attachments: hbase-0.19.3-test.jar, hbase-0.19.3.jar,
> HIVE-705_draft.patch, HIVE-705_revision806905.patch
>
>
> Add a serde over the hbase's tables, so that hive can analyse the data stored
> in hbase easily.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.