回复: hive integrate with hbase, map to existed hbase table report column family not exist
i got the reason, the column mapping section can't have any white space,
including \r\n
Chris Gong
发件人： Chris Gong
发送时间： 2012-11-06 10:56
收件人： user-hive
主题： hive integrate with hbase, map to existed hbase table report column family
not exist
hi all:
now, I'm map to an existed hbase table, i got the following infomation as:
FAILED: Error in metadata: MetaException(message:Column Family
data is not defined in hbase table df_money_files)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
my hive QL is
create external table hbase_money_files (rowkey string,
user_no string,
mon int,
mon_sn int,
group_no int,
sn int,
write_sect_no string,
write_sn int,
business_place_code string,
power_no int,
trans_group int,
price_code string,
ts_flag string,
elec_type_code string,
trade_type_code string,
ms_mode string,
user_ms_type string,
write_power double,
chg_powerdouble,
add_powerdouble,
kb_power double,
share_power double,
total_power double,
total_money double,
num_moneydouble,
add_money1 double,
add_money2 double,
add_money3 double,
add_money4 double,
add_money5 double,
add_money6 double,
add_money7 double,
add_money8 double,
add_money9 double,
add_money10 double,
rp_power double,
rp_money double,
should_money double,
create_date string,
creator string,
warrant_no int,
line_codestring,
trans_no string,
add_taxflag string,
write_date string,
compute_date string,
calculator_id string,
statusstring,
user_type1string,
rela_user_no string,
part_sn int,
have_ext string,
id_fragment string,
check_date string,
check_manstring,
start_date string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,data:user_no,
data:mon,
data:mon_sn,
data:group_no,
data:sn,
data:write_sect_no,
data:write_sn,
data:business_place_code,
data:power_no,
data:trans_group,
data:price_code,
data:ts_flag,
data:elec_type_code,
data:trade_type_code,
data:ms_mode,
data:user_ms_type,
data:write_power,
data:chg_power,
data:add_power,
data:kb_power,
data:share_power,
data:total_power,
data:total_money,
data:num_money,
data:add_money1,
data:add_money2,
data:add_money3,
data:add_money4,
data:add_money5,
data:add_money6,
data:add_money7,
data:add_money8,
data:add_money9,
data:add_money10,
data:rp_power,
data:rp_money,
data:should_money,
data:create_date,
data:creator,
data:warrant_no,
data:line_code,
data:trans_no,
data:add_taxflag,
data:write_date,
data:compute_date,
data:calculator_id,
data:status,
data:user_type1,
data:rela_user_no,
data:part_sn,
data:have_ext,
data:id_fragment,
data:check_date,
data:check_man,
data:start_date")
TBLPROPERTIES("hbase.table.name" = "df_money_files");
however the data column does exist! when i describe the table in hbase shell,
it reported:
hbase(main):001:0> describe 'df_money_files'
DESCRIPTION ENABLED
{NAME => 'df_money_files', FAMILIES => [{NAME => 'd true
ata', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '
0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VER
SIONS => '0', TTL => '2147483647', BLOCKSIZE => '65
536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
1 row(s) in 0.8470 seconds
i am confused now, anyone can give some infomation?
Chris Gong
hive integrate with hbase, map to existed hbase table report column family not exist
hi all:
now, I'm map to an existed hbase table, i got the following infomation as:
FAILED: Error in metadata: MetaException(message:Column Family
data is not defined in hbase table df_money_files)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
my hive QL is
create external table hbase_money_files (rowkey string,
user_no string,
mon int,
mon_sn int,
group_no int,
sn int,
write_sect_no string,
write_sn int,
business_place_code string,
power_no int,
trans_group int,
price_code string,
ts_flag string,
elec_type_code string,
trade_type_code string,
ms_mode string,
user_ms_type string,
write_power double,
chg_powerdouble,
add_powerdouble,
kb_power double,
share_power double,
total_power double,
total_money double,
num_moneydouble,
add_money1 double,
add_money2 double,
add_money3 double,
add_money4 double,
add_money5 double,
add_money6 double,
add_money7 double,
add_money8 double,
add_money9 double,
add_money10 double,
rp_power double,
rp_money double,
should_money double,
create_date string,
creator string,
warrant_no int,
line_codestring,
trans_no string,
add_taxflag string,
write_date string,
compute_date string,
calculator_id string,
statusstring,
user_type1string,
rela_user_no string,
part_sn int,
have_ext string,
id_fragment string,
check_date string,
check_manstring,
start_date string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,data:user_no,
data:mon,
data:mon_sn,
data:group_no,
data:sn,
data:write_sect_no,
data:write_sn,
data:business_place_code,
data:power_no,
data:trans_group,
data:price_code,
data:ts_flag,
data:elec_type_code,
data:trade_type_code,
data:ms_mode,
data:user_ms_type,
data:write_power,
data:chg_power,
data:add_power,
data:kb_power,
data:share_power,
data:total_power,
data:total_money,
data:num_money,
data:add_money1,
data:add_money2,
data:add_money3,
data:add_money4,
data:add_money5,
data:add_money6,
data:add_money7,
data:add_money8,
data:add_money9,
data:add_money10,
data:rp_power,
data:rp_money,
data:should_money,
data:create_date,
data:creator,
data:warrant_no,
data:line_code,
data:trans_no,
data:add_taxflag,
data:write_date,
data:compute_date,
data:calculator_id,
data:status,
data:user_type1,
data:rela_user_no,
data:part_sn,
data:have_ext,
data:id_fragment,
data:check_date,
data:check_man,
data:start_date")
TBLPROPERTIES("hbase.table.name" = "df_money_files");
however the data column does exist! when i describe the table in hbase shell,
it reported:
hbase(main):001:0> describe 'df_money_files'
DESCRIPTION ENABLED
{NAME => 'df_money_files', FAMILIES => [{NAME => 'd true
ata', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '
0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VER
SIONS => '0', TTL => '2147483647', BLOCKSIZE => '65
536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
1 row(s) in 0.8470 seconds
i am confused now, anyone can give some infomation?
Chris Gong
Re: Re: can i define an udf which can process more than one argument?
yes , it works , thank you very much Jan.
Chris Gong
From: Jan Dolinár
Date: 2012-10-19 13:17
To: user; fellowtree
Subject: Re: can i define an udf which can process more than one argument?
Hi Chris,
Of course it is possible to write UDF with as many parameters as you want, even
with variable number of arguments. Have a look at the nice article from Mark
Grover [1] about writing UDFs. It also contains link to a real-life example
[2].
Best regards,
Jan Dolinar
[1] http://mark.thegrovers.ca/1/post/2012/06/how-to-write-a-hive-udf.html
[2]
https://github.com/markgrover/hive-translate/blob/master/GenericUDFTranslate.java
On Fri, Oct 19, 2012 at 5:36 AM, Chris Gong wrote:
i find that UDF can only process one argument like
public long evaluate(String ip) {
}
can i define a udf like COALESCE(T v1, T v2, …) or if(boolean testCondition,
T valueTrue, T valueFalseOrNull)?
Chris Gong
can i define an udf which can process more than one argument?
i find that UDF can only process one argument like
public long evaluate(String ip) {
}
can i define a udf like COALESCE(T v1, T v2, …) or if(boolean testCondition,
T valueTrue, T valueFalseOrNull)?
Chris Gong
