It seems that the snapshotDir is not correctly set. I have set the
hive.hbase.snapshot.restoredir. Is the hive.hbase.snapshot.restoredir to
control the snapshotDir? if not, how can i set the snapshotDir when I use hive
to query on hbase snapshot.
------------------ ???????? ------------------
??????: "????????????";<510688...@qq.com>;
????????: 2015??9??24??(??????) ????7:44
??????: "user"<user@hive.apache.org>;
????: ?????? Hive Query on Hbase snapshot error
Thank you for you reply.
The snapshot should be no problem. Here is the following output for the giving
command.
2015-09-24 19:37:41,981 INFO [main] Configuration.deprecation:
hadoop.native.lib is deprecated. Instead, use io.native.lib.available
Snapshot Info
----------------------------------------
Name: test_snapshot
Type: FLUSH
Table: test_table
Format: 2
Created: 2015-09-24T16:00:11
Table Descriptor
----------------------------------------
'test_table', {TABLE_ATTRIBUTES => {coprocessor$1 =>
'|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2
=>
'|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
coprocessor$3 =>
'|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
coprocessor$4 =>
'|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
coprocessor$5 =>
'|org.apache.hadoop.hbase.regionserver.LocalIndexSplitter|805306366|'}, {NAME
=> 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS
=> '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}
32 HFiles (0 in archive), total size 236.1 G (100.00% 236.1 G shared with the
source table)
0 Logs, total size 0
------------------ ???????? ------------------
??????: "Sandeep Nemuri";<nhsande...@gmail.com>;
????????: 2015??9??24??(??????) ????6:43
??????: "Hive user group"<user@hive.apache.org>;
????: Re: Hive Query on Hbase snapshot error
hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot test_snapshot
-stats -schema
On Thu, Sep 24, 2015 at 3:43 PM, Sandeep Nemuri <nhsande...@gmail.com> wrote:
You can check snapshot state if it is healthy or not using below command.
On Thu, Sep 24, 2015 at 2:55 PM, ???????? <510688...@qq.com> wrote:
Hi all,
I am using hive to query on base snapshot. But I got the following error:
FAILED: IllegalArgumentException
org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read
snapshot info
from:/tmp/hbase-huser/hbase/.hbase-snapshot/test_table_snap0/.snapshotinfo
The following is the steps that I do:
1, on the hbase we already have an table: test_table. And I use command:
snapshot ??test_table?? ????test_table_snap0?? to create a snapshot??
2??on the hive create an external table like??
CREATE EXTERNAL TABLE `test_table_snapshot`(
`id` int,
`alias` string,
`kdt_id` int,
`created_time` string,
`update_time` string,
`title` string,
`price` bigint,
`goods_platform` int,
`buy_url` string,
`class1` int,
`class2` string,
`goods_type` tinyint,
`sold_status` tinyint,
`is_display` tinyint,
`is_delete` tinyint,
`num` bigint,
`buy_way` boolean,
`source` tinyint,
`content` string,
`picture` string,
`is_virtual` tinyint)
ROW FORMAT SERDE
'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY
'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
'field.delim'='\u0001',
'hbase.columns.mapping'=':key,cf1:alalis,cf1:kdt_id,cf1:created_time,cf1:update_time,cf1:title,cf1:price,cf1:goods_platform,cf1:buy_url,cf1:class1,cf1:class2,cf1:goods_type,cf1:sold_stattus,cf1:is_display,cf1:is_delete,cf1:num,cf1:buy_way,cf1:source,cf1:content,cf1:picture,cf1:is_virtual',
'line.delim'='\n',
'serialization.format'='\u0001')
TBLPROPERTIES (
??hbase.table.name'='test_table')
4, on Hive, execute ??select * from test_table_snapshot??, then I get the
above error.
I have set the ??hive.hbase.snapshot.restoredir?? value which is same as the
root dir on hbase, and I also set the zookeep server. I checked the base root
directory on hdfs and I can see the snapshot files. But when I use the above
query on hive. it seems that the base.tmp.dir is used to find the snapshot
info. BTW, if I directly query on hbase, there is no problem. I do not know
what happened?
The following is the hive error log:
FAILED: SemanticException
org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read
snapshot info
from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read
snapshot info
from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:117)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:182)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10207)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:421)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalArgumentException:
org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read
snapshot info
from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo
at
org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:401)
at
org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureInputJobProperties(HBaseStorageHandler.java:317)
at
org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobPropertiesForStorageHandler(PlanUtils.java:809)
at
org.apache.hadoop.hive.ql.plan.PlanUtils.configureInputJobPropertiesForStorageHandler(PlanUtils.java:779)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.convertToWork(SimpleFetchOptimizer.java:379)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.access$000(SimpleFetchOptimizer.java:319)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:135)
at
org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:106)
... 22 more
Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
Couldn't read snapshot info
from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo
at
org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.readSnapshotInfo(SnapshotDescriptionUtils.java:307)
at
org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.copySnapshotForScanner(RestoreSnapshotHelper.java:727)
at
org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl.setInput(TableSnapshotInputFormatImpl.java:364)
at
org.apache.hadoop.hive.hbase.HBaseTableSnapshotInputFormatUtil.configureJob(HBaseTableSnapshotInputFormatUtil.java:77)
at
org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:387)
... 29 more
--
Regards Sandeep Nemuri
--
Regards Sandeep Nemuri
