It seems that the snapshotDir is not correctly set. I have set the hive.hbase.snapshot.restoredir. Is the hive.hbase.snapshot.restoredir to control the snapshotDir? if not, how can i set the snapshotDir when I use hive to query on hbase snapshot.
------------------ ???????? ------------------ ??????: "????????????";<[email protected]>; ????????: 2015??9??24??(??????) ????7:44 ??????: "user"<[email protected]>; ????: ?????? Hive Query on Hbase snapshot error Thank you for you reply. The snapshot should be no problem. Here is the following output for the giving command. 2015-09-24 19:37:41,981 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available Snapshot Info ---------------------------------------- Name: test_snapshot Type: FLUSH Table: test_table Format: 2 Created: 2015-09-24T16:00:11 Table Descriptor ---------------------------------------- 'test_table', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|', coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|', coprocessor$5 => '|org.apache.hadoop.hbase.regionserver.LocalIndexSplitter|805306366|'}, {NAME => 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 32 HFiles (0 in archive), total size 236.1 G (100.00% 236.1 G shared with the source table) 0 Logs, total size 0 ------------------ ???????? ------------------ ??????: "Sandeep Nemuri";<[email protected]>; ????????: 2015??9??24??(??????) ????6:43 ??????: "Hive user group"<[email protected]>; ????: Re: Hive Query on Hbase snapshot error hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot test_snapshot -stats -schema On Thu, Sep 24, 2015 at 3:43 PM, Sandeep Nemuri <[email protected]> wrote: You can check snapshot state if it is healthy or not using below command. On Thu, Sep 24, 2015 at 2:55 PM, ???????? <[email protected]> wrote: Hi all, I am using hive to query on base snapshot. But I got the following error: FAILED: IllegalArgumentException org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read snapshot info from:/tmp/hbase-huser/hbase/.hbase-snapshot/test_table_snap0/.snapshotinfo The following is the steps that I do: 1, on the hbase we already have an table: test_table. And I use command: snapshot ??test_table?? ????test_table_snap0?? to create a snapshot?? 2??on the hive create an external table like?? CREATE EXTERNAL TABLE `test_table_snapshot`( `id` int, `alias` string, `kdt_id` int, `created_time` string, `update_time` string, `title` string, `price` bigint, `goods_platform` int, `buy_url` string, `class1` int, `class2` string, `goods_type` tinyint, `sold_status` tinyint, `is_display` tinyint, `is_delete` tinyint, `num` bigint, `buy_way` boolean, `source` tinyint, `content` string, `picture` string, `is_virtual` tinyint) ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( 'field.delim'='\u0001', 'hbase.columns.mapping'=':key,cf1:alalis,cf1:kdt_id,cf1:created_time,cf1:update_time,cf1:title,cf1:price,cf1:goods_platform,cf1:buy_url,cf1:class1,cf1:class2,cf1:goods_type,cf1:sold_stattus,cf1:is_display,cf1:is_delete,cf1:num,cf1:buy_way,cf1:source,cf1:content,cf1:picture,cf1:is_virtual', 'line.delim'='\n', 'serialization.format'='\u0001') TBLPROPERTIES ( ??hbase.table.name'='test_table') 4, on Hive, execute ??select * from test_table_snapshot??, then I get the above error. I have set the ??hive.hbase.snapshot.restoredir?? value which is same as the root dir on hbase, and I also set the zookeep server. I checked the base root directory on hdfs and I can see the snapshot files. But when I use the above query on hive. it seems that the base.tmp.dir is used to find the snapshot info. BTW, if I directly query on hbase, there is no problem. I do not know what happened? The following is the hive error log: FAILED: SemanticException org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read snapshot info from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo org.apache.hadoop.hive.ql.parse.SemanticException: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read snapshot info from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:117) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:182) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10207) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:421) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.IllegalArgumentException: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read snapshot info from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:401) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureInputJobProperties(HBaseStorageHandler.java:317) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobPropertiesForStorageHandler(PlanUtils.java:809) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureInputJobPropertiesForStorageHandler(PlanUtils.java:779) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.convertToWork(SimpleFetchOptimizer.java:379) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.access$000(SimpleFetchOptimizer.java:319) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:135) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:106) ... 22 more Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read snapshot info from:/tmp/hbase-huser/hbase/.hbase-snapshot/goods_v3_hbase_snap0/.snapshotinfo at org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.readSnapshotInfo(SnapshotDescriptionUtils.java:307) at org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.copySnapshotForScanner(RestoreSnapshotHelper.java:727) at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl.setInput(TableSnapshotInputFormatImpl.java:364) at org.apache.hadoop.hive.hbase.HBaseTableSnapshotInputFormatUtil.configureJob(HBaseTableSnapshotInputFormatUtil.java:77) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:387) ... 29 more -- Regards Sandeep Nemuri -- Regards Sandeep Nemuri
