[
https://issues.apache.org/jira/browse/BEAM-4361?focusedWorklogId=105836&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105836
]
ASF GitHub Bot logged work on BEAM-4361:
----------------------------------------
Author: ASF GitHub Bot
Created on: 25/May/18 08:20
Start Date: 25/May/18 08:20
Worklog Time Spent: 10m
Work Description: asfgit closed pull request #445: [BEAM-4361] Document
usage of HBase TableSnapshotInputFormat
URL: https://github.com/apache/beam-site/pull/445
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/src/documentation/io/built-in-hadoop.md
b/src/documentation/io/built-in-hadoop.md
index 82fc47f5b..bcfa26750 100644
--- a/src/documentation/io/built-in-hadoop.md
+++ b/src/documentation/io/built-in-hadoop.md
@@ -267,6 +267,73 @@ PCollection<Text, DynamoDBItemWritable> dynamoDBData =
.withConfiguration(dynamoDBConf);
```
+```py
+ # The Beam SDK for Python does not support Hadoop InputFormat IO.
+```
+
+### Apache HBase - TableSnapshotInputFormat
+
+To read data from an HBase table snapshot, use
`org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat`.
+Reading from a table snapshot bypasses the HBase region servers, instead
reading HBase data files directly from the filesystem.
+This is useful for cases such as reading historical data or offloading of work
from the HBase cluster.
+There are scenarios when this may prove faster than accessing content through
the region servers using the `HBaseIO`.
+
+A table snapshot can be taken using the HBase shell or programmatically:
+```java
+try (
+ Connection connection = ConnectionFactory.createConnection(hbaseConf);
+ Admin admin = connection.getAdmin()
+ ) {
+ admin.snapshot(
+ "my_snaphshot",
+ TableName.valueOf("my_table"),
+ HBaseProtos.SnapshotDescription.Type.FLUSH);
+}
+```
+
+```py
+ # The Beam SDK for Python does not support Hadoop InputFormat IO.
+```
+
+A `TableSnapshotInputFormat` is configured as follows:
+
+```java
+// Construct a typical HBase scan
+Scan scan = new Scan();
+scan.setCaching(1000);
+scan.setBatch(1000);
+scan.addColumn(Bytes.toBytes("CF"), Bytes.toBytes("col_1"));
+scan.addColumn(Bytes.toBytes("CF"), Bytes.toBytes("col_2"));
+
+Configuration hbaseConf = HBaseConfiguration.create();
+hbaseConf.set(HConstants.ZOOKEEPER_QUORUM, "zk1:2181");
+hbaseConf.set("hbase.rootdir", "/hbase");
+hbaseConf.setClass(
+ "mapreduce.job.inputformat.class", TableSnapshotInputFormat.class,
InputFormat.class);
+hbaseConf.setClass("key.class", ImmutableBytesWritable.class, Writable.class);
+hbaseConf.setClass("value.class", Result.class, Writable.class);
+ClientProtos.Scan proto = ProtobufUtil.toScan(scan);
+hbaseConf.set(TableInputFormat.SCAN, Base64.encodeBytes(proto.toByteArray()));
+
+// Make use of existing utility methods
+Job job = Job.getInstance(hbaseConf); // creates internal clone of hbaseConf
+TableSnapshotInputFormat.setInput(job, "my_snapshot", new
Path("/tmp/snapshot_restore"));
+hbaseConf = job.getConfiguration(); // extract the modified clone
+```
+
+```py
+ # The Beam SDK for Python does not support Hadoop InputFormat IO.
+```
+
+Call Read transform as follows:
+
+```java
+PCollection<ImmutableBytesWritable, Result> hbaseSnapshotData =
+ p.apply("read",
+ HadoopInputFormatIO.<ImmutableBytesWritable, Result>read()
+ .withConfiguration(hbaseConf);
+```
+
```py
# The Beam SDK for Python does not support Hadoop InputFormat IO.
```
\ No newline at end of file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 105836)
Time Spent: 1h 10m (was: 1h)
> Document usage of HBase TableSnapshotInputFormat
> -------------------------------------------------
>
> Key: BEAM-4361
> URL: https://issues.apache.org/jira/browse/BEAM-4361
> Project: Beam
> Issue Type: Task
> Components: website
> Affects Versions: 2.4.0
> Reporter: Tim Robertson
> Assignee: Tim Robertson
> Priority: Trivial
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Add a paragraph demonstrating the usage of {{TableSnapshotInputFormat}} as a
> mechanism for doing efficient full scans over HBase to
> https://beam.apache.org/documentation/io/built-in/hadoop/
> Typically in MR / Spark this yields up to 3-4x improvement over hitting
> region servers directly and keeps load (GC etc) from those services.
> I have it tested and an example ready.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
