[20/22] hbase-site git commit: Published site at ac5bb8155b618194fe9cf1131f0e72c99b7b534c.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/a7f7dba1/book.html -- diff --git a/book.html b/book.html index 03054ef..151f29b 100644 --- a/book.html +++ b/book.html @@ -294,17 +294,41 @@ 175. Protobuf +Procedure Framework (Pv2): https://issues.apache.org/jira/browse/HBASE-12439;>HBASE-12439 + +176. Procedures +177. Subprocedures +178. ProcedureExecutor +179. Nonces +180. Wait/Wake/Suspend/Yield +181. Locking +182. Procedure Types +183. References + + +AMv2 Description for Devs + +184. Background +185. New System +186. Procedures Detail +187. UI +188. Logging +189. Implementation Notes +190. New Configs +191. Tools + + ZooKeeper -176. Using existing ZooKeeper ensemble -177. SASL Authentication with ZooKeeper +192. Using existing ZooKeeper ensemble +193. SASL Authentication with ZooKeeper Community -178. Decisions -179. Community Roles -180. Commit Message format +194. Decisions +195. Community Roles +196. Commit Message format Appendix @@ -322,8 +346,8 @@ Appendix K: HBase and the Apache Software Foundation Appendix L: Apache HBase Orca Appendix M: Enabling Dapper-like Tracing in HBase -181. Client Modifications -182. Tracing from HBase Shell +197. Client Modifications +198. Tracing from HBase Shell Appendix N: 0.95 RPC Specification @@ -33910,6 +33934,509 @@ public API. TODO. +Procedure Framework (Pv2): https://issues.apache.org/jira/browse/HBASE-12439;>HBASE-12439 + + + +Procedure v2 aims to provide a unified way to buildmulti-step procedures with a rollback/roll-forward ability in case of failure (e.g. create/delete table)Matteo Bertozzi, the author of Pv2. + + +With Pv2 you can build and run state machines. It was built by Matteo to make distributed state transitions in HBase resilient in the face of process failures. Previous to Pv2, state transition handling was spread about the codebase with implementation varying by transition-type and context. Pv2 was inspired by https://accumulo.apache.org/1.8/accumulo_user_manual.html#_fault_tolerant_executor_fate;>FATE, of Apache Accumulo. + + +Early Pv2 aspects have been shipping in HBase with a good while now but it has continued to evolve as it takes on more involved scenarios. What we have now is powerful but intricate in operation and incomplete, in need of cleanup and hardening. In this doc we have given overview on the system so you can make use of it (and help with its polishing). + + +This system has the awkward name of Pv2 because HBase already had the notion of a Procedure used in snapshots (see hbase-server org.apache.hadoop.hbase.procedure as opposed to hbase-procedure org.apache.hadoop.hbase.procedure2). Pv2 supercedes and is to replace Procedure. + + + + +176. Procedures + + +A Procedure is a transform made on an HBase entity. Examples of HBase entities would be Regions and Tables. +Procedures are run by a ProcedureExecutor instance. Procedure current state is kept in the ProcedureStore. +The ProcedureExecutor has but a primitive view on what goes on inside a Procedure. From its PoV, Procedures are submitted and then the ProcedureExecutor keeps calling #execute(Object) until the Procedure is done. Execute may be called multiple times in the case of failure or restart, so Procedure code must be idempotent yielding the same result each time it run. Procedure code can also implement rollback so steps can be undone if failure. A call to execute() can result in one of following possibilities: + + + + +execute() returns + + + +null: indicates we are done. + + +this: indicates there is more to do so, persist current procedure state and re-execute(). + + +Array of sub-procedures: indicates a set of procedures needed to be run to completion before we can proceed (after which we expect the framework to call our execute again). + + + + + +execute() throws exception + + + +suspend: indicates execution of procedure is suspended and can be resumed due to some external event. The procedure state is persisted. + + +yield: procedure is added back to scheduler. The procedure state is not persisted. + + +interrupted: currently same as yield. + + +Any exception not listed above: Procedure state is changed to FAILED (after which we expect the framework will attempt rollback). + + + + + + + +The ProcedureExecutor stamps the frameworks notions of Procedure State into the Procedure itself; e.g. it marks Procedures as INITIALIZING on submit. It moves the state to RUNNABLE when it goes to execute. When done, a Procedure gets marked FAILED or SUCCESS depending. Here is the list of all states as of this writing: + + + + +INITIALIZING Procedure in construction, not yet added to the executor + + +RUNNABLE Procedure added to the executor, and ready to be executed. + + +WAITING The procedure is waiting on children (subprocedures) to be completed + + +WAITING_TIMEOUT The procedure is waiting a timeout or an external event +
[20/22] hbase-site git commit: Published site at 46cb5dfa226892fd2580f26ce9ce77225bd7e67c.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/8bc57a1a/book.html -- diff --git a/book.html b/book.html index 8de5bcb..d82af18 100644 --- a/book.html +++ b/book.html @@ -37824,7 +37824,7 @@ The server will return cellblocks compressed using this same compressor as long Version 3.0.0-SNAPSHOT -Last updated 2018-04-20 14:29:52 UTC +Last updated 2018-04-21 14:29:50 UTC http://git-wip-us.apache.org/repos/asf/hbase-site/blob/8bc57a1a/bulk-loads.html -- diff --git a/bulk-loads.html b/bulk-loads.html index 2c97d08..111fe28 100644 --- a/bulk-loads.html +++ b/bulk-loads.html @@ -7,7 +7,7 @@ - + Apache HBase Bulk Loads in Apache HBase (TM) @@ -306,7 +306,7 @@ under the License. --> https://www.apache.org/;>The Apache Software Foundation. All rights reserved. - Last Published: 2018-04-20 + Last Published: 2018-04-21
[20/22] hbase-site git commit: Published site at .
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/34a201e5/checkstyle.rss -- diff --git a/checkstyle.rss b/checkstyle.rss index 40aa3bd..8ea04c0 100644 --- a/checkstyle.rss +++ b/checkstyle.rss @@ -25,8 +25,8 @@ under the License. en-us 2007 - 2017 The Apache Software Foundation - File: 3430, - Errors: 20881, + File: 3431, + Errors: 20865, Warnings: 0, Infos: 0 @@ -15833,7 +15833,7 @@ under the License. 0 - 3 + 0 @@ -16407,7 +16407,7 @@ under the License. 0 - 20 + 0 @@ -16650,6 +16650,20 @@ under the License. + http://hbase.apache.org/checkstyle.html#org.apache.hadoop.hbase.AcidGuaranteesTestTool.java;>org/apache/hadoop/hbase/AcidGuaranteesTestTool.java + + + 0 + + + 0 + + + 7 + + + + http://hbase.apache.org/checkstyle.html#org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.java;>org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java http://git-wip-us.apache.org/repos/asf/hbase-site/blob/34a201e5/coc.html -- diff --git a/coc.html b/coc.html index 957a7d9..fae1b17 100644 --- a/coc.html +++ b/coc.html @@ -7,7 +7,7 @@ - + Apache HBase Code of Conduct Policy @@ -380,7 +380,7 @@ email to mailto:priv...@hbase.apache.org;>the priv https://www.apache.org/;>The Apache Software Foundation. All rights reserved. - Last Published: 2017-11-21 + Last Published: 2017-11-22 http://git-wip-us.apache.org/repos/asf/hbase-site/blob/34a201e5/cygwin.html -- diff --git a/cygwin.html b/cygwin.html index c87bacb..a9e3be0 100644 --- a/cygwin.html +++ b/cygwin.html @@ -7,7 +7,7 @@ - + Apache HBase Installing Apache HBase (TM) on Windows using Cygwin @@ -679,7 +679,7 @@ Now your HBase server is running, start coding and build that next https://www.apache.org/;>The Apache Software Foundation. All rights reserved. - Last Published: 2017-11-21 + Last Published: 2017-11-22 http://git-wip-us.apache.org/repos/asf/hbase-site/blob/34a201e5/dependencies.html -- diff --git a/dependencies.html b/dependencies.html index ed27e32..62423d0 100644 --- a/dependencies.html +++ b/dependencies.html @@ -7,7 +7,7 @@ - + Apache HBase Project Dependencies @@ -445,7 +445,7 @@ https://www.apache.org/;>The Apache Software Foundation. All rights reserved. - Last Published: 2017-11-21 + Last Published: 2017-11-22 http://git-wip-us.apache.org/repos/asf/hbase-site/blob/34a201e5/dependency-convergence.html -- diff --git a/dependency-convergence.html b/dependency-convergence.html index 72b54f6..67e1b39 100644 --- a/dependency-convergence.html +++ b/dependency-convergence.html @@ -7,7 +7,7 @@ - + Apache HBase Reactor Dependency Convergence @@ -1008,7 +1008,7 @@ https://www.apache.org/;>The Apache Software Foundation. All rights reserved. - Last Published: 2017-11-21 + Last Published: 2017-11-22 http://git-wip-us.apache.org/repos/asf/hbase-site/blob/34a201e5/dependency-info.html -- diff --git a/dependency-info.html b/dependency-info.html index bde311c..07b575d 100644 --- a/dependency-info.html +++ b/dependency-info.html @@ -7,7 +7,7 @@ - + Apache HBase Dependency Information @@ -318,7 +318,7 @@ https://www.apache.org/;>The Apache Software Foundation.
[20/22] hbase-site git commit: Published site at .
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/6f0c8299/devapidocs/org/apache/hadoop/hbase/backup/RestoreDriver.html -- diff --git a/devapidocs/org/apache/hadoop/hbase/backup/RestoreDriver.html b/devapidocs/org/apache/hadoop/hbase/backup/RestoreDriver.html index 0884676..bb03b01 100644 --- a/devapidocs/org/apache/hadoop/hbase/backup/RestoreDriver.html +++ b/devapidocs/org/apache/hadoop/hbase/backup/RestoreDriver.html @@ -119,7 +119,7 @@ var activeTableTab = "activeTableTab"; @InterfaceAudience.Private -public class RestoreDriver +public class RestoreDriver extends AbstractHBaseTool Command-line entry point for restore operation @@ -277,7 +277,7 @@ extends LOG -private static finalorg.apache.commons.logging.Log LOG +private static finalorg.apache.commons.logging.Log LOG @@ -286,7 +286,7 @@ extends cmd -privateorg.apache.commons.cli.CommandLine cmd +privateorg.apache.commons.cli.CommandLine cmd @@ -295,7 +295,7 @@ extends USAGE_STRING -private static finalhttp://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">String USAGE_STRING +private static finalhttp://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">String USAGE_STRING See Also: Constant Field Values @@ -308,7 +308,7 @@ extends USAGE_FOOTER -private static finalhttp://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">String USAGE_FOOTER +private static finalhttp://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">String USAGE_FOOTER See Also: Constant Field Values @@ -329,7 +329,7 @@ extends RestoreDriver -protectedRestoreDriver() +protectedRestoreDriver() throws http://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true; title="class or interface in java.io">IOException Throws: @@ -351,7 +351,7 @@ extends init -protectedvoidinit() +protectedvoidinit() throws http://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true; title="class or interface in java.io">IOException Throws: @@ -365,7 +365,7 @@ extends parseAndRun -privateintparseAndRun(http://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">String[]args) +privateintparseAndRun(http://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">String[]args) throws http://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true; title="class or interface in java.io">IOException Throws: @@ -379,7 +379,7 @@ extends getTablesForSet -privatehttp://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">StringgetTablesForSet(Connectionconn, +privatehttp://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">StringgetTablesForSet(Connectionconn, http://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">Stringname, org.apache.hadoop.conf.Configurationconf) throws http://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true; title="class or interface in java.io">IOException @@ -395,7 +395,7 @@ extends addOptions -protectedvoidaddOptions() +protectedvoidaddOptions() Description copied from class:AbstractHBaseTool Override this to add command-line options using AbstractHBaseTool.addOptWithArg(java.lang.String, java.lang.String) and similar methods. @@ -411,7 +411,7 @@ extends processOptions -protectedvoidprocessOptions(org.apache.commons.cli.CommandLinecmd) +protectedvoidprocessOptions(org.apache.commons.cli.CommandLinecmd) Description copied from class:AbstractHBaseTool This method is called to process the options after they have been parsed. @@ -426,7 +426,7 @@ extends doWork -protectedintdoWork() +protectedintdoWork() throws http://docs.oracle.com/javase/8/docs/api/java/lang/Exception.html?is-external=true; title="class or interface in java.lang">Exception Description copied from class:AbstractHBaseTool The "main function" of the tool @@ -444,7 +444,7 @@ extends main -public staticvoidmain(http://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">String[]args) +public staticvoidmain(http://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in java.lang">String[]args) throws
[20/22] hbase-site git commit: Published site at 82d554e3783372cc6b05489452c815b57c06f6cd.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/b9830530/apache_hbase_reference_guide.pdfmarks
--
diff --git a/apache_hbase_reference_guide.pdfmarks
b/apache_hbase_reference_guide.pdfmarks
index 97924a4..13f6680 100644
--- a/apache_hbase_reference_guide.pdfmarks
+++ b/apache_hbase_reference_guide.pdfmarks
@@ -2,8 +2,8 @@
/Author (Apache HBase Team)
/Subject ()
/Keywords ()
- /ModDate (D:20170616144948)
- /CreationDate (D:20170616144948)
+ /ModDate (D:20170617144928)
+ /CreationDate (D:20170617144928)
/Creator (Asciidoctor PDF 1.5.0.alpha.6, based on Prawn 1.2.1)
/Producer ()
/DOCINFO pdfmark
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/b9830530/book.html
--
diff --git a/book.html b/book.html
index b60b371..d388365 100644
--- a/book.html
+++ b/book.html
@@ -26640,6 +26640,59 @@ In these cases, the user may configure the system to
not delete any space quota
+
+136.6. HBase Snapshots with
Space Quotas
+
+One common area of unintended-filesystem-use with HBase is via HBase
snapshots. Because snapshots
+exist outside of the management of HBase tables, it is not uncommon for
administrators to suddenly
+realize that hundreds of gigabytes or terabytes of space is being used by
HBase snapshots which were
+forgotten and never removed.
+
+
+https://issues.apache.org/jira/browse/HBASE-17748;>HBASE-17748
is the umbrella JIRA issue which
+expands on the original space quota functionality to also include HBase
snapshots. While this is a confusing
+subject, the implementation attempts to present this support in as reasonable
and simple of a manner as
+possible for administrators. This feature does not make any changes to
administrator interaction with
+space quotas, only in the internal computation of table/namespace usage. Table
and namespace usage will
+automatically incorporate the size taken by a snapshot per the rules defined
below.
+
+
+As a review, lets cover a snapshots lifecycle: a snapshot is
metadata which points to
+a list of HFiles on the filesystem. This is why creating a snapshot is a very
cheap operation; no HBase
+table data is actually copied to perform a snapshot. Cloning a snapshot into a
new table or restoring
+a table is a cheap operation for the same reason; the new table references the
files which already exist
+on the filesystem without a copy. To include snapshots in space quotas, we
need to define which table
+"owns" a file when a snapshot references the file ("owns" refers to
encompassing the filesystem usage
+of that file).
+
+
+Consider a snapshot which was made against a table. When the snapshot
refers to a file and the table no
+longer refers to that file, the "originating" table "owns" that file. When
multiple snapshots refer to
+the same file and no table refers to that file, the snapshot with the
lowest-sorting name (lexicographically)
+is chosen and the table which that snapshot was created from "owns" that file.
HFiles are not "double-counted"
+ hen a table and one or more snapshots refer to that HFile.
+
+
+When a table is "rematerialized" (via clone_snapshot or
restore_snapshot), a similar problem of file
+ownership arises. In this case, while the rematerialized table references a
file which a snapshot also
+references, the table does not "own" the file. The table from which the
snapshot was created still "owns"
+that file. When the rematerialized table is compacted or the snapshot is
deleted, the rematerialized table
+will uniquely refer to a new file and "own" the usage of that file. Similarly,
when a table is duplicated via a snapshot
+and restore_snapshot, the new table will not consume any quota
size until the original table stops referring
+to the files, either due to a compaction on the original table, a compaction
on the new table, or the
+original table being deleted.
+
+
+One new HBase shell command was added to inspect the computed sizes of each
snapshot in an HBase instance.
+
+
+
+hbase list_snapshot_sizes
+SNAPSHOT SIZE
+ t1.s11159108
+
+
+
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/b9830530/bulk-loads.html
--
diff --git a/bulk-loads.html b/bulk-loads.html
index 86b56de..f757d89 100644
--- a/bulk-loads.html
+++ b/bulk-loads.html
@@ -7,7 +7,7 @@
-
+
Apache HBase
Bulk Loads in Apache HBase (TM)
@@ -311,7 +311,7 @@ under the License. -->
https://www.apache.org/;>The Apache Software
Foundation.
All rights reserved.
- Last Published:
2017-06-16
+ Last Published:
2017-06-17
[20/22] hbase-site git commit: Published site at da5fb27eabed4a4b4d251be973ee945fb52895bf.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/7b1830cf/bulk-loads.html
--
diff --git a/bulk-loads.html b/bulk-loads.html
index 2a9a6c5..dfa368d 100644
--- a/bulk-loads.html
+++ b/bulk-loads.html
@@ -7,7 +7,7 @@
-
+
Apache HBase
Bulk Loads in Apache HBase (TM)
@@ -323,7 +323,7 @@ under the License. -->
https://www.apache.org/;>The Apache Software
Foundation.
All rights reserved.
- Last Published:
2017-04-12
+ Last Published:
2017-04-13
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/7b1830cf/checkstyle-aggregate.html
--
diff --git a/checkstyle-aggregate.html b/checkstyle-aggregate.html
index 733b731..4f1b99a 100644
--- a/checkstyle-aggregate.html
+++ b/checkstyle-aggregate.html
@@ -7,7 +7,7 @@
-
+
Apache HBase Checkstyle Results
@@ -301,7 +301,7 @@
2144
0
0
-14238
+14240
Files
@@ -5979,7 +5979,7 @@
org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
0
0
-17
+19
org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java
0
@@ -7031,7 +7031,7 @@
http://checkstyle.sourceforge.net/config_blocks.html#NeedBraces;>NeedBraces
-1946
+1948
Error
coding
@@ -90419,97 +90419,109 @@
imports
ImportOrder
Wrong order for 'org.apache.hadoop.conf.Configuration' import.
-39
+38
Error
imports
ImportOrder
Wrong order for 'org.apache.hadoop.hbase.HBaseConfiguration' import.
-49
+48
Error
imports
ImportOrder
Wrong order for 'org.apache.hadoop.mapreduce.InputFormat' import.
-71
+70
Error
imports
ImportOrder
Wrong order for
'org.apache.hadoop.hbase.io.hadoopbackport.ThrottledInputStream' import.
-77
+76
Error
blocks
NeedBraces
'if' construct must use '{}'s.
-537
+267
Error
blocks
NeedBraces
'if' construct must use '{}'s.
-540
+268
Error
blocks
NeedBraces
'if' construct must use '{}'s.
-544
+533
Error
blocks
NeedBraces
'if' construct must use '{}'s.
-547
+536
+
+Error
+blocks
+NeedBraces
+'if' construct must use '{}'s.
+540
+
+Error
+blocks
+NeedBraces
+'if' construct must use '{}'s.
+543
Error
indentation
Indentation
'object def rcurly' have incorrect indentation level 4, expected level
should be one of the following: 6, 8.
-595
+591
Error
blocks
LeftCurly
'{' at column 44 should have line break after.
-765
+761
Error
blocks
LeftCurly
'{' at column 45 should have line break after.
-768
+764
Error
blocks
LeftCurly
'{' at column 34 should have line break after.
-771
+767
Error
blocks
NeedBraces
'if' construct must use '{}'s.
-799
+795
Error
blocks
NeedBraces
'if' construct must use '{}'s.
-800
+796
Error
sizes
MethodLength
Method length is 163 lines (max allowed is 150).
-918
+914
Error
sizes
LineLength
Line is longer than 100 characters (found 102).
-958
+954
org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java
@@ -104661,7 +104673,7 @@
https://www.apache.org/;>The Apache Software
Foundation.
All rights reserved.
- Last Published:
2017-04-12
+ Last Published:
2017-04-13
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/7b1830cf/checkstyle.rss
--
diff --git a/checkstyle.rss b/checkstyle.rss
index 3035b6c..9eb8dd4 100644
--- a/checkstyle.rss
+++ b/checkstyle.rss
@@ -26,7 +26,7 @@ under the License.
2007 - 2017 The Apache Software Foundation
File: 2144,
- Errors: 14238,
+ Errors: 14240,
Warnings: 0,
Infos: 0
@@ -14559,7 +14559,7 @@ under the License.
0
- 17
+ 19
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/7b1830cf/coc.html
--
diff --git a/coc.html b/coc.html
index b5db912..4319aab 100644
--- a/coc.html
+++ b/coc.html
@@ -7,7 +7,7 @@
-
+
Apache HBase
Code of Conduct Policy
@@ -392,7 +392,7 @@ email to mailto:priv...@hbase.apache.org;>the priv
https://www.apache.org/;>The Apache Software
Foundation.
All rights reserved.
- Last Published:
2017-04-12
+ Last Published:
2017-04-13
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/7b1830cf/cygwin.html
--
diff --git a/cygwin.html
