svn commit: r31733 [1/2] - in /dev/hbase: hbase-2.1.2RC0/ hbase-2.1.2RC1/
Author: stack Date: Tue Jan 1 00:19:33 2019 New Revision: 31733 Log: Add 2.1.2RC1 and remove 2.1.2RC0 Added: dev/hbase/hbase-2.1.2RC1/ dev/hbase/hbase-2.1.2RC1/CHANGES.md (with props) dev/hbase/hbase-2.1.2RC1/RELEASENOTES.md (with props) dev/hbase/hbase-2.1.2RC1/hbase-2.1.2-bin.tar.gz (with props) dev/hbase/hbase-2.1.2RC1/hbase-2.1.2-bin.tar.gz.asc dev/hbase/hbase-2.1.2RC1/hbase-2.1.2-bin.tar.gz.sha512 dev/hbase/hbase-2.1.2RC1/hbase-2.1.2-client-bin.tar.gz (with props) dev/hbase/hbase-2.1.2RC1/hbase-2.1.2-client-bin.tar.gz.asc dev/hbase/hbase-2.1.2RC1/hbase-2.1.2-client-bin.tar.gz.sha512 dev/hbase/hbase-2.1.2RC1/hbase-2.1.2-src.tar.gz (with props) dev/hbase/hbase-2.1.2RC1/hbase-2.1.2-src.tar.gz.asc dev/hbase/hbase-2.1.2RC1/hbase-2.1.2-src.tar.gz.sha512 Removed: dev/hbase/hbase-2.1.2RC0/ Added: dev/hbase/hbase-2.1.2RC1/CHANGES.md == --- dev/hbase/hbase-2.1.2RC1/CHANGES.md (added) +++ dev/hbase/hbase-2.1.2RC1/CHANGES.md Tue Jan 1 00:19:33 2019 @@ -0,0 +1,639 @@ +# HBASE Changelog + + + +## Release 2.1.2 - Unreleased (as of 2018-12-29) + + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | +|: |: | :--- |: | +| [HBASE-21640](https://issues.apache.org/jira/browse/HBASE-21640) | Remove the TODO when increment zero | Major | . | +| [HBASE-21631](https://issues.apache.org/jira/browse/HBASE-21631) | list\_quotas should print human readable values for LIMIT | Minor | shell | +| [HBASE-21635](https://issues.apache.org/jira/browse/HBASE-21635) | Use maven enforcer to ban imports from illegal packages | Major | build | +| [HBASE-21520](https://issues.apache.org/jira/browse/HBASE-21520) | TestMultiColumnScanner cost long time when using ROWCOL bloom type | Major | test | +| [HBASE-21590](https://issues.apache.org/jira/browse/HBASE-21590) | Optimize trySkipToNextColumn in StoreScanner a bit | Critical | Performance, Scanners | +| [HBASE-21554](https://issues.apache.org/jira/browse/HBASE-21554) | Show replication endpoint classname for replication peer on master web UI | Minor | UI | +| [HBASE-21549](https://issues.apache.org/jira/browse/HBASE-21549) | Add shell command for serial replication peer | Major | . | +| [HBASE-21413](https://issues.apache.org/jira/browse/HBASE-21413) | Empty meta log doesn't get split when restart whole cluster | Major | . | +| [HBASE-21524](https://issues.apache.org/jira/browse/HBASE-21524) | Unnecessary DEBUG log in ConnectionImplementation#isTableEnabled | Major | Client | +| [HBASE-21511](https://issues.apache.org/jira/browse/HBASE-21511) | Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles | Minor | . | +| [HBASE-21480](https://issues.apache.org/jira/browse/HBASE-21480) | Taking snapshot when RS crashes prevent we bring the regions online | Major | snapshots | +| [HBASE-21485](https://issues.apache.org/jira/browse/HBASE-21485) | Add more debug logs for remote procedure execution | Major | proc-v2 | +| [HBASE-21388](https://issues.apache.org/jira/browse/HBASE-21388) | No need to instantiate MemStoreLAB for master which not carry table | Major | . | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | +|: |: | :--- |: | +| [HBASE-21646](https://issues.apache.org/jira/browse/HBASE-21646) | Flakey TestTableSnapshotInputFormat; DisableTable not completing... | Major | test | +| [HBASE-21545](https://issues.apache.org/jira/browse/HBASE-21545) | NEW\_VERSION\_BEHAVIOR breaks Get/Scan with specified columns | Major | API | +| [HBASE-21629](https://issues.apache.org/jira/browse/HBASE-21629) | draining\_servers.rb is broken | Major | scripts | +| [HBASE-21621](https://issues.apache.org/jira/browse/HBASE-21621) | Reversed scan does not return expected number of rows | Critical | scan | +| [HBASE-21620](https://issues.apache.org/jira/browse/HBASE-21620) | Problem in scan query when using more than one column prefix filter in some cases. | Major | scan | +| [HBASE-21618](https://issues.apache.org/jira/browse/HBASE-21618) | Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result | Critical | Client | +| [HBASE-21610](https://issues.apache.org/jira/browse/HBASE-21610) | numOpenConnections metric is set to -1 when zero server channel exist | Minor | metrics | +| [HBASE-21498](https://issues.apache.org/jira/browse/HBASE-21498) | Master OOM when SplitTableRegionProcedure new CacheConfig and instantiate a new BlockCache | Major | . | +| [HBASE-21592](https://issues.apache.org/jira/browse/HBASE-21592) | quota.addGetResult(r) throw NPE | Major | . | +| [HBASE-21589](https://issues.apache.org/jira/browse/HBASE-21589) | TestCleanupMetaWAL fails | Blocker | test, wal | +| [HBASE-21582](https://issues.apache.org/jira/browse/HBASE-21582) | If call HBaseAdmin#snapshotAsync but forget call isSnapshotFinished,
svn commit: r31733 [2/2] - in /dev/hbase: hbase-2.1.2RC0/ hbase-2.1.2RC1/
Added: dev/hbase/hbase-2.1.2RC1/RELEASENOTES.md == --- dev/hbase/hbase-2.1.2RC1/RELEASENOTES.md (added) +++ dev/hbase/hbase-2.1.2RC1/RELEASENOTES.md Tue Jan 1 00:19:33 2019 @@ -0,0 +1,815 @@ +# HBASE 2.1.2 Release Notes + + + + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-21635](https://issues.apache.org/jira/browse/HBASE-21635) | *Major* | **Use maven enforcer to ban imports from illegal packages** + +Use de.skuzzle.enforcer.restrict-imports-enforcer-rule extension for maven enforcer plugin to ban illegal imports at compile time. Now if you use illegal imports, for example, import com.google.common.\*, there will be a compile error, instead of a checkstyle warning. + + +--- + +* [HBASE-21401](https://issues.apache.org/jira/browse/HBASE-21401) | *Critical* | **Sanity check when constructing the KeyValue** + +Add a sanity check when constructing KeyValue from a byte[]. we use the constructor when we're reading kv from socket or HFIle or WAL(replication). the santiy check isn't designed for discovering the bits corruption in network transferring or disk IO. It is designed to detect bugs inside HBase in advance. and HBASE-21459 indicated that there's extremely small performance loss for diff kinds of keyvalue. + + +--- + +* [HBASE-21554](https://issues.apache.org/jira/browse/HBASE-21554) | *Minor* | **Show replication endpoint classname for replication peer on master web UI** + +The replication UI on master will show the replication endpoint classname. + + +--- + +* [HBASE-21549](https://issues.apache.org/jira/browse/HBASE-21549) | *Major* | **Add shell command for serial replication peer** + +Add a SERIAL flag for add\_peer command to identifiy whether or not the replication peer is a serial replication peer. The default serial flag is false. + + +--- + +* [HBASE-21551](https://issues.apache.org/jira/browse/HBASE-21551) | *Blocker* | **Memory leak when use scan with STREAM at server side** + + +### Summary +HBase clusters will experience Region Server failures due to out of memory errors due to a leak given any of the following: + +* User initiates Scan operations set to use the STREAM reading type +* User initiates Scan operations set to use the default reading type that read more than 4 * the block size of column families involved in the scan (e.g. by default 4*64KiB) +* Compactions run + +### Root cause + +When there are long running scans the Region Server process attempts to optimize access by using a different API geared towards sequential access. Due to an error in HBASE-20704 for HBase 2.0+ the Region Server fails to release related resources when those scans finish. That same optimization path is always used for the HBase internal file compaction process. + +### Workaround + +Impact for this error can be minimized by setting the config value âhbase.storescanner.pread.max.bytesâ to MAX_INT to avoid the optimization for default user scans. Clients should also be checked to ensure they do not pass the STREAM read type to the Scan API. This will have a severe impact on performance for long scans. + +Compactions always use this sequential optimized reading mechanism so downstream users will need to periodically restart Region Server roles after compactions have happened. + + +--- + +* [HBASE-21387](https://issues.apache.org/jira/browse/HBASE-21387) | *Major* | **Race condition surrounding in progress snapshot handling in snapshot cache leads to loss of snapshot files** + +To prevent race condition between in progress snapshot (performed by TakeSnapshotHandler) and HFileCleaner which results in data loss, this JIRA introduced mutual exclusion between taking snapshot and running HFileCleaner. That is, at any given moment, either some snapshot can be taken or, HFileCleaner checks hfiles which are not referenced, but not both can be running. + + +--- + +* [HBASE-21423](https://issues.apache.org/jira/browse/HBASE-21423) | *Major* | **Procedures for meta table/region should be able to execute in separate workers** + +The procedure for meta table will be executed in a separate worker thread named 'Urgent Worker' to avoid stuck. A new config named 'hbase.master.urgent.procedure.threads' is added, the default value for it is 1. To disable the separate worker, set it to 0. + + +--- + +* [HBASE-21417](https://issues.apache.org/jira/browse/HBASE-21417) | *Critical* | **Pre commit build is broken due to surefire plugin crashes** + +Add -Djdk.net.URLClassPath.disableClassPathURLCheck=true when executing surefire plugin. + + +--- + +* [HBASE-21237](https://issues.apache.org/jira/browse/HBASE-21237) | *Blocker* | **Use CompatRemoteProcedureResolver to dispatch open/close region requests to RS** + +Use CompatRemoteProcedureResolver instead of ExecuteProceduresRemoteCall to dispatch region open
[hbase] Git Push Summary
Repository: hbase Updated Tags: refs/tags/2.1.2RC1 [created] 68d64f29b
[hbase] Git Push Summary
Repository: hbase Updated Branches: refs/heads/masgter [deleted] e160b5ac8
svn commit: r31731 - /dev/hbase/hbase-1.3.3RC0/ /release/hbase/1.3.3/
Author: apurtell Date: Mon Dec 31 20:31:47 2018 New Revision: 31731 Log: Release Apache HBase 1.3.3 Added: release/hbase/1.3.3/ - copied from r31730, dev/hbase/hbase-1.3.3RC0/ Removed: dev/hbase/hbase-1.3.3RC0/
svn commit: r31732 - /release/hbase/1.3.2.1/
Author: apurtell Date: Mon Dec 31 20:32:02 2018 New Revision: 31732 Log: Remove old artifacts for HBase release 1.3.2.1 Removed: release/hbase/1.3.2.1/
[2/2] hbase git commit: HBASE-21492 CellCodec Written To WAL Before It's Verified
HBASE-21492 CellCodec Written To WAL Before It's Verified Conflicts: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/f7470a8b Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/f7470a8b Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/f7470a8b Branch: refs/heads/branch-1.4 Commit: f7470a8b5734ab2ff05e4fc639f8b8fb8d9d8217 Parents: 6dbb1d4 Author: BELUGA BEHR Authored: Tue Nov 27 08:57:06 2018 -0800 Committer: Andrew Purtell Committed: Mon Dec 31 12:29:17 2018 -0800 -- .../org/apache/hadoop/hbase/mapreduce/WALPlayer.java | 2 +- .../hadoop/hbase/regionserver/wal/ProtobufLogWriter.java | 2 +- .../hadoop/hbase/regionserver/wal/WALCellCodec.java | 8 .../hbase/regionserver/wal/TestCustomWALCellCodec.java | 11 +++ 4 files changed, 17 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/f7470a8b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java index 377b6ea..bff110c 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java @@ -295,7 +295,7 @@ public class WALPlayer extends Configured implements Tool { // No reducers. job.setNumReduceTasks(0); } -String codecCls = WALCellCodec.getWALCellCodecClass(conf); +String codecCls = WALCellCodec.getWALCellCodecClass(conf).getName(); try { TableMapReduceUtil.addDependencyJarsForClasses(job.getConfiguration(), Class.forName(codecCls)); } catch (Exception e) { http://git-wip-us.apache.org/repos/asf/hbase/blob/f7470a8b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java index 42abeae..436df87 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java @@ -69,7 +69,7 @@ public class ProtobufLogWriter extends WriterBase { builder.setWriterClsName(ProtobufLogWriter.class.getSimpleName()); } if (!builder.hasCellCodecClsName()) { - builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf)); + builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf).getName()); } return builder.build(); } http://git-wip-us.apache.org/repos/asf/hbase/blob/f7470a8b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java index 5c62ef2..11b6120 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java @@ -79,8 +79,8 @@ public class WALCellCodec implements Codec { this.compression = compression; } - public static String getWALCellCodecClass(Configuration conf) { -return conf.get(WAL_CELL_CODEC_CLASS_KEY, WALCellCodec.class.getName()); + public static Class getWALCellCodecClass(Configuration conf) { +return conf.getClass(WAL_CELL_CODEC_CLASS_KEY, WALCellCodec.class); } /** @@ -98,7 +98,7 @@ public class WALCellCodec implements Codec { public static WALCellCodec create(Configuration conf, String cellCodecClsName, CompressionContext compression) throws UnsupportedOperationException { if (cellCodecClsName == null) { - cellCodecClsName = getWALCellCodecClass(conf); + cellCodecClsName = getWALCellCodecClass(conf).getName(); } return ReflectionUtils.instantiateWithCustomCtor(cellCodecClsName, new Class[] { Configuration.class, CompressionContext.class }, new Object[] { conf, compression }); @@ -117,7 +117,7 @@ public class WALCellCodec implements Codec { */ public static WALCellCodec create(Configuration conf, CompressionContext compression) throws UnsupportedOperationException { -Stri
[1/2] hbase git commit: HBASE-21492 CellCodec Written To WAL Before It's Verified
Repository: hbase Updated Branches: refs/heads/branch-1 55a775b8d -> beeb0796e refs/heads/branch-1.4 6dbb1d407 -> f7470a8b5 HBASE-21492 CellCodec Written To WAL Before It's Verified Conflicts: hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/beeb0796 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/beeb0796 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/beeb0796 Branch: refs/heads/branch-1 Commit: beeb0796e506dfb758c22dece51527e6247414e6 Parents: 55a775b Author: BELUGA BEHR Authored: Tue Nov 27 08:57:06 2018 -0800 Committer: Andrew Purtell Committed: Mon Dec 31 12:29:04 2018 -0800 -- .../org/apache/hadoop/hbase/mapreduce/WALPlayer.java | 2 +- .../hadoop/hbase/regionserver/wal/ProtobufLogWriter.java | 2 +- .../hadoop/hbase/regionserver/wal/WALCellCodec.java | 8 .../hbase/regionserver/wal/TestCustomWALCellCodec.java | 11 +++ 4 files changed, 17 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/beeb0796/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java index 377b6ea..bff110c 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java @@ -295,7 +295,7 @@ public class WALPlayer extends Configured implements Tool { // No reducers. job.setNumReduceTasks(0); } -String codecCls = WALCellCodec.getWALCellCodecClass(conf); +String codecCls = WALCellCodec.getWALCellCodecClass(conf).getName(); try { TableMapReduceUtil.addDependencyJarsForClasses(job.getConfiguration(), Class.forName(codecCls)); } catch (Exception e) { http://git-wip-us.apache.org/repos/asf/hbase/blob/beeb0796/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java index cb9e5a5..2e4226f 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java @@ -69,7 +69,7 @@ public class ProtobufLogWriter extends WriterBase { builder.setWriterClsName(ProtobufLogWriter.class.getSimpleName()); } if (!builder.hasCellCodecClsName()) { - builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf)); + builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf).getName()); } return builder.build(); } http://git-wip-us.apache.org/repos/asf/hbase/blob/beeb0796/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java index 5c62ef2..11b6120 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java @@ -79,8 +79,8 @@ public class WALCellCodec implements Codec { this.compression = compression; } - public static String getWALCellCodecClass(Configuration conf) { -return conf.get(WAL_CELL_CODEC_CLASS_KEY, WALCellCodec.class.getName()); + public static Class getWALCellCodecClass(Configuration conf) { +return conf.getClass(WAL_CELL_CODEC_CLASS_KEY, WALCellCodec.class); } /** @@ -98,7 +98,7 @@ public class WALCellCodec implements Codec { public static WALCellCodec create(Configuration conf, String cellCodecClsName, CompressionContext compression) throws UnsupportedOperationException { if (cellCodecClsName == null) { - cellCodecClsName = getWALCellCodecClass(conf); + cellCodecClsName = getWALCellCodecClass(conf).getName(); } return ReflectionUtils.instantiateWithCustomCtor(cellCodecClsName, new Class[] { Configuration.class, CompressionContext.class }, new Object[] { conf, compression }); @@ -117,7 +117,7 @@ public class WALCellCodec implements Codec { */ public static
[hbase] Git Push Summary
Repository: hbase Updated Tags: refs/tags/rel/1.3.3 [created] 3327b6417
[39/47] hbase git commit: HBASE-21650 Add DDL operation and some other miscellaneous to thrift2
http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnFamilyDescriptor.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnFamilyDescriptor.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnFamilyDescriptor.java new file mode 100644 index 000..03cb2f6 --- /dev/null +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnFamilyDescriptor.java @@ -0,0 +1,2519 @@ +/** + * Autogenerated by Thrift Compiler (0.9.3) + * + * DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING + * @generated + */ +package org.apache.hadoop.hbase.thrift2.generated; + +import org.apache.thrift.scheme.IScheme; +import org.apache.thrift.scheme.SchemeFactory; +import org.apache.thrift.scheme.StandardScheme; + +import org.apache.thrift.scheme.TupleScheme; +import org.apache.thrift.protocol.TTupleProtocol; +import org.apache.thrift.protocol.TProtocolException; +import org.apache.thrift.EncodingUtils; +import org.apache.thrift.TException; +import org.apache.thrift.async.AsyncMethodCallback; +import org.apache.thrift.server.AbstractNonblockingServer.*; +import java.util.List; +import java.util.ArrayList; +import java.util.Map; +import java.util.HashMap; +import java.util.EnumMap; +import java.util.Set; +import java.util.HashSet; +import java.util.EnumSet; +import java.util.Collections; +import java.util.BitSet; +import java.nio.ByteBuffer; +import java.util.Arrays; +import javax.annotation.Generated; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +@SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"}) +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") +public class TColumnFamilyDescriptor implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { + private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TColumnFamilyDescriptor"); + + private static final org.apache.thrift.protocol.TField NAME_FIELD_DESC = new org.apache.thrift.protocol.TField("name", org.apache.thrift.protocol.TType.STRING, (short)1); + private static final org.apache.thrift.protocol.TField ATTRIBUTES_FIELD_DESC = new org.apache.thrift.protocol.TField("attributes", org.apache.thrift.protocol.TType.MAP, (short)2); + private static final org.apache.thrift.protocol.TField CONFIGURATION_FIELD_DESC = new org.apache.thrift.protocol.TField("configuration", org.apache.thrift.protocol.TType.MAP, (short)3); + private static final org.apache.thrift.protocol.TField BLOCK_SIZE_FIELD_DESC = new org.apache.thrift.protocol.TField("blockSize", org.apache.thrift.protocol.TType.I32, (short)4); + private static final org.apache.thrift.protocol.TField BLOOMN_FILTER_TYPE_FIELD_DESC = new org.apache.thrift.protocol.TField("bloomnFilterType", org.apache.thrift.protocol.TType.I32, (short)5); + private static final org.apache.thrift.protocol.TField COMPRESSION_TYPE_FIELD_DESC = new org.apache.thrift.protocol.TField("compressionType", org.apache.thrift.protocol.TType.I32, (short)6); + private static final org.apache.thrift.protocol.TField DFS_REPLICATION_FIELD_DESC = new org.apache.thrift.protocol.TField("dfsReplication", org.apache.thrift.protocol.TType.I16, (short)7); + private static final org.apache.thrift.protocol.TField DATA_BLOCK_ENCODING_FIELD_DESC = new org.apache.thrift.protocol.TField("dataBlockEncoding", org.apache.thrift.protocol.TType.I32, (short)8); + private static final org.apache.thrift.protocol.TField KEEP_DELETED_CELLS_FIELD_DESC = new org.apache.thrift.protocol.TField("keepDeletedCells", org.apache.thrift.protocol.TType.I32, (short)9); + private static final org.apache.thrift.protocol.TField MAX_VERSIONS_FIELD_DESC = new org.apache.thrift.protocol.TField("maxVersions", org.apache.thrift.protocol.TType.I32, (short)10); + private static final org.apache.thrift.protocol.TField MIN_VERSIONS_FIELD_DESC = new org.apache.thrift.protocol.TField("minVersions", org.apache.thrift.protocol.TType.I32, (short)11); + private static final org.apache.thrift.protocol.TField SCOPE_FIELD_DESC = new org.apache.thrift.protocol.TField("scope", org.apache.thrift.protocol.TType.I32, (short)12); + private static final org.apache.thrift.protocol.TField TIME_TO_LIVE_FIELD_DESC = new org.apache.thrift.protocol.TField("timeToLive", org.apache.thrift.protocol.TType.I32, (short)13); + private static final org.apache.thrift.protocol.TField BLOCK_CACHE_ENABLED_FIELD_DESC = new org.apache.thrift.protocol.TField("blockCacheEnabled", org.apache.thrift.protocol.TType.BOOL, (short)14); + private static final org.apache.thrift.protocol.TField CACHE_BLOOMS_ON_WRITE_FIELD_DESC = new org.apache.thrift.protocol.TField("cacheBloomsOnWrite", org.apache.thrift.protocol
[47/47] hbase git commit: HBASE-21526 Use AsyncClusterConnection in ServerManager for getRsAdmin
HBASE-21526 Use AsyncClusterConnection in ServerManager for getRsAdmin Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/b33b072d Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/b33b072d Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/b33b072d Branch: refs/heads/HBASE-21512 Commit: b33b072de945d5272a3d46c06fd278cd64d11142 Parents: a13292d Author: zhangduo Authored: Thu Dec 6 21:25:34 2018 +0800 Committer: zhangduo Committed: Mon Dec 31 20:34:24 2018 +0800 -- .../hbase/client/AsyncClusterConnection.java| 6 + .../hbase/client/AsyncConnectionImpl.java | 5 + .../hbase/client/AsyncRegionServerAdmin.java| 210 +++ .../apache/hadoop/hbase/util/FutureUtils.java | 60 ++ .../org/apache/hadoop/hbase/master/HMaster.java | 13 +- .../hadoop/hbase/master/ServerManager.java | 67 -- .../master/procedure/RSProcedureDispatcher.java | 44 ++-- 7 files changed, 320 insertions(+), 85 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/b33b072d/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java index c7dea25..1327fd7 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java @@ -17,6 +17,7 @@ */ package org.apache.hadoop.hbase.client; +import org.apache.hadoop.hbase.ServerName; import org.apache.hadoop.hbase.ipc.RpcClient; import org.apache.yetus.audience.InterfaceAudience; @@ -27,6 +28,11 @@ import org.apache.yetus.audience.InterfaceAudience; public interface AsyncClusterConnection extends AsyncConnection { /** + * Get the admin service for the given region server. + */ + AsyncRegionServerAdmin getRegionServerAdmin(ServerName serverName); + + /** * Get the nonce generator for this connection. */ NonceGenerator getNonceGenerator(); http://git-wip-us.apache.org/repos/asf/hbase/blob/b33b072d/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java index 79ec54b..b01c03e 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java @@ -331,4 +331,9 @@ class AsyncConnectionImpl implements AsyncClusterConnection { return new AsyncBufferedMutatorBuilderImpl(connConf, getTableBuilder(tableName, pool), RETRY_TIMER); } + + @Override + public AsyncRegionServerAdmin getRegionServerAdmin(ServerName serverName) { +return new AsyncRegionServerAdmin(serverName, this); + } } http://git-wip-us.apache.org/repos/asf/hbase/blob/b33b072d/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionServerAdmin.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionServerAdmin.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionServerAdmin.java new file mode 100644 index 000..9accd89 --- /dev/null +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionServerAdmin.java @@ -0,0 +1,210 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import java.io.IOException; +import java.util.concurrent.CompletableFuture; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.ipc.HBaseRpcController; +import
[06/47] hbase git commit: HBASE-21590 Optimize trySkipToNextColumn in StoreScanner a bit.
HBASE-21590 Optimize trySkipToNextColumn in StoreScanner a bit. Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/cb1966dc Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/cb1966dc Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/cb1966dc Branch: refs/heads/HBASE-21512 Commit: cb1966dc2d94fba10d9b6af3c5719e03f621df92 Parents: f32d261 Author: Lars Hofhansl Authored: Thu Dec 13 11:57:16 2018 -0800 Committer: Lars Hofhansl Committed: Thu Dec 13 11:57:16 2018 -0800 -- .../apache/hadoop/hbase/regionserver/StoreScanner.java | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/cb1966dc/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java index 736c08a..e7a4528 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java @@ -802,12 +802,16 @@ public class StoreScanner extends NonReversedNonLazyKeyValueScanner @VisibleForTesting protected boolean trySkipToNextRow(Cell cell) throws IOException { Cell nextCell = null; +// used to guard against a changed next indexed key by doing a identity comparison +// when the identity changes we need to compare the bytes again +Cell previousIndexedKey = null; do { Cell nextIndexedKey = getNextIndexedKey(); if (nextIndexedKey != null && nextIndexedKey != KeyValueScanner.NO_NEXT_INDEXED_KEY - && matcher.compareKeyForNextRow(nextIndexedKey, cell) >= 0) { + && (nextIndexedKey == previousIndexedKey || matcher.compareKeyForNextRow(nextIndexedKey, cell) >= 0)) { this.heap.next(); ++kvsScanned; +previousIndexedKey = nextIndexedKey; } else { return false; } @@ -823,12 +827,16 @@ public class StoreScanner extends NonReversedNonLazyKeyValueScanner @VisibleForTesting protected boolean trySkipToNextColumn(Cell cell) throws IOException { Cell nextCell = null; +// used to guard against a changed next indexed key by doing a identity comparison +// when the identity changes we need to compare the bytes again +Cell previousIndexedKey = null; do { Cell nextIndexedKey = getNextIndexedKey(); if (nextIndexedKey != null && nextIndexedKey != KeyValueScanner.NO_NEXT_INDEXED_KEY - && matcher.compareKeyForNextColumn(nextIndexedKey, cell) >= 0) { + && (nextIndexedKey == previousIndexedKey || matcher.compareKeyForNextColumn(nextIndexedKey, cell) >= 0)) { this.heap.next(); ++kvsScanned; +previousIndexedKey = nextIndexedKey; } else { return false; }
[45/47] hbase git commit: HBASE-21515 Also initialize an AsyncClusterConnection in HRegionServer
HBASE-21515 Also initialize an AsyncClusterConnection in HRegionServer Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/f3caa018 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/f3caa018 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/f3caa018 Branch: refs/heads/HBASE-21512 Commit: f3caa0188b5b5f032d4b90a7b75e518a8752e0f4 Parents: 7755d4b Author: zhangduo Authored: Fri Nov 30 08:23:47 2018 +0800 Committer: zhangduo Committed: Mon Dec 31 20:34:24 2018 +0800 -- .../hbase/client/AsyncClusterConnection.java| 38 .../hbase/client/AsyncConnectionImpl.java | 39 ++-- .../hbase/client/ClusterConnectionFactory.java | 63 .../hadoop/hbase/client/ConnectionFactory.java | 5 +- .../hadoop/hbase/util/ReflectionUtils.java | 22 --- .../java/org/apache/hadoop/hbase/Server.java| 20 +++ .../org/apache/hadoop/hbase/master/HMaster.java | 3 + .../hbase/regionserver/HRegionServer.java | 56 - .../regionserver/ReplicationSyncUp.java | 6 ++ .../hadoop/hbase/MockRegionServerServices.java | 5 ++ .../client/TestAsyncNonMetaRegionLocator.java | 2 +- ...syncNonMetaRegionLocatorConcurrenyLimit.java | 2 +- .../client/TestAsyncRegionLocatorTimeout.java | 2 +- ...TestAsyncSingleRequestRpcRetryingCaller.java | 4 +- .../hbase/client/TestAsyncTableNoncedRetry.java | 2 +- .../hbase/master/MockNoopMasterServices.java| 6 ++ .../hadoop/hbase/master/MockRegionServer.java | 5 ++ .../hbase/master/TestActiveMasterManager.java | 6 ++ .../hbase/master/cleaner/TestHFileCleaner.java | 6 ++ .../master/cleaner/TestHFileLinkCleaner.java| 6 ++ .../hbase/master/cleaner/TestLogsCleaner.java | 6 ++ .../cleaner/TestReplicationHFileCleaner.java| 6 ++ .../regionserver/TestHeapMemoryManager.java | 6 ++ .../hbase/regionserver/TestSplitLogWorker.java | 6 ++ .../hbase/regionserver/TestWALLockup.java | 6 ++ .../TestReplicationTrackerZKImpl.java | 6 ++ .../TestReplicationSourceManager.java | 6 ++ .../security/token/TestTokenAuthentication.java | 6 ++ .../apache/hadoop/hbase/util/MockServer.java| 6 ++ 29 files changed, 302 insertions(+), 50 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/f3caa018/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java new file mode 100644 index 000..c7dea25 --- /dev/null +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClusterConnection.java @@ -0,0 +1,38 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import org.apache.hadoop.hbase.ipc.RpcClient; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * The asynchronous connection for internal usage. + */ +@InterfaceAudience.Private +public interface AsyncClusterConnection extends AsyncConnection { + + /** + * Get the nonce generator for this connection. + */ + NonceGenerator getNonceGenerator(); + + /** + * Get the rpc client we used to communicate with other servers. + */ + RpcClient getRpcClient(); +} http://git-wip-us.apache.org/repos/asf/hbase/blob/f3caa018/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java index 078395b..79ec54b 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java @@ -
[03/47] hbase git commit: HBASE-21568 Use CacheConfig.DISABLED where we don't expect to have blockcache running
HBASE-21568 Use CacheConfig.DISABLED where we don't expect to have blockcache running This includes removing the "old way" of disabling blockcache in favor of the new API. Signed-off-by: Guanghao Zhang Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/67d6d508 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/67d6d508 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/67d6d508 Branch: refs/heads/HBASE-21512 Commit: 67d6d5084cf8fc094cda4bd3f091d8a0a9cb1d3e Parents: f88224e Author: Josh Elser Authored: Fri Dec 7 17:18:49 2018 -0500 Committer: Josh Elser Committed: Tue Dec 11 10:02:18 2018 -0500 -- .../org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java | 6 ++ .../src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java | 4 +--- .../org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java | 2 +- .../org/apache/hadoop/hbase/tool/LoadIncrementalHFiles.java| 6 +++--- .../java/org/apache/hadoop/hbase/util/CompressionTest.java | 2 +- .../src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java | 5 ++--- .../apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java | 2 +- 7 files changed, 11 insertions(+), 16 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/67d6d508/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java -- diff --git a/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java b/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java index c911e8c..274a506 100644 --- a/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java +++ b/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java @@ -414,8 +414,6 @@ public class HFileOutputFormat2 DataBlockEncoding encoding = overriddenEncoding; encoding = encoding == null ? datablockEncodingMap.get(tableAndFamily) : encoding; encoding = encoding == null ? DataBlockEncoding.NONE : encoding; -Configuration tempConf = new Configuration(conf); -tempConf.setFloat(HConstants.HFILE_BLOCK_CACHE_SIZE_KEY, 0.0f); HFileContextBuilder contextBuilder = new HFileContextBuilder() .withCompression(compression) .withChecksumType(HStore.getChecksumType(conf)) @@ -430,12 +428,12 @@ public class HFileOutputFormat2 HFileContext hFileContext = contextBuilder.build(); if (null == favoredNodes) { wl.writer = - new StoreFileWriter.Builder(conf, new CacheConfig(tempConf), fs) + new StoreFileWriter.Builder(conf, CacheConfig.DISABLED, fs) .withOutputDir(familydir).withBloomType(bloomType) .withComparator(CellComparator.getInstance()).withFileContext(hFileContext).build(); } else { wl.writer = - new StoreFileWriter.Builder(conf, new CacheConfig(tempConf), new HFileSystem(fs)) + new StoreFileWriter.Builder(conf, CacheConfig.DISABLED, new HFileSystem(fs)) .withOutputDir(familydir).withBloomType(bloomType) .withComparator(CellComparator.getInstance()).withFileContext(hFileContext) .withFavoredNodes(favoredNodes).build(); http://git-wip-us.apache.org/repos/asf/hbase/blob/67d6d508/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java index 5bcaa17..78ebedc 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java @@ -356,9 +356,7 @@ public class HFile { */ public static final WriterFactory getWriterFactoryNoCache(Configuration conf) { -Configuration tempConf = new Configuration(conf); -tempConf.setFloat(HConstants.HFILE_BLOCK_CACHE_SIZE_KEY, 0.0f); -return HFile.getWriterFactory(conf, new CacheConfig(tempConf)); +return HFile.getWriterFactory(conf, CacheConfig.DISABLED); } /** http://git-wip-us.apache.org/repos/asf/hbase/blob/67d6d508/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java index 82e881b..5a6
[28/47] hbase git commit: HBASE-21629 draining_servers.rb is broken
HBASE-21629 draining_servers.rb is broken Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/59f77de7 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/59f77de7 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/59f77de7 Branch: refs/heads/HBASE-21512 Commit: 59f77de723849e4d330167f60e53e44b2763cafc Parents: 97fd647 Author: Nihal Jain Authored: Sun Dec 23 14:29:53 2018 +0530 Committer: stack Committed: Sun Dec 23 20:48:43 2018 -0800 -- bin/draining_servers.rb | 19 --- 1 file changed, 12 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/59f77de7/bin/draining_servers.rb -- diff --git a/bin/draining_servers.rb b/bin/draining_servers.rb index 0d29c19..a8e20f0 100644 --- a/bin/draining_servers.rb +++ b/bin/draining_servers.rb @@ -27,6 +27,7 @@ java_import org.apache.hadoop.hbase.HBaseConfiguration java_import org.apache.hadoop.hbase.client.ConnectionFactory java_import org.apache.hadoop.hbase.client.HBaseAdmin java_import org.apache.hadoop.hbase.zookeeper.ZKUtil +java_import org.apache.hadoop.hbase.zookeeper.ZNodePaths java_import org.slf4j.LoggerFactory # Name of this script @@ -86,11 +87,11 @@ def addServers(_options, hostOrServers) servers = getServerNames(hostOrServers, config) zkw = org.apache.hadoop.hbase.zookeeper.ZKWatcher.new(config, 'draining_servers', nil) - parentZnode = zkw.znodePaths.drainingZNode begin +parentZnode = zkw.getZNodePaths.drainingZNode for server in servers - node = ZKUtil.joinZNode(parentZnode, server) + node = ZNodePaths.joinZNode(parentZnode, server) ZKUtil.createAndFailSilent(zkw, node) end ensure @@ -103,11 +104,11 @@ def removeServers(_options, hostOrServers) servers = getServerNames(hostOrServers, config) zkw = org.apache.hadoop.hbase.zookeeper.ZKWatcher.new(config, 'draining_servers', nil) - parentZnode = zkw.znodePaths.drainingZNode begin +parentZnode = zkw.getZNodePaths.drainingZNode for server in servers - node = ZKUtil.joinZNode(parentZnode, server) + node = ZNodePaths.joinZNode(parentZnode, server) ZKUtil.deleteNodeFailSilent(zkw, node) end ensure @@ -120,10 +121,14 @@ def listServers(_options) config = HBaseConfiguration.create zkw = org.apache.hadoop.hbase.zookeeper.ZKWatcher.new(config, 'draining_servers', nil) - parentZnode = zkw.znodePaths.drainingZNode - servers = ZKUtil.listChildrenNoWatch(zkw, parentZnode) - servers.each { |server| puts server } + begin +parentZnode = zkw.getZNodePaths.drainingZNode +servers = ZKUtil.listChildrenNoWatch(zkw, parentZnode) +servers.each { |server| puts server } + ensure +zkw.close + end end hostOrServers = ARGV[1..ARGV.size]
[10/47] hbase git commit: Update downloads.xml for release 1.4.9
Update downloads.xml for release 1.4.9 Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/2b003c5d Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/2b003c5d Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/2b003c5d Branch: refs/heads/HBASE-21512 Commit: 2b003c5d685160eeaf90387e887b433dadb8695e Parents: 1b08ba7 Author: Andrew Purtell Authored: Fri Dec 14 13:54:57 2018 -0800 Committer: Andrew Purtell Committed: Fri Dec 14 13:54:57 2018 -0800 -- src/site/xdoc/downloads.xml | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/2b003c5d/src/site/xdoc/downloads.xml -- diff --git a/src/site/xdoc/downloads.xml b/src/site/xdoc/downloads.xml index 5d3f2a6..4bb7f51 100644 --- a/src/site/xdoc/downloads.xml +++ b/src/site/xdoc/downloads.xml @@ -88,23 +88,23 @@ under the License. -1.4.8 +1.4.9 -2018/10/08 +2018/12/14 -https://apache.org/dist/hbase/1.4.8/compat-check-report.html";>1.4.7 vs 1.4.8 +https://apache.org/dist/hbase/1.4.9/compat-check-report.html";>1.4.8 vs 1.4.9 -https://github.com/apache/hbase/blob/rel/1.4.8/CHANGES.txt";>Changes +https://github.com/apache/hbase/blob/rel/1.4.9/CHANGES.txt";>Changes -https://s.apache.org/hbase-1.4.8-jira-release-notes";>Release Notes +https://s.apache.org/hbase-1.4.9-jira-release-notes";>Release Notes -https://www.apache.org/dyn/closer.lua/hbase/1.4.8/hbase-1.4.8-src.tar.gz";>src (https://apache.org/dist/hbase/1.4.8/hbase-1.4.8-src.tar.gz.sha512";>sha512 https://apache.org/dist/hbase/1.4.8/hbase-1.4.8-src.tar.gz.asc";>asc) -https://www.apache.org/dyn/closer.lua/hbase/1.4.8/hbase-1.4.8-bin.tar.gz";>bin (https://apache.org/dist/hbase/1.4.8/hbase-1.4.8-bin.tar.gz.sha512";>sha512 https://apache.org/dist/hbase/1.4.8/hbase-1.4.8-bin.tar.gz.asc";>asc) +https://www.apache.org/dyn/closer.lua/hbase/1.4.9/hbase-1.4.9-src.tar.gz";>src (https://apache.org/dist/hbase/1.4.9/hbase-1.4.9-src.tar.gz.sha512";>sha512 https://apache.org/dist/hbase/1.4.9/hbase-1.4.9-src.tar.gz.asc";>asc) +https://www.apache.org/dyn/closer.lua/hbase/1.4.9/hbase-1.4.9-bin.tar.gz";>bin (https://apache.org/dist/hbase/1.4.9/hbase-1.4.9-bin.tar.gz.sha512";>sha512 https://apache.org/dist/hbase/1.4.9/hbase-1.4.9-bin.tar.gz.asc";>asc)
[37/47] hbase git commit: HBASE-21650 Add DDL operation and some other miscellaneous to thrift2
http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java index e8f36a0..7388443 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java @@ -34,7 +34,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"}) -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-07-04") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") public class THBaseService { public interface Iface { @@ -282,6 +282,56 @@ public class THBaseService { */ public boolean checkAndMutate(ByteBuffer table, ByteBuffer row, ByteBuffer family, ByteBuffer qualifier, TCompareOp compareOp, ByteBuffer value, TRowMutations rowMutations) throws TIOError, org.apache.thrift.TException; +public TTableDescriptor getTableDescriptor(TTableName table) throws TIOError, org.apache.thrift.TException; + +public List getTableDescriptors(List tables) throws TIOError, org.apache.thrift.TException; + +public boolean tableExists(TTableName tableName) throws TIOError, org.apache.thrift.TException; + +public List getTableDescriptorsByPattern(String regex, boolean includeSysTables) throws TIOError, org.apache.thrift.TException; + +public List getTableDescriptorsByNamespace(String name) throws TIOError, org.apache.thrift.TException; + +public List getTableNamesByPattern(String regex, boolean includeSysTables) throws TIOError, org.apache.thrift.TException; + +public List getTableNamesByNamespace(String name) throws TIOError, org.apache.thrift.TException; + +public void createTable(TTableDescriptor desc, List splitKeys) throws TIOError, org.apache.thrift.TException; + +public void deleteTable(TTableName tableName) throws TIOError, org.apache.thrift.TException; + +public void truncateTable(TTableName tableName, boolean preserveSplits) throws TIOError, org.apache.thrift.TException; + +public void enableTable(TTableName tableName) throws TIOError, org.apache.thrift.TException; + +public void disableTable(TTableName tableName) throws TIOError, org.apache.thrift.TException; + +public boolean isTableEnabled(TTableName tableName) throws TIOError, org.apache.thrift.TException; + +public boolean isTableDisabled(TTableName tableName) throws TIOError, org.apache.thrift.TException; + +public boolean isTableAvailable(TTableName tableName) throws TIOError, org.apache.thrift.TException; + +public boolean isTableAvailableWithSplit(TTableName tableName, List splitKeys) throws TIOError, org.apache.thrift.TException; + +public void addColumnFamily(TTableName tableName, TColumnFamilyDescriptor column) throws TIOError, org.apache.thrift.TException; + +public void deleteColumnFamily(TTableName tableName, ByteBuffer column) throws TIOError, org.apache.thrift.TException; + +public void modifyColumnFamily(TTableName tableName, TColumnFamilyDescriptor column) throws TIOError, org.apache.thrift.TException; + +public void modifyTable(TTableDescriptor desc) throws TIOError, org.apache.thrift.TException; + +public void createNamespace(TNamespaceDescriptor namespaceDesc) throws TIOError, org.apache.thrift.TException; + +public void modifyNamespace(TNamespaceDescriptor namespaceDesc) throws TIOError, org.apache.thrift.TException; + +public void deleteNamespace(String name) throws TIOError, org.apache.thrift.TException; + +public TNamespaceDescriptor getNamespaceDescriptor(String name) throws TIOError, org.apache.thrift.TException; + +public List listNamespaceDescriptors() throws TIOError, org.apache.thrift.TException; + } public interface AsyncIface { @@ -326,6 +376,56 @@ public class THBaseService { public void checkAndMutate(ByteBuffer table, ByteBuffer row, ByteBuffer family, ByteBuffer qualifier, TCompareOp compareOp, ByteBuffer value, TRowMutations rowMutations, org.apache.thrift.async.AsyncMethodCallback resultHandler) throws org.apache.thrift.TException; +public void getTableDescriptor(TTableName table, org.apache.thrift.async.AsyncMethodCallback resultHandler) throws org.apache.thrift.TException; + +public void getTableDescriptors(List tables, org.apache.thrift.async.AsyncMethodCallback resultHandler) throws org.apache.thrift.TException; + +public void tableExists(TTableName tableName, org.apache.thrift.async.AsyncMethodCallback resultHandler) throws org.apache.thrift.TException; + +public void get
[43/47] hbase git commit: HBASE-21650 Add DDL operation and some other miscellaneous to thrift2 (ADDENDUM add some comments)
HBASE-21650 Add DDL operation and some other miscellaneous to thrift2 (ADDENDUM add some comments) Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/b620334c Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/b620334c Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/b620334c Branch: refs/heads/HBASE-21512 Commit: b620334c20e84a4876226b508213ce11b8b187a0 Parents: 7820ba1 Author: Allan Yang Authored: Fri Dec 28 15:32:50 2018 +0800 Committer: Allan Yang Committed: Fri Dec 28 15:32:50 2018 +0800 -- .../hbase/thrift/generated/AlreadyExists.java | 2 +- .../hbase/thrift/generated/BatchMutation.java | 2 +- .../thrift/generated/ColumnDescriptor.java | 2 +- .../hadoop/hbase/thrift/generated/Hbase.java| 2 +- .../hadoop/hbase/thrift/generated/IOError.java | 2 +- .../hbase/thrift/generated/IllegalArgument.java | 2 +- .../hadoop/hbase/thrift/generated/Mutation.java | 2 +- .../hadoop/hbase/thrift/generated/TAppend.java | 2 +- .../hadoop/hbase/thrift/generated/TCell.java| 2 +- .../hadoop/hbase/thrift/generated/TColumn.java | 2 +- .../hbase/thrift/generated/TIncrement.java | 2 +- .../hbase/thrift/generated/TRegionInfo.java | 2 +- .../hbase/thrift/generated/TRowResult.java | 2 +- .../hadoop/hbase/thrift/generated/TScan.java| 2 +- .../hadoop/hbase/thrift2/generated/TAppend.java | 2 +- .../hbase/thrift2/generated/TAuthorization.java | 2 +- .../thrift2/generated/TBloomFilterType.java | 4 + .../thrift2/generated/TCellVisibility.java | 2 +- .../hadoop/hbase/thrift2/generated/TColumn.java | 2 +- .../generated/TColumnFamilyDescriptor.java | 6 +- .../thrift2/generated/TColumnIncrement.java | 2 +- .../hbase/thrift2/generated/TColumnValue.java | 2 +- .../generated/TCompressionAlgorithm.java| 4 + .../thrift2/generated/TDataBlockEncoding.java | 4 + .../hadoop/hbase/thrift2/generated/TDelete.java | 2 +- .../hadoop/hbase/thrift2/generated/TGet.java| 2 +- .../hbase/thrift2/generated/THBaseService.java | 571 ++- .../hbase/thrift2/generated/THRegionInfo.java | 2 +- .../thrift2/generated/THRegionLocation.java | 2 +- .../hbase/thrift2/generated/TIOError.java | 2 +- .../thrift2/generated/TIllegalArgument.java | 2 +- .../hbase/thrift2/generated/TIncrement.java | 2 +- .../thrift2/generated/TKeepDeletedCells.java| 4 + .../thrift2/generated/TNamespaceDescriptor.java | 6 +- .../hadoop/hbase/thrift2/generated/TPut.java| 2 +- .../hadoop/hbase/thrift2/generated/TResult.java | 2 +- .../hbase/thrift2/generated/TRowMutations.java | 2 +- .../hadoop/hbase/thrift2/generated/TScan.java | 2 +- .../hbase/thrift2/generated/TServerName.java| 2 +- .../thrift2/generated/TTableDescriptor.java | 6 +- .../hbase/thrift2/generated/TTableName.java | 30 +- .../hbase/thrift2/generated/TTimeRange.java | 2 +- .../apache/hadoop/hbase/thrift2/hbase.thrift| 168 +- 43 files changed, 828 insertions(+), 41 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/b620334c/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java index 8ec3e32..4457b9f 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java @@ -38,7 +38,7 @@ import org.slf4j.LoggerFactory; * An AlreadyExists exceptions signals that a table with the specified * name already exists */ -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-28") public class AlreadyExists extends TException implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("AlreadyExists"); http://git-wip-us.apache.org/repos/asf/hbase/blob/b620334c/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/BatchMutation.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/BatchMutation.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/BatchMutation.java index 0872223..f605286 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop
[44/47] hbase git commit: HBASE-21646 Flakey TestTableSnapshotInputFormat; DisableTable not completing... Amendment to fix checkstyle complaint
HBASE-21646 Flakey TestTableSnapshotInputFormat; DisableTable not completing... Amendment to fix checkstyle complaint Includes fix for checkstyle complaint. Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/7755d4be Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/7755d4be Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/7755d4be Branch: refs/heads/HBASE-21512 Commit: 7755d4beeddfee9b72446ccd18d7918278eecc83 Parents: b620334 Author: stack Authored: Fri Dec 28 14:42:22 2018 -0800 Committer: stack Committed: Fri Dec 28 14:48:23 2018 -0800 -- .../hadoop/hbase/mapred/TestTableSnapshotInputFormat.java | 7 +++ .../hbase/mapreduce/TableSnapshotInputFormatTestBase.java | 5 - .../hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java | 5 + 3 files changed, 12 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/7755d4be/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapred/TestTableSnapshotInputFormat.java -- diff --git a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapred/TestTableSnapshotInputFormat.java b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapred/TestTableSnapshotInputFormat.java index b61ed07..c591af6 100644 --- a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapred/TestTableSnapshotInputFormat.java +++ b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapred/TestTableSnapshotInputFormat.java @@ -47,6 +47,7 @@ import org.apache.hadoop.mapred.RunningJob; import org.apache.hadoop.mapred.lib.NullOutputFormat; import org.junit.Assert; import org.junit.ClassRule; +import org.junit.Ignore; import org.junit.Rule; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -310,4 +311,10 @@ public class TestTableSnapshotInputFormat extends TableSnapshotInputFormatTestBa } } } + + @Ignore // Ignored in mapred package because it keeps failing but allowed in mapreduce package. + @Test + public void testWithMapReduceMultipleMappersPerRegion() throws Exception { +testWithMapReduce(UTIL, "testWithMapReduceMultiRegion", 10, 5, 50, false); + } } http://git-wip-us.apache.org/repos/asf/hbase/blob/7755d4be/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java -- diff --git a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java index 744c356..5e7ea7a 100644 --- a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java +++ b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java @@ -108,11 +108,6 @@ public abstract class TableSnapshotInputFormatTestBase { } @Test - public void testWithMapReduceMultipleMappersPerRegion() throws Exception { -testWithMapReduce(UTIL, "testWithMapReduceMultiRegion", 10, 5, 50, false); - } - - @Test // run the MR job while HBase is offline public void testWithMapReduceAndOfflineHBaseMultiRegion() throws Exception { testWithMapReduce(UTIL, "testWithMapReduceAndOfflineHBaseMultiRegion", 10, 1, 8, true); http://git-wip-us.apache.org/repos/asf/hbase/blob/7755d4be/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java -- diff --git a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java index f61c222..358af24 100644 --- a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java +++ b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java @@ -473,4 +473,9 @@ public class TestTableSnapshotInputFormat extends TableSnapshotInputFormatTestBa } } } + + @Test + public void testWithMapReduceMultipleMappersPerRegion() throws Exception { +testWithMapReduce(UTIL, "testWithMapReduceMultiRegion", 10, 5, 50, false); + } }
[12/47] hbase git commit: HBASE-21520 TestMultiColumnScanner cost long time when using ROWCOL bloom type
HBASE-21520 TestMultiColumnScanner cost long time when using ROWCOL bloom type Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/ac0b3bb5 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/ac0b3bb5 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/ac0b3bb5 Branch: refs/heads/HBASE-21512 Commit: ac0b3bb5477612cb8844c4ef10fa2be0f1d1a025 Parents: 4911534 Author: huzheng Authored: Thu Dec 13 15:04:12 2018 +0800 Committer: huzheng Committed: Sat Dec 15 21:08:52 2018 +0800 -- .../regionserver/TestMultiColumnScanner.java| 94 ++-- ...olumnScannerWithAlgoGZAndNoDataEncoding.java | 48 ++ ...lumnScannerWithAlgoGZAndUseDataEncoding.java | 48 ++ ...iColumnScannerWithNoneAndNoDataEncoding.java | 48 ++ ...ColumnScannerWithNoneAndUseDataEncoding.java | 48 ++ 5 files changed, 219 insertions(+), 67 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/ac0b3bb5/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java -- diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java index 2ff0d8c..bb97c9c 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java @@ -32,11 +32,9 @@ import java.util.Map; import java.util.Random; import java.util.Set; import java.util.TreeSet; -import org.apache.commons.lang3.ArrayUtils; import org.apache.hadoop.hbase.Cell; import org.apache.hadoop.hbase.CellComparatorImpl; import org.apache.hadoop.hbase.CellUtil; -import org.apache.hadoop.hbase.HBaseClassTestRule; import org.apache.hadoop.hbase.HBaseTestingUtility; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.KeyValue; @@ -47,29 +45,27 @@ import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.io.compress.Compression; import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; -import org.apache.hadoop.hbase.testclassification.MediumTests; -import org.apache.hadoop.hbase.testclassification.RegionServerTests; import org.apache.hadoop.hbase.util.BloomFilterUtil; import org.apache.hadoop.hbase.util.Bytes; -import org.junit.ClassRule; import org.junit.Test; -import org.junit.experimental.categories.Category; -import org.junit.runner.RunWith; -import org.junit.runners.Parameterized; -import org.junit.runners.Parameterized.Parameters; +import org.junit.runners.Parameterized.Parameter; import org.slf4j.Logger; import org.slf4j.LoggerFactory; /** - * Tests optimized scanning of multiple columns. + * Tests optimized scanning of multiple columns. + * We separated the big test into several sub-class UT, because When in ROWCOL bloom type, we will + * test the row-col bloom filter frequently for saving HDFS seek once we switch from one column to + * another in our UT. It's cpu time consuming (~45s for each case), so moved the ROWCOL case into a + * separated LargeTests to avoid timeout failure. + * + * To be clear: In TestMultiColumnScanner, we will flush 10 (NUM_FLUSHES=10) HFiles here, and the + * table will put ~1000 cells (rows=20, ts=6, qualifiers=8, total=20*6*8 ~ 1000) . Each full table + * scan will check the ROWCOL bloom filter 20 (rows)* 8 (column) * 10 (hfiles)= 1600 times, beside + * it will scan the full table 6*2^8=1536 times, so finally will have 1600*1536=2457600 bloom filter + * testing. (See HBASE-21520) */ -@RunWith(Parameterized.class) -@Category({RegionServerTests.class, MediumTests.class}) -public class TestMultiColumnScanner { - - @ClassRule - public static final HBaseClassTestRule CLASS_RULE = - HBaseClassTestRule.forClass(TestMultiColumnScanner.class); +public abstract class TestMultiColumnScanner { private static final Logger LOG = LoggerFactory.getLogger(TestMultiColumnScanner.class); @@ -104,20 +100,19 @@ public class TestMultiColumnScanner { /** The probability that a column is skipped in a store file. */ private static final double COLUMN_SKIP_IN_STORE_FILE_PROB = 0.7; - /** The probability of skipping a column in a single row */ - private static final double COLUMN_SKIP_IN_ROW_PROB = 0.1; - - /** The probability of skipping a column everywhere */ - private static final double COLUMN_SKIP_EVERYWHERE_PROB = 0.1; - /** The probability to delete a row/column pair */ private static final double DELETE_PROBABILITY = 0.02; private final static HBaseTestingUtility TEST_UTIL = HBaseTestingUtility.c
[24/47] hbase git commit: HBASE-21620 Problem in scan query when using more than one column prefix filter in some cases
HBASE-21620 Problem in scan query when using more than one column prefix filter in some cases Signed-off-by: Guanghao Zhang Signed-off-by: Michael Stack Signed-off-by: Allan Yang Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/e160b5ac Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/e160b5ac Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/e160b5ac Branch: refs/heads/HBASE-21512 Commit: e160b5ac8d82330911ea746e456ea53bf317ace8 Parents: 12786f8 Author: openinx Authored: Thu Dec 20 21:04:10 2018 +0800 Committer: stack Committed: Fri Dec 21 15:21:53 2018 -0800 -- .../hadoop/hbase/filter/FilterListWithOR.java | 65 ++-- .../hadoop/hbase/regionserver/StoreScanner.java | 2 +- .../hadoop/hbase/filter/TestFilterList.java | 62 +-- .../hbase/filter/TestFilterListOnMini.java | 50 ++- 4 files changed, 140 insertions(+), 39 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/e160b5ac/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterListWithOR.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterListWithOR.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterListWithOR.java index 842fdc5..ba4cd88 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterListWithOR.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterListWithOR.java @@ -83,30 +83,40 @@ public class FilterListWithOR extends FilterListBase { * next family for RegionScanner, INCLUDE_AND_NEXT_ROW is the same. so we should pass current cell * to the filter, if row mismatch or row match but column family mismatch. (HBASE-18368) * @see org.apache.hadoop.hbase.filter.Filter.ReturnCode + * @param subFilter which sub-filter to calculate the return code by using previous cell and + * previous return code. + * @param prevCell the previous cell passed to given sub-filter. + * @param currentCell the current cell which will pass to given sub-filter. + * @param prevCode the previous return code for given sub-filter. + * @return return code calculated by using previous cell and previous return code. null means can + * not decide which return code should return, so we will pass the currentCell to + * subFilter for getting currentCell's return code, and it won't impact the sub-filter's + * internal states. */ - private boolean shouldPassCurrentCellToFilter(Cell prevCell, Cell currentCell, - ReturnCode prevCode) throws IOException { + private ReturnCode calculateReturnCodeByPrevCellAndRC(Filter subFilter, Cell currentCell, + Cell prevCell, ReturnCode prevCode) throws IOException { if (prevCell == null || prevCode == null) { - return true; + return null; } switch (prevCode) { case INCLUDE: case SKIP: - return true; +return null; case SEEK_NEXT_USING_HINT: - Cell nextHintCell = getNextCellHint(prevCell); - return nextHintCell == null || this.compareCell(currentCell, nextHintCell) >= 0; +Cell nextHintCell = subFilter.getNextCellHint(prevCell); +return nextHintCell != null && compareCell(currentCell, nextHintCell) < 0 + ? ReturnCode.SEEK_NEXT_USING_HINT : null; case NEXT_COL: case INCLUDE_AND_NEXT_COL: - // Once row changed, reset() will clear prevCells, so we need not to compare their rows - // because rows are the same here. - return !CellUtil.matchingColumn(prevCell, currentCell); +// Once row changed, reset() will clear prevCells, so we need not to compare their rows +// because rows are the same here. +return CellUtil.matchingColumn(prevCell, currentCell) ? ReturnCode.NEXT_COL : null; case NEXT_ROW: case INCLUDE_AND_SEEK_NEXT_ROW: - // As described above, rows are definitely the same, so we only compare the family. - return !CellUtil.matchingFamily(prevCell, currentCell); +// As described above, rows are definitely the same, so we only compare the family. +return CellUtil.matchingFamily(prevCell, currentCell) ? ReturnCode.NEXT_ROW : null; default: - throw new IllegalStateException("Received code is not valid."); +throw new IllegalStateException("Received code is not valid."); } } @@ -240,7 +250,7 @@ public class FilterListWithOR extends FilterListBase { private void updatePrevCellList(int index, Cell currentCell, ReturnCode currentRC) { if (currentCell == null || currentRC == ReturnCode.INCLUDE || currentRC == ReturnCode.SKIP) { // If previous return code is INCLUDE or SKIP, we should al
[29/47] hbase git commit: HBASE-21545 NEW_VERSION_BEHAVIOR breaks Get/Scan with specified columns
HBASE-21545 NEW_VERSION_BEHAVIOR breaks Get/Scan with specified columns Signed-off-by: Duo Zhang Signed-off-by: stack Signed-off-by: Sakthi Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/dbafa1be Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/dbafa1be Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/dbafa1be Branch: refs/heads/HBASE-21512 Commit: dbafa1be83d6f5894f1cc3eadb07ae0c3096de3a Parents: 59f77de Author: Andrey Elenskiy Authored: Tue Dec 4 12:10:38 2018 -0800 Committer: stack Committed: Sun Dec 23 22:01:11 2018 -0800 -- .../querymatcher/NewVersionBehaviorTracker.java | 39 +++ .../hadoop/hbase/HBaseTestingUtility.java | 58 -- ...estGetScanColumnsWithNewVersionBehavior.java | 109 +++ .../TestNewVersionBehaviorTracker.java | 36 ++ 4 files changed, 214 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/dbafa1be/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NewVersionBehaviorTracker.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NewVersionBehaviorTracker.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NewVersionBehaviorTracker.java index 4027766..16ac84c 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NewVersionBehaviorTracker.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NewVersionBehaviorTracker.java @@ -277,26 +277,26 @@ public class NewVersionBehaviorTracker implements ColumnTracker, DeleteTracker { @Override public MatchCode checkColumn(Cell cell, byte type) throws IOException { -if (done()) { - // No more columns left, we are done with this query - return ScanQueryMatcher.MatchCode.SEEK_NEXT_ROW; // done_row +if (columns == null) { +return MatchCode.INCLUDE; } -if (columns != null) { - while (columnIndex < columns.length) { -int c = Bytes.compareTo(columns[columnIndex], 0, columns[columnIndex].length, -cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength()); -if (c < 0) { - columnIndex++; -} else if (c == 0) { - // We drop old version in #isDeleted, so here we must return INCLUDE. - return MatchCode.INCLUDE; -} else { - return MatchCode.SEEK_NEXT_COL; -} + +while (!done()) { + int c = CellUtil.compareQualifiers(cell, +columns[columnIndex], 0, columns[columnIndex].length); + if (c < 0) { +return MatchCode.SEEK_NEXT_COL; } - return MatchCode.SEEK_NEXT_ROW; + + if (c == 0) { +// We drop old version in #isDeleted, so here we must return INCLUDE. +return MatchCode.INCLUDE; + } + + columnIndex++; } -return MatchCode.INCLUDE; +// No more columns left, we are done with this query +return MatchCode.SEEK_NEXT_ROW; } @Override @@ -351,10 +351,7 @@ public class NewVersionBehaviorTracker implements ColumnTracker, DeleteTracker { @Override public boolean done() { -// lastCq* have been updated to this cell. -return !(columns == null || lastCqArray == null) && Bytes -.compareTo(lastCqArray, lastCqOffset, lastCqLength, columns[columnIndex], 0, -columns[columnIndex].length) > 0; +return columns != null && columnIndex >= columns.length; } @Override http://git-wip-us.apache.org/repos/asf/hbase/blob/dbafa1be/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java -- diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java index 7bfbfe1..796dbc3 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java @@ -713,6 +713,18 @@ public class HBaseTestingUtility extends HBaseZKTestingUtility { new Path(root, "mapreduce-am-staging-root-dir").toString()); } + /** + * Check whether the tests should assume NEW_VERSION_BEHAVIOR when creating + * new column families. Default to false. + */ + public boolean isNewVersionBehaviorEnabled(){ +final String propName = "hbase.tests.new.version.behavior"; +String v = System.getProperty(propName); +if (v != null){ + return Boolean.parseBoolean(v); +} +return false; + } /** * Get the HBase setting for dfs.client.read.shortcircuit from
[07/47] hbase git commit: Revert "HIVE-21575 : memstore above high watermark message is logged too much"
Revert "HIVE-21575 : memstore above high watermark message is logged too much" This reverts commit 4640ff5959af4865966126a503a7cd15e26a7408. Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/9a25d0c2 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/9a25d0c2 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/9a25d0c2 Branch: refs/heads/HBASE-21512 Commit: 9a25d0c249e595a1f8aef41cd677b44ff1c72d73 Parents: cb1966d Author: Sergey Shelukhin Authored: Thu Dec 13 12:46:39 2018 -0800 Committer: Sergey Shelukhin Committed: Thu Dec 13 12:46:39 2018 -0800 -- .../apache/hadoop/hbase/regionserver/MemStoreFlusher.java| 8 ++-- 1 file changed, 2 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/9a25d0c2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java index 804a2f8..699c9b6 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java @@ -703,7 +703,6 @@ class MemStoreFlusher implements FlushRequester { if (flushType != FlushType.NORMAL) { TraceUtil.addTimelineAnnotation("Force Flush. We're above high water mark."); long start = EnvironmentEdgeManager.currentTime(); - long nextLogTimeMs = start; synchronized (this.blockSignal) { boolean blocked = false; long startTime = 0; @@ -745,11 +744,8 @@ class MemStoreFlusher implements FlushRequester { LOG.warn("Interrupted while waiting"); interrupted = true; } -long nowMs = EnvironmentEdgeManager.currentTime(); -if (nowMs >= nextLogTimeMs) { - LOG.warn("Memstore is above high water mark and block {} ms", nowMs - start); - nextLogTimeMs = nowMs + 1000; -} +long took = EnvironmentEdgeManager.currentTime() - start; +LOG.warn("Memstore is above high water mark and block " + took + "ms"); flushType = isAboveHighWaterMark(); } } finally {
[08/47] hbase git commit: HBASE-21575 : memstore above high watermark message is logged too much
HBASE-21575 : memstore above high watermark message is logged too much Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/3ff274e2 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/3ff274e2 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/3ff274e2 Branch: refs/heads/HBASE-21512 Commit: 3ff274e22eb5710f4301fb0fce364e22a11288d7 Parents: 9a25d0c Author: Sergey Shelukhin Authored: Wed Dec 12 11:02:25 2018 -0800 Committer: Sergey Shelukhin Committed: Thu Dec 13 12:47:11 2018 -0800 -- .../apache/hadoop/hbase/regionserver/MemStoreFlusher.java| 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/3ff274e2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java index 699c9b6..804a2f8 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java @@ -703,6 +703,7 @@ class MemStoreFlusher implements FlushRequester { if (flushType != FlushType.NORMAL) { TraceUtil.addTimelineAnnotation("Force Flush. We're above high water mark."); long start = EnvironmentEdgeManager.currentTime(); + long nextLogTimeMs = start; synchronized (this.blockSignal) { boolean blocked = false; long startTime = 0; @@ -744,8 +745,11 @@ class MemStoreFlusher implements FlushRequester { LOG.warn("Interrupted while waiting"); interrupted = true; } -long took = EnvironmentEdgeManager.currentTime() - start; -LOG.warn("Memstore is above high water mark and block " + took + "ms"); +long nowMs = EnvironmentEdgeManager.currentTime(); +if (nowMs >= nextLogTimeMs) { + LOG.warn("Memstore is above high water mark and block {} ms", nowMs - start); + nextLogTimeMs = nowMs + 1000; +} flushType = isAboveHighWaterMark(); } } finally {
[34/47] hbase git commit: HBASE-21643 Introduce two new region coprocessor method and deprecated postMutationBeforeWAL
HBASE-21643 Introduce two new region coprocessor method and deprecated postMutationBeforeWAL Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/f5ea00f7 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/f5ea00f7 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/f5ea00f7 Branch: refs/heads/HBASE-21512 Commit: f5ea00f72442e5c80f2a5fc6e99506127fa8d16b Parents: c2d5991 Author: Guanghao Zhang Authored: Wed Dec 26 17:42:02 2018 +0800 Committer: Guanghao Zhang Committed: Thu Dec 27 18:27:06 2018 +0800 -- .../hbase/coprocessor/RegionObserver.java | 47 .../hadoop/hbase/regionserver/HRegion.java | 26 ++- .../regionserver/RegionCoprocessorHost.java | 29 +--- .../hbase/security/access/AccessController.java | 30 ++--- .../visibility/VisibilityController.java| 30 +++-- 5 files changed, 134 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/f5ea00f7/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java index c14cbd1..95b2150 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java @@ -20,6 +20,7 @@ package org.apache.hadoop.hbase.coprocessor; import java.io.IOException; +import java.util.ArrayList; import java.util.List; import java.util.Map; @@ -1029,13 +1030,59 @@ public interface RegionObserver { * @param oldCell old cell containing previous value * @param newCell the new cell containing the computed value * @return the new cell, possibly changed + * @deprecated Use {@link #postIncrementBeforeWAL} or {@link #postAppendBeforeWAL} instead. */ + @Deprecated default Cell postMutationBeforeWAL(ObserverContext ctx, MutationType opType, Mutation mutation, Cell oldCell, Cell newCell) throws IOException { return newCell; } /** + * Called after a list of new cells has been created during an increment operation, but before + * they are committed to the WAL or memstore. + * + * @param ctx the environment provided by the region server + * @param mutation the current mutation + * @param cellPairs a list of cell pair. The first cell is old cell which may be null. + * And the second cell is the new cell. + * @return a list of cell pair, possibly changed. + */ + default List> postIncrementBeforeWAL( + ObserverContext ctx, Mutation mutation, + List> cellPairs) throws IOException { +List> resultPairs = new ArrayList<>(cellPairs.size()); +for (Pair pair : cellPairs) { + resultPairs.add(new Pair<>(pair.getFirst(), + postMutationBeforeWAL(ctx, MutationType.INCREMENT, mutation, pair.getFirst(), + pair.getSecond(; +} +return resultPairs; + } + + /** + * Called after a list of new cells has been created during an append operation, but before + * they are committed to the WAL or memstore. + * + * @param ctx the environment provided by the region server + * @param mutation the current mutation + * @param cellPairs a list of cell pair. The first cell is old cell which may be null. + * And the second cell is the new cell. + * @return a list of cell pair, possibly changed. + */ + default List> postAppendBeforeWAL( + ObserverContext ctx, Mutation mutation, + List> cellPairs) throws IOException { +List> resultPairs = new ArrayList<>(cellPairs.size()); +for (Pair pair : cellPairs) { + resultPairs.add(new Pair<>(pair.getFirst(), + postMutationBeforeWAL(ctx, MutationType.INCREMENT, mutation, pair.getFirst(), + pair.getSecond(; +} +return resultPairs; + } + + /** * Called after the ScanQueryMatcher creates ScanDeleteTracker. Implementing * this hook would help in creating customised DeleteTracker and returning * the newly created DeleteTracker http://git-wip-us.apache.org/repos/asf/hbase/blob/f5ea00f7/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java index 9bf9309..ec222c7 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java +++ b/hbase-serv
[19/47] hbase git commit: HBASE-21535, Zombie Master detector is not working
HBASE-21535, Zombie Master detector is not working Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/fb58a23e Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/fb58a23e Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/fb58a23e Branch: refs/heads/HBASE-21512 Commit: fb58a23e56c8fe85820c97337da887eddf4bb9bb Parents: c448604 Author: Pankaj Authored: Tue Dec 18 00:49:22 2018 +0530 Committer: stack Committed: Tue Dec 18 20:51:01 2018 -0800 -- .../java/org/apache/hadoop/hbase/master/HMaster.java | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/fb58a23e/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java index a16e09d..0bcef59 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java @@ -900,11 +900,6 @@ public class HMaster extends HRegionServer implements MasterServices { */ private void finishActiveMasterInitialization(MonitoredTask status) throws IOException, InterruptedException, KeeperException, ReplicationException { -Thread zombieDetector = new Thread(new InitializationMonitor(this), -"ActiveMasterInitializationMonitor-" + System.currentTimeMillis()); -zombieDetector.setDaemon(true); -zombieDetector.start(); - /* * We are active master now... go initialize components we need to run. */ @@ -1001,6 +996,12 @@ public class HMaster extends HRegionServer implements MasterServices { // Set ourselves as active Master now our claim has succeeded up in zk. this.activeMaster = true; +// Start the Zombie master detector after setting master as active, see HBASE-21535 +Thread zombieDetector = new Thread(new InitializationMonitor(this), +"ActiveMasterInitializationMonitor-" + System.currentTimeMillis()); +zombieDetector.setDaemon(true); +zombieDetector.start(); + // This is for backwards compatibility // See HBASE-11393 status.setStatus("Update TableCFs node in ZNode");
[01/47] hbase git commit: HBASE-21570 Add write buffer periodic flush support for AsyncBufferedMutator [Forced Update!]
Repository: hbase Updated Branches: refs/heads/HBASE-21512 e7a122780 -> b33b072de (forced update) HBASE-21570 Add write buffer periodic flush support for AsyncBufferedMutator Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/b09b87d1 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/b09b87d1 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/b09b87d1 Branch: refs/heads/HBASE-21512 Commit: b09b87d143730db00ec56114a752d3a74f8982c4 Parents: da9508d Author: zhangduo Authored: Tue Dec 11 08:39:43 2018 +0800 Committer: Duo Zhang Committed: Tue Dec 11 14:51:26 2018 +0800 -- .../hbase/client/AsyncBufferedMutator.java | 16 +- .../client/AsyncBufferedMutatorBuilder.java | 19 +++ .../client/AsyncBufferedMutatorBuilderImpl.java | 19 ++- .../hbase/client/AsyncBufferedMutatorImpl.java | 67 +--- .../client/AsyncConnectionConfiguration.java| 37 +++-- .../hbase/client/AsyncConnectionImpl.java | 11 +- .../hbase/client/TestAsyncBufferMutator.java| 161 ++- 7 files changed, 277 insertions(+), 53 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/b09b87d1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutator.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutator.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutator.java index 6fe4b9a..7b21eb5 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutator.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutator.java @@ -18,13 +18,16 @@ package org.apache.hadoop.hbase.client; import java.io.Closeable; +import java.util.Collections; import java.util.List; import java.util.concurrent.CompletableFuture; - +import java.util.concurrent.TimeUnit; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.TableName; import org.apache.yetus.audience.InterfaceAudience; +import org.apache.hbase.thirdparty.com.google.common.collect.Iterables; + /** * Used to communicate with a single HBase table in batches. Obtain an instance from a * {@link AsyncConnection} and call {@link #close()} afterwards. @@ -52,7 +55,9 @@ public interface AsyncBufferedMutator extends Closeable { * part of a batch. Currently only supports {@link Put} and {@link Delete} mutations. * @param mutation The data to send. */ - CompletableFuture mutate(Mutation mutation); + default CompletableFuture mutate(Mutation mutation) { +return Iterables.getOnlyElement(mutate(Collections.singletonList(mutation))); + } /** * Send some {@link Mutation}s to the table. The mutations will be buffered and sent over the wire @@ -81,4 +86,11 @@ public interface AsyncBufferedMutator extends Closeable { * @return The size of the write buffer in bytes. */ long getWriteBufferSize(); + + /** + * Returns the periodical flush interval, 0 means disabled. + */ + default long getPeriodicalFlushTimeout(TimeUnit unit) { +throw new UnsupportedOperationException("Not implemented"); + } } http://git-wip-us.apache.org/repos/asf/hbase/blob/b09b87d1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutatorBuilder.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutatorBuilder.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutatorBuilder.java index 45959bb..c617c8e 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutatorBuilder.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncBufferedMutatorBuilder.java @@ -46,6 +46,25 @@ public interface AsyncBufferedMutatorBuilder { AsyncBufferedMutatorBuilder setRetryPause(long pause, TimeUnit unit); /** + * Set the periodical flush interval. If the data in the buffer has not been flush for a long + * time, i.e, reach this timeout limit, we will flush it automatically. + * + * Notice that, set the timeout to 0 or a negative value means disable periodical flush, not + * 'flush immediately'. If you want to flush immediately then you should not use this class, as it + * is designed to be 'buffered'. + */ + default AsyncBufferedMutatorBuilder setWriteBufferPeriodicFlush(long timeout, TimeUnit unit) { +throw new UnsupportedOperationException("Not implemented"); + } + + /** + * Disable the periodical flush, i.e, set the timeout to 0. + */ + default AsyncBufferedMutatorBuilder disableWriteBufferPeriodicFlush() { +return setWriteBufferPeriodicFlush(0,
[33/47] hbase git commit: HBASE-21642 CopyTable by reading snapshot and bulkloading will save a lot of time
HBASE-21642 CopyTable by reading snapshot and bulkloading will save a lot of time Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/c2d5991b Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/c2d5991b Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/c2d5991b Branch: refs/heads/HBASE-21512 Commit: c2d5991b82e3b807cb11f5735ef5068b73720725 Parents: c552088 Author: huzheng Authored: Wed Dec 26 16:17:55 2018 +0800 Committer: huzheng Committed: Thu Dec 27 18:22:54 2018 +0800 -- .../hadoop/hbase/mapreduce/CopyTable.java | 109 -- .../hadoop/hbase/mapreduce/TestCopyTable.java | 110 --- .../hbase/client/ClientSideRegionScanner.java | 14 ++- .../hadoop/hbase/regionserver/HRegion.java | 2 +- 4 files changed, 187 insertions(+), 48 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/c2d5991b/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java -- diff --git a/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java b/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java index 4e57f54..b59c9e6 100644 --- a/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java +++ b/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java @@ -21,7 +21,7 @@ package org.apache.hadoop.hbase.mapreduce; import java.io.IOException; import java.util.HashMap; import java.util.Map; -import java.util.Random; +import java.util.UUID; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.FileSystem; @@ -29,6 +29,8 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HConstants; import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.mapreduce.Import.CellImporter; +import org.apache.hadoop.hbase.mapreduce.Import.Importer; import org.apache.hadoop.hbase.util.FSUtils; import org.apache.yetus.audience.InterfaceAudience; import org.slf4j.Logger; @@ -70,8 +72,34 @@ public class CopyTable extends Configured implements Tool { boolean bulkload = false; Path bulkloadDir = null; + boolean readingSnapshot = false; + String snapshot = null; + private final static String JOB_NAME_CONF_KEY = "mapreduce.job.name"; + private Path generateUniqTempDir(boolean withDirCreated) throws IOException { +FileSystem fs = FSUtils.getCurrentFileSystem(getConf()); +Path dir = new Path(fs.getWorkingDirectory(), NAME); +if (!fs.exists(dir)) { + fs.mkdirs(dir); +} +Path newDir = new Path(dir, UUID.randomUUID().toString()); +if (withDirCreated) { + fs.mkdirs(newDir); +} +return newDir; + } + + private void initCopyTableMapperReducerJob(Job job, Scan scan) throws IOException { +Class mapper = bulkload ? CellImporter.class : Importer.class; +if (readingSnapshot) { + TableMapReduceUtil.initTableSnapshotMapperJob(snapshot, scan, mapper, null, null, job, true, +generateUniqTempDir(true)); +} else { + TableMapReduceUtil.initTableMapperJob(tableName, scan, mapper, null, null, job); +} + } + /** * Sets up the actual job. * @@ -79,13 +107,13 @@ public class CopyTable extends Configured implements Tool { * @return The newly created job. * @throws IOException When setting up the job fails. */ - public Job createSubmittableJob(String[] args) - throws IOException { + public Job createSubmittableJob(String[] args) throws IOException { if (!doCommandLine(args)) { return null; } -Job job = Job.getInstance(getConf(), getConf().get(JOB_NAME_CONF_KEY, NAME + "_" + tableName)); +String jobName = NAME + "_" + (tableName == null ? snapshot : tableName); +Job job = Job.getInstance(getConf(), getConf().get(JOB_NAME_CONF_KEY, jobName)); job.setJarByClass(CopyTable.class); Scan scan = new Scan(); @@ -107,15 +135,15 @@ public class CopyTable extends Configured implements Tool { job.getConfiguration().set(TableInputFormat.SHUFFLE_MAPS, "true"); } if (versions >= 0) { - scan.setMaxVersions(versions); + scan.readVersions(versions); } if (startRow != null) { - scan.setStartRow(Bytes.toBytesBinary(startRow)); + scan.withStartRow(Bytes.toBytesBinary(startRow)); } if (stopRow != null) { - scan.setStopRow(Bytes.toBytesBinary(stopRow)); + scan.withStopRow(Bytes.toBytesBinary(stopRow)); } if(families != null) { @@ -140,24 +168,13 @@ public class CopyTable extends Configured implements Tool { job.setNumReduceTasks(0); if (bulkload) { - TableMapReduceUtil.initTableMap
[26/47] hbase git commit: HBASE-21621 Reversed scan does not return expected number of rows
HBASE-21621 Reversed scan does not return expected number of rows The unit test is contributed by Nihal Jain Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/7c0a3cc2 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/7c0a3cc2 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/7c0a3cc2 Branch: refs/heads/HBASE-21512 Commit: 7c0a3cc265f1351363dc88f2f70855b3273dd8c6 Parents: b2bf22e Author: Guanghao Zhang Authored: Thu Dec 20 12:34:34 2018 +0800 Committer: Guanghao Zhang Committed: Sun Dec 23 16:19:05 2018 +0800 -- .../regionserver/ReversedStoreScanner.java | 5 +-- .../hadoop/hbase/regionserver/StoreScanner.java | 10 - .../hbase/client/TestFromClientSide3.java | 2 +- .../client/TestScannersFromClientSide.java | 43 .../hbase/regionserver/TestStoreScanner.java| 8 +++- 5 files changed, 61 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/7c0a3cc2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java index 90e1129..491e6ef 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java @@ -59,10 +59,9 @@ public class ReversedStoreScanner extends StoreScanner implements KeyValueScanne } @Override - protected void resetKVHeap(List scanners, + protected KeyValueHeap newKVHeap(List scanners, CellComparator comparator) throws IOException { -// Combine all seeked scanners with a heap -heap = new ReversedKeyValueHeap(scanners, comparator); +return new ReversedKeyValueHeap(scanners, comparator); } @Override http://git-wip-us.apache.org/repos/asf/hbase/blob/7c0a3cc2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java index b318950..d06 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java @@ -403,10 +403,16 @@ public class StoreScanner extends NonReversedNonLazyKeyValueScanner } } + @VisibleForTesting protected void resetKVHeap(List scanners, CellComparator comparator) throws IOException { // Combine all seeked scanners with a heap -heap = new KeyValueHeap(scanners, comparator); +heap = newKVHeap(scanners, comparator); + } + + protected KeyValueHeap newKVHeap(List scanners, + CellComparator comparator) throws IOException { +return new KeyValueHeap(scanners, comparator); } /** @@ -1037,7 +1043,7 @@ public class StoreScanner extends NonReversedNonLazyKeyValueScanner newCurrentScanners = new ArrayList<>(fileScanners.size() + memstoreScanners.size()); newCurrentScanners.addAll(fileScanners); newCurrentScanners.addAll(memstoreScanners); - newHeap = new KeyValueHeap(newCurrentScanners, comparator); + newHeap = newKVHeap(newCurrentScanners, comparator); } catch (Exception e) { LOG.warn("failed to switch to stream read", e); if (fileScanners != null) { http://git-wip-us.apache.org/repos/asf/hbase/blob/7c0a3cc2/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java -- diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java index cbfa1bf..1315d4a 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java @@ -1059,7 +1059,7 @@ public class TestFromClientSide3 { } } - private static byte[] generateHugeValue(int size) { + static byte[] generateHugeValue(int size) { Random rand = ThreadLocalRandom.current(); byte[] value = new byte[size]; for (int i = 0; i < value.length; i++) { http://git-wip-us.apache.org/repos/asf/hbase/blob/7c0a3cc2/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java -
[31/47] hbase git commit: HBASE-21640 Remove the TODO when increment zero
HBASE-21640 Remove the TODO when increment zero Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/4281cb3b Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/4281cb3b Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/4281cb3b Branch: refs/heads/HBASE-21512 Commit: 4281cb3b9574333fab0e7c028c9c0d7e5b320c73 Parents: 44dec60 Author: Guanghao Zhang Authored: Tue Dec 25 17:42:38 2018 +0800 Committer: Guanghao Zhang Committed: Wed Dec 26 21:47:44 2018 +0800 -- .../hadoop/hbase/regionserver/HRegion.java | 21 .../hbase/regionserver/wal/TestDurability.java | 9 - 2 files changed, 8 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/4281cb3b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java index 21458c4..dc0fa22 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java @@ -7963,8 +7963,7 @@ public class HRegion implements HeapSize, PropagatingConfigurationObserver, Regi /** * Reckon the Cells to apply to WAL, memstore, and to return to the Client; these Sets are not - * always the same dependent on whether to write WAL or if the amount to increment is zero (in - * this case we write back nothing, just return latest Cell value to the client). + * always the same dependent on whether to write WAL. * * @param results Fill in here what goes back to the Client if it is non-null (if null, client * doesn't want results). @@ -8006,9 +8005,8 @@ public class HRegion implements HeapSize, PropagatingConfigurationObserver, Regi * @param op Whether Increment or Append * @param mutation The encompassing Mutation object * @param deltas Changes to apply to this Store; either increment amount or data to append - * @param results In here we accumulate all the Cells we are to return to the client; this List - * can be larger than what we return in case where delta is zero; i.e. don't write - * out new values, just return current value. If null, client doesn't want results returned. + * @param results In here we accumulate all the Cells we are to return to the client. If null, + *client doesn't want results returned. * @return Resulting Cells after deltas have been applied to current * values. Side effect is our filling out of the results List. */ @@ -8036,33 +8034,25 @@ public class HRegion implements HeapSize, PropagatingConfigurationObserver, Regi for (int i = 0; i < deltas.size(); i++) { Cell delta = deltas.get(i); Cell currentValue = null; - boolean firstWrite = false; if (currentValuesIndex < currentValues.size() && CellUtil.matchingQualifier(currentValues.get(currentValuesIndex), delta)) { currentValue = currentValues.get(currentValuesIndex); if (i < (deltas.size() - 1) && !CellUtil.matchingQualifier(delta, deltas.get(i + 1))) { currentValuesIndex++; } - } else { -firstWrite = true; } // Switch on whether this an increment or an append building the new Cell to apply. Cell newCell = null; MutationType mutationType = null; - boolean apply = true; switch (op) { case INCREMENT: mutationType = MutationType.INCREMENT; - // If delta amount to apply is 0, don't write WAL or MemStore. long deltaAmount = getLongValue(delta); - // TODO: Does zero value mean reset Cell? For example, the ttl. - apply = deltaAmount != 0; final long newValue = currentValue == null ? deltaAmount : getLongValue(currentValue) + deltaAmount; newCell = reckonDelta(delta, currentValue, columnFamily, now, mutation, (oldCell) -> Bytes.toBytes(newValue)); break; case APPEND: mutationType = MutationType.APPEND; - // Always apply Append. TODO: Does empty delta value mean reset Cell? It seems to. newCell = reckonDelta(delta, currentValue, columnFamily, now, mutation, (oldCell) -> ByteBuffer.wrap(new byte[delta.getValueLength() + oldCell.getValueLength()]) .put(oldCell.getValueArray(), oldCell.getValueOffset(), oldCell.getValueLength()) @@ -8078,10 +8068,7 @@ public class HRegion implements HeapSize, PropagatingConfigurationObserver, Regi newCell = coprocessorHost.postMutationBeforeWAL(muta
[25/47] hbase git commit: HBASE-21631: list_quotas should print human readable values for LIMIT
HBASE-21631: list_quotas should print human readable values for LIMIT Signed-off-by: Guanghao Zhang Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/b2bf22e2 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/b2bf22e2 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/b2bf22e2 Branch: refs/heads/HBASE-21512 Commit: b2bf22e209d2e87121986b35c5749b2b8ae45fa2 Parents: e160b5a Author: Sakthi Authored: Fri Dec 21 16:23:08 2018 -0800 Committer: Guanghao Zhang Committed: Sat Dec 22 22:00:58 2018 +0800 -- .../java/org/apache/hadoop/hbase/quotas/SpaceLimitSettings.java| 2 +- .../org/apache/hadoop/hbase/quotas/GlobalQuotaSettingsImpl.java| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/b2bf22e2/hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitSettings.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitSettings.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitSettings.java index 02bd6e4..8b31e94 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitSettings.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitSettings.java @@ -205,7 +205,7 @@ class SpaceLimitSettings extends QuotaSettings { if (proto.getQuota().getRemove()) { sb.append(", REMOVE => ").append(proto.getQuota().getRemove()); } else { - sb.append(", LIMIT => ").append(proto.getQuota().getSoftLimit()); + sb.append(", LIMIT => ").append(sizeToString(proto.getQuota().getSoftLimit())); sb.append(", VIOLATION_POLICY => ").append(proto.getQuota().getViolationPolicy()); } return sb.toString(); http://git-wip-us.apache.org/repos/asf/hbase/blob/b2bf22e2/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/GlobalQuotaSettingsImpl.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/GlobalQuotaSettingsImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/GlobalQuotaSettingsImpl.java index 0c6cb81..e47e4ff 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/GlobalQuotaSettingsImpl.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/GlobalQuotaSettingsImpl.java @@ -276,7 +276,7 @@ public class GlobalQuotaSettingsImpl extends GlobalQuotaSettings { if (spaceProto.getRemove()) { builder.append(", REMOVE => ").append(spaceProto.getRemove()); } else { -builder.append(", LIMIT => ").append(spaceProto.getSoftLimit()); +builder.append(", LIMIT => ").append(sizeToString(spaceProto.getSoftLimit())); builder.append(", VIOLATION_POLICY => ").append(spaceProto.getViolationPolicy()); } builder.append(" } ");
[21/47] hbase git commit: HBASE-21610, numOpenConnections metric is set to -1 when zero server channel exist
HBASE-21610, numOpenConnections metric is set to -1 when zero server channel exist Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/78756733 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/78756733 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/78756733 Branch: refs/heads/HBASE-21512 Commit: 787567336afb9c5c1e00aaa0326566a5522a5e31 Parents: 8991877 Author: Pankaj Authored: Tue Dec 18 01:31:55 2018 +0530 Committer: stack Committed: Thu Dec 20 16:36:42 2018 -0800 -- .../src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/78756733/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java index 8ea2057..742a728 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java @@ -169,8 +169,9 @@ public class NettyRpcServer extends RpcServer { @Override public int getNumOpenConnections() { +int channelsCount = allChannels.size(); // allChannels also contains the server channel, so exclude that from the count. -return allChannels.size() - 1; +return channelsCount > 0 ? channelsCount - 1 : channelsCount; } @Override
[40/47] hbase git commit: HBASE-21650 Add DDL operation and some other miscellaneous to thrift2
http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TBloomFilterType.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TBloomFilterType.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TBloomFilterType.java new file mode 100644 index 000..601d6b4 --- /dev/null +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TBloomFilterType.java @@ -0,0 +1,69 @@ +/** + * Autogenerated by Thrift Compiler (0.9.3) + * + * DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING + * @generated + */ +package org.apache.hadoop.hbase.thrift2.generated; + + +import java.util.Map; +import java.util.HashMap; +import org.apache.thrift.TEnum; + +public enum TBloomFilterType implements org.apache.thrift.TEnum { + /** + * Bloomfilters disabled + */ + NONE(0), + /** + * Bloom enabled with Table row as Key + */ + ROW(1), + /** + * Bloom enabled with Table row & column (family+qualifier) as Key + */ + ROWCOL(2), + /** + * Bloom enabled with Table row prefix as Key, specify the length of the prefix + */ + ROWPREFIX_FIXED_LENGTH(3), + /** + * Bloom enabled with Table row prefix as Key, specify the delimiter of the prefix + */ + ROWPREFIX_DELIMITED(4); + + private final int value; + + private TBloomFilterType(int value) { +this.value = value; + } + + /** + * Get the integer value of this enum value, as defined in the Thrift IDL. + */ + public int getValue() { +return value; + } + + /** + * Find a the enum type by its integer value, as defined in the Thrift IDL. + * @return null if the value is not found. + */ + public static TBloomFilterType findByValue(int value) { +switch (value) { + case 0: +return NONE; + case 1: +return ROW; + case 2: +return ROWCOL; + case 3: +return ROWPREFIX_FIXED_LENGTH; + case 4: +return ROWPREFIX_DELIMITED; + default: +return null; +} + } +} http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCellVisibility.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCellVisibility.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCellVisibility.java index 7da4dda..464ac12 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCellVisibility.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCellVisibility.java @@ -34,7 +34,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"}) -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2016-05-25") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") public class TCellVisibility implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TCellVisibility"); http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumn.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumn.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumn.java index d0d336c..24a7846 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumn.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumn.java @@ -39,7 +39,7 @@ import org.slf4j.LoggerFactory; * in a HBase table by column family and optionally * a column qualifier and timestamp */ -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2016-05-25") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") public class TColumn implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TColumn");
[05/47] hbase git commit: HBASE-21582 If call HBaseAdmin#snapshotAsync but forget call isSnapshotFinished, then SnapshotHFileCleaner will skip to run every time
HBASE-21582 If call HBaseAdmin#snapshotAsync but forget call isSnapshotFinished, then SnapshotHFileCleaner will skip to run every time Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/f32d2618 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/f32d2618 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/f32d2618 Branch: refs/heads/HBASE-21512 Commit: f32d2618430f70e1b0db92785294b2c7892cc02b Parents: 4640ff5 Author: huzheng Authored: Tue Dec 11 20:27:56 2018 +0800 Committer: huzheng Committed: Thu Dec 13 10:35:20 2018 +0800 -- .../hbase/master/snapshot/SnapshotManager.java | 48 ++-- .../master/cleaner/TestSnapshotFromMaster.java | 27 ++- .../master/snapshot/TestSnapshotManager.java| 36 +-- 3 files changed, 92 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/f32d2618/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java index 2b963b2..05db4ab 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java @@ -28,7 +28,11 @@ import java.util.List; import java.util.Map; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledFuture; import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; import java.util.concurrent.locks.ReadWriteLock; import java.util.concurrent.locks.ReentrantReadWriteLock; @@ -91,6 +95,8 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos.NameStringP import org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos.ProcedureDescription; import org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription; import org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription.Type; +import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting; +import org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder; /** * This class manages the procedure of taking and restoring snapshots. There is only one @@ -120,7 +126,9 @@ public class SnapshotManager extends MasterProcedureManager implements Stoppable * At this point, if the user asks for the snapshot/restore status, the result will be * snapshot done if exists or failed if it doesn't exists. */ - private static final int SNAPSHOT_SENTINELS_CLEANUP_TIMEOUT = 60 * 1000; + public static final String HBASE_SNAPSHOT_SENTINELS_CLEANUP_TIMEOUT_MILLIS = + "hbase.snapshot.sentinels.cleanup.timeoutMillis"; + public static final long SNAPSHOT_SENTINELS_CLEANUP_TIMEOUT_MILLS_DEFAULT = 60 * 1000L; /** Enable or disable snapshot support */ public static final String HBASE_SNAPSHOT_ENABLED = "hbase.snapshot.enabled"; @@ -151,7 +159,11 @@ public class SnapshotManager extends MasterProcedureManager implements Stoppable // The map is always accessed and modified under the object lock using synchronized. // snapshotTable() will insert an Handler in the table. // isSnapshotDone() will remove the handler requested if the operation is finished. - private Map snapshotHandlers = new ConcurrentHashMap<>(); + private final Map snapshotHandlers = new ConcurrentHashMap<>(); + private final ScheduledExecutorService scheduleThreadPool = + Executors.newScheduledThreadPool(1, new ThreadFactoryBuilder() + .setNameFormat("SnapshotHandlerChoreCleaner").setDaemon(true).build()); + private ScheduledFuture snapshotHandlerChoreCleanerTask; // Restore map, with table name as key, procedure ID as value. // The map is always accessed and modified under the object lock using synchronized. @@ -181,17 +193,21 @@ public class SnapshotManager extends MasterProcedureManager implements Stoppable * @param coordinator procedure coordinator instance. exposed for testing. * @param pool HBase ExecutorServcie instance, exposed for testing. */ - public SnapshotManager(final MasterServices master, final MetricsMaster metricsMaster, - ProcedureCoordinator coordinator, ExecutorService pool) + @VisibleForTesting + SnapshotManager(final MasterServices master, ProcedureCoordinator coordinator, + ExecutorService pool, int sentinelCleanInterval) throws IOException, Uns
[14/47] hbase git commit: HBASE-21514 Refactor CacheConfig
http://git-wip-us.apache.org/repos/asf/hbase/blob/1971d02e/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java -- diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java index 444102d..2065c0c 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java @@ -28,12 +28,14 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.Cell; import org.apache.hadoop.hbase.HBaseClassTestRule; import org.apache.hadoop.hbase.HBaseTestingUtility; -import org.apache.hadoop.hbase.HColumnDescriptor; -import org.apache.hadoop.hbase.HRegionInfo; -import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder; import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.RegionInfo; +import org.apache.hadoop.hbase.client.RegionInfoBuilder; import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.TableDescriptor; +import org.apache.hadoop.hbase.client.TableDescriptorBuilder; import org.apache.hadoop.hbase.regionserver.HRegion; import org.apache.hadoop.hbase.regionserver.HStore; import org.apache.hadoop.hbase.regionserver.InternalScanner; @@ -104,17 +106,15 @@ public class TestScannerSelectionUsingTTL { @Test public void testScannerSelection() throws IOException { Configuration conf = TEST_UTIL.getConfiguration(); -CacheConfig.instantiateBlockCache(conf); conf.setBoolean("hbase.store.delete.expired.storefile", false); -HColumnDescriptor hcd = - new HColumnDescriptor(FAMILY_BYTES) - .setMaxVersions(Integer.MAX_VALUE) - .setTimeToLive(TTL_SECONDS); -HTableDescriptor htd = new HTableDescriptor(TABLE); -htd.addFamily(hcd); -HRegionInfo info = new HRegionInfo(TABLE); -HRegion region = HBaseTestingUtility.createRegionAndWAL(info, - TEST_UTIL.getDataTestDir(info.getEncodedName()), conf, htd); +LruBlockCache cache = (LruBlockCache) BlockCacheFactory.createBlockCache(conf); + +TableDescriptor td = TableDescriptorBuilder.newBuilder(TABLE).setColumnFamily( + ColumnFamilyDescriptorBuilder.newBuilder(FAMILY_BYTES).setMaxVersions(Integer.MAX_VALUE) +.setTimeToLive(TTL_SECONDS).build()).build(); +RegionInfo info = RegionInfoBuilder.newBuilder(TABLE).build(); +HRegion region = HBaseTestingUtility +.createRegionAndWAL(info, TEST_UTIL.getDataTestDir(info.getEncodedName()), conf, td, cache); long ts = EnvironmentEdgeManager.currentTime(); long version = 0; //make sure each new set of Put's have a new ts @@ -136,10 +136,7 @@ public class TestScannerSelectionUsingTTL { version++; } -Scan scan = new Scan(); -scan.setMaxVersions(Integer.MAX_VALUE); -CacheConfig cacheConf = new CacheConfig(conf); -LruBlockCache cache = (LruBlockCache) cacheConf.getBlockCache(); +Scan scan = new Scan().readVersions(Integer.MAX_VALUE); cache.clearCache(); InternalScanner scanner = region.getScanner(scan); List results = new ArrayList<>(); http://git-wip-us.apache.org/repos/asf/hbase/blob/1971d02e/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java -- diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java index 844b705..a930d7f 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java @@ -26,6 +26,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Map.Entry; +import java.util.Optional; import java.util.Random; import java.util.TreeMap; import java.util.concurrent.ConcurrentSkipListMap; @@ -48,8 +49,10 @@ import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.client.locking.EntityLock; import org.apache.hadoop.hbase.executor.ExecutorService; +import org.apache.hadoop.hbase.io.hfile.BlockCache; import org.apache.hadoop.hbase.ipc.HBaseRpcController; import org.apache.hadoop.hbase.ipc.RpcServerInterface; +import org.apache.hadoop.hbase.mob.MobFileCache; import org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager; import org.apache.hadoop.hbase.quotas.RegionServerSpaceQuotaManager; import org.apache.hadoop.hbase.quotas.RegionSizeStore; @@ -708,4 +711,14 @@ class MockRegi
[02/47] hbase git commit: HBASE-21453 Convert ReadOnlyZKClient to DEBUG instead of INFO
HBASE-21453 Convert ReadOnlyZKClient to DEBUG instead of INFO Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/f88224ee Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/f88224ee Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/f88224ee Branch: refs/heads/HBASE-21512 Commit: f88224ee34ba2c23f794ec1219ffd93783b20e51 Parents: b09b87d Author: Sakthi Authored: Thu Nov 29 18:52:50 2018 -0800 Committer: Peter Somogyi Committed: Tue Dec 11 08:18:02 2018 +0100 -- .../java/org/apache/hadoop/hbase/zookeeper/ReadOnlyZKClient.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/f88224ee/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ReadOnlyZKClient.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ReadOnlyZKClient.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ReadOnlyZKClient.java index fc2d5f0..09f8984 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ReadOnlyZKClient.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ReadOnlyZKClient.java @@ -136,7 +136,7 @@ public final class ReadOnlyZKClient implements Closeable { this.retryIntervalMs = conf.getInt(RECOVERY_RETRY_INTERVAL_MILLIS, DEFAULT_RECOVERY_RETRY_INTERVAL_MILLIS); this.keepAliveTimeMs = conf.getInt(KEEPALIVE_MILLIS, DEFAULT_KEEPALIVE_MILLIS); -LOG.info( +LOG.debug( "Connect {} to {} with session timeout={}ms, retries {}, " + "retry interval {}ms, keepAlive={}ms", getId(), connectString, sessionTimeoutMs, maxRetries, retryIntervalMs, keepAliveTimeMs); @@ -347,7 +347,7 @@ public final class ReadOnlyZKClient implements Closeable { @Override public void close() { if (closed.compareAndSet(false, true)) { - LOG.info("Close zookeeper connection {} to {}", getId(), connectString); + LOG.debug("Close zookeeper connection {} to {}", getId(), connectString); tasks.add(CLOSE); } }
[32/47] hbase git commit: HBASE-14939 Document bulk loaded hfile replication
HBASE-14939 Document bulk loaded hfile replication Signed-off-by: Ashish Singhi Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/c5520888 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/c5520888 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/c5520888 Branch: refs/heads/HBASE-21512 Commit: c5520888779235a334583f7c369dcee49518e165 Parents: 4281cb3 Author: Wei-Chiu Chuang Authored: Wed Dec 26 20:14:18 2018 +0530 Committer: Ashish Singhi Committed: Wed Dec 26 20:14:18 2018 +0530 -- src/main/asciidoc/_chapters/architecture.adoc | 32 ++ 1 file changed, 26 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/c5520888/src/main/asciidoc/_chapters/architecture.adoc -- diff --git a/src/main/asciidoc/_chapters/architecture.adoc b/src/main/asciidoc/_chapters/architecture.adoc index 17e9e13..27db26a 100644 --- a/src/main/asciidoc/_chapters/architecture.adoc +++ b/src/main/asciidoc/_chapters/architecture.adoc @@ -2543,12 +2543,6 @@ The most straightforward method is to either use the `TableOutputFormat` class f The bulk load feature uses a MapReduce job to output table data in HBase's internal data format, and then directly loads the generated StoreFiles into a running cluster. Using bulk load will use less CPU and network resources than simply using the HBase API. -[[arch.bulk.load.limitations]] -=== Bulk Load Limitations - -As bulk loading bypasses the write path, the WAL doesn't get written to as part of the process. -Replication works by reading the WAL files so it won't see the bulk loaded data â and the same goes for the edits that use `Put.setDurability(SKIP_WAL)`. One way to handle that is to ship the raw files or the HFiles to the other cluster and do the other processing there. - [[arch.bulk.load.arch]] === Bulk Load Architecture @@ -2601,6 +2595,32 @@ To get started doing so, dig into `ImportTsv.java` and check the JavaDoc for HFi The import step of the bulk load can also be done programmatically. See the `LoadIncrementalHFiles` class for more information. +[[arch.bulk.load.replication]] +=== Bulk Loading Replication +HBASE-13153 adds replication support for bulk loaded HFiles, available since HBase 1.3/2.0. This feature is enabled by setting `hbase.replication.bulkload.enabled` to `true` (default is `false`). +You also need to copy the source cluster configuration files to the destination cluster. + +Additional configurations are required too: + +. `hbase.replication.source.fs.conf.provider` ++ +This defines the class which loads the source cluster file system client configuration in the destination cluster. This should be configured for all the RS in the destination cluster. Default is `org.apache.hadoop.hbase.replication.regionserver.DefaultSourceFSConfigurationProvider`. ++ +. `hbase.replication.conf.dir` ++ +This represents the base directory where the file system client configurations of the source cluster are copied to the destination cluster. This should be configured for all the RS in the destination cluster. Default is `$HBASE_CONF_DIR`. ++ +. `hbase.replication.cluster.id` ++ +This configuration is required in the cluster where replication for bulk loaded data is enabled. A source cluster is uniquely identified by the destination cluster using this id. This should be configured for all the RS in the source cluster configuration file for all the RS. ++ + + + +For example: If source cluster FS client configurations are copied to the destination cluster under directory `/home/user/dc1/`, then `hbase.replication.cluster.id` should be configured as `dc1` and `hbase.replication.conf.dir` as `/home/user`. + +NOTE: `DefaultSourceFSConfigurationProvider` supports only `xml` type files. It loads source cluster FS client configuration only once, so if source cluster FS client configuration files are updated, every peer(s) cluster RS must be restarted to reload the configuration. + [[arch.hdfs]] == HDFS
[15/47] hbase git commit: HBASE-21514 Refactor CacheConfig
http://git-wip-us.apache.org/repos/asf/hbase/blob/1971d02e/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java index 6242d36..13f277b 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java @@ -36,6 +36,7 @@ import java.util.List; import java.util.Map; import java.util.Map.Entry; import java.util.Objects; +import java.util.Optional; import java.util.Set; import java.util.SortedMap; import java.util.Timer; @@ -98,7 +99,7 @@ import org.apache.hadoop.hbase.executor.ExecutorType; import org.apache.hadoop.hbase.fs.HFileSystem; import org.apache.hadoop.hbase.http.InfoServer; import org.apache.hadoop.hbase.io.hfile.BlockCache; -import org.apache.hadoop.hbase.io.hfile.CacheConfig; +import org.apache.hadoop.hbase.io.hfile.BlockCacheFactory; import org.apache.hadoop.hbase.io.hfile.HFile; import org.apache.hadoop.hbase.io.util.MemorySizeUtil; import org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils; @@ -114,7 +115,7 @@ import org.apache.hadoop.hbase.log.HBaseMarkers; import org.apache.hadoop.hbase.master.HMaster; import org.apache.hadoop.hbase.master.LoadBalancer; import org.apache.hadoop.hbase.master.RegionState.State; -import org.apache.hadoop.hbase.mob.MobCacheConfig; +import org.apache.hadoop.hbase.mob.MobFileCache; import org.apache.hadoop.hbase.procedure.RegionServerProcedureManagerHost; import org.apache.hadoop.hbase.procedure2.RSProcedureCallable; import org.apache.hadoop.hbase.quotas.FileSystemUtilizationChore; @@ -410,10 +411,10 @@ public class HRegionServer extends HasThread implements private final RegionServerAccounting regionServerAccounting; - // Cache configuration and block cache reference - protected CacheConfig cacheConfig; - // Cache configuration for mob - final MobCacheConfig mobCacheConfig; + // Block cache + private BlockCache blockCache; + // The cache for mob files + private MobFileCache mobFileCache; /** The health check chore. */ private HealthCheckChore healthCheckChore; @@ -591,12 +592,12 @@ public class HRegionServer extends HasThread implements boolean isMasterNotCarryTable = this instanceof HMaster && !LoadBalancer.isTablesOnMaster(conf); - // no need to instantiate global block cache when master not carry table + + // no need to instantiate block cache and mob file cache when master not carry table if (!isMasterNotCarryTable) { -CacheConfig.instantiateBlockCache(conf); +blockCache = BlockCacheFactory.createBlockCache(conf); +mobFileCache = new MobFileCache(conf); } - cacheConfig = new CacheConfig(conf); - mobCacheConfig = new MobCacheConfig(conf); uncaughtExceptionHandler = new UncaughtExceptionHandler() { @Override @@ -1062,10 +1063,12 @@ public class HRegionServer extends HasThread implements } } // Send cache a shutdown. -if (cacheConfig != null && cacheConfig.isBlockCacheEnabled()) { - cacheConfig.getBlockCache().shutdown(); +if (blockCache != null) { + blockCache.shutdown(); +} +if (mobFileCache != null) { + mobFileCache.shutdown(); } -mobCacheConfig.getMobFileCache().shutdown(); if (movedRegionsCleaner != null) { movedRegionsCleaner.stop("Region Server stopping"); @@ -1607,9 +1610,9 @@ public class HRegionServer extends HasThread implements } private void startHeapMemoryManager() { -this.hMemManager = HeapMemoryManager.create(this.conf, this.cacheFlusher, this, -this.regionServerAccounting); -if (this.hMemManager != null) { +if (this.blockCache != null) { + this.hMemManager = + new HeapMemoryManager(this.blockCache, this.cacheFlusher, this, regionServerAccounting); this.hMemManager.start(getChoreService()); } } @@ -3614,10 +3617,23 @@ public class HRegionServer extends HasThread implements } /** - * @return The cache config instance used by the regionserver. + * May be null if this is a master which not carry table. + * + * @return The block cache instance used by the regionserver. + */ + @Override + public Optional getBlockCache() { +return Optional.ofNullable(this.blockCache); + } + + /** + * May be null if this is a master which not carry table. + * + * @return The cache for mob files used by the regionserver. */ - public CacheConfig getCacheConfig() { -return this.cacheConfig; + @Override + public Optional getMobFileCache() { +return Optional.ofNullable(this.mobFileCache); } /** @@ -3646,7 +3662,6 @@ public class HRegionServer extends HasThread imple
[42/47] hbase git commit: HBASE-21650 Add DDL operation and some other miscellaneous to thrift2 (ADDENDUM add some comments)
http://git-wip-us.apache.org/repos/asf/hbase/blob/b620334c/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableDescriptor.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableDescriptor.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableDescriptor.java index 89a8a5e..8e53bdf 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableDescriptor.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableDescriptor.java @@ -34,7 +34,11 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"}) -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") +/** + * Thrift wrapper around + * org.apache.hadoop.hbase.client.TableDescriptor + */ +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-28") public class TTableDescriptor implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TTableDescriptor"); http://git-wip-us.apache.org/repos/asf/hbase/blob/b620334c/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableName.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableName.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableName.java index f2c0743..cec268a 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableName.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableName.java @@ -34,7 +34,11 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"}) -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") +/** + * Thrift wrapper around + * org.apache.hadoop.hbase.TableName + */ +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-28") public class TTableName implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TTableName"); @@ -47,12 +51,24 @@ public class TTableName implements org.apache.thrift.TBase byName = new HashMap(); @@ -157,6 +173,9 @@ public class TTableName implements org.apache.thrift.TBasehttp://git-wip-us.apache.org/repos/asf/hbase/blob/b620334c/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTimeRange.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTimeRange.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTimeRange.java index 1e1898c..8ab746c 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTimeRange.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTimeRange.java @@ -34,7 +34,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"}) -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-28") public class TTimeRange implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TTimeRange"); http://git-wip-us.apache.org/repos/asf/hbase/blob/b620334c/hbase-thrift/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift -- diff --git a/hbase-thrift/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift b/hbase-thrift/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift index 6383329..c1b94ef 100644 --- a/hbase-thrift/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift +++ b/hbase-thrift/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift @@ -315,6 +315,10 @@ enum TCompareOp { NO_OP = 6 } +/** + * Thrift wrapper around + * org.apache.hadoop.hbase.regionserver.BloomType + */ enum TBloomFilterType { /** * Bloomfilters disabled @@ -338,6 +342,10 @@ enum TBloomFilterType { ROWPREFIX_DELIMITED = 4 } +/** + * Thrift wrapper around + * org.apache.hadoop.hbase.io.compress.Algorithm + */ enum TCompressionAlgorithm { LZO = 0, GZ = 1, @@ -348,6 +356,10 @@ enum TCompressionAlgorithm { ZSTD = 6 } +/** + * Thrift wrapper around + * org.apa
[41/47] hbase git commit: HBASE-21650 Add DDL operation and some other miscellaneous to thrift2
HBASE-21650 Add DDL operation and some other miscellaneous to thrift2 Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/7820ba1d Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/7820ba1d Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/7820ba1d Branch: refs/heads/HBASE-21512 Commit: 7820ba1dbdba58b1002cdfde08eb21aa7a0bb6da Parents: f5ea00f Author: Allan Yang Authored: Thu Dec 27 22:25:33 2018 +0800 Committer: Allan Yang Committed: Thu Dec 27 22:25:33 2018 +0800 -- .../hbase/thrift/generated/AlreadyExists.java | 2 +- .../hbase/thrift/generated/BatchMutation.java | 2 +- .../thrift/generated/ColumnDescriptor.java | 2 +- .../hadoop/hbase/thrift/generated/Hbase.java| 2 +- .../hadoop/hbase/thrift/generated/IOError.java | 2 +- .../hbase/thrift/generated/IllegalArgument.java | 2 +- .../hadoop/hbase/thrift/generated/Mutation.java | 2 +- .../hadoop/hbase/thrift/generated/TAppend.java | 2 +- .../hadoop/hbase/thrift/generated/TCell.java| 2 +- .../hadoop/hbase/thrift/generated/TColumn.java | 2 +- .../hbase/thrift/generated/TIncrement.java | 2 +- .../hbase/thrift/generated/TRegionInfo.java | 2 +- .../hbase/thrift/generated/TRowResult.java | 2 +- .../hadoop/hbase/thrift/generated/TScan.java| 2 +- .../thrift2/ThriftHBaseServiceHandler.java | 290 + .../hadoop/hbase/thrift2/ThriftUtilities.java | 411 +- .../thrift2/generated/NamespaceDescriptor.java | 554 + .../hadoop/hbase/thrift2/generated/TAppend.java | 114 +- .../hbase/thrift2/generated/TAuthorization.java | 2 +- .../thrift2/generated/TBloomFilterType.java |69 + .../thrift2/generated/TCellVisibility.java | 2 +- .../hadoop/hbase/thrift2/generated/TColumn.java | 2 +- .../generated/TColumnFamilyDescriptor.java | 2519 + .../thrift2/generated/TColumnIncrement.java | 2 +- .../hbase/thrift2/generated/TColumnValue.java | 110 +- .../generated/TCompressionAlgorithm.java|60 + .../thrift2/generated/TDataBlockEncoding.java |57 + .../hadoop/hbase/thrift2/generated/TDelete.java | 2 +- .../hbase/thrift2/generated/TDurability.java| 3 + .../hadoop/hbase/thrift2/generated/TGet.java| 410 +- .../hbase/thrift2/generated/THBaseService.java | 44644 + .../hbase/thrift2/generated/THRegionInfo.java | 2 +- .../thrift2/generated/THRegionLocation.java | 2 +- .../hbase/thrift2/generated/TIOError.java | 2 +- .../thrift2/generated/TIllegalArgument.java | 2 +- .../hbase/thrift2/generated/TIncrement.java | 114 +- .../thrift2/generated/TKeepDeletedCells.java|63 + .../thrift2/generated/TNamespaceDescriptor.java | 554 + .../hadoop/hbase/thrift2/generated/TPut.java| 2 +- .../hadoop/hbase/thrift2/generated/TResult.java | 112 +- .../hbase/thrift2/generated/TRowMutations.java |38 +- .../hadoop/hbase/thrift2/generated/TScan.java | 2 +- .../hbase/thrift2/generated/TServerName.java| 2 +- .../thrift2/generated/TTableDescriptor.java | 843 + .../hbase/thrift2/generated/TTableName.java | 512 + .../hbase/thrift2/generated/TTimeRange.java | 2 +- .../apache/hadoop/hbase/thrift2/hbase.thrift| 229 +- .../thrift2/TestThriftHBaseServiceHandler.java |96 + 48 files changed, 41553 insertions(+), 10303 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java index 68361c1..8ec3e32 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java @@ -38,7 +38,7 @@ import org.slf4j.LoggerFactory; * An AlreadyExists exceptions signals that a table with the specified * name already exists */ -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2016-05-25") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") public class AlreadyExists extends TException implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("AlreadyExists"); http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/BatchMutation.ja
[27/47] hbase git commit: HBASE-21635 Use maven enforcer to ban imports from illegal packages
HBASE-21635 Use maven enforcer to ban imports from illegal packages Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/97fd647d Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/97fd647d Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/97fd647d Branch: refs/heads/HBASE-21512 Commit: 97fd647de20e6c8df0cd6df248ec1365abc37378 Parents: 7c0a3cc Author: zhangduo Authored: Sun Dec 23 18:25:42 2018 +0800 Committer: Duo Zhang Committed: Mon Dec 24 11:12:25 2018 +0800 -- .../apache/hadoop/hbase/master/DeadServer.java | 17 ++-- .../TestBalancerStatusTagInJMXMetrics.java | 32 pom.xml | 83 +++- 3 files changed, 107 insertions(+), 25 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/97fd647d/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DeadServer.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DeadServer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DeadServer.java index 4183201..0584792 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DeadServer.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DeadServer.java @@ -18,15 +18,6 @@ */ package org.apache.hadoop.hbase.master; -import org.apache.yetus.audience.InterfaceAudience; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.apache.hadoop.hbase.ServerName; -import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; -import org.apache.hadoop.hbase.util.Pair; - -import com.google.common.base.Preconditions; - import java.util.ArrayList; import java.util.Collections; import java.util.Comparator; @@ -37,6 +28,14 @@ import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Set; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; +import org.apache.hadoop.hbase.util.Pair; +import org.apache.yetus.audience.InterfaceAudience; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.hbase.thirdparty.com.google.common.base.Preconditions; /** http://git-wip-us.apache.org/repos/asf/hbase/blob/97fd647d/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBalancerStatusTagInJMXMetrics.java -- diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBalancerStatusTagInJMXMetrics.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBalancerStatusTagInJMXMetrics.java index 9f56621..d23436d 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBalancerStatusTagInJMXMetrics.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBalancerStatusTagInJMXMetrics.java @@ -1,22 +1,25 @@ /** - * Licensed to the Apache Software Foundation (ASF) under one or more contributor license - * agreements. See the NOTICE file distributed with this work for additional information regarding - * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance with the License. You may obtain a - * copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable - * law or agreed to in writing, software distributed under the License is distributed on an "AS IS" - * BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License - * for the specific language governing permissions and limitations under the License. + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. */ - package org.apache.hadoop.hbase.master.balancer; import static org.junit.Assert.assertEquals; import java.util.Random; - -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory;
[22/47] hbase git commit: HBASE-21618 Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result
HBASE-21618 Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/ad819380 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/ad819380 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/ad819380 Branch: refs/heads/HBASE-21512 Commit: ad819380c744678e719431fb8b1b5e1951bc31b6 Parents: 7875673 Author: Guanghao Zhang Authored: Thu Dec 20 11:03:54 2018 +0800 Committer: Guanghao Zhang Committed: Fri Dec 21 09:49:24 2018 +0800 -- .../hadoop/hbase/protobuf/ProtobufUtil.java | 4 +-- .../hbase/shaded/protobuf/ProtobufUtil.java | 4 +-- .../hbase/shaded/protobuf/TestProtobufUtil.java | 1 + .../hbase/client/TestFromClientSide3.java | 4 +-- .../client/TestScannersFromClientSide.java | 38 .../hadoop/hbase/protobuf/TestProtobufUtil.java | 1 + 6 files changed, 44 insertions(+), 8 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/ad819380/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java index 4d54528..a3d49b5 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java @@ -937,9 +937,7 @@ public final class ProtobufUtil { if (!scan.includeStartRow()) { scanBuilder.setIncludeStartRow(false); } -if (scan.includeStopRow()) { - scanBuilder.setIncludeStopRow(true); -} +scanBuilder.setIncludeStopRow(scan.includeStopRow()); if (scan.getReadType() != Scan.ReadType.DEFAULT) { scanBuilder.setReadType(toReadType(scan.getReadType())); } http://git-wip-us.apache.org/repos/asf/hbase/blob/ad819380/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java index cf4c831..fea81f1 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java @@ -1081,9 +1081,7 @@ public final class ProtobufUtil { if (!scan.includeStartRow()) { scanBuilder.setIncludeStartRow(false); } -if (scan.includeStopRow()) { - scanBuilder.setIncludeStopRow(true); -} +scanBuilder.setIncludeStopRow(scan.includeStopRow()); if (scan.getReadType() != Scan.ReadType.DEFAULT) { scanBuilder.setReadType(toReadType(scan.getReadType())); } http://git-wip-us.apache.org/repos/asf/hbase/blob/ad819380/hbase-client/src/test/java/org/apache/hadoop/hbase/shaded/protobuf/TestProtobufUtil.java -- diff --git a/hbase-client/src/test/java/org/apache/hadoop/hbase/shaded/protobuf/TestProtobufUtil.java b/hbase-client/src/test/java/org/apache/hadoop/hbase/shaded/protobuf/TestProtobufUtil.java index be51e96..2d8a74a 100644 --- a/hbase-client/src/test/java/org/apache/hadoop/hbase/shaded/protobuf/TestProtobufUtil.java +++ b/hbase-client/src/test/java/org/apache/hadoop/hbase/shaded/protobuf/TestProtobufUtil.java @@ -246,6 +246,7 @@ public class TestProtobufUtil { scanBuilder.setCacheBlocks(false); scanBuilder.setCaching(1024); scanBuilder.setTimeRange(ProtobufUtil.toTimeRange(TimeRange.allTime())); +scanBuilder.setIncludeStopRow(false); ClientProtos.Scan expectedProto = scanBuilder.build(); ClientProtos.Scan actualProto = ProtobufUtil.toScan( http://git-wip-us.apache.org/repos/asf/hbase/blob/ad819380/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java -- diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java index 0dee20b..cbfa1bf 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java @@ -1094,7 +1094,7 @@ public class TestFromClientSide3 { } Scan scan = new Scan(); -scan.withStartRow(ROW).withStopRow(ROW).addFamily(FAMILY).setBatch(3) +scan.withStartRow(ROW).with
[36/47] hbase git commit: HBASE-21650 Add DDL operation and some other miscellaneous to thrift2
http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionInfo.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionInfo.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionInfo.java index 129ab2e..8450f5b 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionInfo.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionInfo.java @@ -34,7 +34,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"}) -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2016-05-25") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") public class THRegionInfo implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("THRegionInfo"); http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionLocation.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionLocation.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionLocation.java index 94b25ff..b1146e9 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionLocation.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionLocation.java @@ -34,7 +34,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"}) -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2016-05-25") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") public class THRegionLocation implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("THRegionLocation"); http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIOError.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIOError.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIOError.java index 2e50d3d..9569c3f 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIOError.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIOError.java @@ -39,7 +39,7 @@ import org.slf4j.LoggerFactory; * to the HBase master or a HBase region server. Also used to return * more general HBase error conditions. */ -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2016-05-25") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") public class TIOError extends TException implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TIOError"); http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIllegalArgument.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIllegalArgument.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIllegalArgument.java index 9387429..6734dec 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIllegalArgument.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIllegalArgument.java @@ -38,7 +38,7 @@ import org.slf4j.LoggerFactory; * A TIllegalArgument exception indicates an illegal or invalid * argument was passed into a procedure. */ -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2016-05-25") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") public class TIllegalArgument extends TException implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TIllegalArgument"); http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIncrement.java --
[35/47] hbase git commit: HBASE-21650 Add DDL operation and some other miscellaneous to thrift2
http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableName.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableName.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableName.java new file mode 100644 index 000..f2c0743 --- /dev/null +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTableName.java @@ -0,0 +1,512 @@ +/** + * Autogenerated by Thrift Compiler (0.9.3) + * + * DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING + * @generated + */ +package org.apache.hadoop.hbase.thrift2.generated; + +import org.apache.thrift.scheme.IScheme; +import org.apache.thrift.scheme.SchemeFactory; +import org.apache.thrift.scheme.StandardScheme; + +import org.apache.thrift.scheme.TupleScheme; +import org.apache.thrift.protocol.TTupleProtocol; +import org.apache.thrift.protocol.TProtocolException; +import org.apache.thrift.EncodingUtils; +import org.apache.thrift.TException; +import org.apache.thrift.async.AsyncMethodCallback; +import org.apache.thrift.server.AbstractNonblockingServer.*; +import java.util.List; +import java.util.ArrayList; +import java.util.Map; +import java.util.HashMap; +import java.util.EnumMap; +import java.util.Set; +import java.util.HashSet; +import java.util.EnumSet; +import java.util.Collections; +import java.util.BitSet; +import java.nio.ByteBuffer; +import java.util.Arrays; +import javax.annotation.Generated; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +@SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"}) +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") +public class TTableName implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { + private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TTableName"); + + private static final org.apache.thrift.protocol.TField NS_FIELD_DESC = new org.apache.thrift.protocol.TField("ns", org.apache.thrift.protocol.TType.STRING, (short)1); + private static final org.apache.thrift.protocol.TField QUALIFIER_FIELD_DESC = new org.apache.thrift.protocol.TField("qualifier", org.apache.thrift.protocol.TType.STRING, (short)2); + + private static final Map, SchemeFactory> schemes = new HashMap, SchemeFactory>(); + static { +schemes.put(StandardScheme.class, new TTableNameStandardSchemeFactory()); +schemes.put(TupleScheme.class, new TTableNameTupleSchemeFactory()); + } + + public ByteBuffer ns; // required + public ByteBuffer qualifier; // required + + /** The set of fields this struct contains, along with convenience methods for finding and manipulating them. */ + public enum _Fields implements org.apache.thrift.TFieldIdEnum { +NS((short)1, "ns"), +QUALIFIER((short)2, "qualifier"); + +private static final Map byName = new HashMap(); + +static { + for (_Fields field : EnumSet.allOf(_Fields.class)) { +byName.put(field.getFieldName(), field); + } +} + +/** + * Find the _Fields constant that matches fieldId, or null if its not found. + */ +public static _Fields findByThriftId(int fieldId) { + switch(fieldId) { +case 1: // NS + return NS; +case 2: // QUALIFIER + return QUALIFIER; +default: + return null; + } +} + +/** + * Find the _Fields constant that matches fieldId, throwing an exception + * if it is not found. + */ +public static _Fields findByThriftIdOrThrow(int fieldId) { + _Fields fields = findByThriftId(fieldId); + if (fields == null) throw new IllegalArgumentException("Field " + fieldId + " doesn't exist!"); + return fields; +} + +/** + * Find the _Fields constant that matches name, or null if its not found. + */ +public static _Fields findByName(String name) { + return byName.get(name); +} + +private final short _thriftId; +private final String _fieldName; + +_Fields(short thriftId, String fieldName) { + _thriftId = thriftId; + _fieldName = fieldName; +} + +public short getThriftFieldId() { + return _thriftId; +} + +public String getFieldName() { + return _fieldName; +} + } + + // isset id assignments + public static final Map<_Fields, org.apache.thrift.meta_data.FieldMetaData> metaDataMap; + static { +Map<_Fields, org.apache.thrift.meta_data.FieldMetaData> tmpMap = new EnumMap<_Fields, org.apache.thrift.meta_data.FieldMetaData>(_Fields.class); +tmpMap.put(_Fields.NS, new org.apache.thrift.meta_data.FieldMetaData("ns", org.apache.thrift.TFieldRequirementType.REQUIRED, +new org.apache.thrift.meta_data.FieldValueMetaData(org.apache.thrift.protocol.TType.STRING , true))); +
[30/47] hbase git commit: HBASE-21631 (addendum) Fixed TestQuotasShell failure (quotas_test.rb)
HBASE-21631 (addendum) Fixed TestQuotasShell failure (quotas_test.rb) Signed-off-by: Guanghao Zhang Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/44dec600 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/44dec600 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/44dec600 Branch: refs/heads/HBASE-21512 Commit: 44dec60054d1c45880d591c74a023f7a534e6d73 Parents: dbafa1b Author: Sakthi Authored: Sun Dec 23 21:25:07 2018 -0800 Committer: Guanghao Zhang Committed: Mon Dec 24 14:15:59 2018 +0800 -- hbase-shell/src/test/ruby/hbase/quotas_test.rb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/44dec600/hbase-shell/src/test/ruby/hbase/quotas_test.rb -- diff --git a/hbase-shell/src/test/ruby/hbase/quotas_test.rb b/hbase-shell/src/test/ruby/hbase/quotas_test.rb index be6b238..295d545 100644 --- a/hbase-shell/src/test/ruby/hbase/quotas_test.rb +++ b/hbase-shell/src/test/ruby/hbase/quotas_test.rb @@ -99,8 +99,7 @@ module Hbase define_test 'can set and remove quota' do command(:set_quota, TYPE => SPACE, LIMIT => '1G', POLICY => NO_INSERTS, TABLE => @test_name) output = capture_stdout{ command(:list_quotas) } - size = 1024 * 1024 * 1024 - assert(output.include?("LIMIT => #{size}")) + assert(output.include?("LIMIT => 1G")) assert(output.include?("VIOLATION_POLICY => NO_INSERTS")) assert(output.include?("TYPE => SPACE")) assert(output.include?("TABLE => #{@test_name}"))
[16/47] hbase git commit: HBASE-21514 Refactor CacheConfig
HBASE-21514 Refactor CacheConfig Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/1971d02e Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/1971d02e Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/1971d02e Branch: refs/heads/HBASE-21512 Commit: 1971d02e725341fdee79b7ee2308a9870debe2f6 Parents: 68b5df0 Author: Guanghao Zhang Authored: Thu Nov 29 10:30:45 2018 +0800 Committer: Guanghao Zhang Committed: Tue Dec 18 13:43:30 2018 +0800 -- .../tmpl/regionserver/BlockCacheTmpl.jamon | 36 +- .../tmpl/regionserver/BlockCacheViewTmpl.jamon | 3 +- .../hbase/tmpl/regionserver/RSStatusTmpl.jamon | 5 +- .../hbase/io/hfile/BlockCacheFactory.java | 226 + .../hadoop/hbase/io/hfile/CacheConfig.java | 499 --- .../hbase/io/hfile/CombinedBlockCache.java | 12 + .../hadoop/hbase/io/hfile/HFileBlockIndex.java | 17 +- .../hadoop/hbase/io/hfile/HFileReaderImpl.java | 176 +++ .../hadoop/hbase/io/hfile/HFileWriterImpl.java | 9 +- .../hbase/io/hfile/bucket/BucketAllocator.java | 4 +- .../assignment/MergeTableRegionsProcedure.java | 15 +- .../assignment/SplitTableRegionProcedure.java | 15 +- .../apache/hadoop/hbase/mob/MobCacheConfig.java | 64 --- .../apache/hadoop/hbase/mob/MobFileCache.java | 5 +- .../hadoop/hbase/regionserver/HMobStore.java| 28 +- .../hadoop/hbase/regionserver/HRegion.java | 40 +- .../hbase/regionserver/HRegionServer.java | 55 +- .../hadoop/hbase/regionserver/HStore.java | 2 +- .../hbase/regionserver/HeapMemoryManager.java | 30 +- .../MetricsRegionServerWrapperImpl.java | 267 +++--- .../hbase/regionserver/RSRpcServices.java | 3 +- .../regionserver/RegionServerServices.java | 15 +- .../hadoop/hbase/HBaseTestingUtility.java | 48 +- .../hadoop/hbase/MockRegionServerServices.java | 13 + ...estAvoidCellReferencesIntoShippedBlocks.java | 4 +- .../client/TestBlockEvictionFromClient.java | 20 +- .../hadoop/hbase/client/TestFromClientSide.java | 3 +- .../hbase/io/encoding/TestEncodedSeekers.java | 26 +- .../hbase/io/hfile/TestBlockCacheReporting.java | 47 +- .../hadoop/hbase/io/hfile/TestCacheConfig.java | 53 +- .../hadoop/hbase/io/hfile/TestCacheOnWrite.java | 34 +- .../io/hfile/TestForceCacheImportantBlocks.java | 22 +- .../apache/hadoop/hbase/io/hfile/TestHFile.java | 9 +- .../hadoop/hbase/io/hfile/TestHFileBlock.java | 3 +- .../hbase/io/hfile/TestHFileBlockIndex.java | 10 +- .../hfile/TestLazyDataBlockDecompression.java | 20 +- .../hadoop/hbase/io/hfile/TestPrefetch.java | 22 +- .../io/hfile/TestScannerFromBucketCache.java| 58 +-- .../TestScannerSelectionUsingKeyRange.java | 5 +- .../io/hfile/TestScannerSelectionUsingTTL.java | 31 +- .../hadoop/hbase/master/MockRegionServer.java | 13 + .../hbase/master/TestMasterNotCarryTable.java | 8 +- .../hadoop/hbase/mob/TestMobFileCache.java | 22 +- .../regionserver/DataBlockEncodingTool.java | 7 +- .../EncodedSeekPerformanceTest.java | 2 +- .../hbase/regionserver/TestAtomicOperation.java | 10 +- .../hbase/regionserver/TestBlocksRead.java | 66 +-- .../hbase/regionserver/TestBlocksScanned.java | 38 +- .../regionserver/TestCacheOnWriteInSchema.java | 6 +- .../regionserver/TestClearRegionBlockCache.java | 46 +- .../regionserver/TestCompoundBloomFilter.java | 16 +- .../hbase/regionserver/TestHMobStore.java | 84 ++-- .../hbase/regionserver/TestHStoreFile.java | 14 +- .../regionserver/TestMobStoreCompaction.java| 20 +- .../regionserver/TestMultiColumnScanner.java| 15 +- .../hbase/regionserver/TestRSStatusServlet.java | 9 +- .../hbase/regionserver/TestRecoveredEdits.java | 38 +- .../regionserver/TestRowPrefixBloomFilter.java | 2 +- .../regionserver/TestSecureBulkLoadManager.java | 2 +- 59 files changed, 1096 insertions(+), 1276 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/1971d02e/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon -- diff --git a/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon b/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon index 5ea5bcc..a18e6d4 100644 --- a/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon +++ b/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon @@ -20,17 +20,12 @@ Template for rendering Block Cache tabs in RegionServer Status page. <%args> CacheConfig cacheConfig; Configuration config; +BlockCache bc;
[04/47] hbase git commit: HIVE-21575 : memstore above high watermark message is logged too much
HIVE-21575 : memstore above high watermark message is logged too much Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/4640ff59 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/4640ff59 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/4640ff59 Branch: refs/heads/HBASE-21512 Commit: 4640ff5959af4865966126a503a7cd15e26a7408 Parents: 67d6d50 Author: Sergey Shelukhin Authored: Wed Dec 12 11:02:25 2018 -0800 Committer: Sergey Shelukhin Committed: Wed Dec 12 11:02:25 2018 -0800 -- .../apache/hadoop/hbase/regionserver/MemStoreFlusher.java| 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/4640ff59/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java index 699c9b6..804a2f8 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java @@ -703,6 +703,7 @@ class MemStoreFlusher implements FlushRequester { if (flushType != FlushType.NORMAL) { TraceUtil.addTimelineAnnotation("Force Flush. We're above high water mark."); long start = EnvironmentEdgeManager.currentTime(); + long nextLogTimeMs = start; synchronized (this.blockSignal) { boolean blocked = false; long startTime = 0; @@ -744,8 +745,11 @@ class MemStoreFlusher implements FlushRequester { LOG.warn("Interrupted while waiting"); interrupted = true; } -long took = EnvironmentEdgeManager.currentTime() - start; -LOG.warn("Memstore is above high water mark and block " + took + "ms"); +long nowMs = EnvironmentEdgeManager.currentTime(); +if (nowMs >= nextLogTimeMs) { + LOG.warn("Memstore is above high water mark and block {} ms", nowMs - start); + nextLogTimeMs = nowMs + 1000; +} flushType = isAboveHighWaterMark(); } } finally {
[13/47] hbase git commit: HBASE-21589 TestCleanupMetaWAL fails
HBASE-21589 TestCleanupMetaWAL fails Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/68b5df00 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/68b5df00 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/68b5df00 Branch: refs/heads/HBASE-21512 Commit: 68b5df00951d3ee55efaa6068f4530dca17eae1f Parents: ac0b3bb Author: stack Authored: Sun Dec 16 14:15:00 2018 -0800 Committer: stack Committed: Mon Dec 17 09:31:59 2018 -0800 -- .../apache/hadoop/hbase/regionserver/TestCleanupMetaWAL.java | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/68b5df00/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCleanupMetaWAL.java -- diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCleanupMetaWAL.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCleanupMetaWAL.java index 4a723c0..03b3316 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCleanupMetaWAL.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCleanupMetaWAL.java @@ -66,10 +66,13 @@ public class TestCleanupMetaWAL { .getRegionServer(TEST_UTIL.getMiniHBaseCluster().getServerWithMeta()); TEST_UTIL.getAdmin() .move(RegionInfoBuilder.FIRST_META_REGIONINFO.getEncodedNameAsBytes(), null); +LOG.info("KILL"); TEST_UTIL.getMiniHBaseCluster().killRegionServer(serverWithMeta.getServerName()); -TEST_UTIL.waitFor(1, () -> +LOG.info("WAIT"); +TEST_UTIL.waitFor(3, () -> TEST_UTIL.getMiniHBaseCluster().getMaster().getProcedures().stream() .filter(p -> p instanceof ServerCrashProcedure && p.isFinished()).count() > 0); +LOG.info("DONE WAITING"); MasterFileSystem fs = TEST_UTIL.getMiniHBaseCluster().getMaster().getMasterFileSystem(); Path walPath = new Path(fs.getWALRootDir(), HConstants.HREGION_LOGDIR_NAME); for (FileStatus status : FSUtils.listStatus(fs.getFileSystem(), walPath)) { @@ -77,7 +80,5 @@ public class TestCleanupMetaWAL { fail("Should not have splitting wal dir here:" + status); } } - - } }
[23/47] hbase git commit: HBASE-21401 Sanity check when constructing the KeyValue
HBASE-21401 Sanity check when constructing the KeyValue Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/12786f80 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/12786f80 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/12786f80 Branch: refs/heads/HBASE-21512 Commit: 12786f80c14c6f2c3c111a55bbf431fb2e81e828 Parents: ad81938 Author: huzheng Authored: Sat Oct 27 16:57:01 2018 +0800 Committer: huzheng Committed: Fri Dec 21 18:01:35 2018 +0800 -- .../java/org/apache/hadoop/hbase/KeyValue.java | 12 +- .../org/apache/hadoop/hbase/KeyValueUtil.java | 148 +- .../hadoop/hbase/codec/KeyValueCodec.java | 3 +- .../hbase/codec/KeyValueCodecWithTags.java | 2 +- .../org/apache/hadoop/hbase/TestKeyValue.java | 295 +-- .../hadoop/hbase/regionserver/HStore.java | 1 - .../io/encoding/TestDataBlockEncoders.java | 2 +- .../hadoop/hbase/io/hfile/TestCacheOnWrite.java | 4 +- .../hadoop/hbase/io/hfile/TestHFileSeek.java| 9 +- 9 files changed, 290 insertions(+), 186 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/12786f80/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java -- diff --git a/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java b/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java index f7f6c0d..bdaefff 100644 --- a/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java +++ b/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java @@ -252,6 +252,15 @@ public class KeyValue implements ExtendedCell, Cloneable { } /** + * True to indicate that the byte b is a valid type. + * @param b byte to check + * @return true or false + */ +static boolean isValidType(byte b) { + return codeArray[b & 0xff] != null; +} + +/** * Cannot rely on enum ordinals . They change if item is removed or moved. * Do our own codes. * @param b @@ -331,7 +340,8 @@ public class KeyValue implements ExtendedCell, Cloneable { * @param offset offset to start of the KeyValue * @param length length of the KeyValue */ - public KeyValue(final byte [] bytes, final int offset, final int length) { + public KeyValue(final byte[] bytes, final int offset, final int length) { +KeyValueUtil.checkKeyValueBytes(bytes, offset, length, true); this.bytes = bytes; this.offset = offset; this.length = length; http://git-wip-us.apache.org/repos/asf/hbase/blob/12786f80/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java -- diff --git a/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java b/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java index 1b61d1e..16ebdbf 100644 --- a/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java +++ b/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java @@ -518,17 +518,145 @@ public class KeyValueUtil { return (long) length + Bytes.SIZEOF_INT; } + static String bytesToHex(byte[] buf, int offset, int length) { +return ", KeyValueBytesHex=" + Bytes.toStringBinary(buf, offset, length) + ", offset=" + offset ++ ", length=" + length; + } + + static void checkKeyValueBytes(byte[] buf, int offset, int length, boolean withTags) { +int pos = offset, endOffset = offset + length; +// check the key +if (pos + Bytes.SIZEOF_INT > endOffset) { + throw new IllegalArgumentException( + "Overflow when reading key length at position=" + pos + bytesToHex(buf, offset, length)); +} +int keyLen = Bytes.toInt(buf, pos, Bytes.SIZEOF_INT); +pos += Bytes.SIZEOF_INT; +if (keyLen <= 0 || pos + keyLen > endOffset) { + throw new IllegalArgumentException( + "Invalid key length in KeyValue. keyLength=" + keyLen + bytesToHex(buf, offset, length)); +} +// check the value +if (pos + Bytes.SIZEOF_INT > endOffset) { + throw new IllegalArgumentException("Overflow when reading value length at position=" + pos + + bytesToHex(buf, offset, length)); +} +int valLen = Bytes.toInt(buf, pos, Bytes.SIZEOF_INT); +pos += Bytes.SIZEOF_INT; +if (valLen < 0 || pos + valLen > endOffset) { + throw new IllegalArgumentException("Invalid value length in KeyValue, valueLength=" + valLen + + bytesToHex(buf, offset, length)); +} +// check the row +if (pos + Bytes.SIZEOF_SHORT > endOffset) { + throw new IllegalArgumentException( + "Overflow when reading row length at position=" + pos + bytesToHex(buf, offset, length)); +} +short rowLen = Bytes.t
[11/47] hbase git commit: HBASE-21590 Optimize trySkipToNextColumn in StoreScanner a bit. (addendum)
HBASE-21590 Optimize trySkipToNextColumn in StoreScanner a bit. (addendum) Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/49115348 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/49115348 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/49115348 Branch: refs/heads/HBASE-21512 Commit: 491153488ee5b19de22fd72e55dd5039399bb727 Parents: 2b003c5 Author: Sean Busbey Authored: Fri Dec 14 11:23:36 2018 -0600 Committer: Sean Busbey Committed: Fri Dec 14 17:08:22 2018 -0600 -- .../apache/hadoop/hbase/regionserver/StoreScanner.java| 10 ++ 1 file changed, 6 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/49115348/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java index e7a4528..91ca592 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java @@ -807,8 +807,9 @@ public class StoreScanner extends NonReversedNonLazyKeyValueScanner Cell previousIndexedKey = null; do { Cell nextIndexedKey = getNextIndexedKey(); - if (nextIndexedKey != null && nextIndexedKey != KeyValueScanner.NO_NEXT_INDEXED_KEY - && (nextIndexedKey == previousIndexedKey || matcher.compareKeyForNextRow(nextIndexedKey, cell) >= 0)) { + if (nextIndexedKey != null && nextIndexedKey != KeyValueScanner.NO_NEXT_INDEXED_KEY && + (nextIndexedKey == previousIndexedKey || + matcher.compareKeyForNextRow(nextIndexedKey, cell) >= 0)) { this.heap.next(); ++kvsScanned; previousIndexedKey = nextIndexedKey; @@ -832,8 +833,9 @@ public class StoreScanner extends NonReversedNonLazyKeyValueScanner Cell previousIndexedKey = null; do { Cell nextIndexedKey = getNextIndexedKey(); - if (nextIndexedKey != null && nextIndexedKey != KeyValueScanner.NO_NEXT_INDEXED_KEY - && (nextIndexedKey == previousIndexedKey || matcher.compareKeyForNextColumn(nextIndexedKey, cell) >= 0)) { + if (nextIndexedKey != null && nextIndexedKey != KeyValueScanner.NO_NEXT_INDEXED_KEY && + (nextIndexedKey == previousIndexedKey || + matcher.compareKeyForNextColumn(nextIndexedKey, cell) >= 0)) { this.heap.next(); ++kvsScanned; previousIndexedKey = nextIndexedKey;
[17/47] hbase git commit: HBASE-21592 quota.addGetResult(r) throw NPE
HBASE-21592 quota.addGetResult(r) throw NPE Signed-off-by: huzheng Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/f7828468 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/f7828468 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/f7828468 Branch: refs/heads/HBASE-21512 Commit: f78284685fc533230a0395d297ebacff32632396 Parents: 1971d02 Author: xuqinya Authored: Tue Dec 18 08:19:47 2018 +0800 Committer: huzheng Committed: Tue Dec 18 16:15:51 2018 +0800 -- .../hadoop/hbase/regionserver/RSRpcServices.java | 3 ++- .../hadoop/hbase/quotas/TestQuotaThrottle.java | 17 + 2 files changed, 19 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/f7828468/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java index 31df37a..f788a86 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java @@ -2571,7 +2571,8 @@ public class RSRpcServices implements HBaseRPCErrorHandler, } builder.setResult(pbr); } - if (r != null) { + //r.cells is null when an table.exists(get) call + if (r != null && r.rawCells() != null) { quota.addGetResult(r); } return builder.build(); http://git-wip-us.apache.org/repos/asf/hbase/blob/f7828468/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaThrottle.java -- diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaThrottle.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaThrottle.java index e506a08..c069403 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaThrottle.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaThrottle.java @@ -553,6 +553,23 @@ public class TestQuotaThrottle { triggerTableCacheRefresh(true, TABLE_NAMES[0]); } + @Test + public void testTableExistsGetThrottle() throws Exception { +final Admin admin = TEST_UTIL.getAdmin(); + +// Add throttle quota +admin.setQuota(QuotaSettingsFactory.throttleTable(TABLE_NAMES[0], +ThrottleType.REQUEST_NUMBER, 100, TimeUnit.MINUTES)); +triggerTableCacheRefresh(false, TABLE_NAMES[0]); + +Table table = TEST_UTIL.getConnection().getTable(TABLE_NAMES[0]); +// An exists call when having throttle quota +table.exists(new Get(Bytes.toBytes("abc"))); + +admin.setQuota(QuotaSettingsFactory.unthrottleTable(TABLE_NAMES[0])); +triggerTableCacheRefresh(true, TABLE_NAMES[0]); + } + private int doPuts(int maxOps, final Table... tables) throws Exception { return doPuts(maxOps, -1, tables); }
[38/47] hbase git commit: HBASE-21650 Add DDL operation and some other miscellaneous to thrift2
http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnIncrement.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnIncrement.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnIncrement.java index 2fb3f76..0f27519 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnIncrement.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnIncrement.java @@ -37,7 +37,7 @@ import org.slf4j.LoggerFactory; /** * Represents a single cell and the amount to increment it by */ -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2016-05-25") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") public class TColumnIncrement implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TColumnIncrement"); http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnValue.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnValue.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnValue.java index 3ceb4c0..6cded1b 100644 --- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnValue.java +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnValue.java @@ -37,7 +37,7 @@ import org.slf4j.LoggerFactory; /** * Represents a single cell and its value. */ -@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2016-05-25") +@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)", date = "2018-12-27") public class TColumnValue implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable { private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("TColumnValue"); @@ -46,6 +46,7 @@ public class TColumnValue implements org.apache.thrift.TBase, SchemeFactory> schemes = new HashMap, SchemeFactory>(); static { @@ -58,6 +59,7 @@ public class TColumnValue implements org.apache.thrift.TBase byName = new HashMap(); @@ -90,6 +93,8 @@ public class TColumnValue implements org.apache.thrift.TBase metaDataMap; static { Map<_Fields, org.apache.thrift.meta_data.FieldMetaData> tmpMap = new EnumMap<_Fields, org.apache.thrift.meta_data.FieldMetaData>(_Fields.class); @@ -146,6 +152,8 @@ public class TColumnValue implements org.apache.thrift.TBasehttp://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCompressionAlgorithm.java -- diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCompressionAlgorithm.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCompressionAlgorithm.java new file mode 100644 index 000..46799be --- /dev/null +++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCompressionAlgorithm.java @@ -0,0 +1,60 @@ +/** + * Autogenerated by Thrift Compiler (0.9.3) + * + * DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING + * @generated + */ +package org.apache.hadoop.hbase.thrift2.generated; + + +import java.util.Map; +import java.util.HashMap; +import org.apache.thrift.TEnum; + +public enum TCompressionAlgorithm implements org.apache.thrift.TEnum { + LZO(0), + GZ(1), + NONE(2), + SNAPPY(3), + LZ4(4), + BZIP2(5), + ZSTD(6); + + private final int value; + + private TCompressionAlgorithm(int value) { +this.value = value; + } + + /** + * Get the integer value of this enum value, as defined in the Thrift IDL. + */ + public int getValue() { +return value; + } + + /** + * Find a the enum type by its integer value, as defined in the Thrift IDL. + * @return null if the value is not found. + */ + public static TCompressionAlgorithm findByValue(int value) { +switch (value) { + case 0: +return LZO; + case 1: +return GZ; + case 2: +return NONE; + case 3: +return SNAPPY; + case 4: +return LZ4; + case 5: +return BZIP2; + case 6: +return ZSTD; + default: +return null; +} + } +} http://git-wip-us.apache.org/repos/asf/hbase/blob/7820ba1d/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/generated/TDataBlockEncoding.java ---
[09/47] hbase git commit: HBASE-21578 Fix wrong throttling exception for capacity unit
HBASE-21578 Fix wrong throttling exception for capacity unit Signed-off-by: Guanghao Zhang Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/1b08ba73 Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/1b08ba73 Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/1b08ba73 Branch: refs/heads/HBASE-21512 Commit: 1b08ba7385d0dd914a6fb9722b786e4ece116b28 Parents: 3ff274e Author: meiyi Authored: Fri Dec 14 09:42:48 2018 +0800 Committer: Guanghao Zhang Committed: Fri Dec 14 18:17:47 2018 +0800 -- .../hbase/quotas/RpcThrottlingException.java| 21 ++-- .../hadoop/hbase/quotas/TimeBasedLimiter.java | 8 2 files changed, 23 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/1b08ba73/hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/RpcThrottlingException.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/RpcThrottlingException.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/RpcThrottlingException.java index 9baf91f..4c48f65 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/RpcThrottlingException.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/RpcThrottlingException.java @@ -29,13 +29,15 @@ public class RpcThrottlingException extends HBaseIOException { @InterfaceAudience.Public public enum Type { NumRequestsExceeded, RequestSizeExceeded, NumReadRequestsExceeded, NumWriteRequestsExceeded, -WriteSizeExceeded, ReadSizeExceeded, +WriteSizeExceeded, ReadSizeExceeded, RequestCapacityUnitExceeded, ReadCapacityUnitExceeded, +WriteCapacityUnitExceeded } private static final String[] MSG_TYPE = new String[] { "number of requests exceeded", "request size limit exceeded", "number of read requests exceeded", "number of write requests exceeded", -"write size limit exceeded", "read size limit exceeded", }; +"write size limit exceeded", "read size limit exceeded", "request capacity unit exceeded", +"read capacity unit exceeded", "write capacity unit exceeded" }; private static final String MSG_WAIT = " - wait "; @@ -100,6 +102,21 @@ public class RpcThrottlingException extends HBaseIOException { throwThrottlingException(Type.ReadSizeExceeded, waitInterval); } + public static void throwRequestCapacityUnitExceeded(final long waitInterval) + throws RpcThrottlingException { +throwThrottlingException(Type.RequestCapacityUnitExceeded, waitInterval); + } + + public static void throwReadCapacityUnitExceeded(final long waitInterval) + throws RpcThrottlingException { +throwThrottlingException(Type.ReadCapacityUnitExceeded, waitInterval); + } + + public static void throwWriteCapacityUnitExceeded(final long waitInterval) + throws RpcThrottlingException { +throwThrottlingException(Type.WriteCapacityUnitExceeded, waitInterval); + } + private static void throwThrottlingException(final Type type, final long waitInterval) throws RpcThrottlingException { String msg = MSG_TYPE[type.ordinal()] + MSG_WAIT + StringUtils.formatTime(waitInterval); http://git-wip-us.apache.org/repos/asf/hbase/blob/1b08ba73/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TimeBasedLimiter.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TimeBasedLimiter.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TimeBasedLimiter.java index 771eed1..6b5349f 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TimeBasedLimiter.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TimeBasedLimiter.java @@ -148,7 +148,7 @@ public class TimeBasedLimiter implements QuotaLimiter { reqSizeLimiter.waitInterval(estimateWriteSize + estimateReadSize)); } if (!reqCapacityUnitLimiter.canExecute(estimateWriteCapacityUnit + estimateReadCapacityUnit)) { - RpcThrottlingException.throwRequestSizeExceeded( + RpcThrottlingException.throwRequestCapacityUnitExceeded( reqCapacityUnitLimiter.waitInterval(estimateWriteCapacityUnit + estimateReadCapacityUnit)); } @@ -161,7 +161,7 @@ public class TimeBasedLimiter implements QuotaLimiter { writeSizeLimiter.waitInterval(estimateWriteSize)); } if (!writeCapacityUnitLimiter.canExecute(estimateWriteCapacityUnit)) { -RpcThrottlingException.throwWriteSizeExceeded( +RpcThrottlingException.throwWriteCapacityUnitExceeded( writeCapacityUnitLimiter.waitInterval(estimateWriteCapacityUnit)); } } @@ -175,8 +175,8 @@ pu
[46/47] hbase git commit: HBASE-21516 Use AsyncConnection instead of Connection in SecureBulkLoadManager
HBASE-21516 Use AsyncConnection instead of Connection in SecureBulkLoadManager Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/a13292db Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/a13292db Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/a13292db Branch: refs/heads/HBASE-21512 Commit: a13292db13ac8d09940fe8c1e9ec6b3e84b09271 Parents: f3caa01 Author: zhangduo Authored: Sat Dec 1 21:15:48 2018 +0800 Committer: zhangduo Committed: Mon Dec 31 20:34:24 2018 +0800 -- .../hadoop/hbase/protobuf/ProtobufUtil.java | 5 +- .../hbase/shaded/protobuf/ProtobufUtil.java | 7 ++- .../hbase/regionserver/HRegionServer.java | 2 +- .../regionserver/SecureBulkLoadManager.java | 24 + .../hadoop/hbase/security/token/TokenUtil.java | 57 +++- .../hbase/security/token/TestTokenUtil.java | 42 +++ 6 files changed, 96 insertions(+), 41 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/a13292db/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java index a3d49b5..d9e620b 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java @@ -261,13 +261,12 @@ public final class ProtobufUtil { * just {@link ServiceException}. Prefer this method to * {@link #getRemoteException(ServiceException)} because trying to * contain direct protobuf references. - * @param e */ - public static IOException handleRemoteException(Exception e) { + public static IOException handleRemoteException(Throwable e) { return makeIOExceptionOfException(e); } - private static IOException makeIOExceptionOfException(Exception e) { + private static IOException makeIOExceptionOfException(Throwable e) { Throwable t = e; if (e instanceof ServiceException || e instanceof org.apache.hbase.thirdparty.com.google.protobuf.ServiceException) { http://git-wip-us.apache.org/repos/asf/hbase/blob/a13292db/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java -- diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java index fea81f1..de2fb7d 100644 --- a/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java @@ -40,7 +40,6 @@ import java.util.concurrent.TimeUnit; import java.util.function.Function; import java.util.regex.Pattern; import java.util.stream.Collectors; - import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.ByteBufferExtendedCell; @@ -123,6 +122,7 @@ import org.apache.hbase.thirdparty.com.google.protobuf.Service; import org.apache.hbase.thirdparty.com.google.protobuf.ServiceException; import org.apache.hbase.thirdparty.com.google.protobuf.TextFormat; import org.apache.hbase.thirdparty.com.google.protobuf.UnsafeByteOperations; + import org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService; import org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.CloseRegionRequest; import org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.GetOnlineRegionRequest; @@ -343,13 +343,12 @@ public final class ProtobufUtil { * just {@link ServiceException}. Prefer this method to * {@link #getRemoteException(ServiceException)} because trying to * contain direct protobuf references. - * @param e */ - public static IOException handleRemoteException(Exception e) { + public static IOException handleRemoteException(Throwable e) { return makeIOExceptionOfException(e); } - private static IOException makeIOExceptionOfException(Exception e) { + private static IOException makeIOExceptionOfException(Throwable e) { Throwable t = e; if (e instanceof ServiceException) { t = e.getCause(); http://git-wip-us.apache.org/repos/asf/hbase/blob/a13292db/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServ
[20/47] hbase git commit: HBASE-21514: Refactor CacheConfig(addendum)
HBASE-21514: Refactor CacheConfig(addendum) Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/8991877b Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/8991877b Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/8991877b Branch: refs/heads/HBASE-21512 Commit: 8991877bb250ee1fe66c2b9a491645973927d674 Parents: fb58a23 Author: Guanghao Zhang Authored: Tue Dec 18 16:46:34 2018 +0800 Committer: Guanghao Zhang Committed: Wed Dec 19 13:55:13 2018 +0800 -- .../java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/8991877b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java index 0fc9576..d095ceb 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java @@ -1300,8 +1300,8 @@ public class HFileReaderImpl implements HFile.Reader, Configurable { boolean isCompaction, boolean updateCacheMetrics, BlockType expectedBlockType, DataBlockEncoding expectedDataBlockEncoding) throws IOException { // Check cache for block. If found return. -if (cacheConf.getBlockCache().isPresent()) { - BlockCache cache = cacheConf.getBlockCache().get(); +BlockCache cache = cacheConf.getBlockCache().orElse(null); +if (cache != null) { HFileBlock cachedBlock = (HFileBlock) cache.getBlock(cacheKey, cacheBlock, useLock, updateCacheMetrics); if (cachedBlock != null) {
[18/47] hbase git commit: HBASE-21565 Delete dead server from dead server list too early leads to concurrent Server Crash Procedures(SCP) for a same server
HBASE-21565 Delete dead server from dead server list too early leads to concurrent Server Crash Procedures(SCP) for a same server Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/c448604c Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/c448604c Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/c448604c Branch: refs/heads/HBASE-21512 Commit: c448604ceb987d113913f0583452b2abce04db0d Parents: f782846 Author: Jingyun Tian Authored: Mon Dec 17 19:32:23 2018 +0800 Committer: Jingyun Tian Committed: Tue Dec 18 16:57:11 2018 +0800 -- .../hbase/master/RegionServerTracker.java | 3 + .../hadoop/hbase/master/ServerManager.java | 25 .../master/assignment/AssignmentManager.java| 28 ++--- .../hbase/master/assignment/RegionStates.java | 3 +- .../hbase/master/assignment/ServerState.java| 2 +- .../master/assignment/ServerStateNode.java | 2 +- .../master/procedure/ServerCrashProcedure.java | 16 ++--- .../hadoop/hbase/HBaseTestingUtility.java | 7 ++- .../hadoop/hbase/master/TestRestartCluster.java | 65 .../procedure/TestServerCrashProcedure.java | 38 10 files changed, 155 insertions(+), 34 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hbase/blob/c448604c/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java index f419732..9d33a21 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java @@ -128,6 +128,9 @@ public class RegionServerTracker extends ZKListener { // '-SPLITTING'. Each splitting server should have a corresponding SCP. Log if not. splittingServersFromWALDir.stream().filter(s -> !deadServersFromPE.contains(s)). forEach(s -> LOG.error("{} has no matching ServerCrashProcedure", s)); +//create ServerNode for all possible live servers from wal directory +liveServersFromWALDir.stream() +.forEach(sn -> server.getAssignmentManager().getRegionStates().getOrCreateServer(sn)); watcher.registerListener(this); synchronized (this) { List servers = http://git-wip-us.apache.org/repos/asf/hbase/blob/c448604c/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java index dc76d72..86d72d1 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java @@ -602,19 +602,22 @@ public class ServerManager { return false; } LOG.info("Processing expiration of " + serverName + " on " + this.master.getServerName()); -master.getAssignmentManager().submitServerCrash(serverName, true); - -// Tell our listeners that a server was removed -if (!this.listeners.isEmpty()) { - for (ServerListener listener : this.listeners) { -listener.serverRemoved(serverName); +long pid = master.getAssignmentManager().submitServerCrash(serverName, true); +if(pid <= 0) { + return false; +} else { + // Tell our listeners that a server was removed + if (!this.listeners.isEmpty()) { +for (ServerListener listener : this.listeners) { + listener.serverRemoved(serverName); +} } + // trigger a persist of flushedSeqId + if (flushedSeqIdFlusher != null) { +flushedSeqIdFlusher.triggerNow(); + } + return true; } -// trigger a persist of flushedSeqId -if (flushedSeqIdFlusher != null) { - flushedSeqIdFlusher.triggerNow(); -} -return true; } @VisibleForTesting http://git-wip-us.apache.org/repos/asf/hbase/blob/c448604c/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java -- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java index a564ea9..b7c2203 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master