[GitHub] [hbase] ZhaoBQ commented on a change in pull request #633: HBASE-22890 Verify the file integrity in persistent IOEngine
ZhaoBQ commented on a change in pull request #633: HBASE-22890 Verify the file
integrity in persistent IOEngine
URL: https://github.com/apache/hbase/pull/633#discussion_r325510742
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java
##
@@ -267,6 +339,75 @@ void refreshFileConnection(int accessFileNum, IOException
ioe) throws IOExceptio
}
}
+ /**
+ * Delete bucketcache files
+ */
+ private void deleteCacheDataFile() {
+if (filePaths == null) {
+ return;
+}
+for (String file : filePaths) {
+ new File(file).delete();
+}
+ }
+
+ @Override
+ public byte[] calculateChecksum()
+throws IOException, NoSuchAlgorithmException {
+if (filePaths == null) {
+ return null;
+}
+StringBuilder sb = new StringBuilder();
+for (String filePath : filePaths){
+ File file = new File(filePath);
+ if (file.exists()){
+sb.append(filePath);
+sb.append(getFileSize(filePath));
+sb.append(file.lastModified());
+ } else {
+throw new IOException("Cache file: " + filePath + " is not exists.");
+ }
+}
+MessageDigest messageDigest = MessageDigest.getInstance(algorithmName);
+messageDigest.update(Bytes.toBytes(sb.toString()));
+return messageDigest.digest();
+ }
+
+ @Override
+ public boolean isOldVersion() {
Review comment:
I got your point.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932156#comment-16932156
]
Viraj Jasani commented on HBASE-22978:
--
In this case, is it fine to have "optionally dumping logs to HDFS" as a
sub-task to this?
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Reporter: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility classes and methods for accomplishing that.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
> as additional argument. The TABLE filter, which expects a string containing a
> table name, will include only entries pertaining to that table. The REGION
> filter, which expects a string containing an encoded region name, will
> include only entries pertaining to that region. The CLIENT_IP filter, which
> expects a string containing an IP address, will include only entries
> pertaining to that client. The USER filter, which expects a string containing
> a user
[jira] [Assigned] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reassigned HBASE-22978:
--
Assignee: Andrew Purtell
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility classes and methods for accomplishing that.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
> as additional argument. The TABLE filter, which expects a string containing a
> table name, will include only entries pertaining to that table. The REGION
> filter, which expects a string containing an encoded region name, will
> include only entries pertaining to that region. The CLIENT_IP filter, which
> expects a string containing an IP address, will include only entries
> pertaining to that client. The USER filter, which expects a string containing
> a user name, will include only entries pertaining to that user. Filters are
>
[jira] [Updated] (HBASE-23039) HBCK2 bypass -r command does not work
[
https://issues.apache.org/jira/browse/HBASE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi updated HBASE-23039:
--
Component/s: hbase-operator-tools
> HBCK2 bypass -r command does not work
> -
>
> Key: HBASE-23039
> URL: https://issues.apache.org/jira/browse/HBASE-23039
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Attachments: HBASE-23039.001.patch
>
>
>
> The recursiveFlag is wrong:
> {code:java}
> boolean overrideFlag = commandLine.hasOption(override.getOpt());
> boolean recursiveFlag = commandLine.hasOption(override.getOpt());
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23039) HBCK2 bypass -r command does not work
[
https://issues.apache.org/jira/browse/HBASE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi updated HBASE-23039:
--
Fix Version/s: hbase-operator-tools-2.0.0
> HBCK2 bypass -r command does not work
> -
>
> Key: HBASE-23039
> URL: https://issues.apache.org/jira/browse/HBASE-23039
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Affects Versions: hbase-operator-tools-1.0.0
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Fix For: hbase-operator-tools-2.0.0
>
> Attachments: HBASE-23039.001.patch
>
>
>
> The recursiveFlag is wrong:
> {code:java}
> boolean overrideFlag = commandLine.hasOption(override.getOpt());
> boolean recursiveFlag = commandLine.hasOption(override.getOpt());
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23039) HBCK2 bypass -r command does not work
[
https://issues.apache.org/jira/browse/HBASE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi updated HBASE-23039:
--
Affects Version/s: hbase-operator-tools-1.0.0
> HBCK2 bypass -r command does not work
> -
>
> Key: HBASE-23039
> URL: https://issues.apache.org/jira/browse/HBASE-23039
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Affects Versions: hbase-operator-tools-1.0.0
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Attachments: HBASE-23039.001.patch
>
>
>
> The recursiveFlag is wrong:
> {code:java}
> boolean overrideFlag = commandLine.hasOption(override.getOpt());
> boolean recursiveFlag = commandLine.hasOption(override.getOpt());
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] infraio commented on issue #595: HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature …
infraio commented on issue #595: HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature … URL: https://github.com/apache/hbase/pull/595#issuecomment-532562066 The master ui show RSGroup info by the coprocessor "RSGroupEndpoint" config. Need to update too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-22847) Release 2.2.1
[ https://issues.apache.org/jira/browse/HBASE-22847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-22847. Fix Version/s: 2.2.1 Resolution: Fixed > Release 2.2.1 > - > > Key: HBASE-22847 > URL: https://issues.apache.org/jira/browse/HBASE-22847 > Project: HBase > Issue Type: Umbrella >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 2.2.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23028) Add 2.2.1 to download page
[ https://issues.apache.org/jira/browse/HBASE-23028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23028. Fix Version/s: 3.0.0 Resolution: Fixed > Add 2.2.1 to download page > -- > > Key: HBASE-23028 > URL: https://issues.apache.org/jira/browse/HBASE-23028 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23042) Parameters are incorrect in procedures jsp
Yi Mei created HBASE-23042: -- Summary: Parameters are incorrect in procedures jsp Key: HBASE-23042 URL: https://issues.apache.org/jira/browse/HBASE-23042 Project: HBase Issue Type: Bug Reporter: Yi Mei In procedures jps, the parameters of table name, region start end keys are wrong, please see the first picture. This is because all bytes params are encoded in base64. It is confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-21056) Findbugs false positive: BucketCache.persistToFile may fail to clean up java.io.OutputStream
[
https://issues.apache.org/jira/browse/HBASE-21056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reopened HBASE-21056:
---
Reopening to backport to branch-2.
> Findbugs false positive: BucketCache.persistToFile may fail to clean up
> java.io.OutputStream
> -
>
> Key: HBASE-21056
> URL: https://issues.apache.org/jira/browse/HBASE-21056
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-21056.0.patch
>
>
> Found by the nightly job via FindBugs:
> {code}
> FindBugs module:hbase-server
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.persistToFile() may fail
> to clean up java.io.OutputStream Obligation to clean up resource created at
> BucketCache.java:up java.io.OutputStream Obligation to clean up resource
> created at BucketCache.java:[line 1089] is not discharged
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-21056) Findbugs false positive: BucketCache.persistToFile may fail to clean up java.io.OutputStream
[
https://issues.apache.org/jira/browse/HBASE-21056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932177#comment-16932177
]
Peter Somogyi commented on HBASE-21056:
---
Nightly build on branch-2 failed with the same findbugs warning:
||Code||Warning||
|OBL|org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.persistToFile() may
fail to clean up java.io.OutputStream|
> Findbugs false positive: BucketCache.persistToFile may fail to clean up
> java.io.OutputStream
> -
>
> Key: HBASE-21056
> URL: https://issues.apache.org/jira/browse/HBASE-21056
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-21056.0.patch
>
>
> Found by the nightly job via FindBugs:
> {code}
> FindBugs module:hbase-server
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.persistToFile() may fail
> to clean up java.io.OutputStream Obligation to clean up resource created at
> BucketCache.java:up java.io.OutputStream Obligation to clean up resource
> created at BucketCache.java:[line 1089] is not discharged
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23042) Parameters are incorrect in procedures jsp
[ https://issues.apache.org/jira/browse/HBASE-23042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Mei updated HBASE-23042: --- Attachment: 1.png > Parameters are incorrect in procedures jsp > -- > > Key: HBASE-23042 > URL: https://issues.apache.org/jira/browse/HBASE-23042 > Project: HBase > Issue Type: Bug >Reporter: Yi Mei >Priority: Major > Attachments: 1.png > > > In procedures jps, the parameters of table name, region start end keys are > wrong, please see the first picture. > This is because all bytes params are encoded in base64. It is confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932179#comment-16932179
]
Andrew Purtell commented on HBASE-22978:
If need be but seems premature because the work hasn’t even started yet.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
> is fine. Response too slow logging is best effort. If we can detect this make
> a note of it in the log file. Provide a tool for parsing, dumping, filtering,
> and pretty printing the slow logs written to HDFS. The tool and the shell can
> share and reuse some utility classes and methods for accomplishing that.
> —
> New shell commands:
> {{get_slow_responses [ ... , ] [ , \{
> } ]}}
> Retrieve, decode, and pretty print the contents of the too slow response ring
> buffer maintained by the given list of servers; or all servers in the cluster
> if no list is provided. Optionally provide a map of parameters for filtering
> as additional argument. The TABLE filter, which expects a string containing a
> table name, will include only entries pertaining to that table. The REGION
> filter, which expects a string containing an encoded region name, will
> include only entries pertaining to that region. The CLIENT_IP filter, which
> expects a string containing an IP address, will include only entries
> pertaining to that client. The USER filter, which expects a str
[jira] [Resolved] (HBASE-21056) Findbugs false positive: BucketCache.persistToFile may fail to clean up java.io.OutputStream
[
https://issues.apache.org/jira/browse/HBASE-21056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-21056.
---
Fix Version/s: 2.3.0
Resolution: Fixed
Cherry-picked d159b1f8bb to branch-2.
> Findbugs false positive: BucketCache.persistToFile may fail to clean up
> java.io.OutputStream
> -
>
> Key: HBASE-21056
> URL: https://issues.apache.org/jira/browse/HBASE-21056
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-21056.0.patch
>
>
> Found by the nightly job via FindBugs:
> {code}
> FindBugs module:hbase-server
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.persistToFile() may fail
> to clean up java.io.OutputStream Obligation to clean up resource created at
> BucketCache.java:up java.io.OutputStream Obligation to clean up resource
> created at BucketCache.java:[line 1089] is not discharged
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23043) TestWALEntryStream times out
Peter Somogyi created HBASE-23043: - Summary: TestWALEntryStream times out Key: HBASE-23043 URL: https://issues.apache.org/jira/browse/HBASE-23043 Project: HBase Issue Type: Bug Components: wal Affects Versions: 3.0.0, 2.3.0, 2.1.7, 2.2.2 Reporter: Peter Somogyi Assignee: Peter Somogyi Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 TestWALEntryStream#testDifferentCounts times out almost every time (90%+). On my machine the test runs in 9,5 minutes but on ASF infra it reaches the 720s timeout. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #634: HBASE-23041 Should not show split parent regions in HBCK report's unk…
Apache-HBase commented on issue #634: HBASE-23041 Should not show split parent regions in HBCK report's unk… URL: https://github.com/apache/hbase/pull/634#issuecomment-532592772 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :yellow_heart: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 5m 51s | master passed | | :green_heart: | compile | 1m 0s | master passed | | :green_heart: | checkstyle | 1m 29s | master passed | | :green_heart: | shadedjars | 4m 55s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 37s | master passed | | :blue_heart: | spotbugs | 4m 32s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 4m 30s | master passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 5m 23s | the patch passed | | :green_heart: | compile | 0m 56s | the patch passed | | :green_heart: | javac | 0m 56s | the patch passed | | :green_heart: | checkstyle | 1m 29s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 4m 57s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 17m 15s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 36s | the patch passed | | :green_heart: | findbugs | 4m 34s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 260m 28s | hbase-server in the patch failed. | | :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 322m 41s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-634/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/634 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 96a11ffb6a5f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-634/out/precommit/personality/provided.sh | | git revision | master / cb62f73406 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-634/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-634/1/testReport/ | | Max. process+thread count | 4726 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-634/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ZhaoBQ commented on issue #633: HBASE-22890 Verify the file integrity in persistent IOEngine
ZhaoBQ commented on issue #633: HBASE-22890 Verify the file integrity in persistent IOEngine URL: https://github.com/apache/hbase/pull/633#issuecomment-532597228 Hi, @anoopsjohn, I have modified. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #633: HBASE-22890 Verify the file integrity in persistent IOEngine
Apache-HBase commented on issue #633: HBASE-22890 Verify the file integrity in persistent IOEngine URL: https://github.com/apache/hbase/pull/633#issuecomment-532612182 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 42s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-1 Compile Tests _ | | :green_heart: | mvninstall | 8m 4s | branch-1 passed | | :green_heart: | compile | 1m 58s | branch-1 passed with JDK v1.8.0_222 | | :broken_heart: | compile | 0m 22s | hbase-server in branch-1 failed with JDK v1.7.0_232. | | :green_heart: | checkstyle | 1m 33s | branch-1 passed | | :green_heart: | shadedjars | 2m 46s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 37s | branch-1 passed with JDK v1.8.0_222 | | :green_heart: | javadoc | 0m 42s | branch-1 passed with JDK v1.7.0_232 | | :blue_heart: | spotbugs | 3m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 3m 9s | branch-1 passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 0m 56s | the patch passed | | :green_heart: | compile | 1m 46s | the patch passed with JDK v1.8.0_222 | | :green_heart: | javac | 1m 46s | the patch passed | | :broken_heart: | compile | 0m 21s | hbase-server in the patch failed with JDK v1.7.0_232. | | :broken_heart: | javac | 0m 21s | hbase-server in the patch failed with JDK v1.7.0_232. | | :broken_heart: | checkstyle | 1m 29s | hbase-server: The patch generated 2 new + 48 unchanged - 4 fixed = 50 total (was 52) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 2m 45s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 5m 3s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | :green_heart: | javadoc | 0m 34s | the patch passed with JDK v1.8.0_222 | | :green_heart: | javadoc | 0m 41s | the patch passed with JDK v1.7.0_232 | | :broken_heart: | findbugs | 3m 5s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | :broken_heart: | unit | 32m 4s | hbase-server in the patch failed. | | :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 72m 27s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.retrieveFromFile(int[]) ignores result of java.io.ObjectInputStream.read(byte[]) At BucketCache.java: At BucketCache.java:[line 1084] | | Failed junit tests | hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/633 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 3a9df52e28e5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-633/out/precommit/personality/provided.sh | | git revision | branch-1 / 5c4d8e0 | | Default Java | 1.7.0_232 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_222 /usr/lib/jvm/zulu-7-amd64:1.7.0_232 | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/3/artifact/out/branch-compile-hbase-server-jdk1.7.0_232.txt | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/3/artifact/out/patch-compile-hbase-server-jdk1.7.0_232.txt | | javac | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/3/artifact/out/patch-compile-hbase-server-jdk1.7.0_232.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/3/artifact/out/diff-checkstyle-hbase-server.txt | | findbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/3/artifact/out/new-findbugs-hbase-server.ht
[jira] [Created] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
Guanghao Zhang created HBASE-23044:
--
Summary: CatalogJanitor#cleanMergeQualifier may clean wrong parent
regions
Key: HBASE-23044
URL: https://issues.apache.org/jira/browse/HBASE-23044
Project: HBase
Issue Type: Improvement
Reporter: Guanghao Zhang
2019-09-17,19:42:40,539 INFO [PEWorker-1]
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
state=SUCCESS; GCMultipleMergedRegionsProcedure
child={color:red}647600d28633bb2fe06b40682bab0593{color},
parents:[81b6fc3c560a00692bc7c3cd266a626a], [472500358997b0dc8f0002ec86593dcf]
in 2.6470sec
2019-09-17,19:59:54,179 INFO [PEWorker-6]
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
state=SUCCESS; GCMultipleMergedRegionsProcedure
child={color:red}647600d28633bb2fe06b40682bab0593{color},
parents:[9c52f24e0a9cc9b4959c1ebdfea29d64], [a623f298870df5581bcfae7f83311b33]
in 1.0340sec
The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color} but
the parent regions are different.
MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
for the regions to merge.
{code:java}
for (RegionInfo ri: this.regionsToMerge) {
if (!catalogJanitor.cleanMergeQualifier(ri)) {
String msg = "Skip merging " +
RegionInfo.getShortNameToLog(regionsToMerge) +
", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
merge qualifier";
LOG.warn(msg);
throw new MergeRegionException(msg);
}
{code}
If region A and B merge to C, region D and E merge to F. When merge C and F, it
will try to cleanMergeQualifier for C and F. catalogJanitor.cleanMergeQualifier
for region C succeed but catalogJanitor.cleanMergeQualifier for region F failed
as there are references in region F.
When merge C and F again, it will try to cleanMergeQualifier for C and F again.
But MetaTableAccessor.getMergeRegions will get wrong parents now. It use scan
with filter to scan result. But region C's MergeQualifier already was deleted
before. Then the scan will return a wrong result, may be anther region..
{code:java}
public boolean cleanMergeQualifier(final RegionInfo region) throws IOException {
// Get merge regions if it is a merged region and already has merge
qualifier
List parents =
MetaTableAccessor.getMergeRegions(this.services.getConnection(),
region.getRegionName());
if (parents == null || parents.isEmpty()) {
// It doesn't have merge qualifier, no need to clean
return true;
}
return cleanMergeRegion(region, parents);
}
public static List getMergeRegions(Connection connection, byte[]
regionName)
throws IOException {
return getMergeRegions(getMergeRegionsRaw(connection, regionName));
}
private static Cell [] getMergeRegionsRaw(Connection connection, byte []
regionName)
throws IOException {
Scan scan = new Scan().withStartRow(regionName).
setOneRowLimit().
readVersions(1).
addFamily(HConstants.CATALOG_FAMILY).
setFilter(new QualifierFilter(CompareOperator.EQUAL,
new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
".*")));
try (Table m = getMetaHTable(connection); ResultScanner scanner =
m.getScanner(scan)) {
// Should be only one result in this scanner if any.
Result result = scanner.next();
if (result == null) {
return null;
}
// Should be safe to just return all Cells found since we had filter in
place.
// All values should be RegionInfos or something wrong.
return result.rawCells();
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-23044:
---
Priority: Critical (was: Major)
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Priority: Critical
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-23045) currentPath may be stitched in a loop in replication source code.
[ https://issues.apache.org/jira/browse/HBASE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23045 started by kangkang.guo. > currentPath may be stitched in a loop in replication source code. > -- > > Key: HBASE-23045 > URL: https://issues.apache.org/jira/browse/HBASE-23045 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.6.1 >Reporter: kangkang.guo >Assignee: kangkang.guo >Priority: Critical > Fix For: 1.2.6.1 > > Attachments: 0001-fix-ReplicationSource-bug.patch > > > When the openReader encounters a FileNotFoundException, we may go to all > possible directories to find the current hlog. When found, the path may be > wrong, and it is looped together. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23045) currentPath may be stitched in a loop in replication source code.
kangkang.guo created HBASE-23045: Summary: currentPath may be stitched in a loop in replication source code. Key: HBASE-23045 URL: https://issues.apache.org/jira/browse/HBASE-23045 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 1.2.6.1 Reporter: kangkang.guo Assignee: kangkang.guo Fix For: 1.2.6.1 Attachments: 0001-fix-ReplicationSource-bug.patch When the openReader encounters a FileNotFoundException, we may go to all possible directories to find the current hlog. When found, the path may be wrong, and it is looped together. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #635: HBASE-23040 give friendly message about unknown server when running unload for RegionMover.
Apache-HBase commented on issue #635: HBASE-23040 give friendly message about unknown server when running unload for RegionMover. URL: https://github.com/apache/hbase/pull/635#issuecomment-532617061 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 2m 32s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :yellow_heart: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 9m 27s | master passed | | :green_heart: | compile | 1m 19s | master passed | | :green_heart: | checkstyle | 1m 42s | master passed | | :green_heart: | shadedjars | 6m 14s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 52s | master passed | | :blue_heart: | spotbugs | 5m 27s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 5m 25s | master passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 7m 35s | the patch passed | | :green_heart: | compile | 1m 18s | the patch passed | | :green_heart: | javac | 1m 18s | the patch passed | | :broken_heart: | checkstyle | 1m 51s | hbase-server: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 5m 47s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 22m 58s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 48s | the patch passed | | :green_heart: | findbugs | 5m 2s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 232m 52s | hbase-server in the patch failed. | | :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 314m 9s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-635/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/635 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 863d6fdc2361 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-635/out/precommit/personality/provided.sh | | git revision | master / cb62f73406 | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-635/1/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-635/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-635/1/testReport/ | | Max. process+thread count | 5073 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-635/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23045) currentPath may be stitched in a loop in replication source code.
[ https://issues.apache.org/jira/browse/HBASE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kangkang.guo updated HBASE-23045: - Attachment: 0001-fix-ReplicationSource-bug.patch Status: Patch Available (was: In Progress) > currentPath may be stitched in a loop in replication source code. > -- > > Key: HBASE-23045 > URL: https://issues.apache.org/jira/browse/HBASE-23045 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.6.1 >Reporter: kangkang.guo >Assignee: kangkang.guo >Priority: Critical > Fix For: 1.2.6.1 > > Attachments: 0001-fix-ReplicationSource-bug.patch, > 0001-fix-ReplicationSource-bug.patch > > > When the openReader encounters a FileNotFoundException, we may go to all > possible directories to find the current hlog. When found, the path may be > wrong, and it is looped together. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23045) currentPath may be stitched in a loop in replication source code.
[
https://issues.apache.org/jira/browse/HBASE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932277#comment-16932277
]
HBase QA commented on HBASE-23045:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HBASE-23045 does not apply to master. Rebase required? Wrong
Branch? See
https://yetus.apache.org/documentation/in-progress/precommit-patchnames for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-23045 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12980596/0001-fix-ReplicationSource-bug.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/907/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.0 https://yetus.apache.org |
This message was automatically generated.
> currentPath may be stitched in a loop in replication source code.
> --
>
> Key: HBASE-23045
> URL: https://issues.apache.org/jira/browse/HBASE-23045
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.2.6.1
>Reporter: kangkang.guo
>Assignee: kangkang.guo
>Priority: Critical
> Fix For: 1.2.6.1
>
> Attachments: 0001-fix-ReplicationSource-bug.patch,
> 0001-fix-ReplicationSource-bug.patch
>
>
> When the openReader encounters a FileNotFoundException, we may go to all
> possible directories to find the current hlog. When found, the path may be
> wrong, and it is looped together.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] meszibalu commented on a change in pull request #592: HBASE-22982: region server suspend/resume and graceful rolling restart actions
meszibalu commented on a change in pull request #592: HBASE-22982: region
server suspend/resume and graceful rolling restart actions
URL: https://github.com/apache/hbase/pull/592#discussion_r325592713
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/DistributedHBaseCluster.java
##
@@ -126,6 +126,20 @@ public void waitForRegionServerToStop(ServerName
serverName, long timeout) throw
waitForServiceToStop(ServiceType.HBASE_REGIONSERVER, serverName, timeout);
}
+ @Override
+ public void suspendRegionServer(ServerName serverName) throws IOException {
+LOG.info("Suspend RS: " + serverName.getServerName());
Review comment:
Please use parametrized ({}) logging **everywhere.**
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] meszibalu commented on a change in pull request #592: HBASE-22982: region server suspend/resume and graceful rolling restart actions
meszibalu commented on a change in pull request #592: HBASE-22982: region
server suspend/resume and graceful rolling restart actions
URL: https://github.com/apache/hbase/pull/592#discussion_r325593445
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/Action.java
##
@@ -269,6 +298,19 @@ protected void forceBalancer() throws Exception {
}
}
+ protected void setBalancer(boolean onOrOff, boolean synchronous) throws
Exception {
+Admin admin = this.context.getHBaseIntegrationTestingUtility().getAdmin();
+boolean result = false;
+try {
+ result = admin.balancerSwitch(onOrOff, synchronous);
+} catch (Exception e) {
+ LOG.warn("Got exception while switching balancee ", e);
Review comment:
Typo (balancee).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] meszibalu commented on a change in pull request #592: HBASE-22982: region server suspend/resume and graceful rolling restart actions
meszibalu commented on a change in pull request #592: HBASE-22982: region
server suspend/resume and graceful rolling restart actions
URL: https://github.com/apache/hbase/pull/592#discussion_r325595283
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/RollingBatchSuspendResumeRsAction.java
##
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.chaos.actions;
+
+import java.io.IOException;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.chaos.monkies.PolicyBasedChaosMonkey;
+import org.apache.hadoop.hbase.util.Threads;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Suspend then resume a ratio of the regionservers in a rolling fashion. At
each step, either
+ * suspend a server, or resume one, sleeping (sleepTime) in between steps. The
parameter
+ * maxSuspendedServers limits the maximum number of servers that can be down
at the same time
+ * during rolling restarts.
+ */
+public class RollingBatchSuspendResumeRsAction extends Action {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RollingBatchSuspendResumeRsAction.class);
+ private float ratio;
+ private long sleepTime;
+ private int maxSuspendedServers; // number of maximum suspended servers at
any given time.
+
+ public RollingBatchSuspendResumeRsAction(long sleepTime, float ratio) {
+this(sleepTime, ratio, 5);
+ }
+
+ public RollingBatchSuspendResumeRsAction(long sleepTime, float ratio, int
maxSuspendedServers) {
+this.ratio = ratio;
+this.sleepTime = sleepTime;
+this.maxSuspendedServers = maxSuspendedServers;
+ }
+
+ enum SuspendOrResume {
+SUSPEND, RESUME
+ }
+
+ @Override public void perform() throws Exception {
+LOG.info(String.format("Performing action: Rolling batch restarting %d%%
of region servers",
+(int) (ratio * 100)));
+List selectedServers = selectServers();
+
+Queue serversToBeSuspended = new LinkedList<>(selectedServers);
+Queue suspendedServers = new LinkedList<>();
+
+// loop while there are servers to be suspended or suspended servers to be
resumed
+while ((!serversToBeSuspended.isEmpty() || !suspendedServers.isEmpty()) &&
!context
+.isStopping()) {
+ SuspendOrResume action;
+
+ if (serversToBeSuspended.isEmpty()) { // no more servers to suspend
+action = SuspendOrResume.RESUME;
+ } else if (suspendedServers.isEmpty()) {
+action = SuspendOrResume.SUSPEND; // no more servers to resume
+ } else if (suspendedServers.size() >= maxSuspendedServers) {
+// we have too many suspended servers. Don't suspend any more
+action = SuspendOrResume.RESUME;
+ } else {
+// do a coin toss
+action = RandomUtils.nextBoolean() ? SuspendOrResume.SUSPEND :
SuspendOrResume.RESUME;
+ }
+
+ ServerName server;
+ switch (action) {
+case SUSPEND:
+ server = serversToBeSuspended.remove();
+ try {
+suspendRs(server);
+ } catch (org.apache.hadoop.util.Shell.ExitCodeException e) {
+LOG.info("Problem suspending but presume successful; code=" +
e.getExitCode(), e);
+ }
+ suspendedServers.add(server);
+ break;
+case RESUME:
+ server = suspendedServers.remove();
+ try {
+resumeRs(server);
+ } catch (org.apache.hadoop.util.Shell.ExitCodeException e) {
Review comment:
Add import?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reassigned HBASE-23044:
--
Assignee: Guanghao Zhang
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #603: HBASE-22965 RS Crash due to DBE reference to an reused ByteBuff
Apache-HBase commented on issue #603: HBASE-22965 RS Crash due to DBE reference to an reused ByteBuff URL: https://github.com/apache/hbase/pull/603#issuecomment-532629721 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | :blue_heart: | mvndep | 0m 36s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 5m 17s | master passed | | :green_heart: | compile | 1m 20s | master passed | | :green_heart: | checkstyle | 1m 45s | master passed | | :green_heart: | shadedjars | 4m 42s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 1m 1s | master passed | | :blue_heart: | spotbugs | 4m 16s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 5m 4s | master passed | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 15s | Maven dependency ordering for patch | | :green_heart: | mvninstall | 4m 53s | the patch passed | | :green_heart: | compile | 1m 19s | the patch passed | | :green_heart: | javac | 1m 19s | the patch passed | | :green_heart: | checkstyle | 1m 45s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 4m 30s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 15m 34s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 56s | the patch passed | | :green_heart: | findbugs | 5m 3s | the patch passed | ||| _ Other Tests _ | | :green_heart: | unit | 3m 8s | hbase-common in the patch passed. | | :broken_heart: | unit | 170m 36s | hbase-server in the patch failed. | | :green_heart: | asflicense | 1m 2s | The patch does not generate ASF License warnings. | | | | 235m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-603/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/603 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux bab7f54bf1a2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-603/out/precommit/personality/provided.sh | | git revision | master / cb62f73406 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-603/4/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-603/4/testReport/ | | Max. process+thread count | 5118 (vs. ulimit of 1) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-603/4/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #361: HBase-22027: Split non-MR related parts of TokenUtil off into a Clien…
Apache-HBase commented on issue #361: HBase-22027: Split non-MR related parts of TokenUtil off into a Clien… URL: https://github.com/apache/hbase/pull/361#issuecomment-532633235 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ master Compile Tests _ | | :blue_heart: | mvndep | 0m 35s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 5m 43s | master passed | | :green_heart: | compile | 1m 22s | master passed | | :green_heart: | checkstyle | 2m 3s | master passed | | :green_heart: | shadedjars | 5m 16s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 1m 6s | master passed | | :blue_heart: | spotbugs | 4m 24s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 5m 40s | master passed | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 13s | Maven dependency ordering for patch | | :green_heart: | mvninstall | 5m 33s | the patch passed | | :green_heart: | compile | 1m 26s | the patch passed | | :green_heart: | javac | 1m 26s | the patch passed | | :broken_heart: | checkstyle | 0m 35s | hbase-client: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 5m 1s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 16m 58s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 57s | the patch passed | | :green_heart: | findbugs | 5m 48s | the patch passed | ||| _ Other Tests _ | | :green_heart: | unit | 1m 49s | hbase-client in the patch passed. | | :green_heart: | unit | 156m 57s | hbase-server in the patch passed. | | :green_heart: | asflicense | 0m 47s | The patch does not generate ASF License warnings. | | | | 226m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-361/154/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/361 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4580648ed206 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-361/out/precommit/personality/provided.sh | | git revision | master / cb62f73406 | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-361/154/artifact/out/diff-checkstyle-hbase-client.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-361/154/testReport/ | | Max. process+thread count | 4496 (vs. ulimit of 1) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-361/154/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23043) TestWALEntryStream times out
[
https://issues.apache.org/jira/browse/HBASE-23043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932303#comment-16932303
]
Peter Somogyi commented on HBASE-23043:
---
After HBASE-22963 the test runtime went up drastically from 82 seconds to 550.
The performance issue is caused by these flags in pom.xml:
{{-Dorg.apache.hbase.thirdparty.io.netty.leakDetection.level=paranoid}}
[~Apache9], what do you think about removing the PARANOID level?
> TestWALEntryStream times out
>
>
> Key: HBASE-23043
> URL: https://issues.apache.org/jira/browse/HBASE-23043
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> TestWALEntryStream#testDifferentCounts times out almost every time (90%+).
> On my machine the test runs in 9,5 minutes but on ASF infra it reaches the
> 720s timeout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22514) Move rsgroup feature into core of HBase
[
https://issues.apache.org/jira/browse/HBASE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932304#comment-16932304
]
Hudson commented on HBASE-22514:
Results for branch HBASE-22514
[build #115 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/115/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/115//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/115//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/115//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/115//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Move rsgroup feature into core of HBase
> ---
>
> Key: HBASE-22514
> URL: https://issues.apache.org/jira/browse/HBASE-22514
> Project: HBase
> Issue Type: Umbrella
> Components: Admin, Client, rsgroup
>Reporter: Yechao Chen
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-22514.master.001.patch,
> image-2019-05-31-18-25-38-217.png
>
>
> The class RSGroupAdminClient is not public
> we need to use java api RSGroupAdminClient to manager RSG
> so RSGroupAdminClient should be public
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23043) TestWALEntryStream times out
[ https://issues.apache.org/jira/browse/HBASE-23043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932308#comment-16932308 ] Peter Somogyi commented on HBASE-23043: --- With ADVANCED level the performance is not impacted. Description of the different levels: https://netty.io/wiki/reference-counted-objects.html#leak-detection-levels > TestWALEntryStream times out > > > Key: HBASE-23043 > URL: https://issues.apache.org/jira/browse/HBASE-23043 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 3.0.0, 2.3.0, 2.1.7, 2.2.2 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > > TestWALEntryStream#testDifferentCounts times out almost every time (90%+). > On my machine the test runs in 9,5 minutes but on ASF infra it reaches the > 720s timeout. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] petersomogyi opened a new pull request #636: HBASE-23043 Use 'advanced' Netty leak detection
petersomogyi opened a new pull request #636: HBASE-23043 Use 'advanced' Netty leak detection URL: https://github.com/apache/hbase/pull/636 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] infraio opened a new pull request #637: HBASE-23044 CatalogJanitor#cleanMergeQualifier may clean wrong parent…
infraio opened a new pull request #637: HBASE-23044 CatalogJanitor#cleanMergeQualifier may clean wrong parent… URL: https://github.com/apache/hbase/pull/637 … regions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #592: HBASE-22982: region server suspend/resume and graceful rolling restart actions
BukrosSzabolcs commented on a change in pull request #592: HBASE-22982: region
server suspend/resume and graceful rolling restart actions
URL: https://github.com/apache/hbase/pull/592#discussion_r325618363
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/DistributedHBaseCluster.java
##
@@ -126,6 +126,20 @@ public void waitForRegionServerToStop(ServerName
serverName, long timeout) throw
waitForServiceToStop(ServiceType.HBASE_REGIONSERVER, serverName, timeout);
}
+ @Override
+ public void suspendRegionServer(ServerName serverName) throws IOException {
+LOG.info("Suspend RS: " + serverName.getServerName());
Review comment:
Thanks, I'll change this in every affected class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #592: HBASE-22982: region server suspend/resume and graceful rolling restart actions
BukrosSzabolcs commented on a change in pull request #592: HBASE-22982: region
server suspend/resume and graceful rolling restart actions
URL: https://github.com/apache/hbase/pull/592#discussion_r325618733
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/RollingBatchSuspendResumeRsAction.java
##
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.chaos.actions;
+
+import java.io.IOException;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.chaos.monkies.PolicyBasedChaosMonkey;
+import org.apache.hadoop.hbase.util.Threads;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Suspend then resume a ratio of the regionservers in a rolling fashion. At
each step, either
+ * suspend a server, or resume one, sleeping (sleepTime) in between steps. The
parameter
+ * maxSuspendedServers limits the maximum number of servers that can be down
at the same time
+ * during rolling restarts.
+ */
+public class RollingBatchSuspendResumeRsAction extends Action {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RollingBatchSuspendResumeRsAction.class);
+ private float ratio;
+ private long sleepTime;
+ private int maxSuspendedServers; // number of maximum suspended servers at
any given time.
+
+ public RollingBatchSuspendResumeRsAction(long sleepTime, float ratio) {
+this(sleepTime, ratio, 5);
+ }
+
+ public RollingBatchSuspendResumeRsAction(long sleepTime, float ratio, int
maxSuspendedServers) {
+this.ratio = ratio;
+this.sleepTime = sleepTime;
+this.maxSuspendedServers = maxSuspendedServers;
+ }
+
+ enum SuspendOrResume {
+SUSPEND, RESUME
+ }
+
+ @Override public void perform() throws Exception {
+LOG.info(String.format("Performing action: Rolling batch restarting %d%%
of region servers",
+(int) (ratio * 100)));
+List selectedServers = selectServers();
+
+Queue serversToBeSuspended = new LinkedList<>(selectedServers);
+Queue suspendedServers = new LinkedList<>();
+
+// loop while there are servers to be suspended or suspended servers to be
resumed
+while ((!serversToBeSuspended.isEmpty() || !suspendedServers.isEmpty()) &&
!context
+.isStopping()) {
+ SuspendOrResume action;
+
+ if (serversToBeSuspended.isEmpty()) { // no more servers to suspend
+action = SuspendOrResume.RESUME;
+ } else if (suspendedServers.isEmpty()) {
+action = SuspendOrResume.SUSPEND; // no more servers to resume
+ } else if (suspendedServers.size() >= maxSuspendedServers) {
+// we have too many suspended servers. Don't suspend any more
+action = SuspendOrResume.RESUME;
+ } else {
+// do a coin toss
+action = RandomUtils.nextBoolean() ? SuspendOrResume.SUSPEND :
SuspendOrResume.RESUME;
+ }
+
+ ServerName server;
+ switch (action) {
+case SUSPEND:
+ server = serversToBeSuspended.remove();
+ try {
+suspendRs(server);
+ } catch (org.apache.hadoop.util.Shell.ExitCodeException e) {
+LOG.info("Problem suspending but presume successful; code=" +
e.getExitCode(), e);
+ }
+ suspendedServers.add(server);
+ break;
+case RESUME:
+ server = suspendedServers.remove();
+ try {
+resumeRs(server);
+ } catch (org.apache.hadoop.util.Shell.ExitCodeException e) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] BukrosSzabolcs commented on a change in pull request #592: HBASE-22982: region server suspend/resume and graceful rolling restart actions
BukrosSzabolcs commented on a change in pull request #592: HBASE-22982: region
server suspend/resume and graceful rolling restart actions
URL: https://github.com/apache/hbase/pull/592#discussion_r325618522
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/Action.java
##
@@ -269,6 +298,19 @@ protected void forceBalancer() throws Exception {
}
}
+ protected void setBalancer(boolean onOrOff, boolean synchronous) throws
Exception {
+Admin admin = this.context.getHBaseIntegrationTestingUtility().getAdmin();
+boolean result = false;
+try {
+ result = admin.balancerSwitch(onOrOff, synchronous);
+} catch (Exception e) {
+ LOG.warn("Got exception while switching balancee ", e);
Review comment:
Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] busbey commented on issue #635: HBASE-23040 give friendly message about unknown server when running unload for RegionMover.
busbey commented on issue #635: HBASE-23040 give friendly message about unknown server when running unload for RegionMover. URL: https://github.com/apache/hbase/pull/635#issuecomment-532642144 I'll clean up checkstyle on commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] asfgit closed pull request #635: HBASE-23040 give friendly message about unknown server when running unload for RegionMover.
asfgit closed pull request #635: HBASE-23040 give friendly message about unknown server when running unload for RegionMover. URL: https://github.com/apache/hbase/pull/635 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23046) Remove compatibility case from truncate command
Peter Somogyi created HBASE-23046:
-
Summary: Remove compatibility case from truncate command
Key: HBASE-23046
URL: https://issues.apache.org/jira/browse/HBASE-23046
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 3.0.0, 2.3.0
Reporter: Peter Somogyi
Assignee: Peter Somogyi
The truncate and truncate_preserve commands in shell have a compatibility block
to handle the case when Master does not have truncate command.
This was added in HBASE-8332 for HBase 0.99 so it is safe to remove it now.
The current compatibility block catches DoNotRetryIOException which can hide
different kind of errors and just drops and recreates the table.
{code:ruby}
begin
puts 'Truncating table...'
@admin.truncateTable(table_name, false)
rescue => e
# Handle the compatibility case, where the truncate method doesn't exists on
the Master
raise e unless e.respond_to?(:cause) && !e.cause.nil?
rootCause = e.cause
if rootCause.is_a?(org.apache.hadoop.hbase.DoNotRetryIOException)
# Handle the compatibility case, where the truncate method doesn't exists
on the Master
puts 'Dropping table...'
@admin.deleteTable(table_name)
puts 'Creating table...'
@admin.createTable(table_description)
else
raise e
end
end
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #633: HBASE-22890 Verify the file integrity in persistent IOEngine
Apache-HBase commented on issue #633: HBASE-22890 Verify the file integrity in persistent IOEngine URL: https://github.com/apache/hbase/pull/633#issuecomment-532652827 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 46s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-1 Compile Tests _ | | :green_heart: | mvninstall | 8m 9s | branch-1 passed | | :green_heart: | compile | 1m 53s | branch-1 passed with JDK v1.8.0_222 | | :broken_heart: | compile | 0m 21s | hbase-server in branch-1 failed with JDK v1.7.0_232. | | :green_heart: | checkstyle | 1m 34s | branch-1 passed | | :green_heart: | shadedjars | 2m 46s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 37s | branch-1 passed with JDK v1.8.0_222 | | :green_heart: | javadoc | 0m 41s | branch-1 passed with JDK v1.7.0_232 | | :blue_heart: | spotbugs | 2m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 2m 49s | branch-1 passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 1m 55s | the patch passed | | :green_heart: | compile | 1m 42s | the patch passed with JDK v1.8.0_222 | | :green_heart: | javac | 1m 42s | the patch passed | | :broken_heart: | compile | 0m 23s | hbase-server in the patch failed with JDK v1.7.0_232. | | :broken_heart: | javac | 0m 23s | hbase-server in the patch failed with JDK v1.7.0_232. | | :broken_heart: | checkstyle | 1m 31s | hbase-server: The patch generated 2 new + 48 unchanged - 4 fixed = 50 total (was 52) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 2m 42s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 4m 52s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | :green_heart: | javadoc | 0m 30s | the patch passed with JDK v1.8.0_222 | | :green_heart: | javadoc | 0m 43s | the patch passed with JDK v1.7.0_232 | | :green_heart: | findbugs | 2m 49s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 31m 2s | hbase-server in the patch failed. | | :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 71m 38s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/633 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 004a9f2bb8e2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-633/out/precommit/personality/provided.sh | | git revision | branch-1 / 5c4d8e0 | | Default Java | 1.7.0_232 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_222 /usr/lib/jvm/zulu-7-amd64:1.7.0_232 | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/4/artifact/out/branch-compile-hbase-server-jdk1.7.0_232.txt | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/4/artifact/out/patch-compile-hbase-server-jdk1.7.0_232.txt | | javac | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/4/artifact/out/patch-compile-hbase-server-jdk1.7.0_232.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/4/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/4/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-633/4/testReport/ | | Max. process+thread count | 741 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/j
[jira] [Commented] (HBASE-23043) TestWALEntryStream times out
[ https://issues.apache.org/jira/browse/HBASE-23043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932365#comment-16932365 ] Duo Zhang commented on HBASE-23043: --- Seems a bit overkill, let’s change it to advanced. > TestWALEntryStream times out > > > Key: HBASE-23043 > URL: https://issues.apache.org/jira/browse/HBASE-23043 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 3.0.0, 2.3.0, 2.1.7, 2.2.2 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > > TestWALEntryStream#testDifferentCounts times out almost every time (90%+). > On my machine the test runs in 9,5 minutes but on ASF infra it reaches the > 720s timeout. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on issue #636: HBASE-23043 Use 'advanced' Netty leak detection
Apache9 commented on issue #636: HBASE-23043 Use 'advanced' Netty leak detection URL: https://github.com/apache/hbase/pull/636#issuecomment-532664540 Let's merge this soon as it effects the pre commit a lot. And checked the pre commit job https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/view/change-requests/job/PR-636/1/console No problem, the mvn test is fine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 merged pull request #636: HBASE-23043 Use 'advanced' Netty leak detection
Apache9 merged pull request #636: HBASE-23043 Use 'advanced' Netty leak detection URL: https://github.com/apache/hbase/pull/636 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-23043) TestWALEntryStream times out
[ https://issues.apache.org/jira/browse/HBASE-23043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-23043. --- Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.1+. Thanks [~psomogyi] for helping. > TestWALEntryStream times out > > > Key: HBASE-23043 > URL: https://issues.apache.org/jira/browse/HBASE-23043 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 3.0.0, 2.3.0, 2.1.7, 2.2.2 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > > TestWALEntryStream#testDifferentCounts times out almost every time (90%+). > On my machine the test runs in 9,5 minutes but on ASF infra it reaches the > 720s timeout. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #632: HBASE-23037 Make the split WAL related log more readable
Apache-HBase commented on issue #632: HBASE-23037 Make the split WAL related log more readable URL: https://github.com/apache/hbase/pull/632#issuecomment-532673014 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 1m 9s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 1s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :yellow_heart: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 5m 59s | master passed | | :green_heart: | compile | 0m 57s | master passed | | :green_heart: | checkstyle | 1m 30s | master passed | | :green_heart: | shadedjars | 4m 55s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 39s | master passed | | :blue_heart: | spotbugs | 4m 37s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 4m 35s | master passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 5m 29s | the patch passed | | :green_heart: | compile | 0m 58s | the patch passed | | :green_heart: | javac | 0m 58s | the patch passed | | :green_heart: | checkstyle | 1m 30s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 4m 55s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 17m 42s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 39s | the patch passed | | :green_heart: | findbugs | 5m 10s | the patch passed | ||| _ Other Tests _ | | :green_heart: | unit | 264m 28s | hbase-server in the patch passed. | | :green_heart: | asflicense | 2m 4s | The patch does not generate ASF License warnings. | | | | 329m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-632/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/632 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux f14bc15c73a4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-632/out/precommit/personality/provided.sh | | git revision | master / cb62f73406 | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-632/2/testReport/ | | Max. process+thread count | 5116 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-632/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932416#comment-16932416
]
Duo Zhang commented on HBASE-23044:
---
All 2.x releases are effect? This is a critical problem...
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #593: HBASE-22927 Upgrade Mockito version for jdk11
Apache9 merged pull request #593: HBASE-22927 Upgrade Mockito version for jdk11 URL: https://github.com/apache/hbase/pull/593 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-21056) Findbugs false positive: BucketCache.persistToFile may fail to clean up java.io.OutputStream
[
https://issues.apache.org/jira/browse/HBASE-21056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932457#comment-16932457
]
Hudson commented on HBASE-21056:
Results for branch branch-2
[build #2271 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2271/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2271//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2271//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2271//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Findbugs false positive: BucketCache.persistToFile may fail to clean up
> java.io.OutputStream
> -
>
> Key: HBASE-21056
> URL: https://issues.apache.org/jira/browse/HBASE-21056
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-21056.0.patch
>
>
> Found by the nightly job via FindBugs:
> {code}
> FindBugs module:hbase-server
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.persistToFile() may fail
> to clean up java.io.OutputStream Obligation to clean up resource created at
> BucketCache.java:up java.io.OutputStream Obligation to clean up resource
> created at BucketCache.java:[line 1089] is not discharged
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23046) Remove compatibility case from truncate command
[
https://issues.apache.org/jira/browse/HBASE-23046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi updated HBASE-23046:
--
Status: Patch Available (was: Open)
> Remove compatibility case from truncate command
> ---
>
> Key: HBASE-23046
> URL: https://issues.apache.org/jira/browse/HBASE-23046
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Minor
>
> The truncate and truncate_preserve commands in shell have a compatibility
> block to handle the case when Master does not have truncate command.
> This was added in HBASE-8332 for HBase 0.99 so it is safe to remove it now.
> The current compatibility block catches DoNotRetryIOException which can hide
> different kind of errors and just drops and recreates the table.
> {code:ruby}
> begin
> puts 'Truncating table...'
> @admin.truncateTable(table_name, false)
> rescue => e
> # Handle the compatibility case, where the truncate method doesn't exists
> on the Master
> raise e unless e.respond_to?(:cause) && !e.cause.nil?
> rootCause = e.cause
> if rootCause.is_a?(org.apache.hadoop.hbase.DoNotRetryIOException)
> # Handle the compatibility case, where the truncate method doesn't exists
> on the Master
> puts 'Dropping table...'
> @admin.deleteTable(table_name)
> puts 'Creating table...'
> @admin.createTable(table_description)
> else
> raise e
> end
> end
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] petersomogyi opened a new pull request #638: HBASE-23046 Remove compatibility case from truncate command
petersomogyi opened a new pull request #638: HBASE-23046 Remove compatibility case from truncate command URL: https://github.com/apache/hbase/pull/638 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] infraio merged pull request #632: HBASE-23037 Make the split WAL related log more readable
infraio merged pull request #632: HBASE-23037 Make the split WAL related log more readable URL: https://github.com/apache/hbase/pull/632 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932480#comment-16932480
]
Guanghao Zhang commented on HBASE-23044:
Yes, may cause data loss.. Introduced by HBASE-22777, only the latest
release was effected.
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-23044:
---
Affects Version/s: 2.0.6
2.2.1
2.1.6
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-22927) Upgrade mockito version for Java 11 compatibility
[ https://issues.apache.org/jira/browse/HBASE-22927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-22927. --- Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.1+. Thanks [~rabikumar.kc] for contributing, and thanks all for reviewing. > Upgrade mockito version for Java 11 compatibility > - > > Key: HBASE-22927 > URL: https://issues.apache.org/jira/browse/HBASE-22927 > Project: HBase > Issue Type: Sub-task >Reporter: Sakthi >Assignee: Rabi Kumar K C >Priority: Major > Labels: jdk11 > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > > Pasting the discussion from HBASE-22534 here: > "Currently mockito-core version is at 2.1.0. According to > [https://github.com/mockito/mockito/blob/release/2.x/doc/release-notes/official.md], > looks like Java 11 compatibility was introduced in 2.19+. And 2.23.2 claims > to have full java 11 support after byte-buddy fix etc." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22927) Upgrade mockito version for Java 11 compatibility
[ https://issues.apache.org/jira/browse/HBASE-22927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-22927: -- Fix Version/s: 2.2.2 2.1.7 2.3.0 3.0.0 > Upgrade mockito version for Java 11 compatibility > - > > Key: HBASE-22927 > URL: https://issues.apache.org/jira/browse/HBASE-22927 > Project: HBase > Issue Type: Sub-task >Reporter: Sakthi >Assignee: Rabi Kumar K C >Priority: Major > Labels: jdk11 > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > > Pasting the discussion from HBASE-22534 here: > "Currently mockito-core version is at 2.1.0. According to > [https://github.com/mockito/mockito/blob/release/2.x/doc/release-notes/official.md], > looks like Java 11 compatibility was introduced in 2.19+. And 2.23.2 claims > to have full java 11 support after byte-buddy fix etc." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22975) Add read and write QPS metrics at server level and table level
[ https://issues.apache.org/jira/browse/HBASE-22975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932497#comment-16932497 ] chenxu commented on HBASE-22975: Does hbase-15518 already contain similar features? > Add read and write QPS metrics at server level and table level > -- > > Key: HBASE-22975 > URL: https://issues.apache.org/jira/browse/HBASE-22975 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 2.2.0, 1.4.10, master >Reporter: zbq.dean >Priority: Major > Attachments: readQPS.png, writeQPS.png > > > Use HBase‘s existing class DropwizardMeter to collect read and write QPS. The > collected location is the same as metrics readRequestsCount and > writeRequestsCount. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] petersomogyi commented on issue #638: HBASE-23046 Remove compatibility case from truncate command
petersomogyi commented on issue #638: HBASE-23046 Remove compatibility case from truncate command URL: https://github.com/apache/hbase/pull/638#issuecomment-532711314 I don't think upgrade works from pre-1.0 to 2+ cluster. I can add a release note to the Jira if you think it's needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23045) currentPath may be stitched in a loop in replication source code.
[ https://issues.apache.org/jira/browse/HBASE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932515#comment-16932515 ] Wellington Chevreuil commented on HBASE-23045: -- Hi [~gk_coder], can you provide a UT or IT that simulates the error scenario described? Also, please name your patch properly as [described in the ref guide|https://hbase.apache.org/book.html#submitting.patches], so that it can trigger the proper validation jobs. > currentPath may be stitched in a loop in replication source code. > -- > > Key: HBASE-23045 > URL: https://issues.apache.org/jira/browse/HBASE-23045 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.6.1 >Reporter: kangkang.guo >Assignee: kangkang.guo >Priority: Critical > Fix For: 1.2.6.1 > > Attachments: 0001-fix-ReplicationSource-bug.patch, > 0001-fix-ReplicationSource-bug.patch > > > When the openReader encounters a FileNotFoundException, we may go to all > possible directories to find the current hlog. When found, the path may be > wrong, and it is looped together. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group
management methods in Admin and AsyncAdmin
URL: https://github.com/apache/hbase/pull/613#discussion_r325717439
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
##
@@ -2833,4 +2864,270 @@ private boolean shouldSubmitSCP(ServerName serverName)
{
}
return true;
}
+
+
+ @Override
+ public GetRSGroupInfoResponse getRSGroupInfo(RpcController controller,
+ GetRSGroupInfoRequest request) throws ServiceException {
+GetRSGroupInfoResponse.Builder builder =
GetRSGroupInfoResponse.newBuilder();
+String groupName = request.getRSGroupName();
+LOG.info(
+master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval,
group=" + groupName);
+try {
+ if (master.getMasterCoprocessorHost() != null) {
+master.getMasterCoprocessorHost().preGetRSGroupInfo(groupName);
+ }
+ RSGroupInfo rsGroupInfo =
master.getRSRSGroupInfoManager().getRSGroup(groupName);
+ if (rsGroupInfo != null) {
+
builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(rsGroupInfo)));
+ }
+ if (master.getMasterCoprocessorHost() != null) {
+master.getMasterCoprocessorHost().postGetRSGroupInfo(groupName);
+ }
+} catch (IOException e) {
+ throw new ServiceException(e);
+}
+return builder.build();
+ }
+
+ @Override
+ public GetRSGroupInfoOfServerResponse getRSGroupInfoOfServer(RpcController
controller,
+ GetRSGroupInfoOfServerRequest request) throws ServiceException {
+GetRSGroupInfoOfServerResponse.Builder builder =
GetRSGroupInfoOfServerResponse.newBuilder();
+Address hp = Address.fromParts(request.getServer().getHostName(),
+request.getServer().getPort());
+LOG.info(master.getClientIdAuditPrefix() + " initiates rsgroup info
retrieval, server=" + hp);
+try {
+ if (master.getMasterCoprocessorHost() != null) {
+master.getMasterCoprocessorHost().preGetRSGroupInfoOfServer(hp);
+ }
+ RSGroupInfo info =
master.getRSRSGroupInfoManager().getRSGroupOfServer(hp);
+ if (info != null) {
+
builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(info)));
+ }
+ if (master.getMasterCoprocessorHost() != null) {
+master.getMasterCoprocessorHost().postGetRSGroupInfoOfServer(hp);
+ }
+} catch (IOException e) {
+ throw new ServiceException(e);
+}
+return builder.build();
+ }
+
+ private RSGroupInfo fillTables(RSGroupInfo rsGroupInfo) throws IOException {
+return RSGroupUtil.fillTables(rsGroupInfo,
master.getTableDescriptors().getAll().values());
+ }
+
+ @Override
+ public MoveServersResponse moveServers(RpcController controller,
MoveServersRequest request)
+ throws ServiceException {
+Set hostPorts = Sets.newHashSet();
+MoveServersResponse.Builder builder = MoveServersResponse.newBuilder();
+for (HBaseProtos.ServerName el : request.getServersList()) {
+ hostPorts.add(Address.fromParts(el.getHostName(), el.getPort()));
+}
+LOG.info(master.getClientIdAuditPrefix() + " move servers " + hostPorts +
" to rsgroup " +
+request.getTargetGroup());
+try {
+ if (master.getMasterCoprocessorHost() != null) {
+master.getMasterCoprocessorHost().preMoveServers(hostPorts,
request.getTargetGroup());
+ }
+ master.getRSRSGroupInfoManager().moveServers(hostPorts,
request.getTargetGroup());
+ if (master.getMasterCoprocessorHost() != null) {
+master.getMasterCoprocessorHost().postMoveServers(hostPorts,
request.getTargetGroup());
+ }
+} catch (IOException e) {
+ throw new ServiceException(e);
+}
+return builder.build();
+ }
+
+ @Deprecated
+ @Override
+ public MoveTablesResponse moveTables(RpcController controller,
MoveTablesRequest request)
+ throws ServiceException {
+return null;
+ }
+
+ @Override
+ public AddRSGroupResponse addRSGroup(RpcController controller,
AddRSGroupRequest request)
+ throws ServiceException {
+AddRSGroupResponse.Builder builder = AddRSGroupResponse.newBuilder();
+LOG.info(master.getClientIdAuditPrefix() + " add rsgroup " +
request.getRSGroupName());
+try {
+ if (master.getMasterCoprocessorHost() != null) {
+
master.getMasterCoprocessorHost().preAddRSGroup(request.getRSGroupName());
+ }
+ master.getRSRSGroupInfoManager().addRSGroup(new
RSGroupInfo(request.getRSGroupName()));
+ if (master.getMasterCoprocessorHost() != null) {
+
master.getMasterCoprocessorHost().postAddRSGroup(request.getRSGroupName());
+ }
+} catch (IOException e) {
+ throw new ServiceException(e);
+}
+return builder.build();
+ }
+
+ @Override
+ public RemoveRSGroupResponse removeRSGroup(RpcController controller,
RemoveRSGroupRequest request)
+ throws ServiceException {
+RemoveRSGroupResponse.Bui
[GitHub] [hbase] Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group
management methods in Admin and AsyncAdmin
URL: https://github.com/apache/hbase/pull/613#discussion_r325714982
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
##
@@ -1506,5 +1508,63 @@
* The return value will be wrapped by a {@link CompletableFuture}.
*/
CompletableFuture isSnapshotCleanupEnabled();
+
+ /**
+ * Gets group info for the given group name
+ * @param groupName the group name
+ * @return group info
+ */
+ CompletableFuture getRSGroupInfo(String groupName);
+
+ /**
+ * Move given set of servers to the specified target RegionServer group
+ * @param servers set of servers to move
+ * @param targetGroup the group to move servers to
+ */
+ CompletableFuture moveServers(Set servers, String
targetGroup);
+
+ /**
+ * Creates a new RegionServer group with the given name
+ * @param groupName the name of the group
+ */
+ CompletableFuture addRSGroup(String groupName);
+
+ /**
+ * Removes RegionServer group associated with the given name
+ * @param groupName the group name
+ */
+ CompletableFuture removeRSGroup(String groupName);
+
+ /**
+ * Balance regions in the given RegionServer group
+ * @param groupName the group name
+ * @return boolean Whether balance ran or not
+ */
+ CompletableFuture balanceRSGroup(String groupName);
+
+ /**
+ * Lists current set of RegionServer groups
+ */
+ CompletableFuture> listRSGroups();
+
+ /**
+ * Retrieve the RSGroupInfo a server is affiliated to
+ * @param hostPort HostPort to get RSGroupInfo for
+ */
+ CompletableFuture getRSGroupOfServer(Address
hostPort);
+
+ /**
+ * Remove decommissioned servers from group
+ * 1. Sometimes we may find the server aborted due to some hardware failure
and we must offline
+ * the server for repairing. Or we need to move some servers to join other
clusters.
+ * So we need to remove these servers from the group.
+ * 2. Dead/recovering/live servers will be disallowed.
+ * @param servers set of servers to remove
+ */
+ CompletableFuture removeServers(Set servers);
+
+ CompletableFuture getRSGroupInfoOfTable(TableName tableName);
Review comment:
Can just make a default implementation here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin URL: https://github.com/apache/hbase/pull/613#discussion_r325717964 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java ## @@ -38,34 +58,858 @@ import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoRequest; import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoResponse; import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.ListRSGroupInfosRequest; -import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersAndTablesRequest; import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersRequest; -import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveTablesRequest; import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RSGroupAdminService; import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupRequest; import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveServersRequest; +import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.SetRSGroupForTablesRequest; import org.apache.hadoop.hbase.protobuf.generated.RSGroupProtos; +import org.apache.hadoop.hbase.quotas.QuotaFilter; +import org.apache.hadoop.hbase.quotas.QuotaSettings; +import org.apache.hadoop.hbase.quotas.SpaceQuotaSnapshotView; +import org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException; +import org.apache.hadoop.hbase.replication.ReplicationPeerConfig; +import org.apache.hadoop.hbase.replication.ReplicationPeerDescription; +import org.apache.hadoop.hbase.replication.SyncReplicationState; +import org.apache.hadoop.hbase.security.access.GetUserPermissionsRequest; +import org.apache.hadoop.hbase.security.access.Permission; +import org.apache.hadoop.hbase.security.access.UserPermission; +import org.apache.hadoop.hbase.snapshot.HBaseSnapshotException; +import org.apache.hadoop.hbase.snapshot.RestoreSnapshotException; +import org.apache.hadoop.hbase.snapshot.SnapshotCreationException; +import org.apache.hadoop.hbase.snapshot.UnknownSnapshotException; +import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting; +import org.apache.hbase.thirdparty.com.google.common.collect.Sets; import org.apache.yetus.audience.InterfaceAudience; +import com.google.protobuf.ServiceException; + +/** + * Client used for managing region server group information. + */ +@InterfaceAudience.Private Review comment: We do not need this class any more? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group
management methods in Admin and AsyncAdmin
URL: https://github.com/apache/hbase/pull/613#discussion_r325714479
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
##
@@ -1506,5 +1508,63 @@
* The return value will be wrapped by a {@link CompletableFuture}.
*/
CompletableFuture isSnapshotCleanupEnabled();
+
+ /**
+ * Gets group info for the given group name
+ * @param groupName the group name
+ * @return group info
+ */
+ CompletableFuture getRSGroupInfo(String groupName);
+
+ /**
+ * Move given set of servers to the specified target RegionServer group
+ * @param servers set of servers to move
+ * @param targetGroup the group to move servers to
+ */
+ CompletableFuture moveServers(Set servers, String
targetGroup);
+
+ /**
+ * Creates a new RegionServer group with the given name
+ * @param groupName the name of the group
+ */
+ CompletableFuture addRSGroup(String groupName);
+
+ /**
+ * Removes RegionServer group associated with the given name
+ * @param groupName the group name
+ */
+ CompletableFuture removeRSGroup(String groupName);
+
+ /**
+ * Balance regions in the given RegionServer group
+ * @param groupName the group name
+ * @return boolean Whether balance ran or not
+ */
+ CompletableFuture balanceRSGroup(String groupName);
+
+ /**
+ * Lists current set of RegionServer groups
+ */
+ CompletableFuture> listRSGroups();
+
+ /**
+ * Retrieve the RSGroupInfo a server is affiliated to
+ * @param hostPort HostPort to get RSGroupInfo for
+ */
+ CompletableFuture getRSGroupOfServer(Address
hostPort);
+
+ /**
+ * Remove decommissioned servers from group
+ * 1. Sometimes we may find the server aborted due to some hardware failure
and we must offline
Review comment:
Use html tags.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group
management methods in Admin and AsyncAdmin
URL: https://github.com/apache/hbase/pull/613#discussion_r325714233
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
##
@@ -1506,5 +1508,63 @@
* The return value will be wrapped by a {@link CompletableFuture}.
*/
CompletableFuture isSnapshotCleanupEnabled();
+
+ /**
+ * Gets group info for the given group name
+ * @param groupName the group name
+ * @return group info
+ */
+ CompletableFuture getRSGroupInfo(String groupName);
+
+ /**
+ * Move given set of servers to the specified target RegionServer group
+ * @param servers set of servers to move
+ * @param targetGroup the group to move servers to
+ */
+ CompletableFuture moveServers(Set servers, String
targetGroup);
+
+ /**
+ * Creates a new RegionServer group with the given name
+ * @param groupName the name of the group
+ */
+ CompletableFuture addRSGroup(String groupName);
+
+ /**
+ * Removes RegionServer group associated with the given name
+ * @param groupName the group name
+ */
+ CompletableFuture removeRSGroup(String groupName);
+
+ /**
+ * Balance regions in the given RegionServer group
+ * @param groupName the group name
+ * @return boolean Whether balance ran or not
+ */
+ CompletableFuture balanceRSGroup(String groupName);
Review comment:
What is the behavior if there are misplaced regions?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group
management methods in Admin and AsyncAdmin
URL: https://github.com/apache/hbase/pull/613#discussion_r325716358
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##
@@ -3864,23 +3886,130 @@ private void getProcedureResult(long procId,
CompletableFuture future, int
@Override
public CompletableFuture snapshotCleanupSwitch(final boolean on,
final boolean sync) {
-return this.newMasterCaller()
-.action((controller, stub) -> this
-.call(controller, stub,
-RequestConverter.buildSetSnapshotCleanupRequest(on, sync),
-MasterService.Interface::switchSnapshotCleanup,
-SetSnapshotCleanupResponse::getPrevSnapshotCleanup))
-.call();
+return this.newMasterCaller().action((controller, stub) -> this
+.call(controller, stub,
RequestConverter.buildSetSnapshotCleanupRequest(on, sync),
+MasterService.Interface::switchSnapshotCleanup,
+SetSnapshotCleanupResponse::getPrevSnapshotCleanup)).call();
}
@Override
public CompletableFuture isSnapshotCleanupEnabled() {
-return this.newMasterCaller()
+return this.newMasterCaller().action((controller, stub) -> this
+.call(controller, stub,
RequestConverter.buildIsSnapshotCleanupEnabledRequest(),
+MasterService.Interface::isSnapshotCleanupEnabled,
+IsSnapshotCleanupEnabledResponse::getEnabled)).call();
+ }
+
+ @Override
+ public CompletableFuture getRSGroupInfo(String groupName) {
+return this. newMasterCaller()
+.action(((controller, stub) -> this.
+
call(controller, stub,
+RequestConverter.buildGetRSGroupInfoRequest(groupName),
+ (s, c, req, done) -> s.getRSGroupInfo(c, req, done),
+ resp -> resp.hasRSGroupInfo() ?
+ ProtobufUtil.toGroupInfo(resp.getRSGroupInfo()) : null)))
+.call();
+ }
+
+ @Override
+ public CompletableFuture moveServers(Set servers, String
targetGroup) {
+return this. newMasterCaller()
+.action((controller, stub) -> this.
+ call(controller,
stub,
+RequestConverter.buildMoveServersRequest(servers, targetGroup),
+ (s, c, req, done) -> s.moveServers(c, req, done), resp -> null))
+.call();
+ }
+
+ @Override
+ public CompletableFuture addRSGroup(String groupName) {
+return this. newMasterCaller()
+.action(((controller, stub) -> this.
+ call(controller,
stub,
+
AddRSGroupRequest.newBuilder().setRSGroupName(groupName).build(),
+ (s, c, req, done) -> s.addRSGroup(c, req, done), resp -> null)))
+.call();
+ }
+
+ @Override
+ public CompletableFuture removeRSGroup(String groupName) {
+return this. newMasterCaller()
+.action((controller, stub) -> this.
+
call(controller, stub,
+
RemoveRSGroupRequest.newBuilder().setRSGroupName(groupName).build(),
+ (s, c, req, done) -> s.removeRSGroup(c, req, done), resp ->
null))
+.call();
+ }
+
+ @Override
+ public CompletableFuture balanceRSGroup(String groupName) {
+return this. newMasterCaller()
+.action((controller, stub) -> this.
+
call(controller, stub,
+
BalanceRSGroupRequest.newBuilder().setRSGroupName(groupName).build(),
+ (s, c, req, done) -> s.balanceRSGroup(c, req, done),
+resp -> resp.getBalanceRan()))
+.call();
+ }
+
+ @Override
+ public CompletableFuture> listRSGroups() {
+return this.> newMasterCaller()
.action((controller, stub) -> this
-.call(controller, stub,
-RequestConverter.buildIsSnapshotCleanupEnabledRequest(),
-MasterService.Interface::isSnapshotCleanupEnabled,
-IsSnapshotCleanupEnabledResponse::getEnabled))
+.> call(controller,
+stub, ListRSGroupInfosRequest.getDefaultInstance(),
+(s, c, req, done) -> s.listRSGroupInfos(c, req, done),
+resp -> resp.getRSGroupInfoList().stream()
+.map(r -> ProtobufUtil.toGroupInfo(r))
+.collect(Collectors.toList(
+.call();
+ }
+
+ @Override
+ public CompletableFuture getRSGroupOfServer(Address hostPort) {
+return this. newMasterCaller()
+.action((controller, stub) -> this.
+ call(
+controller, stub,
+ RequestConverter.buildGetRSGroupInfoOfServerRequest(hostPort),
+ (s, c, req, done) -> s.getRSGroupInfoOfServer(c, req, done),
+ resp -> resp.hasRSGroupInfo() ?
+ ProtobufUtil.toGroupInfo(resp.getRSGroupInfo()) : null))
+.call();
+ }
+
+ @Override
+ public CompletableFuture removeServers(Set servers) {
+
[GitHub] [hbase] Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group management methods in Admin and AsyncAdmin
Apache9 commented on a change in pull request #613: HBASE-22932 Add rs group
management methods in Admin and AsyncAdmin
URL: https://github.com/apache/hbase/pull/613#discussion_r325712959
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
##
@@ -1506,5 +1508,63 @@
* The return value will be wrapped by a {@link CompletableFuture}.
*/
CompletableFuture isSnapshotCleanupEnabled();
+
+ /**
+ * Gets group info for the given group name
+ * @param groupName the group name
+ * @return group info
+ */
+ CompletableFuture getRSGroupInfo(String groupName);
Review comment:
Why extends? IIRC we do not have any sub classes which extend RSGroupInfo?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932520#comment-16932520
]
Duo Zhang commented on HBASE-23044:
---
So maybe we should roll new releases then... And also remove the download link
of 2.0.6 on our download page.
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #595: HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature …
Apache9 commented on a change in pull request #595: HBASE-22971 Deprecated
RSGroupAdminEndpoint and make RSGroup feature …
URL: https://github.com/apache/hbase/pull/595#discussion_r325726717
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -324,6 +325,11 @@ private LoadBalancer getBalancer() {
return master.getLoadBalancer();
}
+ private FavoredNodesPromoter getFavoredNodePromoter() {
+return (FavoredNodesPromoter) ((RSGroupBasedLoadBalancer)
master.getLoadBalancer())
Review comment:
The return value for the MasterServices interface is still LoadBalancer so...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on issue #595: HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature …
Apache9 commented on issue #595: HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature … URL: https://github.com/apache/hbase/pull/595#issuecomment-532725949 > The master ui show RSGroup info by the coprocessor "RSGroupEndpoint" config. Need to update too. It uses RSGroupTableAccessor to access the rsgroup table directly. I think the problem here is that we will use the coprocessor to test whether we should show rs group on master web page. This can be done in another issue, as I think maybe we still need a flag about whether to enable rs group. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #638: HBASE-23046 Remove compatibility case from truncate command
Apache-HBase commented on issue #638: HBASE-23046 Remove compatibility case from truncate command URL: https://github.com/apache/hbase/pull/638#issuecomment-532726949 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 8m 1s | master passed | | :green_heart: | javadoc | 0m 17s | master passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 6m 23s | the patch passed | | :broken_heart: | rubocop | 0m 14s | The patch generated 4 new + 436 unchanged - 14 fixed = 440 total (was 450) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | :green_heart: | unit | 7m 6s | hbase-shell in the patch passed. | | :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 24m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-638/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/638 | | Optional Tests | dupname asflicense javac javadoc unit rubocop | | uname | Linux 23f37109836c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-638/out/precommit/personality/provided.sh | | git revision | master / 25bcc91dab | | Default Java | 1.8.0_181 | | rubocop | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-638/1/artifact/out/diff-patch-rubocop.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-638/1/testReport/ | | Max. process+thread count | 2625 (vs. ulimit of 1) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-638/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) rubocop=0.74.0 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #637: HBASE-23044 CatalogJanitor#cleanMergeQualifier may clean wrong parent…
Apache-HBase commented on issue #637: HBASE-23044 CatalogJanitor#cleanMergeQualifier may clean wrong parent… URL: https://github.com/apache/hbase/pull/637#issuecomment-532731741 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | :blue_heart: | mvndep | 0m 33s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 5m 37s | master passed | | :green_heart: | compile | 1m 21s | master passed | | :green_heart: | checkstyle | 2m 1s | master passed | | :green_heart: | shadedjars | 4m 53s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 57s | master passed | | :blue_heart: | spotbugs | 4m 32s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 5m 38s | master passed | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 13s | Maven dependency ordering for patch | | :green_heart: | mvninstall | 5m 23s | the patch passed | | :green_heart: | compile | 1m 22s | the patch passed | | :green_heart: | javac | 1m 22s | the patch passed | | :broken_heart: | checkstyle | 0m 33s | hbase-client: The patch generated 2 new + 46 unchanged - 0 fixed = 48 total (was 46) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 4m 56s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 17m 38s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 58s | the patch passed | | :green_heart: | findbugs | 5m 57s | the patch passed | ||| _ Other Tests _ | | :green_heart: | unit | 1m 57s | hbase-client in the patch passed. | | :green_heart: | unit | 165m 11s | hbase-server in the patch passed. | | :green_heart: | asflicense | 0m 46s | The patch does not generate ASF License warnings. | | | | 235m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-637/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/637 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 0411e3f2ad8c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-637/out/precommit/personality/provided.sh | | git revision | master / cb62f73406 | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-637/1/artifact/out/diff-checkstyle-hbase-client.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-637/1/testReport/ | | Max. process+thread count | 4548 (vs. ulimit of 1) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-637/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #592: HBASE-22982: region server suspend/resume and graceful rolling restart actions
Apache-HBase commented on issue #592: HBASE-22982: region server suspend/resume and graceful rolling restart actions URL: https://github.com/apache/hbase/pull/592#issuecomment-532732556 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 13 new or modified test files. | ||| _ master Compile Tests _ | | :blue_heart: | mvndep | 0m 34s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 5m 18s | master passed | | :green_heart: | compile | 1m 28s | master passed | | :green_heart: | checkstyle | 1m 38s | master passed | | :green_heart: | shadedjars | 4m 36s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 0m 49s | master passed | | :blue_heart: | spotbugs | 4m 17s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :green_heart: | findbugs | 4m 16s | master passed | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 15s | Maven dependency ordering for patch | | :green_heart: | mvninstall | 5m 8s | the patch passed | | :green_heart: | compile | 1m 25s | the patch passed | | :green_heart: | javac | 1m 25s | the patch passed | | :green_heart: | checkstyle | 1m 16s | The patch passed checkstyle in hbase-server | | :green_heart: | checkstyle | 0m 15s | hbase-it: The patch generated 0 new + 11 unchanged - 64 fixed = 11 total (was 75) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | shadedjars | 4m 35s | patch has no errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 16m 32s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 0m 53s | the patch passed | | :green_heart: | findbugs | 4m 21s | the patch passed | ||| _ Other Tests _ | | :green_heart: | unit | 162m 40s | hbase-server in the patch passed. | | :green_heart: | unit | 1m 10s | hbase-it in the patch passed. | | :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | | | | 226m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-592/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/592 | | JIRA Issue | HBASE-22982 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 6a18c5fb65b7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-592/out/precommit/personality/provided.sh | | git revision | master / ab076b0731 | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-592/3/testReport/ | | Max. process+thread count | 4348 (vs. ulimit of 1) | | modules | C: hbase-server hbase-it U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-592/3/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22982) Send SIGSTOP to hang or SIGCONT to resume rs and add graceful rolling restart
[
https://issues.apache.org/jira/browse/HBASE-22982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932546#comment-16932546
]
HBase QA commented on HBASE-22982:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
36s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m
17s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} The patch passed checkstyle in hbase-server {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} hbase-it: The patch generated 0 new + 11 unchanged -
64 fixed = 11 total (was 75) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 32s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}162m
40s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
10s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
6s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}226m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-592/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hbase/pull/592 |
| JIRA Iss
[jira] [Resolved] (HBASE-23040) region mover gives NullPointerException instead of saying a host isn't in the cluster
[
https://issues.apache.org/jira/browse/HBASE-23040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-23040.
-
Fix Version/s: 2.2.2
2.1.7
2.3.0
3.0.0
Release Note:
giving the region mover "unload" command a region server name that isn't
recognized by the cluster results in a "I don't know about that host" message
instead of a NPE.
set log level to DEBUG if you'd like the region mover to log the set of region
server names it got back from the cluster.
Resolution: Fixed
> region mover gives NullPointerException instead of saying a host isn't in the
> cluster
> -
>
> Key: HBASE-23040
> URL: https://issues.apache.org/jira/browse/HBASE-23040
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 2.2.0, 2.3.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> if passed a host that isn't in the cluster we get a NPE
> {code}
> 2019-09-17 21:50:45,595 ERROR [pool-2-thread-1] util.RegionMover: Error while
> unloading regions
> java.lang.NullPointerException: serverName is null
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:897)
> at
> org.apache.hadoop.hbase.client.AsyncRpcRetryingCallerFactory$AdminRequestCallerBuilder.build(AsyncRpcRetryingCallerFactory.java:518)
> at
> org.apache.hadoop.hbase.client.AsyncRpcRetryingCallerFactory$AdminRequestCallerBuilder.call(AsyncRpcRetryingCallerFactory.java:522)
> at
> org.apache.hadoop.hbase.client.RawAsyncHBaseAdmin.getRegions(RawAsyncHBaseAdmin.java:842)
> at
> org.apache.hadoop.hbase.client.AdminOverAsyncAdmin.getRegions(AdminOverAsyncAdmin.java:229)
> at
> org.apache.hadoop.hbase.util.RegionMover.unloadRegions(RegionMover.java:448)
> at
> org.apache.hadoop.hbase.util.RegionMover.lambda$unload$1(RegionMover.java:431)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Should instead give some kind of "I don't know what this host is
> 'foobar.example.com'" and maybe a debug message with what hosts were in the
> cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] anoopsjohn commented on a change in pull request #633: HBASE-22890 Verify the file integrity in persistent IOEngine
anoopsjohn commented on a change in pull request #633: HBASE-22890 Verify the
file integrity in persistent IOEngine
URL: https://github.com/apache/hbase/pull/633#discussion_r325760729
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
##
@@ -1021,41 +1038,60 @@ void doDrain(final List entries) throws
InterruptedException {
private void persistToFile() throws IOException {
assert !cacheEnabled;
-FileOutputStream fos = null;
-ObjectOutputStream oos = null;
-try {
+try (ObjectOutputStream oos = new ObjectOutputStream(
+ new FileOutputStream(persistencePath, false))){
if (!ioEngine.isPersistent()) {
throw new IOException("Attempt to persist non-persistent cache
mappings!");
}
- fos = new FileOutputStream(persistencePath, false);
- oos = new ObjectOutputStream(fos);
+ oos.write(ProtobufUtil.PB_MAGIC);
+ byte[] checksum = ((PersistentIOEngine)
ioEngine).calculateChecksum(algorithm);
+ oos.writeInt(checksum.length);
+ oos.write(checksum);
oos.writeLong(cacheCapacity);
oos.writeUTF(ioEngine.getClass().getName());
oos.writeUTF(backingMap.getClass().getName());
oos.writeObject(deserialiserMap);
oos.writeObject(backingMap);
-} finally {
- if (oos != null) oos.close();
- if (fos != null) fos.close();
+} catch (NoSuchAlgorithmException e) {
+ LOG.error("No such algorithm : " + algorithm + "! Failed to persist data
on exit",e);
}
}
@SuppressWarnings("unchecked")
- private void retrieveFromFile(int[] bucketSizes) throws IOException,
BucketAllocatorException,
+ private void retrieveFromFile(int[] bucketSizes) throws IOException,
ClassNotFoundException {
File persistenceFile = new File(persistencePath);
if (!persistenceFile.exists()) {
return;
}
assert !cacheEnabled;
-FileInputStream fis = null;
ObjectInputStream ois = null;
try {
if (!ioEngine.isPersistent())
throw new IOException(
"Attempt to restore non-persistent cache mappings!");
- fis = new FileInputStream(persistencePath);
- ois = new ObjectInputStream(fis);
+ ois = new ObjectInputStream(new FileInputStream(persistencePath));
+ int pblen = ProtobufUtil.lengthOfPBMagic();
+ byte[] pbuf = new byte[pblen];
+ int read = ois.read(pbuf);
+ if (read != pblen) {
+throw new IOException("Incorrect number of bytes read while checking
for protobuf magic "
+ + "number. Requested=" + pblen + ", Received= " + read + ", File=" +
persistencePath);
+ }
+ if (Bytes.equals(ProtobufUtil.PB_MAGIC, pbuf)) {
+int length = ois.readInt();
+byte[] persistentChecksum = new byte[length];
+int readLen = ois.read(persistentChecksum);
+if (readLen != length || !((PersistentIOEngine)
ioEngine).verifyFileIntegrity(
+persistentChecksum, algorithm)) {
+ return;
Review comment:
We can add an INFO log here? Or we have the log saying the verify fail in
FileIOE?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] anoopsjohn commented on a change in pull request #633: HBASE-22890 Verify the file integrity in persistent IOEngine
anoopsjohn commented on a change in pull request #633: HBASE-22890 Verify the
file integrity in persistent IOEngine
URL: https://github.com/apache/hbase/pull/633#discussion_r325762374
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
##
@@ -288,14 +302,17 @@ public BucketCache(String ioEngineName, long capacity,
int blockSize, int[] buck
this.ramCache = new ConcurrentHashMap();
this.backingMap = new ConcurrentHashMap((int)
blockNumCapacity);
-
-if (ioEngine.isPersistent() && persistencePath != null) {
+if (ioEngine.isPersistent()) {
try {
-retrieveFromFile(bucketSizes);
+if (persistencePath != null) {
+ retrieveFromFile(bucketSizes);
+} else {
+ ((PersistentIOEngine) ioEngine).deleteCacheDataFile();
Review comment:
If we delete data file here not, there wont be any functional diff I
believe. We were not doing it before. Any reason to do so now? Or we do this
delete in master branch now? Not remembering exactly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] anoopsjohn commented on a change in pull request #633: HBASE-22890 Verify the file integrity in persistent IOEngine
anoopsjohn commented on a change in pull request #633: HBASE-22890 Verify the
file integrity in persistent IOEngine
URL: https://github.com/apache/hbase/pull/633#discussion_r325763798
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java
##
@@ -59,6 +65,30 @@ public FileIOEngine(long capacity, String... filePaths)
throws IOException {
this.fileChannels = new FileChannel[filePaths.length];
this.rafs = new RandomAccessFile[filePaths.length];
this.channelLocks = new ReentrantLock[filePaths.length];
+init();
+ }
+
+ @Override
+ public boolean verifyFileIntegrity(byte[] persistentChecksum, String
algorithm)
+throws IOException {
+try {
+ byte[] calculateChecksum = calculateChecksum(algorithm);
+ if (!Bytes.equals(persistentChecksum, calculateChecksum)) {
+throw new IOException("The persistent checksum is " +
Bytes.toString(persistentChecksum) +
Review comment:
The response from this method is in 2 ways for IOE or mismatch of checksum.
In caller place we check for boolean return value basically. Can we just log
the issue here and return boolean in any case?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] anoopsjohn commented on a change in pull request #633: HBASE-22890 Verify the file integrity in persistent IOEngine
anoopsjohn commented on a change in pull request #633: HBASE-22890 Verify the
file integrity in persistent IOEngine
URL: https://github.com/apache/hbase/pull/633#discussion_r325765169
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java
##
@@ -59,6 +65,30 @@ public FileIOEngine(long capacity, String... filePaths)
throws IOException {
this.fileChannels = new FileChannel[filePaths.length];
this.rafs = new RandomAccessFile[filePaths.length];
this.channelLocks = new ReentrantLock[filePaths.length];
+init();
+ }
+
+ @Override
+ public boolean verifyFileIntegrity(byte[] persistentChecksum, String
algorithm)
+throws IOException {
+try {
+ byte[] calculateChecksum = calculateChecksum(algorithm);
+ if (!Bytes.equals(persistentChecksum, calculateChecksum)) {
+throw new IOException("The persistent checksum is " +
Bytes.toString(persistentChecksum) +
+ ", but the calculate checksum is " +
Bytes.toString(calculateChecksum));
+ }
+ return true;
+} catch (IOException ioex) {
+ LOG.error("File verification failed because of ", ioex);
+ reinit();
+ return false;
+} catch (NoSuchAlgorithmException nsae) {
+ LOG.error("No such algorithm " + algorithm, nsae);
+ throw new RuntimeException(nsae);
Review comment:
What will happen if we throw RTE here? The FileIOE create itself will get
failed or even more? Retrieve from persistent cached fail need not be a reason
for the FileIOE to be failed. But here we have configured this to be a
persistent FileIOE. Later when we try persist during RS shutdown, we will get
failure any way (?). Are u considering that here for this RTE? Am not really
sure this throw RTE is correct here. Thoughts
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] anoopsjohn commented on a change in pull request #633: HBASE-22890 Verify the file integrity in persistent IOEngine
anoopsjohn commented on a change in pull request #633: HBASE-22890 Verify the
file integrity in persistent IOEngine
URL: https://github.com/apache/hbase/pull/633#discussion_r325766107
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java
##
@@ -59,6 +65,30 @@ public FileIOEngine(long capacity, String... filePaths)
throws IOException {
this.fileChannels = new FileChannel[filePaths.length];
this.rafs = new RandomAccessFile[filePaths.length];
this.channelLocks = new ReentrantLock[filePaths.length];
+init();
+ }
+
+ @Override
+ public boolean verifyFileIntegrity(byte[] persistentChecksum, String
algorithm)
+throws IOException {
+try {
+ byte[] calculateChecksum = calculateChecksum(algorithm);
+ if (!Bytes.equals(persistentChecksum, calculateChecksum)) {
+throw new IOException("The persistent checksum is " +
Bytes.toString(persistentChecksum) +
+ ", but the calculate checksum is " +
Bytes.toString(calculateChecksum));
+ }
+ return true;
+} catch (IOException ioex) {
+ LOG.error("File verification failed because of ", ioex);
+ reinit();
Review comment:
Why this reinit only in this case? In case of checksum mismatch no such
thing happening no?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (HBASE-23047) ChecksumUtil.validateChecksum logs an INFO message inside a "if(LOG.isTraceEnabled())" block.
Wellington Chevreuil created HBASE-23047:
Summary: ChecksumUtil.validateChecksum logs an INFO message inside
a "if(LOG.isTraceEnabled())" block.
Key: HBASE-23047
URL: https://issues.apache.org/jira/browse/HBASE-23047
Project: HBase
Issue Type: Improvement
Affects Versions: 2.1.6, 2.2.1, 3.0.0, 2.3.0
Reporter: Wellington Chevreuil
Assignee: Wellington Chevreuil
Noticed this while analysing another potential checksum issue. Despite doing a
check for TRACE level, we log an INFO message inside the if block:
{noformat}
if (LOG.isTraceEnabled()) {
LOG.info("dataLength=" + buf.capacity() + ", sizeWithHeader=" +
onDiskDataSizeWithHeader
+ ", checksumType=" + ctype.getName() + ", file=" + pathName + ",
offset=" + offset
+ ", headerSize=" + hdrSize + ", bytesPerChecksum=" +
bytesPerChecksum);
}
{noformat}
Uploading a patch that logs a TRACE message and switch to parameterising
logging. Since there's no extra computation on the param passing, we shouldn't
need the extra if either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23047) ChecksumUtil.validateChecksum logs an INFO message inside a "if(LOG.isTraceEnabled())" block.
[
https://issues.apache.org/jira/browse/HBASE-23047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-23047:
-
Attachment: HBASE-23047.master.001.patch
> ChecksumUtil.validateChecksum logs an INFO message inside a
> "if(LOG.isTraceEnabled())" block.
> -
>
> Key: HBASE-23047
> URL: https://issues.apache.org/jira/browse/HBASE-23047
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.0, 2.2.1, 2.1.6
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HBASE-23047.master.001.patch
>
>
> Noticed this while analysing another potential checksum issue. Despite doing
> a check for TRACE level, we log an INFO message inside the if block:
> {noformat}
> if (LOG.isTraceEnabled()) {
> LOG.info("dataLength=" + buf.capacity() + ", sizeWithHeader=" +
> onDiskDataSizeWithHeader
> + ", checksumType=" + ctype.getName() + ", file=" + pathName + ",
> offset=" + offset
> + ", headerSize=" + hdrSize + ", bytesPerChecksum=" +
> bytesPerChecksum);
> }
> {noformat}
> Uploading a patch that logs a TRACE message and switch to parameterising
> logging. Since there's no extra computation on the param passing, we
> shouldn't need the extra if either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23047) ChecksumUtil.validateChecksum logs an INFO message inside a "if(LOG.isTraceEnabled())" block.
[
https://issues.apache.org/jira/browse/HBASE-23047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-23047:
-
Status: Patch Available (was: Open)
> ChecksumUtil.validateChecksum logs an INFO message inside a
> "if(LOG.isTraceEnabled())" block.
> -
>
> Key: HBASE-23047
> URL: https://issues.apache.org/jira/browse/HBASE-23047
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.6, 2.2.1, 3.0.0, 2.3.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HBASE-23047.master.001.patch
>
>
> Noticed this while analysing another potential checksum issue. Despite doing
> a check for TRACE level, we log an INFO message inside the if block:
> {noformat}
> if (LOG.isTraceEnabled()) {
> LOG.info("dataLength=" + buf.capacity() + ", sizeWithHeader=" +
> onDiskDataSizeWithHeader
> + ", checksumType=" + ctype.getName() + ", file=" + pathName + ",
> offset=" + offset
> + ", headerSize=" + hdrSize + ", bytesPerChecksum=" +
> bytesPerChecksum);
> }
> {noformat}
> Uploading a patch that logs a TRACE message and switch to parameterising
> logging. Since there's no extra computation on the param passing, we
> shouldn't need the extra if either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23048) Adapt dev-support release-related scripts to function for artifacts from our various repositories
Nick Dimiduk created HBASE-23048:
Summary: Adapt dev-support release-related scripts to function for
artifacts from our various repositories
Key: HBASE-23048
URL: https://issues.apache.org/jira/browse/HBASE-23048
Project: HBase
Issue Type: Improvement
Components: tooling
Reporter: Nick Dimiduk
Looks like there's just a few assumptions we can unwind between the output of
{{create-release}} and the input to {{hbase-vote.sh}} that will allow for these
tools to be used for the operator-tools repo. We should work through these for
all of our relevant side-repos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ZhaoBQ commented on a change in pull request #633: HBASE-22890 Verify the file integrity in persistent IOEngine
ZhaoBQ commented on a change in pull request #633: HBASE-22890 Verify the file
integrity in persistent IOEngine
URL: https://github.com/apache/hbase/pull/633#discussion_r325781271
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java
##
@@ -68,15 +98,15 @@ public FileIOEngine(long capacity, String... filePaths)
throws IOException {
// The next setting length will throw exception,logging this message
// is just used for the detail reason of exception，
String msg = "Only " + StringUtils.byteDesc(totalSpace)
- + " total space under " + filePath + ", not enough for requested
"
- + StringUtils.byteDesc(sizePerFile);
++ " total space under " + filePath + ", not enough for requested "
++ StringUtils.byteDesc(sizePerFile);
LOG.warn(msg);
}
rafs[i].setLength(sizePerFile);
Review comment:
setLength() can change the last modified time of the file, so the persistent
checksum and calculate checksum are not equal and the unit test failed. But I
don't know why my local test can pass.. Anyway, I will fix this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-22969) A new binary component comparator(BinaryComponentComparator) to perform comparison of arbitrary length and position
[
https://issues.apache.org/jira/browse/HBASE-22969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932653#comment-16932653
]
Clay B. commented on HBASE-22969:
-
Hi [~udaikashyap]; thanks for getting this in to JIRA! A few review questions
against 0008 patch:
h3. BinaryComponentComparator.java:
{code:java}
58 + /**
59 + * offset of component from beginning.
60 + */
61 +
{code}
Nit: I'd say you can do an inline comment:
{code:java}
private int offset; // offset of component from beginning.
{code}
Nit: extraneous blank lines in line 81, 98?
{code:java}
104 + result = 31 * result + offset;
{code}
How did you arrive at 31?
h3. TestComparators.java
{code:java}
171 +comparable = new BinaryComponentComparator(component,1);
176 +comparable = new BinaryComponentComparator(component,2);
183 +comparable = new BinaryComponentComparator(component,1);
193 +comparable = new BinaryComponentComparator(component,1);
197 +comparable = new BinaryComponentComparator(component,2);
{code}
Nit: spaces needed around {{,}}'s
{code:java}
185 +assertTrue(PrivateCellUtil.compareRow(bbCell, comparable)>0);
186 +assertTrue(PrivateCellUtil.compareRow(kv, comparable)>0);
188 +assertTrue(PrivateCellUtil.compareValue(bbCell, comparable)>0);
189 +assertTrue(PrivateCellUtil.compareValue(kv, comparable)>0);
{code}
Nit: spaces around {{>}}'s
I'll iterate with you offline on your tests, as I see {{mvn test
-Dtest=org.apache.hadoop.hbase.filter.TestComparators}} passes after applying
your patch but I am unsure if you are testing all the comparisons you add.
> A new binary component comparator(BinaryComponentComparator) to perform
> comparison of arbitrary length and position
> ---
>
> Key: HBASE-22969
> URL: https://issues.apache.org/jira/browse/HBASE-22969
> Project: HBase
> Issue Type: Improvement
> Components: Filters
>Reporter: Udai Bhan Kashyap
>Assignee: Udai Bhan Kashyap
>Priority: Minor
> Attachments: HBASE-22969.0003.patch, HBASE-22969.0004.patch,
> HBASE-22969.0005.patch, HBASE-22969.0006.patch, HBASE-22969.0007.patch,
> HBASE-22969.0008.patch, HBASE-22969.HBASE-22969.0001.patch,
> HBASE-22969.master.0001.patch
>
>
> Lets say you have composite key: a+b+c+d. And for simplicity assume that
> a,b,c, and d all are 4 byte integers.
> Now, if you want to execute a query which is semantically same to following
> sql:
> {{"SELECT * from table where a=1 and b > 10 and b < 20 and c > 90 and c < 100
> and d=1"}}
> The only choice you have is to do client side filtering. That could be lots
> of unwanted data going through various software components and network.
> Solution:
> We can create a "component" comparator which takes the value of the
> "component" and its relative position in the key to pass the 'Filter'
> subsystem of the server:
> {code}
> FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL);
> int bOffset = 4;
> byte[] b10 = Bytes.toBytes(10);
> Filter b10Filter = new RowFilter(CompareFilter.CompareOp.GREATER,
> new BinaryComponentComparator(b10,bOffset));
> filterList.addFilter(b10Filter);
> byte[] b20 = Bytes.toBytes(20);
> Filter b20Filter = new RowFilter(CompareFilter.CompareOp.LESS,
> new BinaryComponentComparator(b20,bOffset));
> filterList.addFilter(b20Filter);
> int cOffset = 8;
> byte[] c90 = Bytes.toBytes(90);
> Filter c90Filter = new RowFilter(CompareFilter.CompareOp.GREATER,
> new BinaryComponentComparator(c90,cOffset));
> filterList.addFilter(c90Filter);
> byte[] c100 = Bytes.toBytes(100);
> Filter c100Filter = new RowFilter(CompareFilter.CompareOp.LESS,
> new BinaryComponentComparator(c100,cOffset));
> filterList.addFilter(c100Filter);
> in dOffset = 12;
> byte[] d1 = Bytes.toBytes(1);
> Filter dFilter = new RowFilter(CompareFilter.CompareOp.EQUAL,
> new BinaryComponentComparator(d1,dOffset));
> filterList.addFilter(dFilter);
> //build start and end key for scan
> int aOffset = 0;
> byte[] startKey = new byte[16]; //key size with four ints
> Bytes.putInt(startKey,aOffset,1); //a=1
> Bytes.putInt(startKey,bOffset,11); //b=11, takes care of b > 10
> Bytes.putInt(startKey,cOffset,91); //c=91,
> Bytes.putInt(startKey,dOffset,1); //d=1,
> byte[] endKey = new byte[16];
> Bytes.putInt(endKey,aOffset,1); //a=1
> Bytes.putInt(endKey,bOffset,20); /
[jira] [Commented] (HBASE-23048) Adapt dev-support release-related scripts to function for artifacts from our various repositories
[
https://issues.apache.org/jira/browse/HBASE-23048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932667#comment-16932667
]
Nick Dimiduk commented on HBASE-23048:
--
Have a look at discussion on dev@ under "[DISCUSS] Multiplicity of repos and
release tooling".
> Adapt dev-support release-related scripts to function for artifacts from our
> various repositories
> --
>
> Key: HBASE-23048
> URL: https://issues.apache.org/jira/browse/HBASE-23048
> Project: HBase
> Issue Type: Improvement
> Components: tooling
>Reporter: Nick Dimiduk
>Priority: Major
>
> Looks like there's just a few assumptions we can unwind between the output of
> {{create-release}} and the input to {{hbase-vote.sh}} that will allow for
> these tools to be used for the operator-tools repo. We should work through
> these for all of our relevant side-repos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/623#discussion_r325795904
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java
##
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.CompactionState;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.mob.MobConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MobFileCompactionChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCompactionChore.class);
+ private final Configuration conf;
+ private final HMaster master;
+ private volatile boolean running = false;
+ private int regionBatchSize = 0;// not set - compact all
+
+ public MobFileCompactionChore(HMaster master) {
+super(master.getServerName() + "-MobFileCompactionChore", master,
master.getConfiguration()
+ .getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), master
+ .getConfiguration().getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), TimeUnit.SECONDS);
+this.master = master;
+this.conf = master.getConfiguration();
+this.regionBatchSize =
+
master.getConfiguration().getInt(MobConstants.MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE,
+ MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE);
+
+ }
+
+ @Override
+ protected void chore() {
+
+boolean reported = false;
+
+try (Connection conn = ConnectionFactory.createConnection(conf);
+ Admin admin = conn.getAdmin(); ) {
+
+ if (running) {
+LOG.warn(getName() +" is running already, skipping this attempt.");
+return;
+ }
+ running = true;
+ TableDescriptors htds = master.getTableDescriptors();
+ Map map = htds.getAll();
+ for (TableDescriptor htd : map.values()) {
+if (!master.getTableStateManager().isTableState(htd.getTableName(),
+ TableState.State.ENABLED)) {
+ continue;
+}
+for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
+ if (hcd.isMobEnabled()) {
+if (!reported) {
+ master.reportMobCompactionStart(htd.getTableName());
+ reported = true;
+}
+LOG.info(" Major compacting "+ htd.getTableName() + " cf=" +
hcd.getNameAsString());
+if (regionBatchSize ==
MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE) {
+ admin.majorCompact(htd.getTableName(), hcd.getName());
+} else {
+ performMajorCompactionInBatches(admin, htd, hcd);
+}
+ }
+}
+if (reported) {
+ master.reportMobCompactionEnd(htd.getTableName());
+ reported = false;
+}
+ }
+} catch (Exception e) {
+ LOG.error("Failed to compact", e);
+} finally {
+ running = false;
+}
+ }
+
+ private void performMajorCompactionInBatches(Admin admin, TableDescriptor
htd,
+ ColumnFamilyDescriptor hcd) throws IOException {
+
+List regions = admin.getRegions(htd.getTableName());
+if (regions.size() <= this.regionBatchSize) {
+
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/623#discussion_r325795513
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java
##
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.CompactionState;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.mob.MobConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MobFileCompactionChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCompactionChore.class);
+ private final Configuration conf;
+ private final HMaster master;
+ private volatile boolean running = false;
+ private int regionBatchSize = 0;// not set - compact all
+
+ public MobFileCompactionChore(HMaster master) {
+super(master.getServerName() + "-MobFileCompactionChore", master,
master.getConfiguration()
+ .getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), master
+ .getConfiguration().getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), TimeUnit.SECONDS);
+this.master = master;
+this.conf = master.getConfiguration();
+this.regionBatchSize =
+
master.getConfiguration().getInt(MobConstants.MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE,
+ MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE);
+
+ }
+
+ @Override
+ protected void chore() {
+
+boolean reported = false;
+
+try (Connection conn = ConnectionFactory.createConnection(conf);
+ Admin admin = conn.getAdmin(); ) {
+
+ if (running) {
+LOG.warn(getName() +" is running already, skipping this attempt.");
+return;
+ }
+ running = true;
+ TableDescriptors htds = master.getTableDescriptors();
+ Map map = htds.getAll();
+ for (TableDescriptor htd : map.values()) {
+if (!master.getTableStateManager().isTableState(htd.getTableName(),
+ TableState.State.ENABLED)) {
+ continue;
+}
+for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
+ if (hcd.isMobEnabled()) {
+if (!reported) {
+ master.reportMobCompactionStart(htd.getTableName());
+ reported = true;
+}
+LOG.info(" Major compacting "+ htd.getTableName() + " cf=" +
hcd.getNameAsString());
+if (regionBatchSize ==
MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE) {
+ admin.majorCompact(htd.getTableName(), hcd.getName());
+} else {
+ performMajorCompactionInBatches(admin, htd, hcd);
+}
+ }
+}
+if (reported) {
+ master.reportMobCompactionEnd(htd.getTableName());
+ reported = false;
+}
+ }
+} catch (Exception e) {
+ LOG.error("Failed to compact", e);
+} finally {
+ running = false;
+}
+ }
+
+ private void performMajorCompactionInBatches(Admin admin, TableDescriptor
htd,
+ ColumnFamilyDescriptor hcd) throws IOException {
+
+List regions = admin.getRegions(htd.getTableName());
+if (regions.size() <= this.regionBatchSize) {
+
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/623#discussion_r325770323
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
##
@@ -1744,10 +1742,13 @@ public CompactRegionResponse compactRegion(final
RpcController controller,
master.checkInitialized();
byte[] regionName = request.getRegion().getValue().toByteArray();
TableName tableName = RegionInfo.getTable(regionName);
+ // TODO: support CompactType.MOB
// if the region is a mob region, do the mob file compaction.
if (MobUtils.isMobRegionName(tableName, regionName)) {
checkHFileFormatVersionForMob();
-return compactMob(request, tableName);
+//return compactMob(request, tableName);
+//TODO: support CompactType.MOB
Review comment:
we should log here. maybe at INFO? instead the TODO comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/623#discussion_r325790581
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java
##
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.CompactionState;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.mob.MobConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MobFileCompactionChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCompactionChore.class);
+ private final Configuration conf;
+ private final HMaster master;
+ private volatile boolean running = false;
+ private int regionBatchSize = 0;// not set - compact all
+
+ public MobFileCompactionChore(HMaster master) {
+super(master.getServerName() + "-MobFileCompactionChore", master,
master.getConfiguration()
+ .getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), master
+ .getConfiguration().getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), TimeUnit.SECONDS);
+this.master = master;
+this.conf = master.getConfiguration();
+this.regionBatchSize =
+
master.getConfiguration().getInt(MobConstants.MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE,
+ MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE);
+
+ }
+
+ @Override
+ protected void chore() {
+
+boolean reported = false;
+
+try (Connection conn = ConnectionFactory.createConnection(conf);
+ Admin admin = conn.getAdmin(); ) {
+
+ if (running) {
+LOG.warn(getName() +" is running already, skipping this attempt.");
+return;
+ }
+ running = true;
+ TableDescriptors htds = master.getTableDescriptors();
+ Map map = htds.getAll();
+ for (TableDescriptor htd : map.values()) {
+if (!master.getTableStateManager().isTableState(htd.getTableName(),
+ TableState.State.ENABLED)) {
+ continue;
+}
+for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
+ if (hcd.isMobEnabled()) {
+if (!reported) {
+ master.reportMobCompactionStart(htd.getTableName());
+ reported = true;
+}
+LOG.info(" Major compacting "+ htd.getTableName() + " cf=" +
hcd.getNameAsString());
+if (regionBatchSize ==
MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE) {
+ admin.majorCompact(htd.getTableName(), hcd.getName());
+} else {
+ performMajorCompactionInBatches(admin, htd, hcd);
+}
+ }
+}
+if (reported) {
+ master.reportMobCompactionEnd(htd.getTableName());
+ reported = false;
+}
+ }
+} catch (Exception e) {
+ LOG.error("Failed to compact", e);
+} finally {
+ running = false;
+}
+ }
+
+ private void performMajorCompactionInBatches(Admin admin, TableDescriptor
htd,
+ ColumnFamilyDescriptor hcd) throws IOException {
+
+List regions = admin.getRegions(htd.getTableName());
+if (regions.size() <= this.regionBatchSize) {
+
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/623#discussion_r325786170
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java
##
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.CompactionState;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.mob.MobConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MobFileCompactionChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCompactionChore.class);
+ private final Configuration conf;
+ private final HMaster master;
+ private volatile boolean running = false;
+ private int regionBatchSize = 0;// not set - compact all
+
+ public MobFileCompactionChore(HMaster master) {
+super(master.getServerName() + "-MobFileCompactionChore", master,
master.getConfiguration()
+ .getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), master
+ .getConfiguration().getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), TimeUnit.SECONDS);
+this.master = master;
+this.conf = master.getConfiguration();
+this.regionBatchSize =
+
master.getConfiguration().getInt(MobConstants.MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE,
+ MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE);
+
+ }
+
+ @Override
+ protected void chore() {
+
+boolean reported = false;
+
+try (Connection conn = ConnectionFactory.createConnection(conf);
+ Admin admin = conn.getAdmin(); ) {
+
+ if (running) {
+LOG.warn(getName() +" is running already, skipping this attempt.");
+return;
+ }
+ running = true;
+ TableDescriptors htds = master.getTableDescriptors();
+ Map map = htds.getAll();
+ for (TableDescriptor htd : map.values()) {
+if (!master.getTableStateManager().isTableState(htd.getTableName(),
+ TableState.State.ENABLED)) {
+ continue;
+}
+for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
+ if (hcd.isMobEnabled()) {
+if (!reported) {
+ master.reportMobCompactionStart(htd.getTableName());
+ reported = true;
+}
+LOG.info(" Major compacting "+ htd.getTableName() + " cf=" +
hcd.getNameAsString());
+if (regionBatchSize ==
MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE) {
+ admin.majorCompact(htd.getTableName(), hcd.getName());
+} else {
+ performMajorCompactionInBatches(admin, htd, hcd);
+}
+ }
+}
+if (reported) {
+ master.reportMobCompactionEnd(htd.getTableName());
+ reported = false;
+}
+ }
+} catch (Exception e) {
+ LOG.error("Failed to compact", e);
+} finally {
+ running = false;
+}
+ }
+
+ private void performMajorCompactionInBatches(Admin admin, TableDescriptor
htd,
+ ColumnFamilyDescriptor hcd) throws IOException {
+
+List regions = admin.getRegions(htd.getTableName());
+if (regions.size() <= this.regionBatchSize) {
Re
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/623#discussion_r325779514
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java
##
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.CompactionState;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.mob.MobConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MobFileCompactionChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCompactionChore.class);
+ private final Configuration conf;
+ private final HMaster master;
+ private volatile boolean running = false;
+ private int regionBatchSize = 0;// not set - compact all
+
+ public MobFileCompactionChore(HMaster master) {
+super(master.getServerName() + "-MobFileCompactionChore", master,
master.getConfiguration()
+ .getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), master
+ .getConfiguration().getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), TimeUnit.SECONDS);
+this.master = master;
+this.conf = master.getConfiguration();
+this.regionBatchSize =
+
master.getConfiguration().getInt(MobConstants.MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE,
+ MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE);
+
+ }
+
+ @Override
+ protected void chore() {
+
+boolean reported = false;
+
+try (Connection conn = ConnectionFactory.createConnection(conf);
+ Admin admin = conn.getAdmin(); ) {
+
+ if (running) {
+LOG.warn(getName() +" is running already, skipping this attempt.");
+return;
+ }
+ running = true;
+ TableDescriptors htds = master.getTableDescriptors();
+ Map map = htds.getAll();
+ for (TableDescriptor htd : map.values()) {
+if (!master.getTableStateManager().isTableState(htd.getTableName(),
+ TableState.State.ENABLED)) {
+ continue;
+}
+for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
+ if (hcd.isMobEnabled()) {
+if (!reported) {
+ master.reportMobCompactionStart(htd.getTableName());
+ reported = true;
+}
+LOG.info(" Major compacting "+ htd.getTableName() + " cf=" +
hcd.getNameAsString());
Review comment:
Please use slf4j substitutions instead of string concatenation.
also can we include in this message wether or not we're going to use the
batched call
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions URL: https://github.com/apache/hbase/pull/623#discussion_r325777898 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java ## @@ -0,0 +1,179 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.master; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.TimeUnit; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.ScheduledChore; +import org.apache.hadoop.hbase.TableDescriptors; +import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.client.Admin; +import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor; +import org.apache.hadoop.hbase.client.CompactionState; +import org.apache.hadoop.hbase.client.Connection; +import org.apache.hadoop.hbase.client.ConnectionFactory; +import org.apache.hadoop.hbase.client.RegionInfo; +import org.apache.hadoop.hbase.client.TableDescriptor; +import org.apache.hadoop.hbase.client.TableState; +import org.apache.hadoop.hbase.mob.MobConstants; +import org.apache.yetus.audience.InterfaceAudience; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +@InterfaceAudience.Private Review comment: this class would benefit from a class comment that explained the planned scope of work, specifically that it's requesting that region servers do the actual compaction. (and that it's trying to spread this work out) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/623#discussion_r325776570
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCleanerChore.java
##
@@ -78,9 +84,15 @@ protected void chore() {
}
}
}
+// Now clean obsolete files for a table
+LOG.info("Cleaning obsolete MOB files ...");
+MobUtils.cleanupObsoleteMobFiles(master.getConfiguration(),
htd.getTableName());
Review comment:
Why is the logic for this broken out into MobUtils? The only other uses I
see are in test code that could just instantiate this chore instead of
reimplementing it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/623#discussion_r325782059
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java
##
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.CompactionState;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.mob.MobConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MobFileCompactionChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCompactionChore.class);
+ private final Configuration conf;
+ private final HMaster master;
+ private volatile boolean running = false;
+ private int regionBatchSize = 0;// not set - compact all
+
+ public MobFileCompactionChore(HMaster master) {
+super(master.getServerName() + "-MobFileCompactionChore", master,
master.getConfiguration()
+ .getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), master
+ .getConfiguration().getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), TimeUnit.SECONDS);
+this.master = master;
+this.conf = master.getConfiguration();
+this.regionBatchSize =
+
master.getConfiguration().getInt(MobConstants.MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE,
+ MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE);
+
+ }
+
+ @Override
+ protected void chore() {
+
+boolean reported = false;
+
+try (Connection conn = ConnectionFactory.createConnection(conf);
+ Admin admin = conn.getAdmin(); ) {
+
+ if (running) {
+LOG.warn(getName() +" is running already, skipping this attempt.");
+return;
+ }
+ running = true;
+ TableDescriptors htds = master.getTableDescriptors();
+ Map map = htds.getAll();
+ for (TableDescriptor htd : map.values()) {
+if (!master.getTableStateManager().isTableState(htd.getTableName(),
+ TableState.State.ENABLED)) {
+ continue;
+}
+for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
+ if (hcd.isMobEnabled()) {
+if (!reported) {
+ master.reportMobCompactionStart(htd.getTableName());
+ reported = true;
+}
+LOG.info(" Major compacting "+ htd.getTableName() + " cf=" +
hcd.getNameAsString());
+if (regionBatchSize ==
MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE) {
+ admin.majorCompact(htd.getTableName(), hcd.getName());
+} else {
+ performMajorCompactionInBatches(admin, htd, hcd);
+}
+ }
+}
+if (reported) {
+ master.reportMobCompactionEnd(htd.getTableName());
+ reported = false;
+}
+ }
+} catch (Exception e) {
Review comment:
we should have a tighter try/catch here so that failure to compact a given
table doesn't prevent trying to compact other tables.
probably should be wrapped at the cf level?
This is an automated message from the Apache Git Service.
To respond to the message
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/623#discussion_r325789978
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java
##
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.CompactionState;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.mob.MobConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MobFileCompactionChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCompactionChore.class);
+ private final Configuration conf;
+ private final HMaster master;
+ private volatile boolean running = false;
+ private int regionBatchSize = 0;// not set - compact all
+
+ public MobFileCompactionChore(HMaster master) {
+super(master.getServerName() + "-MobFileCompactionChore", master,
master.getConfiguration()
+ .getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), master
+ .getConfiguration().getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), TimeUnit.SECONDS);
+this.master = master;
+this.conf = master.getConfiguration();
+this.regionBatchSize =
+
master.getConfiguration().getInt(MobConstants.MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE,
+ MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE);
+
+ }
+
+ @Override
+ protected void chore() {
+
+boolean reported = false;
+
+try (Connection conn = ConnectionFactory.createConnection(conf);
+ Admin admin = conn.getAdmin(); ) {
+
+ if (running) {
+LOG.warn(getName() +" is running already, skipping this attempt.");
+return;
+ }
+ running = true;
+ TableDescriptors htds = master.getTableDescriptors();
+ Map map = htds.getAll();
+ for (TableDescriptor htd : map.values()) {
+if (!master.getTableStateManager().isTableState(htd.getTableName(),
+ TableState.State.ENABLED)) {
+ continue;
+}
+for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
+ if (hcd.isMobEnabled()) {
+if (!reported) {
+ master.reportMobCompactionStart(htd.getTableName());
+ reported = true;
+}
+LOG.info(" Major compacting "+ htd.getTableName() + " cf=" +
hcd.getNameAsString());
+if (regionBatchSize ==
MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE) {
+ admin.majorCompact(htd.getTableName(), hcd.getName());
+} else {
+ performMajorCompactionInBatches(admin, htd, hcd);
+}
+ }
+}
+if (reported) {
+ master.reportMobCompactionEnd(htd.getTableName());
+ reported = false;
+}
+ }
+} catch (Exception e) {
+ LOG.error("Failed to compact", e);
+} finally {
+ running = false;
+}
+ }
+
+ private void performMajorCompactionInBatches(Admin admin, TableDescriptor
htd,
+ ColumnFamilyDescriptor hcd) throws IOException {
+
+List regions = admin.getRegions(htd.getTableName());
+if (regions.size() <= this.regionBatchSize) {
+
[GitHub] [hbase] busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB compactions
busbey commented on a change in pull request #623: HBASE-22749: Distributed MOB
compactions
URL: https://github.com/apache/hbase/pull/623#discussion_r325783912
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java
##
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.CompactionState;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.mob.MobConstants;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MobFileCompactionChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCompactionChore.class);
+ private final Configuration conf;
+ private final HMaster master;
+ private volatile boolean running = false;
+ private int regionBatchSize = 0;// not set - compact all
+
+ public MobFileCompactionChore(HMaster master) {
+super(master.getServerName() + "-MobFileCompactionChore", master,
master.getConfiguration()
+ .getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), master
+ .getConfiguration().getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD), TimeUnit.SECONDS);
+this.master = master;
+this.conf = master.getConfiguration();
+this.regionBatchSize =
+
master.getConfiguration().getInt(MobConstants.MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE,
+ MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE);
+
+ }
+
+ @Override
+ protected void chore() {
+
+boolean reported = false;
+
+try (Connection conn = ConnectionFactory.createConnection(conf);
+ Admin admin = conn.getAdmin(); ) {
+
+ if (running) {
+LOG.warn(getName() +" is running already, skipping this attempt.");
+return;
+ }
+ running = true;
+ TableDescriptors htds = master.getTableDescriptors();
+ Map map = htds.getAll();
+ for (TableDescriptor htd : map.values()) {
+if (!master.getTableStateManager().isTableState(htd.getTableName(),
+ TableState.State.ENABLED)) {
+ continue;
+}
+for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
+ if (hcd.isMobEnabled()) {
+if (!reported) {
+ master.reportMobCompactionStart(htd.getTableName());
+ reported = true;
+}
+LOG.info(" Major compacting "+ htd.getTableName() + " cf=" +
hcd.getNameAsString());
+if (regionBatchSize ==
MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE) {
+ admin.majorCompact(htd.getTableName(), hcd.getName());
+} else {
+ performMajorCompactionInBatches(admin, htd, hcd);
+}
+ }
+}
+if (reported) {
Review comment:
I think we need to do this even if the compaction fails? It's not apparent
from `reportMobCompactionEnd` if it's meant to signify success.
it looks like the state gets use for `getMobCompactionState` and if it still
makes sense to get that information from the master then I don't know what
purpose an indefinitely incrementing number for outstanding mob compactions
would be.
This is an autom
