[jira] [Updated] (HBASE-21717) Implement Connection based on AsyncConnection
[ https://issues.apache.org/jira/browse/HBASE-21717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21717: -- Attachment: HBASE-21717-HBASE-21512.patch > Implement Connection based on AsyncConnection > - > > Key: HBASE-21717 > URL: https://issues.apache.org/jira/browse/HBASE-21717 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > Attachments: HBASE-21717-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19889) Revert Workaround: Purge User API building from branch-2 so can make a beta-1
[ https://issues.apache.org/jira/browse/HBASE-19889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766894#comment-16766894 ] Peter Somogyi commented on HBASE-19889: --- There are 2 report sets where the configuration was disabled. Can you change both? These are under userapi and testuserapi reportSet. The change is needed in from branch-2.0 to branch-2, but only patch is enough, I can cherry-pick it to the remaining branches. > Revert Workaround: Purge User API building from branch-2 so can make a beta-1 > - > > Key: HBASE-19889 > URL: https://issues.apache.org/jira/browse/HBASE-19889 > Project: HBase > Issue Type: Sub-task > Components: website >Reporter: stack >Assignee: stack >Priority: Major > Attachments: hbase-19889.branch-2.0.001.patch, > hbase-19889.branch-2.1.001.patch > > > Root fix looks to be -HBASE-19780- > Let me try it -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21717) Implement Connection based on AsyncConnection
[ https://issues.apache.org/jira/browse/HBASE-21717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766893#comment-16766893 ] Duo Zhang commented on HBASE-21717: --- Implement Table based on AsyncTable. The coprocessor related method may reference the blocking protobuf interface so it can not be executed asynchronously, mark them as deprecated and will be removed in next major release. Implement a Connection based on AsyncConnection. Since we only implemented the Table interface, we still need to keep the ConnectionImplementation to support other methods, such as getAdmin, getBufferedMutator, etc. Will replace them one by one. Introduce a toConnection method in AsyncConnection and a toAsyncConnection method in Connection, to make it more flexible. > Implement Connection based on AsyncConnection > - > > Key: HBASE-21717 > URL: https://issues.apache.org/jira/browse/HBASE-21717 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21717-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21717) Implement Connection based on AsyncConnection
[ https://issues.apache.org/jira/browse/HBASE-21717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21717: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > Implement Connection based on AsyncConnection > - > > Key: HBASE-21717 > URL: https://issues.apache.org/jira/browse/HBASE-21717 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21717-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21888) Add a isClosed method to AsyncConnection
Duo Zhang created HBASE-21888: - Summary: Add a isClosed method to AsyncConnection Key: HBASE-21888 URL: https://issues.apache.org/jira/browse/HBASE-21888 Project: HBase Issue Type: Task Components: asyncclient, Client Reporter: Duo Zhang Fix For: 3.0.0, 2.2.0, 2.3.0 Align with Connection interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21057) upgrade to latest spotbugs
[
https://issues.apache.org/jira/browse/HBASE-21057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766868#comment-16766868
]
Hadoop QA commented on HBASE-21057:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 42s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}216m
3s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}262m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21057 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958495/HBASE-21057.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit shadedjars
hadoopcheck xml compile |
| uname | Linux 72bcc4bc6ddf 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f1e5999ad2 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15945/testReport/ |
| Max. process+thread count | 5004 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15945/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> upgrade to latest spotbugs
> --
>
> Key: HBASE-21057
> URL: https://issues.apache.org/jira/browse/HBASE-21057
> Project: HBase
> Issue Type: Task
> Components: community, test
>Reporter: Sean Busbey
>Assignee: kevin su
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 1.6.0, 2.2.0
>
> Attachments: HBASE-21057.master.001.patch,
[jira] [Created] (HBASE-21887) HBaseTestingUtility should not be IA.Public
Duo Zhang created HBASE-21887: - Summary: HBaseTestingUtility should not be IA.Public Key: HBASE-21887 URL: https://issues.apache.org/jira/browse/HBASE-21887 Project: HBase Issue Type: Task Components: Client, test Reporter: Duo Zhang It exposes too many internal stuffs to end user, and it is not easy to keep the API stable, as it is also used by us to implementing UTs. For end users, we should only exposes API to start a in process mini hbase cluster, and then user can create Connections to communicate with the cluster. And could exposes several APIs to start/stop/restart master regionservers. I think this is enough? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21884) Fix box/unbox findbugs warning in secure bulk load
[
https://issues.apache.org/jira/browse/HBASE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766846#comment-16766846
]
Hadoop QA commented on HBASE-21884:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
32s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
41s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
36s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}136m 59s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}161m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapred.TestTableSnapshotInputFormat |
| | hadoop.hbase.coprocessor.TestMetaTableMetrics |
| | hadoop.hbase.client.TestAdmin1 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21884 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-21879) Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
[
https://issues.apache.org/jira/browse/HBASE-21879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766838#comment-16766838
]
Anoop Sam John commented on HBASE-21879:
This direct copy is possible iff it is an off heap DRAM bucket cache right?
For other types of IOEngine, still the same path has to be there. Need to see
how complicated the code will be then. Or should we try use the BB pool (off
heap) available in RS? We use this pool at the RPC layer now. If we have
buffers available, make use of them? Then the Q is what if the size of the
block > one pooled buffer. Use multi buffers? Am just trying to give diff
options here.. Agree that we can try to solve this young GC issue.
> Read HFile's block to ByteBuffer directly instead of to byte for reducing
> young gc purpose
> --
>
> Key: HBASE-21879
> URL: https://issues.apache.org/jira/browse/HBASE-21879
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.1.4
>
> Attachments: QPS-latencies-before-HBASE-21879.png,
> gc-data-before-HBASE-21879.png
>
>
> In HFileBlock#readBlockDataInternal, we have the following:
> {code}
> @VisibleForTesting
> protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset,
> long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum,
> boolean updateMetrics)
> throws IOException {
> // .
> // TODO: Make this ByteBuffer-based. Will make it easier to go to HDFS with
> BBPool (offheap).
> byte [] onDiskBlock = new byte[onDiskSizeWithHeader + hdrSize];
> int nextBlockOnDiskSize = readAtOffset(is, onDiskBlock, preReadHeaderSize,
> onDiskSizeWithHeader - preReadHeaderSize, true, offset +
> preReadHeaderSize, pread);
> if (headerBuf != null) {
> // ...
> }
> // ...
> }
> {code}
> In the read path, we still read the block from hfile to on-heap byte[], then
> copy the on-heap byte[] to offheap bucket cache asynchronously, and in my
> 100% get performance test, I also observed some frequent young gc, The
> largest memory footprint in the young gen should be the on-heap block byte[].
> In fact, we can read HFile's block to ByteBuffer directly instead of to
> byte[] for reducing young gc purpose. we did not implement this before,
> because no ByteBuffer reading interface in the older HDFS client, but 2.7+
> has supported this now, so we can fix this now. I think.
> Will provide an patch and some perf-comparison for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21814) Remove the TODO in AccessControlLists#addUserPermission
[
https://issues.apache.org/jira/browse/HBASE-21814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766833#comment-16766833
]
Anoop Sam John commented on HBASE-21814:
Which jira added this TODO? The comment is bit confusing so let me read the
jira history to remember things back. :-)
> Remove the TODO in AccessControlLists#addUserPermission
> ---
>
> Key: HBASE-21814
> URL: https://issues.apache.org/jira/browse/HBASE-21814
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.3.0
>
> Attachments: HBASE-21814.master.001.patch,
> HBASE-21814.master.001.patch, HBASE-21814.master.002.patch,
> HBASE-21814.master.002.patch
>
>
> The TODO was added by me. Because this method happens within the RS. The old
> impl use a login user(User.runAsLoginUser where the login user is the user
> who started RS process) to call Table.put(). And it will check the permission
> when put record to ACL table. At RpcServer we have a ThreadLocal where we
> keep the CallContext and inside that the current RPC called user info is set.
> We need Table.put(List) to change to a new thread and and so old
> ThreadLocal variable is not accessible and so it looks as if no Rpc context
> and we were relying on the super user who starts the RS process.
>
> {code:java}
> User.runAsLoginUser(new PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
>
> AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm,
> regionEnv.getTable(AccessControlLists.ACL_TABLE_NAME),
> request.getMergeExistingPermissions());
> return null;
> }
> });
> {code}
>
> But after HBASE-21739, no need to User.runAsLoginUser. Because we will call
> Admin method to grant/revoke. And this will be execute in master and use the
> master user(the user who started master process) to call Table.put. So this
> is not a problem now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21814) Remove the TODO in AccessControlLists#addUserPermission
[
https://issues.apache.org/jira/browse/HBASE-21814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766836#comment-16766836
]
Guanghao Zhang commented on HBASE-21814:
{quote}bq. Which jira added this TODO?
{quote}
Please check HBASE-18500. Thanks.
> Remove the TODO in AccessControlLists#addUserPermission
> ---
>
> Key: HBASE-21814
> URL: https://issues.apache.org/jira/browse/HBASE-21814
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.3.0
>
> Attachments: HBASE-21814.master.001.patch,
> HBASE-21814.master.001.patch, HBASE-21814.master.002.patch,
> HBASE-21814.master.002.patch
>
>
> The TODO was added by me. Because this method happens within the RS. The old
> impl use a login user(User.runAsLoginUser where the login user is the user
> who started RS process) to call Table.put(). And it will check the permission
> when put record to ACL table. At RpcServer we have a ThreadLocal where we
> keep the CallContext and inside that the current RPC called user info is set.
> We need Table.put(List) to change to a new thread and and so old
> ThreadLocal variable is not accessible and so it looks as if no Rpc context
> and we were relying on the super user who starts the RS process.
>
> {code:java}
> User.runAsLoginUser(new PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
>
> AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm,
> regionEnv.getTable(AccessControlLists.ACL_TABLE_NAME),
> request.getMergeExistingPermissions());
> return null;
> }
> });
> {code}
>
> But after HBASE-21739, no need to User.runAsLoginUser. Because we will call
> Admin method to grant/revoke. And this will be execute in master and use the
> master user(the user who started master process) to call Table.put. So this
> is not a problem now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21875) Change the retry logic in RSProcedureDispatcher to 'retry by default, only if xxx'
[
https://issues.apache.org/jira/browse/HBASE-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766831#comment-16766831
]
Hadoop QA commented on HBASE-21875:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 25s{color}
| {color:red} hbase-server generated 3 new + 185 unchanged - 3 fixed = 188
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
2s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 51s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}143m 57s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}189m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.TestSyncReplicationStandbyKillRS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21875 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958499/HBASE-21875-v1.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 709b4d04e333 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / f1e5999ad2 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/15948/artifact/patchprocess/diff-compile-javac-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/15948/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test
[jira] [Updated] (HBASE-21884) Fix box/unbox findbugs warning in secure bulk load
[
https://issues.apache.org/jira/browse/HBASE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21884:
Fix Version/s: 2.0.6
2.1.4
2.3.0
2.2.0
3.0.0
> Fix box/unbox findbugs warning in secure bulk load
> --
>
> Key: HBASE-21884
> URL: https://issues.apache.org/jira/browse/HBASE-21884
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 1.5.0, 2.2.0, 2.1.1, 2.0.3, 2.1.2, 2.0.4, 1.4.10,
> 1.3.4, 1.2.11, 2.3.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 1.4.10, 1.3.4, 1.2.11, 2.3.0, 2.1.4,
> 2.0.6, 1.5.1
>
> Attachments: HBASE-21884-branch-1.v0.patch
>
>
> {code}
> ReasonTests
> FindBugs module:hbase-server
> Boxed value is unboxed and then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:[line 268]
> {code}
> Looking at branch-2 and master I suspect we're doing the same wasteful
> operation but findbugs can't see it through the lambda definition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21814) Remove the TODO in AccessControlLists#addUserPermission
[
https://issues.apache.org/jira/browse/HBASE-21814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766805#comment-16766805
]
Guanghao Zhang commented on HBASE-21814:
Ping [~Apache9] [~stack] [~anoop.hbase] for reviewing.
> Remove the TODO in AccessControlLists#addUserPermission
> ---
>
> Key: HBASE-21814
> URL: https://issues.apache.org/jira/browse/HBASE-21814
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.3.0
>
> Attachments: HBASE-21814.master.001.patch,
> HBASE-21814.master.001.patch, HBASE-21814.master.002.patch,
> HBASE-21814.master.002.patch
>
>
> The TODO was added by me. Because this method happens within the RS. The old
> impl use a login user(User.runAsLoginUser where the login user is the user
> who started RS process) to call Table.put(). And it will check the permission
> when put record to ACL table. At RpcServer we have a ThreadLocal where we
> keep the CallContext and inside that the current RPC called user info is set.
> We need Table.put(List) to change to a new thread and and so old
> ThreadLocal variable is not accessible and so it looks as if no Rpc context
> and we were relying on the super user who starts the RS process.
>
> {code:java}
> User.runAsLoginUser(new PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
>
> AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm,
> regionEnv.getTable(AccessControlLists.ACL_TABLE_NAME),
> request.getMergeExistingPermissions());
> return null;
> }
> });
> {code}
>
> But after HBASE-21739, no need to User.runAsLoginUser. Because we will call
> Admin method to grant/revoke. And this will be execute in master and use the
> master user(the user who started master process) to call Table.put. So this
> is not a problem now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21884) Fix box/unbox findbugs warning in secure bulk load
[
https://issues.apache.org/jira/browse/HBASE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766789#comment-16766789
]
Sean Busbey commented on HBASE-21884:
-
okay I have a patch for master/branches-2 that removes similar autoboxing
there. I'll attach it once QABot has a go at this branches-1 version.
> Fix box/unbox findbugs warning in secure bulk load
> --
>
> Key: HBASE-21884
> URL: https://issues.apache.org/jira/browse/HBASE-21884
> Project: HBase
> Issue Type: Task
>Affects Versions: 1.5.0, 1.4.10, 1.3.4, 1.2.11
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 1.4.10, 1.3.4, 1.2.11, 1.5.1
>
> Attachments: HBASE-21884-branch-1.v0.patch
>
>
> {code}
> ReasonTests
> FindBugs module:hbase-server
> Boxed value is unboxed and then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:[line 268]
> {code}
> Looking at branch-2 and master I suspect we're doing the same wasteful
> operation but findbugs can't see it through the lambda definition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21884) Fix box/unbox findbugs warning in secure bulk load
[
https://issues.apache.org/jira/browse/HBASE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21884:
Affects Version/s: 2.3.0
2.2.0
3.0.0
2.1.1
2.0.3
2.1.2
2.0.4
> Fix box/unbox findbugs warning in secure bulk load
> --
>
> Key: HBASE-21884
> URL: https://issues.apache.org/jira/browse/HBASE-21884
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 1.5.0, 2.2.0, 2.1.1, 2.0.3, 2.1.2, 2.0.4, 1.4.10,
> 1.3.4, 1.2.11, 2.3.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 1.4.10, 1.3.4, 1.2.11, 1.5.1
>
> Attachments: HBASE-21884-branch-1.v0.patch
>
>
> {code}
> ReasonTests
> FindBugs module:hbase-server
> Boxed value is unboxed and then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:[line 268]
> {code}
> Looking at branch-2 and master I suspect we're doing the same wasteful
> operation but findbugs can't see it through the lambda definition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21814) Remove the TODO in AccessControlLists#addUserPermission
[
https://issues.apache.org/jira/browse/HBASE-21814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766784#comment-16766784
]
Hadoop QA commented on HBASE-21814:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
19s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
6s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 21s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}140m
7s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}186m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21814 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958489/HBASE-21814.master.002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9199a9c51936 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 9ef6bc4323 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15944/testReport/ |
| Max. process+thread count | 5264 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Updated] (HBASE-21884) Fix box/unbox findbugs warning in secure bulk load
[
https://issues.apache.org/jira/browse/HBASE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21884:
Fix Version/s: 1.5.1
1.2.11
1.3.4
1.4.10
> Fix box/unbox findbugs warning in secure bulk load
> --
>
> Key: HBASE-21884
> URL: https://issues.apache.org/jira/browse/HBASE-21884
> Project: HBase
> Issue Type: Task
>Affects Versions: 1.5.0, 1.4.10, 1.3.4, 1.2.11
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 1.4.10, 1.3.4, 1.2.11, 1.5.1
>
> Attachments: HBASE-21884-branch-1.v0.patch
>
>
> {code}
> ReasonTests
> FindBugs module:hbase-server
> Boxed value is unboxed and then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:[line 268]
> {code}
> Looking at branch-2 and master I suspect we're doing the same wasteful
> operation but findbugs can't see it through the lambda definition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20060) Add details of off heap memstore into book.
[ https://issues.apache.org/jira/browse/HBASE-20060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766745#comment-16766745 ] Zheng Hu commented on HBASE-20060: -- Well, assigned to myself. > Add details of off heap memstore into book. > --- > > Key: HBASE-20060 > URL: https://issues.apache.org/jira/browse/HBASE-20060 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Anoop Sam John >Assignee: Zheng Hu >Priority: Critical > Fix For: 2.2.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20060) Add details of off heap memstore into book.
[ https://issues.apache.org/jira/browse/HBASE-20060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu reassigned HBASE-20060: Assignee: Zheng Hu (was: Anoop Sam John) > Add details of off heap memstore into book. > --- > > Key: HBASE-20060 > URL: https://issues.apache.org/jira/browse/HBASE-20060 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Anoop Sam John >Assignee: Zheng Hu >Priority: Critical > Fix For: 2.2.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21886) Run ITBLL for branch-2.2
[ https://issues.apache.org/jira/browse/HBASE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-21886: -- Assignee: Guanghao Zhang > Run ITBLL for branch-2.2 > > > Key: HBASE-21886 > URL: https://issues.apache.org/jira/browse/HBASE-21886 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21785) master reports open regions as RITs and also messes up rit age metric
[
https://issues.apache.org/jira/browse/HBASE-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766738#comment-16766738
]
Hudson commented on HBASE-21785:
Results for branch branch-2.2
[build #37 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/37/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/37//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/37//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/37//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> master reports open regions as RITs and also messes up rit age metric
> -
>
> Key: HBASE-21785
> URL: https://issues.apache.org/jira/browse/HBASE-21785
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21785.01.patch, HBASE-21785.patch
>
>
> {noformat}
> RegionState RIT time (ms) Retries
> dba183f0dadfcc9dc8ae0a6dd59c84e6 dba183f0dadfcc9dc8ae0a6dd59c84e6.
> state=OPEN, ts=Wed Dec 31 16:00:00 PST 1969 (1548453918s ago),
> server=server,17020,1548452922054 1548453918735 0
> {noformat}
> RIT age metric also gets set to a bogus value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21854) Race condition in TestProcedureSkipPersistence
[
https://issues.apache.org/jira/browse/HBASE-21854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766736#comment-16766736
]
Hudson commented on HBASE-21854:
Results for branch branch-2.0
[build #1341 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1341/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1341//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1341//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1341//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Race condition in TestProcedureSkipPersistence
> ---
>
> Key: HBASE-21854
> URL: https://issues.apache.org/jira/browse/HBASE-21854
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Affects Versions: 2.1.3
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.0.5, 2.3.0, 2.1.4
>
> Attachments: HBASE-21854.patch
>
>
> There is a race condition in TestProcedureSkipPersistence. After the
> procedure is added, the test stops ProcedureExecutor. In some cases the
> procedure is not added to the queue in time.
> Failing execution:
> {noformat}
> 2019-02-06 14:18:11,133 INFO [Time-limited test]
> procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=3,
> timestamp=1549491521133
> 2019-02-06 14:18:11,135 INFO [PEWorker-1]
> procedure2.TimeoutExecutorThread(82): ADDED pid=1, state=WAITING_TIMEOUT,
> locked=true;
> org.apache.hadoop.hbase.procedure2.TestProcedureSkipPersistence$TestProcedure;
> timeout=2000, timestamp=1549491493135
> 2019-02-06 14:18:11,137 INFO [Time-limited test] hbase.Waiter(189): Waiting
> up to [30,000] milli-secs(wait.for.ratio=[1])
> 2019-02-06 14:18:11,139 INFO [Time-limited test]
> procedure2.ProcedureTestingUtility(125): RESTART - Stop
> 2019-02-06 14:18:11,139 INFO [Time-limited test]
> procedure2.ProcedureExecutor(702): Stopping
> 2019-02-06 14:18:11,139 INFO [Time-limited test] wal.WALProcedureStore(331):
> Stopping the WAL Procedure Store, isAbort=false
> 2019-02-06 14:18:11,140 DEBUG [PEWorker-1]
> procedure2.RootProcedureState(153): Add procedure pid=1,
> state=WAITING_TIMEOUT, locked=true;
> org.apache.hadoop.hbase.procedure2.TestProcedureSkipPersistence$TestProcedure
> as the 0th rollback step
> 2019-02-06 14:18:11,141 WARN [PEWorker-1]
> procedure2.ProcedureExecutor$WorkerThread(2074): Worker terminating
> UNNATURALLY null
> java.lang.RuntimeException: the store must be running before inserting data
>at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:710)
>at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.update(WALProcedureStore.java:603)
>at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.updateStoreOnExec(ProcedureExecutor.java:1943)
>at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1809)
>at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1481)
>at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1200(ProcedureExecutor.java:78)
>at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2058)
> 2019-02-06 14:18:11,145 INFO [Time-limited test]
> procedure2.ProcedureTestingUtility(137): RESTART - Start{noformat}
> In a successful run the ProcExecutor is stopped AFTER the procedure is
> actually in the queue.
> Successful:
> {noformat}
> 2019-02-07 15:48:08,731 INFO [Time-limited test]
> procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=3,
> timestamp=1549550918731
> 2019-02-07 15:48:08,731 INFO [PEWorker-1]
> procedure2.TimeoutExecutorThread(82): ADDED pid=1, state=WAITING_TIMEOUT,
> locked=true;
> org.apache.hadoop.hbase.procedure2.TestProcedureSkipPersistence$TestProcedure;
> timeout=2000, timestamp=1549550890731
> 2019-02-07 15:48:08,732 DEBUG [PEWorker-1]
> procedure2.RootProcedureState(153): Add procedure pid=1,
> state=WAITING_TIMEOUT, locked=true;
>
[jira] [Created] (HBASE-21886) Run ITBLL for branch-2.2
Guanghao Zhang created HBASE-21886: -- Summary: Run ITBLL for branch-2.2 Key: HBASE-21886 URL: https://issues.apache.org/jira/browse/HBASE-21886 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21885) Cancel remote procedure call if the remote procedure is succeeded
[ https://issues.apache.org/jira/browse/HBASE-21885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766735#comment-16766735 ] Duo Zhang commented on HBASE-21885: --- [~sershe] [~stack] FYI. > Cancel remote procedure call if the remote procedure is succeeded > - > > Key: HBASE-21885 > URL: https://issues.apache.org/jira/browse/HBASE-21885 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Priority: Major > > I used to think it could rarely rarely happen that a region server can report > back to master but master can not get the response from region server, only > if there are strange network errors. But when implementing HBASE-21875, I > found a way to reproduce the problem without any strange network issues. > First time, we send the request to region server, and it accept the request, > but before returning, there is a network error cause the connection to be > broken, so master will try to send the request to the region server again. > But then the region server gets too busy, and always returns > CallQueueTooBigException, then the master will retry forever, even if the > region has already been opened on the region server. > And this is not only waste more resources, as later we may close the region > on the region server, and if the region server is back, we will receive an > open region requst and a close region request at the same time. Not sure if > this will cause any problems but at least, we haven't thought this condition > yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21884) Fix box/unbox findbugs warning in secure bulk load
[
https://issues.apache.org/jira/browse/HBASE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21884:
Status: Patch Available (was: In Progress)
v0 for branches-1
- keep the same structure, but use common-lang's {{MutableInt}} to avoid
autoboxing
> Fix box/unbox findbugs warning in secure bulk load
> --
>
> Key: HBASE-21884
> URL: https://issues.apache.org/jira/browse/HBASE-21884
> Project: HBase
> Issue Type: Task
>Affects Versions: 1.5.0, 1.4.10, 1.3.4, 1.2.11
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Attachments: HBASE-21884-branch-1.v0.patch
>
>
> {code}
> ReasonTests
> FindBugs module:hbase-server
> Boxed value is unboxed and then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:[line 268]
> {code}
> Looking at branch-2 and master I suspect we're doing the same wasteful
> operation but findbugs can't see it through the lambda definition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21884) Fix box/unbox findbugs warning in secure bulk load
[
https://issues.apache.org/jira/browse/HBASE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21884:
Attachment: HBASE-21884-branch-1.v0.patch
> Fix box/unbox findbugs warning in secure bulk load
> --
>
> Key: HBASE-21884
> URL: https://issues.apache.org/jira/browse/HBASE-21884
> Project: HBase
> Issue Type: Task
>Affects Versions: 1.5.0, 1.4.10, 1.3.4, 1.2.11
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Attachments: HBASE-21884-branch-1.v0.patch
>
>
> {code}
> ReasonTests
> FindBugs module:hbase-server
> Boxed value is unboxed and then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:[line 268]
> {code}
> Looking at branch-2 and master I suspect we're doing the same wasteful
> operation but findbugs can't see it through the lambda definition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20623) Introduce the helper method "getCellBuilder()" to Mutation
[
https://issues.apache.org/jira/browse/HBASE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20623:
---
Fix Version/s: (was: 2.2.0)
2.3.0
> Introduce the helper method "getCellBuilder()" to Mutation
> --
>
> Key: HBASE-20623
> URL: https://issues.apache.org/jira/browse/HBASE-20623
> Project: HBase
> Issue Type: Task
> Components: API
>Reporter: Chia-Ping Tsai
>Assignee: maoling
>Priority: Minor
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-20623.master.001.patch,
> HBASE-20623.master.002.patch, HBASE-20623.master.003.patch,
> HBASE-20623.master.004.patch, HBASE-20623.master.005.patch,
> HBASE-20623.master.006.patch, HBASE-20623.master.007.patch,
> HBASE-20623.master.008.patch, HBASE-20623.master.009.patch
>
>
> see
> [https://lists.apache.org/thread.html/d05bfaa0134502a47f6e1aca56cb0b096d4dd32ddefbbdf28db4952a@%3Cdev.hbase.apache.org%3E]
> for more details.
> {code:java}
> How about a "getCellBuilder" or "getCellBuilderFactory" method for
> Mutation implementations that gives you a CellBuilder instance that
> already has relevant parts set? Like for a Put instance it should be
> able to already have the Type and Row set.{code}
> mentioned a day or so ago by [~busbey]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21885) Cancel remote procedure call if the remote procedure is succeeded
Duo Zhang created HBASE-21885: - Summary: Cancel remote procedure call if the remote procedure is succeeded Key: HBASE-21885 URL: https://issues.apache.org/jira/browse/HBASE-21885 Project: HBase Issue Type: Improvement Components: proc-v2 Reporter: Duo Zhang I used to think it could rarely rarely happen that a region server can report back to master but master can not get the response from region server, only if there are strange network errors. But when implementing HBASE-21875, I found a way to reproduce the problem without any strange network issues. First time, we send the request to region server, and it accept the request, but before returning, there is a network error cause the connection to be broken, so master will try to send the request to the region server again. But then the region server gets too busy, and always returns CallQueueTooBigException, then the master will retry forever, even if the region has already been opened on the region server. And this is not only waste more resources, as later we may close the region on the region server, and if the region server is back, we will receive an open region requst and a close region request at the same time. Not sure if this will cause any problems but at least, we haven't thought this condition yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20587) Replace Jackson with shaded thirdparty gson in client
[ https://issues.apache.org/jira/browse/HBASE-20587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766729#comment-16766729 ] Guanghao Zhang commented on HBASE-20587: [~elserj] Any updates here, sir? If not, will move this to 2.3.0. Thanks. > Replace Jackson with shaded thirdparty gson in client > - > > Key: HBASE-20587 > URL: https://issues.apache.org/jira/browse/HBASE-20587 > Project: HBase > Issue Type: Bug > Components: dependencies >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20587.001.patch > > > HBASE-20582 got me looking at how we use Jackson. It appears that we moved > some JSON code from hbase-server into hbase-common via HBASE-19053. But, > there seems to be no good reason why this code should live there and not in > hbase-http instead. Keeping Jackson off the user's classpath is a nice goal. > FYI [~appy], [~mdrob] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20305) Add option to SyncTable that skip deletes on target cluster
[ https://issues.apache.org/jira/browse/HBASE-20305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20305: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Add option to SyncTable that skip deletes on target cluster > --- > > Key: HBASE-20305 > URL: https://issues.apache.org/jira/browse/HBASE-20305 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 2.0.0-alpha-4 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 3.0.0, 2.3.0, 1.5.1 > > Attachments: 0001-HBASE-20305.master.001.patch, > HBASE-20305.master.002.patch > > > We had a situation where two clusters with active-active replication got out > of sync, but both had data that should be kept. The tables in question never > have data deleted, but ingestion had happened on the two different clusters, > some rows had been even updated. > In this scenario, a cell that is present in one of the table clusters should > not be deleted, but replayed on the other. Also, for cells with same > identifier but different values, the most recent value should be kept. > Current version of SyncTable would not be applicable here, because it would > simply copy the whole state from source to target, then losing any additional > rows that might be only in target, as well as cell values that got most > recent update. This could be solved by adding an option to skip deletes for > SyncTable. This way, the additional cells not present on source would still > be kept. For cells with same identifier but different values, it would just > perform a Put for the cell version from source, but client scans would still > fetch the most recent timestamp. > I'm attaching a patch with this additional option shortly. Please share your > thoughts. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20553) Add dependency CVE checking to nightly tests
[
https://issues.apache.org/jira/browse/HBASE-20553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20553:
---
Fix Version/s: (was: 2.2.0)
2.3.0
> Add dependency CVE checking to nightly tests
>
>
> Key: HBASE-20553
> URL: https://issues.apache.org/jira/browse/HBASE-20553
> Project: HBase
> Issue Type: Umbrella
> Components: dependencies
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Sakthi
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> We should proactively work to flag dependencies with known CVEs so that we
> can then update them early in our development instead of near a release.
> YETUS-441 is working to add a plugin for this, we should grab a copy early to
> make sure it works for us.
> Rough outline:
> 1. [install yetus locally|http://yetus.apache.org/downloads/]
> 2. [install the dependency-check
> cli|https://www.owasp.org/index.php/OWASP_Dependency_Check] (homebrew
> instructions on right hand margin)
> 3. Get a local copy of the OWASP datafile ({{dependency-check --updateonly
> --data /some/local/path/to/dir}})
> 4. Run {{hbase_nightly_yetus.sh}} using matching environment variables from
> the “yetus general check” (currently [line #126 in our nightly
> Jenkinsfile|https://github.com/apache/hbase/blob/master/dev-support/Jenkinsfile#L126])
> 5. Grab the plugin definition and suppression file from from YETUS-441
> 6. put the plugin definition either in a directory of dev-support or into the
> hbase-personality.sh directly
> 7. Re-run {{hbase_nightly_yetus.sh}} to verify that the plugin results show
> up. (Probably this will involve adding new pointers for “where is the
> suppression file”, “where is the OWASP datafile” and pointing them somewhere
> locally.)
> Once all of that is in place we’ll get the changes needed into a branch that
> we can test out. Over in YETUS-441 I’ll need to add a jenkins job that’ll
> handle periodically updating a copy of the datafile for the OWASP dependency
> checker. Presuming I have that in place by the time we have a nightly branch
> to check this out, then we’ll also need to update our nightly Jenkinsfile to
> fetch the data file from that job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20586) SyncTable tool: Add support for cross-realm remote clusters
[
https://issues.apache.org/jira/browse/HBASE-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20586:
---
Fix Version/s: (was: 2.2.0)
2.3.0
> SyncTable tool: Add support for cross-realm remote clusters
> ---
>
> Key: HBASE-20586
> URL: https://issues.apache.org/jira/browse/HBASE-20586
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce, Operability, Replication
>Affects Versions: 1.2.0, 2.0.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 2.3.0, 1.5.1
>
> Attachments: HBASE-20586.master.001.patch
>
>
> One possible scenario for HashTable/SyncTable is for synchronize different
> clusters, for instance, when replication has been enabled but data existed
> already, or due replication issues that may had caused long lags in the
> replication.
> For secured clusters under different kerberos realms (with cross-realm
> properly set), though, current SyncTable version would fail to authenticate
> with the remote cluster when trying to read HashTable outputs (when
> *sourcehashdir* is remote) and also when trying to read table data on the
> remote cluster (when *sourcezkcluster* is remote).
> The hdfs error would look like this:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_105392_m_00_0, Status
> : FAILED
> Error: java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is: "local-host/1.1.1.1";
> destination host is: "remote-nn":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1506)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy13.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
> ...
> at
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.readPropertiesFile(HashTable.java:144)
> at
> org.apache.hadoop.hbase.mapreduce.HashTable$TableHash.read(HashTable.java:105)
> at
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.setup(SyncTable.java:188)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> ...
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]{noformat}
> The above can be sorted if the SyncTable job acquires a DT for the remote NN.
> Once hdfs related authentication is done, it's also necessary to authenticate
> against remote HBase, as the below error would arise:
> {noformat}
> INFO mapreduce.Job: Task Id : attempt_1524358175778_172414_m_00_0, Status
> : FAILED
> Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get
> the location
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:326)
> ...
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:867)
> at
> org.apache.hadoop.hbase.mapreduce.SyncTable$SyncMapper.syncRange(SyncTable.java:331)
> ...
> Caused by: java.io.IOException: Could not set up IO Streams to
> remote-rs-host/1.1.1.2:60020
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:786)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most
> likely cause is missing or invalid credentials. Consider 'kinit'.
> ...
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> ...{noformat}
> The above would need additional authentication logic against the remote hbase
> cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20455) [IHC] Workloadx
[
https://issues.apache.org/jira/browse/HBASE-20455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20455:
---
Fix Version/s: (was: 2.2.0)
2.3.0
> [IHC] Workloadx
> ---
>
> Key: HBASE-20455
> URL: https://issues.apache.org/jira/browse/HBASE-20455
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> I tried workloadx from parent issue, the ycsb workload that is meant to make
> IHC shine. I'm doing something wrong. I tried just plugging it in and
> doing this:
> {code}
> ycsb run hbase12 -P /home/stack/ycsb/workloads/workloadx -p table=ycsb
> -threads 48 -cp /home/stack/conf_hbase -p columnfamily=family -p
> clientbuffering=true -p writebuffersize=2097152 -s -p maxexecutiontime=1200
> -jvm-args=-Xmx8192
> m -Djava.security.egd=file:/dev/./urandom -p recordcount=2500 -p
> operationcount=2500 -p
> exportfile=logs/ycsb-workloadx-measurements-ve0524-20180419T02:58:14Z.json -p
> exporter=com.yahoo.ycsb.measurements.exporter.JSONArrayMeasurementsExporter
> {code}
> It completes in a minute and a half. Claims stuff like this for throughput:
> 236825 ops/second. In this case I have in-memory-compaction set to NONE. I
> was going to compare before and after.
> [~eshcar]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19952) Find tests which are declared with wrong category
[ https://issues.apache.org/jira/browse/HBASE-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-19952: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Find tests which are declared with wrong category > - > > Key: HBASE-19952 > URL: https://issues.apache.org/jira/browse/HBASE-19952 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.3.0 > > Attachments: category-report > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20448) update ref guide to expressly use shaded clients for examples
[ https://issues.apache.org/jira/browse/HBASE-20448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766728#comment-16766728 ] Guanghao Zhang commented on HBASE-20448: Ping [~busbey]. Any updates? Can finish it before release 2.2.0? > update ref guide to expressly use shaded clients for examples > - > > Key: HBASE-20448 > URL: https://issues.apache.org/jira/browse/HBASE-20448 > Project: HBase > Issue Type: Improvement > Components: Client, documentation, mapreduce >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Blocker > Fix For: 2.2.0 > > > the whole mapreduce section, especially, should be using the shaded version. > Specifically include an example of running the ITBLL we ship -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20300) Minor refactor for RpcExecutor
[
https://issues.apache.org/jira/browse/HBASE-20300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766725#comment-16766725
]
Hadoop QA commented on HBASE-20300:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-20300 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-20300 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12916749/HBASE-20300.v0.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15949/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Minor refactor for RpcExecutor
> --
>
> Key: HBASE-20300
> URL: https://issues.apache.org/jira/browse/HBASE-20300
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-20300.v0.patch, HBASE-20300.v0.patch.patch
>
>
> Plan to do the following changes.
> # make Handler be static class
> # move the threadlocal variables of MonitoredRPCHandler from RpcServer to
> FifoRpcScheduler since only FifoRpcScheduler use it
> # create MonitoredRPCHandler in Handler constuction instead of saving the
> MonitoredRPCHandler in threadlocal variables. In FPBQ mode, the web UI can
> display all Handlers info even if the rpc Handlers are not used yet.
> # Threshhold -> Threshold
> # make RpcExecutor extend ConfigurationObserver
> # don't create task filter repeatly
> # add a ut to check whether each Handler has created own MonitoredTask even
> if no ops
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20383) [AMv2] AssignmentManager: Failed transition XYZ is not OPEN
[
https://issues.apache.org/jira/browse/HBASE-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20383:
---
Fix Version/s: (was: 2.2.0)
2.3.0
> [AMv2] AssignmentManager: Failed transition XYZ is not OPEN
> ---
>
> Key: HBASE-20383
> URL: https://issues.apache.org/jira/browse/HBASE-20383
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-20383.master.001.patch
>
>
> Seeing a bunch of this testing 2.0.0:
> {code}
> 2018-04-10 13:57:09,430 WARN
> [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000]
> assignment.AssignmentManager: Failed transition
>
>
> org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> 19a2cd6f88abae0036415ee1ea041c2e is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:193)
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.(SplitTableRegionProcedure.java:112)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createSplitProcedure(AssignmentManager.java:769)
>
>
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionSplitTransition(AssignmentManager.java:911)
>
>
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:819)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1538)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:11093)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>
>
>at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> Looks like report back from Master OK'ing a split to go ahead but the split
> is already running. Figure how to shut these down. They are noisy at least.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20281) [DOC] upgrade section needs an explanation of changes to Bucket Cache
[
https://issues.apache.org/jira/browse/HBASE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20281:
---
Fix Version/s: (was: 2.2.0)
2.3.0
> [DOC] upgrade section needs an explanation of changes to Bucket Cache
> -
>
> Key: HBASE-20281
> URL: https://issues.apache.org/jira/browse/HBASE-20281
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache, documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> the first pass version of the upgrade docs has a brief note about the bucket
> cache changing:
> {quote}
> * hbase.bucketcache.combinedcache.enabled
> * hbase.bucketcache.ioengine no longer supports the 'heap' value.
> {quote}
> But the changes are substantial and warrant a section of their own under the
> "Changes of Note!" banner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21875) Change the retry logic in RSProcedureDispatcher to 'retry by default, only if xxx'
[ https://issues.apache.org/jira/browse/HBASE-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21875: -- Attachment: HBASE-21875-v1.patch > Change the retry logic in RSProcedureDispatcher to 'retry by default, only if > xxx' > -- > > Key: HBASE-21875 > URL: https://issues.apache.org/jira/browse/HBASE-21875 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.3.0 > > Attachments: HBASE-21875-v1.patch, HBASE-21875.patch > > > For now it is not retry by default, only if xxx. > In executeProcedures, we will only throw a fixed set of exception, so we > should change to retry by default, and check for the exceptions which we do > not need to retry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20300) Minor refactor for RpcExecutor
[ https://issues.apache.org/jira/browse/HBASE-20300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20300: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Minor refactor for RpcExecutor > -- > > Key: HBASE-20300 > URL: https://issues.apache.org/jira/browse/HBASE-20300 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-20300.v0.patch, HBASE-20300.v0.patch.patch > > > Plan to do the following changes. > # make Handler be static class > # move the threadlocal variables of MonitoredRPCHandler from RpcServer to > FifoRpcScheduler since only FifoRpcScheduler use it > # create MonitoredRPCHandler in Handler constuction instead of saving the > MonitoredRPCHandler in threadlocal variables. In FPBQ mode, the web UI can > display all Handlers info even if the rpc Handlers are not used yet. > # Threshhold -> Threshold > # make RpcExecutor extend ConfigurationObserver > # don't create task filter repeatly > # add a ut to check whether each Handler has created own MonitoredTask even > if no ops > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20266) Review current set of ignored tests
[
https://issues.apache.org/jira/browse/HBASE-20266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20266:
---
Fix Version/s: (was: 2.2.0)
2.3.0
> Review current set of ignored tests
> ---
>
> Key: HBASE-20266
> URL: https://issues.apache.org/jira/browse/HBASE-20266
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> [~Apache9] turned up a list of currently ignored tests. At first blush, its
> fine to ignore some such as TestHTraceHooks and TestRegionsOnMaster but
> others could do with a review. This issue is about looking at list to make
> sure nothing important missed for hbase2 and that we for sure marked why a
> test was ignored with comment and that there is a follow-on to enable JIRA.
> {code}
> TestRpcHandlerException
> TestRSKilledWhenInitializing
> TestHTraceHooks
> TestAsyncTableGetMultiThreadedWithEagerCompaction
> TestStochasticBalancerJmxMetrics
> TestReplicator
> TestQuotaThrottle
> TestFavoredStochasticLoadBalancer
> TestAsyncTableGetMultiThreadedWithBasicCompaction
> TestRegionPlacement
> TestMasterTransitions
> TestMemstoreLABWithoutPool
> TestRegionsOnMasterOptions
> TestRestoreSnapshotFromClientWithRegionReplicas
> TestMasterBalanceThrottling
> TestMasterProcedureWalLease
> TestRegionServerReadRequestMetrics
> TestHttpServerLifecycle
> TestHRegionServerBulkLoadWithOldSecureEndpoint
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21875) Change the retry logic in RSProcedureDispatcher to 'retry by default, only if xxx'
[ https://issues.apache.org/jira/browse/HBASE-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766720#comment-16766720 ] Duo Zhang commented on HBASE-21875: --- And think again, it is not safe to always quit if we hit CallQueueTooBigException. We can only quit for the first time. So maybe we truly need what you suggest, that a way to cancel a remote call, let me file a issue. [~sershe]. Thanks. > Change the retry logic in RSProcedureDispatcher to 'retry by default, only if > xxx' > -- > > Key: HBASE-21875 > URL: https://issues.apache.org/jira/browse/HBASE-21875 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.3.0 > > Attachments: HBASE-21875-v1.patch, HBASE-21875.patch > > > For now it is not retry by default, only if xxx. > In executeProcedures, we will only throw a fixed set of exception, so we > should change to retry by default, and check for the exceptions which we do > not need to retry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766721#comment-16766721
]
Guanghao Zhang commented on HBASE-20151:
Any updates here?
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.0.1, 1.4.5
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 1.5.1
>
> Attachments: HBASE-20151.master.001.patch,
> HBASE-20151.master.002.patch, HBASE-20151.master.003.patch,
> HBASE-20151.master.004.patch, HBASE-20151.master.004.patch,
> HBASE-20151.master.005.patch, HBASE-20151.master.006.patch,
> filter-list-type.v1.txt
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20252) Admin.move will not fail if we move region to a nonexistent region server
[ https://issues.apache.org/jira/browse/HBASE-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20252: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Admin.move will not fail if we move region to a nonexistent region server > -- > > Key: HBASE-20252 > URL: https://issues.apache.org/jira/browse/HBASE-20252 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-20252-UT.patch > > > The region will just be reopened on the source regionserver... > This is a bit confusing I think... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19562) Purge mirror writing of region and table info into fs at .tableinfo and .regioninfo
[ https://issues.apache.org/jira/browse/HBASE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-19562: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Purge mirror writing of region and table info into fs at .tableinfo and > .regioninfo > --- > > Key: HBASE-19562 > URL: https://issues.apache.org/jira/browse/HBASE-19562 > Project: HBase > Issue Type: Bug > Components: fs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.3.0 > > Attachments: > 0002-HBASE-19562-Purge-mirror-writing-of-region-and-table.patch > > > We don't use these files in hbase2 yet we keep writing them when we create a > table or region. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19452) Turn ON off heap Bucket Cache by default
[
https://issues.apache.org/jira/browse/HBASE-19452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766712#comment-16766712
]
Hadoop QA commented on HBASE-19452:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-19452 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-19452 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12901671/HBASE-19452_V3.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15947/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Turn ON off heap Bucket Cache by default
>
>
> Key: HBASE-19452
> URL: https://issues.apache.org/jira/browse/HBASE-19452
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: HBASE-19452.patch, HBASE-19452_V2.patch,
> HBASE-19452_V3.patch
>
>
> BC's hbase.bucketcache.ioengine by default is empty now. Means now BC.
> Make this default to be 'offheap'. And the default off heap size for the BC
> also to be provided. This can be 8 GB? Better to make it also a % of the
> Xmx. Lets continue to be 40% of Xmx as LRU cache default size.
> When user has to disable BC, configure size as 0. An empty value of this
> config will be treated as default size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20064) Disable MOB threads that are running whether you use MOB or not
[ https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20064: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Disable MOB threads that are running whether you use MOB or not > --- > > Key: HBASE-20064 > URL: https://issues.apache.org/jira/browse/HBASE-20064 > Project: HBase > Issue Type: Improvement >Reporter: stack >Assignee: Reid Chan >Priority: Major > Fix For: 2.3.0 > > Attachments: HBASE-20064.master.001.patch, > HBASE-20064.master.002.patch > > > Master starts up some cleaner and compacting threads even though no MOB. > Disable them and have users explicitly enable it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20064) Disable MOB threads that are running whether you use MOB or not
[ https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20064: --- Issue Type: Improvement (was: Bug) > Disable MOB threads that are running whether you use MOB or not > --- > > Key: HBASE-20064 > URL: https://issues.apache.org/jira/browse/HBASE-20064 > Project: HBase > Issue Type: Improvement >Reporter: stack >Assignee: Reid Chan >Priority: Major > Fix For: 2.2.0 > > Attachments: HBASE-20064.master.001.patch, > HBASE-20064.master.002.patch > > > Master starts up some cleaner and compacting threads even though no MOB. > Disable them and have users explicitly enable it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19222) update jruby to 9.1.14.0+
[ https://issues.apache.org/jira/browse/HBASE-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-19222: --- Fix Version/s: (was: 2.2.0) 2.3.0 > update jruby to 9.1.14.0+ > - > > Key: HBASE-19222 > URL: https://issues.apache.org/jira/browse/HBASE-19222 > Project: HBase > Issue Type: Sub-task > Components: shell >Reporter: Sean Busbey >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.5.1 > > > JRuby's 9.1.14.0 release is described as "things generally work in jdk9" > https://twitter.com/headius/status/928405094407827457 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20060) Add details of off heap memstore into book.
[ https://issues.apache.org/jira/browse/HBASE-20060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766705#comment-16766705 ] Guanghao Zhang commented on HBASE-20060: Ping [~openinx]. Any updates? > Add details of off heap memstore into book. > --- > > Key: HBASE-20060 > URL: https://issues.apache.org/jira/browse/HBASE-20060 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.2.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19952) Find tests which are declared with wrong category
[ https://issues.apache.org/jira/browse/HBASE-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766706#comment-16766706 ] Duo Zhang commented on HBASE-19952: --- Do not have enough time to do this, can move to 2.3.0... > Find tests which are declared with wrong category > - > > Key: HBASE-19952 > URL: https://issues.apache.org/jira/browse/HBASE-19952 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.2.0 > > Attachments: category-report > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21858) report more metrics for exceptions; DFS error categories; RS abort reasons
[
https://issues.apache.org/jira/browse/HBASE-21858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766704#comment-16766704
]
Hadoop QA commented on HBASE-21858:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
41s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} hbase-hadoop2-compat in master has 18 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hbase-hadoop2-compat: The patch generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 47s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}144m
58s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
25s{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}208m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21858 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958465/HBASE-21858.03.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d7bdfce9923f
[jira] [Commented] (HBASE-19952) Find tests which are declared with wrong category
[ https://issues.apache.org/jira/browse/HBASE-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766701#comment-16766701 ] Guanghao Zhang commented on HBASE-19952: Any update here? Can we finish this before release 2.2.0? > Find tests which are declared with wrong category > - > > Key: HBASE-19952 > URL: https://issues.apache.org/jira/browse/HBASE-19952 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.2.0 > > Attachments: category-report > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19452) Turn ON off heap Bucket Cache by default
[ https://issues.apache.org/jira/browse/HBASE-19452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-19452: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Turn ON off heap Bucket Cache by default > > > Key: HBASE-19452 > URL: https://issues.apache.org/jira/browse/HBASE-19452 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Fix For: 2.3.0 > > Attachments: HBASE-19452.patch, HBASE-19452_V2.patch, > HBASE-19452_V3.patch > > > BC's hbase.bucketcache.ioengine by default is empty now. Means now BC. > Make this default to be 'offheap'. And the default off heap size for the BC > also to be provided. This can be 8 GB? Better to make it also a % of the > Xmx. Lets continue to be 40% of Xmx as LRU cache default size. > When user has to disable BC, configure size as 0. An empty value of this > config will be treated as default size. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20657) Retrying RPC call for ModifyTableProcedure may get stuck
[
https://issues.apache.org/jira/browse/HBASE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766695#comment-16766695
]
Hadoop QA commented on HBASE-20657:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HBASE-20657 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-20657 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933276/HBASE-20657-4-master.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15946/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Retrying RPC call for ModifyTableProcedure may get stuck
>
>
> Key: HBASE-20657
> URL: https://issues.apache.org/jira/browse/HBASE-20657
> Project: HBase
> Issue Type: Bug
> Components: Client, proc-v2
>Affects Versions: 2.0.0
>Reporter: Sergey Soldatov
>Assignee: stack
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-20657-1-branch-2.patch,
> HBASE-20657-2-branch-2.patch, HBASE-20657-3-branch-2.patch,
> HBASE-20657-4-master.patch, HBASE-20657-testcase-branch2.patch
>
>
> Env: 2 masters, 1 RS.
> Steps to reproduce: Active master is killed while ModifyTableProcedure is
> executed.
> If the table has enough regions it may come that when the secondary master
> get active some of the regions may be closed, so once client retries the call
> to the new active master, a new ModifyTableProcedure is created and get stuck
> during MODIFY_TABLE_REOPEN_ALL_REGIONS state handling. That happens because:
> 1. When we are retrying from client side, we call modifyTableAsync which
> create a procedure with a new nonce key:
> {noformat}
> ModifyTableRequest request =
> RequestConverter.buildModifyTableRequest(
> td.getTableName(), td, ng.getNonceGroup(), ng.newNonce());
> {noformat}
> So on the server side, it's considered as a new procedure and starts
> executing immediately.
> 2. When we are processing MODIFY_TABLE_REOPEN_ALL_REGIONS we create
> MoveRegionProcedure for each region, but it checks whether the region is
> online (and it's not), so it fails immediately, forcing the procedure to
> restart.
> [~an...@apache.org] saw a similar case when two concurrent ModifyTable
> procedures were running and got stuck in the similar way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18405) Track scope for HBase-Spark module
[ https://issues.apache.org/jira/browse/HBASE-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18405: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Track scope for HBase-Spark module > -- > > Key: HBASE-18405 > URL: https://issues.apache.org/jira/browse/HBASE-18405 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.5.1 > > Attachments: Apache HBase - Apache Spark Integration Scope - update > 1.pdf, Apache HBase - Apache Spark Integration Scope.pdf > > > Start with [\[DISCUSS\] status of and plans for our hbase-spark integration > |https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E] > and formalize into a scope document for bringing this feature into a release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18326) Fix and reenable TestMasterProcedureWalLease
[ https://issues.apache.org/jira/browse/HBASE-18326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18326: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Fix and reenable TestMasterProcedureWalLease > > > Key: HBASE-18326 > URL: https://issues.apache.org/jira/browse/HBASE-18326 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Priority: Blocker > Fix For: 3.0.0, 2.3.0 > > > Fix and reenable flakey important test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766693#comment-16766693 ] Guanghao Zhang commented on HBASE-15728: Ping [~busbey]. > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-15728.branch-1.001.patch, > HBASE-15728.branch-1.001.patch, HBASE-15728.branch-1.002.patch, > HBASE-15728.branch-2.addendum.001.patch, HBASE-15728.master.001.patch, > HBASE-15728.master.addendum.001.patch, hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20657) Retrying RPC call for ModifyTableProcedure may get stuck
[
https://issues.apache.org/jira/browse/HBASE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20657:
---
Fix Version/s: (was: 2.2.0)
2.3.0
> Retrying RPC call for ModifyTableProcedure may get stuck
>
>
> Key: HBASE-20657
> URL: https://issues.apache.org/jira/browse/HBASE-20657
> Project: HBase
> Issue Type: Bug
> Components: Client, proc-v2
>Affects Versions: 2.0.0
>Reporter: Sergey Soldatov
>Assignee: stack
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-20657-1-branch-2.patch,
> HBASE-20657-2-branch-2.patch, HBASE-20657-3-branch-2.patch,
> HBASE-20657-4-master.patch, HBASE-20657-testcase-branch2.patch
>
>
> Env: 2 masters, 1 RS.
> Steps to reproduce: Active master is killed while ModifyTableProcedure is
> executed.
> If the table has enough regions it may come that when the secondary master
> get active some of the regions may be closed, so once client retries the call
> to the new active master, a new ModifyTableProcedure is created and get stuck
> during MODIFY_TABLE_REOPEN_ALL_REGIONS state handling. That happens because:
> 1. When we are retrying from client side, we call modifyTableAsync which
> create a procedure with a new nonce key:
> {noformat}
> ModifyTableRequest request =
> RequestConverter.buildModifyTableRequest(
> td.getTableName(), td, ng.getNonceGroup(), ng.newNonce());
> {noformat}
> So on the server side, it's considered as a new procedure and starts
> executing immediately.
> 2. When we are processing MODIFY_TABLE_REOPEN_ALL_REGIONS we create
> MoveRegionProcedure for each region, but it checks whether the region is
> online (and it's not), so it fails immediately, forcing the procedure to
> restart.
> [~an...@apache.org] saw a similar case when two concurrent ModifyTable
> procedures were running and got stuck in the similar way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20682) MoveProcedure can be subtask of ModifyTableProcedure/ReopenTableRegionsProcedure; ensure all kosher.
[ https://issues.apache.org/jira/browse/HBASE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20682: --- Fix Version/s: (was: 2.2.0) 2.3.0 > MoveProcedure can be subtask of > ModifyTableProcedure/ReopenTableRegionsProcedure; ensure all kosher. > > > Key: HBASE-20682 > URL: https://issues.apache.org/jira/browse/HBASE-20682 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.3.0 > > > MoveProcedure is used in at least two places as a means of 'reopening' a > region after a config has changed. This issue is about review of MP so this > usage is first-class (and that MP is good procedure citizen running as a > subprocedure. In question is what should happen if the source server or the > target servers are offline when we go to run. As of the parent issue, we'll > skip to the end. Should we instead go ahead with the move (internally, if we > are asked to assign to an offline server, we'll pick another -- do same for > unassign? Would help in this reopen use case). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20719) HTable.batch() doesn't handle TableNotFound correctly.
[
https://issues.apache.org/jira/browse/HBASE-20719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20719:
---
Fix Version/s: (was: 2.2.0)
2.3.0
> HTable.batch() doesn't handle TableNotFound correctly.
> --
>
> Key: HBASE-20719
> URL: https://issues.apache.org/jira/browse/HBASE-20719
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.1.0
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Minor
> Fix For: 2.3.0
>
>
> batch() as well as delete() are processing using AsyncRequest. To report
> about problems we are using RetriesExhaustedWithDetailsException and there is
> no special handling for TableNotFound exception. So, the final result for
> running batch or delete operations against not existing table looks really
> weird and missleading:
> {noformat}
> hbase(main):003:0> delete 't1', 'r1', 'c1'
> 2018-06-12 15:02:50,742 ERROR [main] client.AsyncRequestFutureImpl: Cannot
> get replica 0 location for
> {"totalColumns":1,"row":"r1","families":{"c1":[{"qualifier":"","vlen":0,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807}
> ERROR: Failed 1 action: t1: 1 time, servers with issues: null
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20897) Port HBASE-20866 "HBase 1.x scan performance degradation compared to 0.98 version" to branch-2 and up
[ https://issues.apache.org/jira/browse/HBASE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20897: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Port HBASE-20866 "HBase 1.x scan performance degradation compared to 0.98 > version" to branch-2 and up > - > > Key: HBASE-20897 > URL: https://issues.apache.org/jira/browse/HBASE-20897 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Andrew Purtell >Assignee: Vikas Vishwakarma >Priority: Major > Fix For: 3.0.0, 2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20926) IntegrationTestRSGroup is broken
[ https://issues.apache.org/jira/browse/HBASE-20926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20926: --- Fix Version/s: (was: 2.2.0) 2.3.0 > IntegrationTestRSGroup is broken > - > > Key: HBASE-20926 > URL: https://issues.apache.org/jira/browse/HBASE-20926 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.1.0 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.5.1 > > > There are several problems: > 1. It doesn't work in minicluster mode because in afterMethod() it using > IntegrationTestingUtility.restoreCluster() which just shutdown minicluster in > the not distributed mode > 2. It uses tests from TestRSGroups which was supposed to be common for both > unit and integration tests, but for last two years, there were a number of > tests added that are using internal API, not compatible with the distributed > mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20934) Create an hbase-connectors repository; commit new kafka connect here
[ https://issues.apache.org/jira/browse/HBASE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20934: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Create an hbase-connectors repository; commit new kafka connect here > > > Key: HBASE-20934 > URL: https://issues.apache.org/jira/browse/HBASE-20934 > Project: HBase > Issue Type: Bug > Components: kafka, mapreduce, REST, spark, Thrift >Affects Versions: 2.1.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.3.0 > > > Create a new repository at hbase.apache.org and commit the new kafka proxy > here (HBASE-15320). Make sure it plays nicely with hbase core making use of > public-apis only (It does as best as I can see... but saying this anyways). > Once the kafka proxy is working, as subissue, move REST and thrift over... > SPARK too. > This might be better done for an hbase3 target. I filed it against hbase2.2 > for now. > See discussion up on dev list, "[DISCUSS] Kafka Connection, HBASE-15320", > https://s.apache.org/RQcC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21057) upgrade to latest spotbugs
[ https://issues.apache.org/jira/browse/HBASE-21057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21057: --- Attachment: HBASE-21057.master.001.patch > upgrade to latest spotbugs > -- > > Key: HBASE-21057 > URL: https://issues.apache.org/jira/browse/HBASE-21057 > Project: HBase > Issue Type: Task > Components: community, test >Reporter: Sean Busbey >Assignee: kevin su >Priority: Minor > Labels: beginner > Fix For: 3.0.0, 1.6.0, 2.2.0 > > Attachments: HBASE-21057.master.001.patch, HBASE-21057.v0.patch, > HBASE-21057.v1.patch, HBASE-21057.v1.patch > > > we currently rely on [spotbugs definitions from > 3.1.0-RC3|https://github.com/spotbugs/spotbugs/releases/tag/3.1.0_RC3], which > was a pre-release candidate from Jun 2017. > [spotbugs version > 3.1.6|https://github.com/spotbugs/spotbugs/releases/tag/3.1.6] came out about > a month ago. We should update to the latest. > they also have their own maven plugin now. as a stretch goal we could switch > over to that, if it works with yetus. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21057) upgrade to latest spotbugs
[ https://issues.apache.org/jira/browse/HBASE-21057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766684#comment-16766684 ] Guanghao Zhang commented on HBASE-21057: The latest release version of SpotBugs is 3.1.11. Lets upgrade it for branch-2.2+? > upgrade to latest spotbugs > -- > > Key: HBASE-21057 > URL: https://issues.apache.org/jira/browse/HBASE-21057 > Project: HBase > Issue Type: Task > Components: community, test >Reporter: Sean Busbey >Assignee: kevin su >Priority: Minor > Labels: beginner > Fix For: 3.0.0, 1.6.0, 2.2.0 > > Attachments: HBASE-21057.v0.patch, HBASE-21057.v1.patch, > HBASE-21057.v1.patch > > > we currently rely on [spotbugs definitions from > 3.1.0-RC3|https://github.com/spotbugs/spotbugs/releases/tag/3.1.0_RC3], which > was a pre-release candidate from Jun 2017. > [spotbugs version > 3.1.6|https://github.com/spotbugs/spotbugs/releases/tag/3.1.6] came out about > a month ago. We should update to the latest. > they also have their own maven plugin now. as a stretch goal we could switch > over to that, if it works with yetus. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21859) Add clearRegionLocationCache method for AsyncConnection
[ https://issues.apache.org/jira/browse/HBASE-21859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21859: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-2.2+. Thanks [~zghaobac] for reviewing. > Add clearRegionLocationCache method for AsyncConnection > --- > > Key: HBASE-21859 > URL: https://issues.apache.org/jira/browse/HBASE-21859 > Project: HBase > Issue Type: Task > Components: asyncclient, Client >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.3.0 > > Attachments: HBASE-21859.patch > > > As we have this method for Connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21873) region can be assigned to 2 servers due to a timed-out call or an unknown exception
[ https://issues.apache.org/jira/browse/HBASE-21873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766677#comment-16766677 ] Sergey Shelukhin edited comment on HBASE-21873 at 2/13/19 2:33 AM: --- This is not about changing from procedures etc - that is a separate JIRA, out of the scope of this JIRA. Also, double checking AssignRegionHandlerI do see that it checks the state, so that should be fine. What I'm saying in this JIRA is that, keeping procedures and everything, the current state management in this case is incorrect. It's not about different assignment model; the specific part of the spec for the existing one is simply invalid, it doesn't account for this state, and it needs a small change to the state machine. I gave examples above why retry, while making the situation safer, doesn't solve the core problem (and also takes longer). Complicating state machine is not a reason to not have correct state machine. It's the same as if we say "why have CONFIRM_CLOSED, it's just complicating state machine, we'll just send the close message, and assume region is closed"... it would just cause bugs, and while it could be "solved" in most cases with many hacks (e.g. sleep for 10 minutes) the proper solution is to add an extra state/transition and to confirm region is closed. It's the same here... retries can get e.g. network timeout, then on retry they can get CallQueueTooBig, or some other error, and it will assume that open didn't happen based on CallQueueToBig, but actually the first attempt started the open. So, the master has to retry until server dies or opens the region to be safe. But after master restart, new master will not know that it already got into this state and that it needs to retry accounting for it, for example. So we can pile hacks to solve each new special case, or we can fix state machine properly... was (Author: sershe): This is not about changing from procedures etc - that is a separate JIRA, out of the scope of this JIRA. Also, double checking AssignRegionHandlerI do see that it checks the state, so that should be fine. What I'm saying in this JIRA is that, keeping procedures and everything, the current state management in this case is incorrect. It's not about different assignment model; the specific part of the spec for the existing one is simply invalid, it doesn't account for this state, and it needs a small change to the state machine. I gave examples above why retry, while making the situation safer, doesn't solve the core problem (and also takes longer). Complicating state machine is not a reason to not have correct state machine. It's the same as if we say "why have CONFIRM_CLOSED, it's just complicating state machine, we'll just send the close message, and assume region is closed"... it would just cause bugs, and while it could be "solved" in most cases with many hacks (e.g. sleep for 10 minutes) the proper solution is to add an extra state/transition and to confirm region is closed. It's the same here... retries can get e.g. network timeout, then on retry they can get CallQueueTooBig, or some other error, and it will assume that open didn't happen based on CallQueueToBig, but actually the first attempt started the open. So, the master has to retry until server dies or opens the region to be safe. But after master restart, new master will not know that it already got into this state and that it needs to retry accounting for it, for example. > region can be assigned to 2 servers due to a timed-out call or an unknown > exception > --- > > Key: HBASE-21873 > URL: https://issues.apache.org/jira/browse/HBASE-21873 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Fix For: 3.0.0, 2.2.0, 2.3.0 > > Attachments: HBASE-21862-forUT.patch, HBASE-21862-v1.patch, > HBASE-21862-v2.patch, HBASE-21862.patch > > > It's a classic bug, sort of... the call times out to open the region, but RS > actually processes it alright. It could also happen if the response didn't > make it back due to a network issue. > As a result region is opened on two servers. > There are some mitigations possible to narrow down the race window. > 1) Don't process expired open calls, fail them. Won't help for network issues. > 2) Don't ignore invalid RS state, kill it (YouAreDead exception) - but that > will require fixing other network races where master kills RS, which would > require adding state versioning to the protocol. > The fundamental fix though would require either > 1) an unknown failure from open to ascertain the state of the region from the > server. Again, this would
[jira] [Updated] (HBASE-21352) Polish the rollback implementation in ProcedureExecutor
[ https://issues.apache.org/jira/browse/HBASE-21352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21352: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Polish the rollback implementation in ProcedureExecutor > --- > > Key: HBASE-21352 > URL: https://issues.apache.org/jira/browse/HBASE-21352 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-21352.patch > > > Need to confirm that the way we make sure that there is one PE worker which > rollback the whole procedure stack is correct. Now the comment is not enough. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21082) Reimplement assign/unassign related procedure metrics
[ https://issues.apache.org/jira/browse/HBASE-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21082: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Reimplement assign/unassign related procedure metrics > - > > Key: HBASE-21082 > URL: https://issues.apache.org/jira/browse/HBASE-21082 > Project: HBase > Issue Type: Sub-task > Components: amv2, metrics >Reporter: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > As after HBASE-20881, we only have one TRSP procedure to handle all the > assign/unassign/move/reopen operations, we need to reimplement(redefine?) the > metrics for these procedures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21873) region can be assigned to 2 servers due to a timed-out call or an unknown exception
[
https://issues.apache.org/jira/browse/HBASE-21873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766677#comment-16766677
]
Sergey Shelukhin commented on HBASE-21873:
--
This is not about changing from procedures etc - that is a separate JIRA, out
of the scope of this JIRA.
Also, double checking AssignRegionHandlerI do see that it checks the state, so
that should be fine.
What I'm saying in this JIRA is that, keeping procedures and everything, the
current state management in this case is incorrect. It's not about different
assignment model; the specific part of the spec for the existing one is simply
invalid, it doesn't account for this state, and it needs a small change to the
state machine. I gave examples above why retry, while making the situation
safer, doesn't solve the core problem (and also takes longer).
Complicating state machine is not a reason to not have correct state machine.
It's the same as if we say "why have CONFIRM_CLOSED, it's just complicating
state machine, we'll just send the close message, and assume region is
closed"... it would just cause bugs, and while it could be "solved" in most
cases with many hacks (e.g. sleep for 10 minutes) the proper solution is to add
an extra state/transition and to confirm region is closed. It's the same
here... retries can get e.g. network timeout, then on retry they can get
CallQueueTooBig, or some other error, and it will assume that open didn't
happen based on CallQueueToBig, but actually the first attempt started the
open. So, the master has to retry until server dies or opens the region to be
safe. But after master restart, new master will not know that it already got
into this state and that it needs to retry accounting for it, for example.
> region can be assigned to 2 servers due to a timed-out call or an unknown
> exception
> ---
>
> Key: HBASE-21873
> URL: https://issues.apache.org/jira/browse/HBASE-21873
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Fix For: 3.0.0, 2.2.0, 2.3.0
>
> Attachments: HBASE-21862-forUT.patch, HBASE-21862-v1.patch,
> HBASE-21862-v2.patch, HBASE-21862.patch
>
>
> It's a classic bug, sort of... the call times out to open the region, but RS
> actually processes it alright. It could also happen if the response didn't
> make it back due to a network issue.
> As a result region is opened on two servers.
> There are some mitigations possible to narrow down the race window.
> 1) Don't process expired open calls, fail them. Won't help for network issues.
> 2) Don't ignore invalid RS state, kill it (YouAreDead exception) - but that
> will require fixing other network races where master kills RS, which would
> require adding state versioning to the protocol.
> The fundamental fix though would require either
> 1) an unknown failure from open to ascertain the state of the region from the
> server. Again, this would probably require protocol changes to make sure we
> ascertain the region is not opened, and also that the
> already-failed-on-master open is NOT going to be processed if it's some queue
> or even in transit on the network (via a nonce-like mechanism)?
> 2) some form of a distributed lock per region, e.g. in ZK
> 3) some form of 2PC? but the participant list cannot be determined in a
> manner that's both scalable and guaranteed correct. Theoretically it could be
> all RSes.
> {noformat}
> 2019-02-08 03:21:31,715 INFO [PEWorker-7]
> procedure.MasterProcedureScheduler: Took xlock for pid=260626, ppid=260595,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false;
> TransitRegionStateProcedure table=table,
> region=d0214809147e43dc6870005742d5d204, ASSIGN
> 2019-02-08 03:21:31,758 INFO [PEWorker-7]
> assignment.TransitRegionStateProcedure: Starting pid=260626, ppid=260595,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true;
> TransitRegionStateProcedure table=table,
> region=d0214809147e43dc6870005742d5d204, ASSIGN; rit=OPEN,
> location=server1,17020,1549567999303; forceNewPlan=false, retain=true
> 2019-02-08 03:21:31,984 INFO [PEWorker-13] assignment.RegionStateStore:
> pid=260626 updating hbase:meta row=d0214809147e43dc6870005742d5d204,
> regionState=OPENING, regionLocation=server1,17020,1549623714617
> 2019-02-08 03:22:32,552 WARN [RSProcedureDispatcher-pool4-t3451]
> assignment.RegionRemoteProcedureBase: The remote operation pid=260637,
> ppid=260626, state=RUNNABLE, hasLock=false;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure for region ...
> to server server1,17020,1549623714617 failed
> java.io.IOException: Call to server1/...:17020
[jira] [Updated] (HBASE-21350) Forward-port HBASE-21242 [amv2] Miscellaneous minor log and assign procedure create improvements
[ https://issues.apache.org/jira/browse/HBASE-21350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21350: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Forward-port HBASE-21242 [amv2] Miscellaneous minor log and assign procedure > create improvements > > > Key: HBASE-21350 > URL: https://issues.apache.org/jira/browse/HBASE-21350 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Sub-issue to forward port the parent. Its acting up and the parent has been > open long enough. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21301) Heatmap for key access patterns
[ https://issues.apache.org/jira/browse/HBASE-21301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21301: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Heatmap for key access patterns > --- > > Key: HBASE-21301 > URL: https://issues.apache.org/jira/browse/HBASE-21301 > Project: HBase > Issue Type: Improvement >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.5.1 > > Attachments: HBASE-21301.v0.master.patch > > > Google recently released a beta feature for Cloud Bigtable which presents a > heat map of the keyspace. *Given how hotspotting comes up now and again here, > this is a good idea for giving HBase ops a tool to be proactive about it.* > >>> > Additionally, we are announcing the beta version of Key Visualizer, a > visualization tool for Cloud Bigtable key access patterns. Key Visualizer > helps debug performance issues due to unbalanced access patterns across the > key space, or single rows that are too large or receiving too much read or > write activity. With Key Visualizer, you get a heat map visualization of > access patterns over time, along with the ability to zoom into specific key > or time ranges, or select a specific row to find the full row key ID that's > responsible for a hotspot. Key Visualizer is automatically enabled for Cloud > Bigtable clusters with sufficient data or activity, and does not affect Cloud > Bigtable cluster performance. > <<< > From > [https://cloudplatform.googleblog.com/2018/07/on-gcp-your-database-your-way.html] > (Copied this description from the write-up by [~apurtell], thanks Andrew.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21112) [Auth] IPC client fallback to simple auth (forward-port to master)
[ https://issues.apache.org/jira/browse/HBASE-21112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21112: --- Fix Version/s: (was: 2.2.0) 2.3.0 > [Auth] IPC client fallback to simple auth (forward-port to master) > -- > > Key: HBASE-21112 > URL: https://issues.apache.org/jira/browse/HBASE-21112 > Project: HBase > Issue Type: Bug > Components: Client, IPC/RPC, security >Affects Versions: 2.1.0, 2.0.2 >Reporter: Jack Bearden >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21785) master reports open regions as RITs and also messes up rit age metric
[
https://issues.apache.org/jira/browse/HBASE-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766672#comment-16766672
]
Hudson commented on HBASE-21785:
Results for branch master
[build #791 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/791/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/791//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/791//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/791//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> master reports open regions as RITs and also messes up rit age metric
> -
>
> Key: HBASE-21785
> URL: https://issues.apache.org/jira/browse/HBASE-21785
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21785.01.patch, HBASE-21785.patch
>
>
> {noformat}
> RegionState RIT time (ms) Retries
> dba183f0dadfcc9dc8ae0a6dd59c84e6 dba183f0dadfcc9dc8ae0a6dd59c84e6.
> state=OPEN, ts=Wed Dec 31 16:00:00 PST 1969 (1548453918s ago),
> server=server,17020,1548452922054 1548453918735 0
> {noformat}
> RIT age metric also gets set to a bogus value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21884) Fix box/unbox findbugs warning in secure bulk load
Sean Busbey created HBASE-21884:
---
Summary: Fix box/unbox findbugs warning in secure bulk load
Key: HBASE-21884
URL: https://issues.apache.org/jira/browse/HBASE-21884
Project: HBase
Issue Type: Task
Affects Versions: 1.5.0, 1.4.10, 1.3.4, 1.2.11
Reporter: Sean Busbey
Assignee: Sean Busbey
{code}
Reason Tests
FindBugsmodule:hbase-server
Boxed value is unboxed and then immediately reboxed in
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
At SecureBulkLoadEndpoint.java:then immediately reboxed in
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
At SecureBulkLoadEndpoint.java:[line 268]
{code}
Looking at branch-2 and master I suspect we're doing the same wasteful
operation but findbugs can't see it through the lambda definition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HBASE-21884) Fix box/unbox findbugs warning in secure bulk load
[
https://issues.apache.org/jira/browse/HBASE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-21884 started by Sean Busbey.
---
> Fix box/unbox findbugs warning in secure bulk load
> --
>
> Key: HBASE-21884
> URL: https://issues.apache.org/jira/browse/HBASE-21884
> Project: HBase
> Issue Type: Task
>Affects Versions: 1.5.0, 1.4.10, 1.3.4, 1.2.11
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
>
> {code}
> ReasonTests
> FindBugs module:hbase-server
> Boxed value is unboxed and then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:[line 268]
> {code}
> Looking at branch-2 and master I suspect we're doing the same wasteful
> operation but findbugs can't see it through the lambda definition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19889) Revert Workaround: Purge User API building from branch-2 so can make a beta-1
[
https://issues.apache.org/jira/browse/HBASE-19889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1679#comment-1679
]
Hadoop QA commented on HBASE-19889:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
16s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
48s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
7s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 23m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 53s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}176m 27s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}263m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-19889 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958451/hbase-19889.branch-2.1.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit shadedjars
hadoopcheck xml compile |
| uname | Linux 36bc3807c74b 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.1 / fca1ece03c |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/15941/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15941/testReport/ |
| Max. process+thread count | 5048 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15941/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Revert Workaround: Purge User API building from branch-2 so can make a beta-1
> -
>
> Key: HBASE-19889
> URL:
[jira] [Commented] (HBASE-21875) Change the retry logic in RSProcedureDispatcher to 'retry by default, only if xxx'
[
https://issues.apache.org/jira/browse/HBASE-21875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766670#comment-16766670
]
Guanghao Zhang commented on HBASE-21875:
{quote}bq.And we should release 2.2.0 ASAP.
{quote}
Sorry for the late. Will start release work this week.
> Change the retry logic in RSProcedureDispatcher to 'retry by default, only if
> xxx'
> --
>
> Key: HBASE-21875
> URL: https://issues.apache.org/jira/browse/HBASE-21875
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.3.0
>
> Attachments: HBASE-21875.patch
>
>
> For now it is not retry by default, only if xxx.
> In executeProcedures, we will only throw a fixed set of exception, so we
> should change to retry by default, and check for the exceptions which we do
> not need to retry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-21374) Backport HBASE-21342 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-21374. - Resolution: Fixed I'm going to fix this in a follow-on, since there are fix version(s) nearing RC. > Backport HBASE-21342 to branch-1 > > > Key: HBASE-21374 > URL: https://issues.apache.org/jira/browse/HBASE-21374 > Project: HBase > Issue Type: Sub-task >Reporter: Mike Drob >Assignee: mazhenlin >Priority: Major > Fix For: 1.4.10, 1.3.4, 1.2.11, 1.5.0 > > Attachments: HBASE-21374.branch-1.001.patch, > HBASE-21374.branch-1.002.patch, HBASE-21374.branch-1.003.patch, > HBASE-21374.branch-1.003.patch, HBASE-21374.branch-1.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21858) report more metrics for exceptions; DFS error categories; RS abort reasons
[
https://issues.apache.org/jira/browse/HBASE-21858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766657#comment-16766657
]
Hadoop QA commented on HBASE-21858:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
56s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
21s{color} | {color:blue} hbase-hadoop2-compat in master has 18 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} hbase-hadoop2-compat: The patch generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
24s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}145m 7s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
40s{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}232m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHdfsSnapshotHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21858 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958455/HBASE-21858.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit
[jira] [Updated] (HBASE-21814) Remove the TODO in AccessControlLists#addUserPermission
[
https://issues.apache.org/jira/browse/HBASE-21814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-21814:
---
Attachment: HBASE-21814.master.002.patch
> Remove the TODO in AccessControlLists#addUserPermission
> ---
>
> Key: HBASE-21814
> URL: https://issues.apache.org/jira/browse/HBASE-21814
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.3.0
>
> Attachments: HBASE-21814.master.001.patch,
> HBASE-21814.master.001.patch, HBASE-21814.master.002.patch,
> HBASE-21814.master.002.patch
>
>
> The TODO was added by me. Because this method happens within the RS. The old
> impl use a login user(User.runAsLoginUser where the login user is the user
> who started RS process) to call Table.put(). And it will check the permission
> when put record to ACL table. At RpcServer we have a ThreadLocal where we
> keep the CallContext and inside that the current RPC called user info is set.
> We need Table.put(List) to change to a new thread and and so old
> ThreadLocal variable is not accessible and so it looks as if no Rpc context
> and we were relying on the super user who starts the RS process.
>
> {code:java}
> User.runAsLoginUser(new PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
>
> AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm,
> regionEnv.getTable(AccessControlLists.ACL_TABLE_NAME),
> request.getMergeExistingPermissions());
> return null;
> }
> });
> {code}
>
> But after HBASE-21739, no need to User.runAsLoginUser. Because we will call
> Admin method to grant/revoke. And this will be execute in master and use the
> master user(the user who started master process) to call Table.put. So this
> is not a problem now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-15560: --- Fix Version/s: (was: 2.2.0) 2.3.0 > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.5.1 > > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets, run_ycsb_c.sh, > run_ycsb_loading.sh, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-12081) Considering Java 9
[ https://issues.apache.org/jira/browse/HBASE-12081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-12081: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Considering Java 9 > -- > > Key: HBASE-12081 > URL: https://issues.apache.org/jira/browse/HBASE-12081 > Project: HBase > Issue Type: Umbrella >Reporter: Andrew Purtell >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.5.1 > > > Java 9 will ship in 2016. This will be the first Java release that makes a > significant compatibility departure from earlier runtimes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21879) Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
[
https://issues.apache.org/jira/browse/HBASE-21879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-21879:
-
Attachment: (was: QPS-latencies-before-HBASE-21879.png)
> Read HFile's block to ByteBuffer directly instead of to byte for reducing
> young gc purpose
> --
>
> Key: HBASE-21879
> URL: https://issues.apache.org/jira/browse/HBASE-21879
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.1.4
>
> Attachments: QPS-latencies-before-HBASE-21879.png,
> gc-data-before-HBASE-21879.png
>
>
> In HFileBlock#readBlockDataInternal, we have the following:
> {code}
> @VisibleForTesting
> protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset,
> long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum,
> boolean updateMetrics)
> throws IOException {
> // .
> // TODO: Make this ByteBuffer-based. Will make it easier to go to HDFS with
> BBPool (offheap).
> byte [] onDiskBlock = new byte[onDiskSizeWithHeader + hdrSize];
> int nextBlockOnDiskSize = readAtOffset(is, onDiskBlock, preReadHeaderSize,
> onDiskSizeWithHeader - preReadHeaderSize, true, offset +
> preReadHeaderSize, pread);
> if (headerBuf != null) {
> // ...
> }
> // ...
> }
> {code}
> In the read path, we still read the block from hfile to on-heap byte[], then
> copy the on-heap byte[] to offheap bucket cache asynchronously, and in my
> 100% get performance test, I also observed some frequent young gc, The
> largest memory footprint in the young gen should be the on-heap block byte[].
> In fact, we can read HFile's block to ByteBuffer directly instead of to
> byte[] for reducing young gc purpose. we did not implement this before,
> because no ByteBuffer reading interface in the older HDFS client, but 2.7+
> has supported this now, so we can fix this now. I think.
> Will provide an patch and some perf-comparison for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-16594) ROW_INDEX_V2 DBE
[ https://issues.apache.org/jira/browse/HBASE-16594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-16594: --- Fix Version/s: (was: 2.2.0) 2.3.0 > ROW_INDEX_V2 DBE > > > Key: HBASE-16594 > URL: https://issues.apache.org/jira/browse/HBASE-16594 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: binlijin >Assignee: binlijin >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.5.1 > > Attachments: HBASE-16594-master_v1.patch, HBASE-16594-master_v2.patch > > > See HBASE-16213, ROW_INDEX_V1 DataBlockEncoding. > ROW_INDEX_V1 is the first version which have no storage optimization, > ROW_INDEX_V2 do storage optimization: store every row only once, store column > family only once in a HFileBlock. > ROW_INDEX_V1 is : > /** > * Store cells following every row's start offset, so we can binary search to > a row's cells. > * > * Format: > * flat cells > * integer: number of rows > * integer: row0's offset > * integer: row1's offset > * > * integer: dataSize > * > */ > ROW_INDEX_V2 is : > * row1 qualifier timestamp type value tag > * qualifier timestamp type value tag > * qualifier timestamp type value tag > * row2 qualifier timestamp type value tag > * row3 qualifier timestamp type value tag > * qualifier timestamp type value tag > * > * integer: number of rows > * integer: row0's offset > * integer: row1's offset > * > * column family > * integer: dataSize -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.
[ https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18822: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Create table for peer cluster automatically when creating table in source > cluster of using namespace replication. > - > > Key: HBASE-18822 > URL: https://issues.apache.org/jira/browse/HBASE-18822 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0-alpha-2 >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-18822.master.001.patch, HBASE-18822.v1.patch, > HBASE-18822.v1.patch > > > In our cluster of using namespace replication, we always forget to create > table in peer cluster, which lead to replication get stuck. > We have implemented the feature in our cluster: create table for peer > cluster automatically when creating table in source cluster of using > namespace replication. > > I'm not sure if someone else needs this feature, so create an issue here for > discussing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20188: --- Fix Version/s: (was: 2.2.0) 2.3.0 > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Blocker > Fix For: 3.0.0, 2.3.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, HBase 2.0 performance evaluation - throughput > SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, hits_with_fp_scheduler.png, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21838) Create a special ReplicationEndpoint just for verifying the WAL entries are fine
[
https://issues.apache.org/jira/browse/HBASE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766659#comment-16766659
]
Hudson commented on HBASE-21838:
Results for branch branch-1
[build #677 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/677/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/677//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/677//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/677//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Create a special ReplicationEndpoint just for verifying the WAL entries are
> fine
>
>
> Key: HBASE-21838
> URL: https://issues.apache.org/jira/browse/HBASE-21838
> Project: HBase
> Issue Type: Sub-task
> Components: Replication, wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.3.0
>
> Attachments: HBASE-21838-v1.patch, HBASE-21838.patch
>
>
> This is a missing part in our ITBLL. We can config a dummy replication
> endpoint which replicates nothing but only reads and verifies that all the
> wal files are readable and the entries are all fine.
> With this I think it will be much easier to catch the problem in the parent
> issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21585) Remove ClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21585: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HBASE-21512 Status: Resolved (was: Patch Available) Pushed to branch HBASE-21512. Thanks [~stack] for reviewing. > Remove ClusterConnection > > > Key: HBASE-21585 > URL: https://issues.apache.org/jira/browse/HBASE-21585 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-21512 > > Attachments: HBASE-21585-HBASE-21512-v1.patch, > HBASE-21585-HBASE-21512-v2.patch, HBASE-21585-HBASE-21512-v3.patch, > HBASE-21585-HBASE-21512-v4.patch, HBASE-21585-HBASE-21512-v5.patch, > HBASE-21585-HBASE-21512-v5.patch, HBASE-21585-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21488) Reimplement proc-v1 based procedures with proc-v2
[ https://issues.apache.org/jira/browse/HBASE-21488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21488: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Reimplement proc-v1 based procedures with proc-v2 > - > > Key: HBASE-21488 > URL: https://issues.apache.org/jira/browse/HBASE-21488 > Project: HBase > Issue Type: Umbrella > Components: proc-v2 >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > 1. Flush table. > 2. Snapshot. > 3. Log rolling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21649) Complete Thrift2
[ https://issues.apache.org/jira/browse/HBASE-21649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21649: --- Fix Version/s: (was: 2.2.0) 2.3.0 > Complete Thrift2 > > > Key: HBASE-21649 > URL: https://issues.apache.org/jira/browse/HBASE-21649 > Project: HBase > Issue Type: Umbrella >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Thrift1 and Thrift2 coexists in our project for a very long time. > Functionality is more complete in thrift1 but its interface design is bad for > adding new features（so we have get(), getVer()，getVerTs,getRowWithColumns() > and so many other methods for a single get request, this is bad). Thrift2 has > a more clean interface and structure definition, making our user more easy to > use. But, it has not been updated for a long time, lacking of DDL method is a > major weakness. > I think we should complete Thrift2 and supersede Thrift1, making Thrift2 as > the standard multi language definition. This is a umbrella issue to make it > happen. > The plan would be: > 1. Complete the DDL interface of thrift2 > 2. Making thrift2 server be able to handle thrift1 requests, user don't have > to choose which thrift server they need to start > 3. Deprecate thrift1(need to discuss and only go to 3.x) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21548) WAL writer for recovered.edits file in WalSplitting should not require hflush from filesystem
[ https://issues.apache.org/jira/browse/HBASE-21548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21548: --- Fix Version/s: (was: 2.2.0) 2.3.0 > WAL writer for recovered.edits file in WalSplitting should not require hflush > from filesystem > - > > Key: HBASE-21548 > URL: https://issues.apache.org/jira/browse/HBASE-21548 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.3.0 > > > We ran into a problem running HBase on top of Azure filesystems as described > in HBASE-21544 > The quick solution was to backport HBASE-20734 to branch-2.0 to solve this > issue. However, it is incorrect for HBase to have the recovered.edits writer > asserting more stringent requirements than it actually needs (does not need > hflush). > This is to track fixing up the writers such that we are not requiring more > than we actually need. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21585) Remove ClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766658#comment-16766658 ] Duo Zhang commented on HBASE-21585: --- TestJMXListener passed locally for me. And the failure on jenkins is 'Caused by: java.net.BindException: Address already in use (Bind failed)', which should be an environment issue. Let me commit. > Remove ClusterConnection > > > Key: HBASE-21585 > URL: https://issues.apache.org/jira/browse/HBASE-21585 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21585-HBASE-21512-v1.patch, > HBASE-21585-HBASE-21512-v2.patch, HBASE-21585-HBASE-21512-v3.patch, > HBASE-21585-HBASE-21512-v4.patch, HBASE-21585-HBASE-21512-v5.patch, > HBASE-21585-HBASE-21512-v5.patch, HBASE-21585-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21859) Add clearRegionLocationCache method for AsyncConnection
[ https://issues.apache.org/jira/browse/HBASE-21859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766655#comment-16766655 ] Guanghao Zhang commented on HBASE-21859: +1 > Add clearRegionLocationCache method for AsyncConnection > --- > > Key: HBASE-21859 > URL: https://issues.apache.org/jira/browse/HBASE-21859 > Project: HBase > Issue Type: Task > Components: asyncclient, Client >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.3.0 > > Attachments: HBASE-21859.patch > > > As we have this method for Connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21879) Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
[
https://issues.apache.org/jira/browse/HBASE-21879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-21879:
-
Attachment: QPS-latencies-before-HBASE-21879.png
> Read HFile's block to ByteBuffer directly instead of to byte for reducing
> young gc purpose
> --
>
> Key: HBASE-21879
> URL: https://issues.apache.org/jira/browse/HBASE-21879
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.1.4
>
> Attachments: QPS-latencies-before-HBASE-21879.png,
> gc-data-before-HBASE-21879.png
>
>
> In HFileBlock#readBlockDataInternal, we have the following:
> {code}
> @VisibleForTesting
> protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset,
> long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum,
> boolean updateMetrics)
> throws IOException {
> // .
> // TODO: Make this ByteBuffer-based. Will make it easier to go to HDFS with
> BBPool (offheap).
> byte [] onDiskBlock = new byte[onDiskSizeWithHeader + hdrSize];
> int nextBlockOnDiskSize = readAtOffset(is, onDiskBlock, preReadHeaderSize,
> onDiskSizeWithHeader - preReadHeaderSize, true, offset +
> preReadHeaderSize, pread);
> if (headerBuf != null) {
> // ...
> }
> // ...
> }
> {code}
> In the read path, we still read the block from hfile to on-heap byte[], then
> copy the on-heap byte[] to offheap bucket cache asynchronously, and in my
> 100% get performance test, I also observed some frequent young gc, The
> largest memory footprint in the young gen should be the on-heap block byte[].
> In fact, we can read HFile's block to ByteBuffer directly instead of to
> byte[] for reducing young gc purpose. we did not implement this before,
> because no ByteBuffer reading interface in the older HDFS client, but 2.7+
> has supported this now, so we can fix this now. I think.
> Will provide an patch and some perf-comparison for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21817) handle corrupted cells like other corrupted WAL cases
[ https://issues.apache.org/jira/browse/HBASE-21817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766654#comment-16766654 ] Sergey Shelukhin commented on HBASE-21817: -- Actually looks like this needs a change in approach after HBASE-21401, because that adds sanity checks that causes getNextLogLine to fail and for the read to stop. Would need to modify some stuff... > handle corrupted cells like other corrupted WAL cases > - > > Key: HBASE-21817 > URL: https://issues.apache.org/jira/browse/HBASE-21817 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Critical > Attachments: HBASE-21817.01.patch, HBASE-21817.02.patch, > HBASE-21817.03.patch, HBASE-21817.patch > > > See HBASE-21601 for context. > I looked at the code a bit but it will take a while to understand, so for now > I'm going to mitigate it by handling such cases like any other corrupted WAL > (via either skipErrors or CorruptedHLog exception) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21879) Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
[
https://issues.apache.org/jira/browse/HBASE-21879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766650#comment-16766650
]
Zheng Hu commented on HBASE-21879:
--
bq. Is this about HDFS-2834 that added DFSInputStream#read(ByteBuffer)? But
that was resolved in 2.0.2-alpha
OK, I did not check the HDFS client's API before, and misunderstood that.
Thanks for pointing this out.
> Read HFile's block to ByteBuffer directly instead of to byte for reducing
> young gc purpose
> --
>
> Key: HBASE-21879
> URL: https://issues.apache.org/jira/browse/HBASE-21879
> Project: HBase
> Issue Type: Improvement
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.1.4
>
> Attachments: QPS-latencies-before-HBASE-21879.png,
> gc-data-before-HBASE-21879.png
>
>
> In HFileBlock#readBlockDataInternal, we have the following:
> {code}
> @VisibleForTesting
> protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset,
> long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum,
> boolean updateMetrics)
> throws IOException {
> // .
> // TODO: Make this ByteBuffer-based. Will make it easier to go to HDFS with
> BBPool (offheap).
> byte [] onDiskBlock = new byte[onDiskSizeWithHeader + hdrSize];
> int nextBlockOnDiskSize = readAtOffset(is, onDiskBlock, preReadHeaderSize,
> onDiskSizeWithHeader - preReadHeaderSize, true, offset +
> preReadHeaderSize, pread);
> if (headerBuf != null) {
> // ...
> }
> // ...
> }
> {code}
> In the read path, we still read the block from hfile to on-heap byte[], then
> copy the on-heap byte[] to offheap bucket cache asynchronously, and in my
> 100% get performance test, I also observed some frequent young gc, The
> largest memory footprint in the young gen should be the on-heap block byte[].
> In fact, we can read HFile's block to ByteBuffer directly instead of to
> byte[] for reducing young gc purpose. we did not implement this before,
> because no ByteBuffer reading interface in the older HDFS client, but 2.7+
> has supported this now, so we can fix this now. I think.
> Will provide an patch and some perf-comparison for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-16228) Add read/write HDFS size metrics for flush/compact/handler
[ https://issues.apache.org/jira/browse/HBASE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766645#comment-16766645 ] binlijin commented on HBASE-16228: -- [~brfrn169] you can take over it, sorry i am busy with other things. > Add read/write HDFS size metrics for flush/compact/handler > -- > > Key: HBASE-16228 > URL: https://issues.apache.org/jira/browse/HBASE-16228 > Project: HBase > Issue Type: New Feature >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Attachments: HBASE-16228.patch > > > Flush/Compact/Handler and other threads read or write to HDFS, we can get > this metrics to see read/write amplification in hbase, and test how a > compaction algorithm will affect it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
