[jira] [Updated] (HBASE-20731) Incorrect folders in documentation
[ https://issues.apache.org/jira/browse/HBASE-20731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Aggarwal updated HBASE-20731: --- Attachment: HBASE-20731.master.001.patch > Incorrect folders in documentation > -- > > Key: HBASE-20731 > URL: https://issues.apache.org/jira/browse/HBASE-20731 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Peter Somogyi >Priority: Minor > Labels: beginner > Attachments: HBASE-20731.master.001.patch > > > Unexpected Filesystem Growth chapter in Reference Guide mentions .snapshots > and .archive directories. Both of these were changed long ago. > /hbase/.snapshots -> /hbase/.hbase-snapshot > /hbase/.archive -> /hbase/archive > [https://hbase.apache.org/book.html#_unexpected_filesystem_growth] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20578) Support region server group in target cluster
[ https://issues.apache.org/jira/browse/HBASE-20578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561013#comment-16561013 ] Albert Lee commented on HBASE-20578: Sorry, busying for job last few weeks. And yes, I am blocked by something. I found most methods I used in HBaseReplicationEndpoint are static methods. So if I implement a RSGroupReplicationEndpoint, maybe I need to copy these static methods to all subclasses of HBaseReplicationEndpoint. So I want to copy the logic of RSGroupInfoManagerImpl.getRSGroupOfServer to HBaseReplicationEndpoint to replace use RSGroupAdminClient in hbase-server to avoid the dependency issue and also can reduce the number of the copy code. Any better ideas? > Support region server group in target cluster > - > > Key: HBASE-20578 > URL: https://issues.apache.org/jira/browse/HBASE-20578 > Project: HBase > Issue Type: Improvement > Components: Replication, rsgroup >Reporter: Ted Yu >Assignee: Albert Lee >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20578-001.patch > > > When source tables belong to non-default region server group(s) and there are > region server group counterpart in the target cluster, we should support > replicating to target cluster using the region server group mapping. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20753) reference guide should direct security related issues to priv...@hbase.apache.org
[
https://issues.apache.org/jira/browse/HBASE-20753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Aggarwal updated HBASE-20753:
---
Attachment: HBASE-20753.master.001.patch
> reference guide should direct security related issues to
> priv...@hbase.apache.org
> -
>
> Key: HBASE-20753
> URL: https://issues.apache.org/jira/browse/HBASE-20753
> Project: HBase
> Issue Type: Bug
> Components: documentation, security
>Reporter: Sean Busbey
>Priority: Critical
> Labels: beginner
> Attachments: HBASE-20753.master.001.patch
>
>
> the reference guide currently directs folks to send security issues to
> priv...@apache.org:
> {quote}
> To protect existing HBase installations from new vulnerabilities, please do
> not use JIRA to report security-related bugs. Instead, send your report to
> the mailing list priv...@apache.org, which allows anyone to send messages,
> but restricts who can read them. Someone on that list will contact you to
> follow up on your report.
> {quote}
> This address does not exist. It should tell folks to send the email to
> priv...@hbase.apache.org.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20968) list_procedures_test fails due to no matching regex
[
https://issues.apache.org/jira/browse/HBASE-20968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561010#comment-16561010
]
Jack Bearden commented on HBASE-20968:
--
[~yuzhih...@gmail.com] I wasn't able to reproduce the error you are getting. My
test passes with 16 rows of output and each are passing the regex check. Does
the following sequence match your steps to produce the bug?
{code:java}
git checkout branch-2
git pull origin branch-2
mvn -Dmaven.javadoc.skip -DskipTests -Dhadoop.profile=3.0
-Dhadoop-three.version=3.0.0 clean install
cd hbase-shell
mvn test -PrunAllTests -Dtest=TestShell{code}
> list_procedures_test fails due to no matching regex
> ---
>
> Key: HBASE-20968
> URL: https://issues.apache.org/jira/browse/HBASE-20968
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Jack Bearden
>Priority: Major
>
> From test output against hadoop3:
> {code}
> 2018-07-28 12:04:24,838 DEBUG [Time-limited test]
> procedure2.ProcedureExecutor(948): Stored pid=12, state=RUNNABLE,
> hasLock=false; org.apache.hadoop.hbase.client.procedure.
> ShellTestProcedure
> 2018-07-28 12:04:24,864 INFO [RS-EventLoopGroup-1-3]
> ipc.ServerRpcConnection(556): Connection from 172.18.128.12:46918,
> version=3.0.0-SNAPSHOT, sasl=false, ugi=hbase (auth: SIMPLE),
> service=MasterService
> 2018-07-28 12:04:24,900 DEBUG [Thread-114] master.MasterRpcServices(1157):
> Checking to see if procedure is done pid=11
> ^[[38;5;196mF^[[0m
> ===
> Failure:
> ^[[48;5;124;38;5;231;1mtest_list_procedures(Hbase::ListProceduresTest)^[[0m
> src/test/ruby/shell/list_procedures_test.rb:65:in `block in
> test_list_procedures'
> 62: end
> 63: end
> 64:
> ^[[48;5;124;38;5;231;1m => 65: assert_equal(1, matching_lines)^[[0m
> 66: end
> 67: end
> 68: end
> <^[[48;5;34;38;5;231;1m1^[[0m> expected but was
> <^[[48;5;124;38;5;231;1m0^[[0m>
> ===
> ...
> 2018-07-28 12:04:25,374 INFO [PEWorker-9]
> procedure2.ProcedureExecutor(1316): Finished pid=12, state=SUCCESS,
> hasLock=false; org.apache.hadoop.hbase.client.procedure.
> ShellTestProcedure in 336msec
> {code}
> The completion of the ShellTestProcedure was after the assertion was raised.
> {code}
> def create_procedure_regexp(table_name)
> regexp_string = '[0-9]+ .*ShellTestProcedure SUCCESS.*' \
> {code}
> The regex used by the test isn't found in test output either.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20766) Verify Replication Tool Has Typo "remove cluster"
[ https://issues.apache.org/jira/browse/HBASE-20766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Aggarwal updated HBASE-20766: --- Attachment: HBASE-20766.master.001.patch > Verify Replication Tool Has Typo "remove cluster" > - > > Key: HBASE-20766 > URL: https://issues.apache.org/jira/browse/HBASE-20766 > Project: HBase > Issue Type: Bug >Reporter: Clay B. >Priority: Trivial > Labels: beginner > Attachments: HBASE-20766.master.001.patch > > > The verify replication tool has a trivial typo "remove cluster" instead of > "remote cluster": > https://github.com/apache/hbase/blob/a6eeb26cc0b4d0af3fff50b5b931b6847df1f9d2/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java#L355 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20851) Change rubocop config for max line length of 100
[
https://issues.apache.org/jira/browse/HBASE-20851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561009#comment-16561009
]
Sahil Aggarwal commented on HBASE-20851:
Running rubocop currently shows many offenses and even after adding above
configuration there multiple line length exceeding offenses. Scope of this task
is just to add the config or even fix the violations?
> Change rubocop config for max line length of 100
>
>
> Key: HBASE-20851
> URL: https://issues.apache.org/jira/browse/HBASE-20851
> Project: HBase
> Issue Type: Bug
> Components: community, shell
>Affects Versions: 2.0.1
>Reporter: Umesh Agashe
>Priority: Minor
> Labels: beginner, beginners
>
> Existing ruby and Java code uses max line length of 100 characters. Change
> rubocop config with:
> {code:java}
> Metrics/LineLength:
> Max: 100
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20925) Canary test to expose table availability rate
[
https://issues.apache.org/jira/browse/HBASE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561005#comment-16561005
]
Hadoop QA commented on HBASE-20925:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} hbase-server: The patch generated 0 new + 13
unchanged - 36 fixed = 13 total (was 49) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 45s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}168m
41s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}208m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20925 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933493/HBASE-20925.master.004.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b123e717661e 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e963694259 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13842/testReport/ |
| Max. process+thread count | 4511 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13842/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This
[jira] [Updated] (HBASE-20925) Canary test to expose table availability rate
[ https://issues.apache.org/jira/browse/HBASE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-20925: Attachment: HBASE-20925.master.004.patch > Canary test to expose table availability rate > -- > > Key: HBASE-20925 > URL: https://issues.apache.org/jira/browse/HBASE-20925 > Project: HBase > Issue Type: Improvement > Components: canary >Affects Versions: 3.0.0, 2.0.0, 1.4.6 >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Major > Labels: Canary > Attachments: HBASE-20925.master.001.patch, > HBASE-20925.master.002.patch, HBASE-20925.master.003.patch, > HBASE-20925.master.004.patch > > > Canary test to expose table availability rate. > > It will print table availability rate such as below. > > > *2018-07-27 17:11:06,823 INFO [CanaryMonitor-1532736665083] tool.Canary: > * > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: > === Summary: ===* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : MyTable is: 1.0 .* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : mytable3 is: 0.9* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : mytable2 is: 0.8* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : mytable4 is: 1.0* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: > ===END==* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20895) NPE in RpcServer#readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560926#comment-16560926
]
Hudson commented on HBASE-20895:
Results for branch branch-1.3
[build #406 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/406/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/406//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/406//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/406//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> NPE in RpcServer#readAndProcess
> ---
>
> Key: HBASE-20895
> URL: https://issues.apache.org/jira/browse/HBASE-20895
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 1.3.2
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Major
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.7
>
> Attachments: HBASE-20895-branch-1.patch, HBASE-20895-branch-1.patch
>
>
> {noformat}
> 2018-07-10 16:25:55,005 DEBUG [.sfdc.net,port=60020] ipc.RpcServer -
> RpcServer.listener,port=60020: Caught exception while reading:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1761)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:949)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:730)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:706)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This looks like it could be a use after close problem if there is concurrent
> access to a Connection.
> In process() we might store a null back to the 'data' field.
> Meanwhile in readAndProcess() we have a case where we might be blocked on a
> channel read and then after coming back from the read we go to use 'data'
> after a null has been written back, leading to a NPE.
> {quote}count = channelRead(channel, data);
> 1761 ---> if (count >= 0 && *data.remaining()* == 0)
> \{ process(); }{quote}
> Whether a NPE happens or not is going to depend on the timing of the store
> back to 'data' in another thread and use of 'data' in this thread and whether
> or not the JVM has optimized away a reload of 'data' (it's not declared
> volatile)
> We should do a null check here just to be defensive. We should also look at
> whether concurrent access to the Connection is happening and intended.The
> above is just a theory. We should also look at other execution sequences that
> could lead to 'data' being null in this location. At a glance I didn't find
> one but the store to 'data' happens behind conditionals so it is possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20968) list_procedures_test fails due to no matching regex
[
https://issues.apache.org/jira/browse/HBASE-20968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jack Bearden reassigned HBASE-20968:
Assignee: Jack Bearden
> list_procedures_test fails due to no matching regex
> ---
>
> Key: HBASE-20968
> URL: https://issues.apache.org/jira/browse/HBASE-20968
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Jack Bearden
>Priority: Major
>
> From test output against hadoop3:
> {code}
> 2018-07-28 12:04:24,838 DEBUG [Time-limited test]
> procedure2.ProcedureExecutor(948): Stored pid=12, state=RUNNABLE,
> hasLock=false; org.apache.hadoop.hbase.client.procedure.
> ShellTestProcedure
> 2018-07-28 12:04:24,864 INFO [RS-EventLoopGroup-1-3]
> ipc.ServerRpcConnection(556): Connection from 172.18.128.12:46918,
> version=3.0.0-SNAPSHOT, sasl=false, ugi=hbase (auth: SIMPLE),
> service=MasterService
> 2018-07-28 12:04:24,900 DEBUG [Thread-114] master.MasterRpcServices(1157):
> Checking to see if procedure is done pid=11
> ^[[38;5;196mF^[[0m
> ===
> Failure:
> ^[[48;5;124;38;5;231;1mtest_list_procedures(Hbase::ListProceduresTest)^[[0m
> src/test/ruby/shell/list_procedures_test.rb:65:in `block in
> test_list_procedures'
> 62: end
> 63: end
> 64:
> ^[[48;5;124;38;5;231;1m => 65: assert_equal(1, matching_lines)^[[0m
> 66: end
> 67: end
> 68: end
> <^[[48;5;34;38;5;231;1m1^[[0m> expected but was
> <^[[48;5;124;38;5;231;1m0^[[0m>
> ===
> ...
> 2018-07-28 12:04:25,374 INFO [PEWorker-9]
> procedure2.ProcedureExecutor(1316): Finished pid=12, state=SUCCESS,
> hasLock=false; org.apache.hadoop.hbase.client.procedure.
> ShellTestProcedure in 336msec
> {code}
> The completion of the ShellTestProcedure was after the assertion was raised.
> {code}
> def create_procedure_regexp(table_name)
> regexp_string = '[0-9]+ .*ShellTestProcedure SUCCESS.*' \
> {code}
> The regex used by the test isn't found in test output either.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19369) HBase Should use Builder Pattern to Create Log Files while using WAL on Erasure Coding
[
https://issues.apache.org/jira/browse/HBASE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560911#comment-16560911
]
Hadoop QA commented on HBASE-19369:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
33s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
55s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} The patch hbase-common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch hbase-procedure passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} hbase-server: The patch generated 0 new + 0
unchanged - 2 fixed = 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
54s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 51s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
32s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
43s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}131m
29s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
7s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:6f01af0 |
| JIRA Issue | HBASE-19369 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933485/HBASE-19369.branch-2.0.001.patch
|
[jira] [Created] (HBASE-20969) Create an hbase-operator-tools repo to host hbck2 and later, other toolings
stack created HBASE-20969: - Summary: Create an hbase-operator-tools repo to host hbck2 and later, other toolings Key: HBASE-20969 URL: https://issues.apache.org/jira/browse/HBASE-20969 Project: HBase Issue Type: Sub-task Components: hbck2 Affects Versions: 2.0.1 Reporter: stack Let me make a new repo to host hbck2 and any other operator tools that make sense to break off from core. See the discusion thread on dev [1] that blesses this project. 1. http://apache-hbase.679495.n3.nabble.com/DISCUSS-Separate-Git-Repository-for-HBCK2-td4096319.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20894) Move BucketCache from java serialization to protobuf
[
https://issues.apache.org/jira/browse/HBASE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560907#comment-16560907
]
Hadoop QA commented on HBASE-20894:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
42s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
41s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
20s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
26s{color} | {color:red} hbase-server: The patch generated 1 new + 0 unchanged
- 54 fixed = 1 total (was 54) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 3m
11s{color} | {color:red} patch has 48 errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m
34s{color} | {color:red} The patch causes 48 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
14s{color} | {color:red} The patch causes 48 errors with Hadoop v3.0.0. {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} hbase-server in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 19s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20894 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933490/0001-Write-the-CacheableDeserializerIdManager-index-into-.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux eab995ff3184 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e963694259 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HBASE-Build/13841/artifact/patchprocess/patch-mvninstall-root.txt
|
| compile |
[jira] [Commented] (HBASE-20968) list_procedures_test fails due to no matching regex
[
https://issues.apache.org/jira/browse/HBASE-20968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560895#comment-16560895
]
Ted Yu commented on HBASE-20968:
I was able to reproduce this locally on branch-2.
{code}
mvn test -PrunAllTests -Dtest=TestShell
{code}
> list_procedures_test fails due to no matching regex
> ---
>
> Key: HBASE-20968
> URL: https://issues.apache.org/jira/browse/HBASE-20968
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Major
>
> From test output against hadoop3:
> {code}
> 2018-07-28 12:04:24,838 DEBUG [Time-limited test]
> procedure2.ProcedureExecutor(948): Stored pid=12, state=RUNNABLE,
> hasLock=false; org.apache.hadoop.hbase.client.procedure.
> ShellTestProcedure
> 2018-07-28 12:04:24,864 INFO [RS-EventLoopGroup-1-3]
> ipc.ServerRpcConnection(556): Connection from 172.18.128.12:46918,
> version=3.0.0-SNAPSHOT, sasl=false, ugi=hbase (auth: SIMPLE),
> service=MasterService
> 2018-07-28 12:04:24,900 DEBUG [Thread-114] master.MasterRpcServices(1157):
> Checking to see if procedure is done pid=11
> ^[[38;5;196mF^[[0m
> ===
> Failure:
> ^[[48;5;124;38;5;231;1mtest_list_procedures(Hbase::ListProceduresTest)^[[0m
> src/test/ruby/shell/list_procedures_test.rb:65:in `block in
> test_list_procedures'
> 62: end
> 63: end
> 64:
> ^[[48;5;124;38;5;231;1m => 65: assert_equal(1, matching_lines)^[[0m
> 66: end
> 67: end
> 68: end
> <^[[48;5;34;38;5;231;1m1^[[0m> expected but was
> <^[[48;5;124;38;5;231;1m0^[[0m>
> ===
> ...
> 2018-07-28 12:04:25,374 INFO [PEWorker-9]
> procedure2.ProcedureExecutor(1316): Finished pid=12, state=SUCCESS,
> hasLock=false; org.apache.hadoop.hbase.client.procedure.
> ShellTestProcedure in 336msec
> {code}
> The completion of the ShellTestProcedure was after the assertion was raised.
> {code}
> def create_procedure_regexp(table_name)
> regexp_string = '[0-9]+ .*ShellTestProcedure SUCCESS.*' \
> {code}
> The regex used by the test isn't found in test output either.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20894) Move BucketCache from java serialization to protobuf
[ https://issues.apache.org/jira/browse/HBASE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560894#comment-16560894 ] stack commented on HBASE-20894: --- Added a suggestion (See [^0001-Write-the-CacheableDeserializerIdManager-index-into-.patch] ). It is incomplete. Missing is some serialization juggling (TODO). Patch removes an indirection that we have in naming BucketEntry deserializers. If we are breaking serialization -- i.e. moving from one serialization type to another -- this is an opportunity for simplifying this bit of cryptic code. Perhaps integrate this patch? I'll finish it up if wanted (This suggestion comes of an innocent [~mdrob] question asking what is UniqueIndexMap for?). > Move BucketCache from java serialization to protobuf > > > Key: HBASE-20894 > URL: https://issues.apache.org/jira/browse/HBASE-20894 > Project: HBase > Issue Type: Task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-Write-the-CacheableDeserializerIdManager-index-into-.patch, > HBASE-20894.WIP-2.patch, HBASE-20894.WIP.patch, HBASE-20894.master.001.patch, > HBASE-20894.master.002.patch, HBASE-20894.master.003.patch > > > We should use a better serialization format instead of Java Serialization for > the BucketCache entry persistence. > Suggested by Chris McCown, who does not appear to have a JIRA account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20894) Move BucketCache from java serialization to protobuf
[ https://issues.apache.org/jira/browse/HBASE-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20894: -- Attachment: 0001-Write-the-CacheableDeserializerIdManager-index-into-.patch > Move BucketCache from java serialization to protobuf > > > Key: HBASE-20894 > URL: https://issues.apache.org/jira/browse/HBASE-20894 > Project: HBase > Issue Type: Task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-Write-the-CacheableDeserializerIdManager-index-into-.patch, > HBASE-20894.WIP-2.patch, HBASE-20894.WIP.patch, HBASE-20894.master.001.patch, > HBASE-20894.master.002.patch, HBASE-20894.master.003.patch > > > We should use a better serialization format instead of Java Serialization for > the BucketCache entry persistence. > Suggested by Chris McCown, who does not appear to have a JIRA account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20968) list_procedures_test fails due to no matching regex
[
https://issues.apache.org/jira/browse/HBASE-20968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560890#comment-16560890
]
Jack Bearden commented on HBASE-20968:
--
Hi [~yuzhih...@gmail.com], I am trying to reproduce this locally. Could you
kindly send me the command you are using to generate this error? Not new to
ruby but still trying to figure out how to isolate the execution of these ruby
tests in jruby.
> list_procedures_test fails due to no matching regex
> ---
>
> Key: HBASE-20968
> URL: https://issues.apache.org/jira/browse/HBASE-20968
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Major
>
> From test output against hadoop3:
> {code}
> 2018-07-28 12:04:24,838 DEBUG [Time-limited test]
> procedure2.ProcedureExecutor(948): Stored pid=12, state=RUNNABLE,
> hasLock=false; org.apache.hadoop.hbase.client.procedure.
> ShellTestProcedure
> 2018-07-28 12:04:24,864 INFO [RS-EventLoopGroup-1-3]
> ipc.ServerRpcConnection(556): Connection from 172.18.128.12:46918,
> version=3.0.0-SNAPSHOT, sasl=false, ugi=hbase (auth: SIMPLE),
> service=MasterService
> 2018-07-28 12:04:24,900 DEBUG [Thread-114] master.MasterRpcServices(1157):
> Checking to see if procedure is done pid=11
> ^[[38;5;196mF^[[0m
> ===
> Failure:
> ^[[48;5;124;38;5;231;1mtest_list_procedures(Hbase::ListProceduresTest)^[[0m
> src/test/ruby/shell/list_procedures_test.rb:65:in `block in
> test_list_procedures'
> 62: end
> 63: end
> 64:
> ^[[48;5;124;38;5;231;1m => 65: assert_equal(1, matching_lines)^[[0m
> 66: end
> 67: end
> 68: end
> <^[[48;5;34;38;5;231;1m1^[[0m> expected but was
> <^[[48;5;124;38;5;231;1m0^[[0m>
> ===
> ...
> 2018-07-28 12:04:25,374 INFO [PEWorker-9]
> procedure2.ProcedureExecutor(1316): Finished pid=12, state=SUCCESS,
> hasLock=false; org.apache.hadoop.hbase.client.procedure.
> ShellTestProcedure in 336msec
> {code}
> The completion of the ShellTestProcedure was after the assertion was raised.
> {code}
> def create_procedure_regexp(table_name)
> regexp_string = '[0-9]+ .*ShellTestProcedure SUCCESS.*' \
> {code}
> The regex used by the test isn't found in test output either.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20895) NPE in RpcServer#readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560889#comment-16560889
]
Hudson commented on HBASE-20895:
Results for branch branch-1.2
[build #411 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/411/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/411//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/411//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/411//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> NPE in RpcServer#readAndProcess
> ---
>
> Key: HBASE-20895
> URL: https://issues.apache.org/jira/browse/HBASE-20895
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 1.3.2
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Major
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.7
>
> Attachments: HBASE-20895-branch-1.patch, HBASE-20895-branch-1.patch
>
>
> {noformat}
> 2018-07-10 16:25:55,005 DEBUG [.sfdc.net,port=60020] ipc.RpcServer -
> RpcServer.listener,port=60020: Caught exception while reading:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1761)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:949)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:730)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:706)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This looks like it could be a use after close problem if there is concurrent
> access to a Connection.
> In process() we might store a null back to the 'data' field.
> Meanwhile in readAndProcess() we have a case where we might be blocked on a
> channel read and then after coming back from the read we go to use 'data'
> after a null has been written back, leading to a NPE.
> {quote}count = channelRead(channel, data);
> 1761 ---> if (count >= 0 && *data.remaining()* == 0)
> \{ process(); }{quote}
> Whether a NPE happens or not is going to depend on the timing of the store
> back to 'data' in another thread and use of 'data' in this thread and whether
> or not the JVM has optimized away a reload of 'data' (it's not declared
> volatile)
> We should do a null check here just to be defensive. We should also look at
> whether concurrent access to the Connection is happening and intended.The
> above is just a theory. We should also look at other execution sequences that
> could lead to 'data' being null in this location. At a glance I didn't find
> one but the store to 'data' happens behind conditionals so it is possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20967) TestFromClientSide3 fails with NPE
[
https://issues.apache.org/jira/browse/HBASE-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560888#comment-16560888
]
Jack Bearden commented on HBASE-20967:
--
Hey [~Apache9], I pulled down today's master and was attempting to reproduce
this issue but the test passes for me. Are you able to get it to fire the NPE?
> TestFromClientSide3 fails with NPE
> --
>
> Key: HBASE-20967
> URL: https://issues.apache.org/jira/browse/HBASE-20967
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Priority: Major
> Attachments:
> org.apache.hadoop.hbase.client.TestFromClientSide3-output.txt
>
>
> https://builds.apache.org/job/HBASE-Flaky-Tests/35375/testReport/junit/org.apache.hadoop.hbase.client/TestFromClientSide3/testLockLeakWithDelta/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.find(TestFromClientSide3.java:995)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.find(TestFromClientSide3.java:1002)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.testLockLeakWithDelta(TestFromClientSide3.java:783)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20895) NPE in RpcServer#readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560872#comment-16560872
]
Hudson commented on HBASE-20895:
Results for branch branch-1.4
[build #400 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/400/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/400//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/400//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/400//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> NPE in RpcServer#readAndProcess
> ---
>
> Key: HBASE-20895
> URL: https://issues.apache.org/jira/browse/HBASE-20895
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 1.3.2
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Major
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.7
>
> Attachments: HBASE-20895-branch-1.patch, HBASE-20895-branch-1.patch
>
>
> {noformat}
> 2018-07-10 16:25:55,005 DEBUG [.sfdc.net,port=60020] ipc.RpcServer -
> RpcServer.listener,port=60020: Caught exception while reading:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1761)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:949)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:730)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:706)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This looks like it could be a use after close problem if there is concurrent
> access to a Connection.
> In process() we might store a null back to the 'data' field.
> Meanwhile in readAndProcess() we have a case where we might be blocked on a
> channel read and then after coming back from the read we go to use 'data'
> after a null has been written back, leading to a NPE.
> {quote}count = channelRead(channel, data);
> 1761 ---> if (count >= 0 && *data.remaining()* == 0)
> \{ process(); }{quote}
> Whether a NPE happens or not is going to depend on the timing of the store
> back to 'data' in another thread and use of 'data' in this thread and whether
> or not the JVM has optimized away a reload of 'data' (it's not declared
> volatile)
> We should do a null check here just to be defensive. We should also look at
> whether concurrent access to the Connection is happening and intended.The
> above is just a theory. We should also look at other execution sequences that
> could lead to 'data' being null in this location. At a glance I didn't find
> one but the store to 'data' happens behind conditionals so it is possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)
[
https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560863#comment-16560863
]
stack commented on HBASE-19121:
---
I think I've asked for this above but here is more detail.
A corrupt Master proc WAL file was responsible for two regions being stuck in
OPENING. It looks like this in Master log:
{code}
2018-07-28 12:33:49,724 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OPENING,
location=ve0530.halxg.cloudera.com,16020,1532716446468,
table=IntegrationTestBigLinkedList, region=8198218a4532a0ee544cb069970f9a77
2018-07-28 12:33:49,724 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OPENING,
location=ve0530.halxg.cloudera.com,16020,1532716446468,
table=IntegrationTestBigLinkedList, region=4459746bcff48c116337e732ac4df705
2018-07-28 12:34:17,532 WARN [PEWorker-2]
assignment.RegionTransitionProcedure: Failed transition, suspend 3600secs
pid=14482, ppid=14168, state=RUNNABLE:REGION_TRANSITION_DISPATCH;
UnassignProcedure table=IntegrationTestBigLinkedList,
region=4459746bcff48c116337e732ac4df705,
server=ve0530.halxg.cloudera.com,16020,1532716446468; rit=OPENING,
location=ve0530.halxg.cloudera.com,16020,1532716446468; waiting on rectified
condition fixed by other Procedure or operator intervention
org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Expected
[SPLITTING, SPLIT, MERGING, OPEN, CLOSING] so could move to CLOSING but current
state=OPENING
at
org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode.transitionState(RegionStates.java:164)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1542)
at
org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:204)
at
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:345)
at
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:95)
at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1474)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1249)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1763)
2018-07-28 12:34:17,533 INFO [PEWorker-2] procedure2.TimeoutExecutorThread:
ADDED pid=14482, ppid=14168, state=WAITING_TIMEOUT:REGION_TRANSITION_DISPATCH;
UnassignProcedure table=IntegrationTestBigLinkedList,
region=4459746bcff48c116337e732ac4df705,
server=ve0530.halxg.cloudera.com,16020,1532716446468; timeout=360,
timestamp=1532810057533
2018-07-28 12:34:19,078 WARN [PEWorker-12]
assignment.RegionTransitionProcedure: Failed transition, suspend 3600secs
pid=14373, ppid=14168, state=RUNNABLE:REGION_TRANSITION_DISPATCH;
UnassignProcedure table=IntegrationTestBigLinkedList,
region=8198218a4532a0ee544cb069970f9a77,
server=ve0530.halxg.cloudera.com,16020,1532716446468; rit=OPENING,
location=ve0530.halxg.cloudera.com,16020,1532716446468; waiting on rectified
condition fixed by other Procedure or operator intervention
org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Expected
[SPLITTING, SPLIT, MERGING, OPEN, CLOSING] so could move to CLOSING but current
state=OPENING
at
org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode.transitionState(RegionStates.java:164)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1542)
at
org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:204)
at
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:345)
at
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:95)
at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1474)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1249)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1763)
{code}
At least the log is clear no what has to be done. We are seeing the STUCK
messages.. and then out comes the prescription on a period.
The Locks UI shows that there is an exclusive lock on the two
regions making it so no other Procedure can run to do fixup:
{code}
Locks
[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters
[
https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560852#comment-16560852
]
Hudson commented on HBASE-18477:
Results for branch HBASE-18477
[build #278 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/278/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/278//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/278//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/278//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/278//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Umbrella JIRA for HBase Read Replica clusters
> -
>
> Key: HBASE-18477
> URL: https://issues.apache.org/jira/browse/HBASE-18477
> Project: HBase
> Issue Type: New Feature
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase
> Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope
> doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf
>
>
> Recently, changes (such as HBASE-17437) have unblocked HBase to run with a
> root directory external to the cluster (such as in Amazon S3). This means
> that the data is stored outside of the cluster and can be accessible after
> the cluster has been terminated. One use case that is often asked about is
> pointing multiple clusters to one root directory (sharing the data) to have
> read resiliency in the case of a cluster failure.
>
> This JIRA is an umbrella JIRA to contain all the tasks necessary to create a
> read-replica HBase cluster that is pointed at the same root directory.
>
> This requires making the Read-Replica cluster Read-Only (no metadata
> operation or data operations).
> Separating the hbase:meta table for each cluster (Otherwise HBase gets
> confused with multiple clusters trying to update the meta table with their ip
> addresses)
> Adding refresh functionality for the meta table to ensure new metadata is
> picked up on the read replica cluster.
> Adding refresh functionality for HFiles for a given table to ensure new data
> is picked up on the read replica cluster.
>
> This can be used with any existing cluster that is backed by an external
> filesystem.
>
> Please note that this feature is still quite manual (with the potential for
> automation later).
>
> More information on this particular feature can be found here:
> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19369) HBase Should use Builder Pattern to Create Log Files while using WAL on Erasure Coding
[ https://issues.apache.org/jira/browse/HBASE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-19369: -- Attachment: HBASE-19369.branch-2.0.001.patch > HBase Should use Builder Pattern to Create Log Files while using WAL on > Erasure Coding > -- > > Key: HBASE-19369 > URL: https://issues.apache.org/jira/browse/HBASE-19369 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Alex Leblang >Assignee: Mike Drob >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1 > > Attachments: HBASE-19369.branch-2.0.001.patch, > HBASE-19369.master.001.patch, HBASE-19369.master.002.patch, > HBASE-19369.master.003.patch, HBASE-19369.master.004.patch, > HBASE-19369.v10.patch, HBASE-19369.v11.patch, HBASE-19369.v12.patch, > HBASE-19369.v13.patch, HBASE-19369.v5.patch, HBASE-19369.v6.patch, > HBASE-19369.v7.patch, HBASE-19369.v8.patch, HBASE-19369.v9.patch > > > Right now an HBase instance using the WAL won't function properly in an > Erasure Coded environment. We should change the following line to use the > hdfs.DistributedFileSystem builder pattern > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java#L92 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20749) Upgrade our use of checkstyle to 8.6+
[ https://issues.apache.org/jira/browse/HBASE-20749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560844#comment-16560844 ] Mike Drob commented on HBASE-20749: --- Master nightly: https://builds.apache.org/job/HBase%20Nightly/job/master/411/artifact/output-general/console-report.html -0 checkstyle 2m 47s root: The source tree has 15630 issues. Branch nightly: https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20749/6/artifact/output-general/console-report.html -0 checkstyle 2m 15s root: The source tree has 15595 issues. I think this is good to commit > Upgrade our use of checkstyle to 8.6+ > - > > Key: HBASE-20749 > URL: https://issues.apache.org/jira/browse/HBASE-20749 > Project: HBase > Issue Type: Improvement > Components: build, community >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-20749.master.001.patch > > > We should upgrade our checkstyle version to 8.6 or later so we can use the > "match violation message to this regex" feature for suppression. That will > allow us to make sure we don't regress on HTrace v3 vs v4 APIs (came up in > HBASE-20332). > We're currently blocked on upgrading to 8.3+ by [checkstyle > #5279|https://github.com/checkstyle/checkstyle/issues/5279], a regression > that flags our use of both the "separate import groups" and "put static > imports over here" configs as an error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20749) Upgrade our use of checkstyle to 8.6+
[
https://issues.apache.org/jira/browse/HBASE-20749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560831#comment-16560831
]
Hudson commented on HBASE-20749:
Results for branch HBASE-20749
[build #6 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20749/6/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20749/6//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20749/6//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20749/6//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Upgrade our use of checkstyle to 8.6+
> -
>
> Key: HBASE-20749
> URL: https://issues.apache.org/jira/browse/HBASE-20749
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Mike Drob
>Priority: Minor
> Attachments: HBASE-20749.master.001.patch
>
>
> We should upgrade our checkstyle version to 8.6 or later so we can use the
> "match violation message to this regex" feature for suppression. That will
> allow us to make sure we don't regress on HTrace v3 vs v4 APIs (came up in
> HBASE-20332).
> We're currently blocked on upgrading to 8.3+ by [checkstyle
> #5279|https://github.com/checkstyle/checkstyle/issues/5279], a regression
> that flags our use of both the "separate import groups" and "put static
> imports over here" configs as an error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560802#comment-16560802 ] Sahil Aggarwal commented on HBASE-20716: [~anoop.hbase] Changed the BBUtils too. > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: Sahil Aggarwal >Priority: Critical > Labels: beginner > Attachments: HBASE-20716.master.001.patch, > HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, > HBASE-20716.master.004.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png > > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Aggarwal updated HBASE-20716: --- Attachment: HBASE-20716.master.004.patch > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: Sahil Aggarwal >Priority: Critical > Labels: beginner > Attachments: HBASE-20716.master.001.patch, > HBASE-20716.master.002.patch, HBASE-20716.master.003.patch, > HBASE-20716.master.004.patch, Screen Shot 2018-06-26 at 11.37.49 AM.png > > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560792#comment-16560792
]
Reid Chan commented on HBASE-20886:
---
Changed to "New Feature" and added RN.
Let's wait if more comments.
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: New Feature
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch,
> HBASE-20886.master.006.patch, HBASE-20886.master.007.patch,
> HBASE-20886.master.008.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Fix Version/s: 2.2.0
3.0.0
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: New Feature
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch,
> HBASE-20886.master.006.patch, HBASE-20886.master.007.patch,
> HBASE-20886.master.008.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Release Note: From 2.2.0, hbase supports client login via keytab. To use
this feature, client should specify `hbase.client.keytab.file` and
`hbase.client.keytab.principal` in hbase-site.xml, then the connection will
contain the needed credentials which be renewed periodically to communicate
with kerberized hbase cluster.
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: New Feature
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch,
> HBASE-20886.master.006.patch, HBASE-20886.master.007.patch,
> HBASE-20886.master.008.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20893) Data loss if splitting region while ServerCrashProcedure executing
[ https://issues.apache.org/jira/browse/HBASE-20893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560784#comment-16560784 ] stack commented on HBASE-20893: --- Thank you for taking a look [~allan163] To be clear, what was here did not cause my long-running test breakage (as I suppose). On deeper digging, it was a case of HBASE-18152, corruption, which I think is now fixed by HBASE-20939 (I hope!). What was here 'worked' but lets commit the addendum because I think it puts the original fix up on a better basis. I filed an issue to address your finding on miscounts if an exception thrown as subtask here. Thanks [~allan163]. Let me try a test w/ the addendum in place before commit. > Data loss if splitting region while ServerCrashProcedure executing > -- > > Key: HBASE-20893 > URL: https://issues.apache.org/jira/browse/HBASE-20893 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1 > > Attachments: HBASE-20893-branch-2.0.addendum.patch, > HBASE-20893.branch-2.0.001.patch, HBASE-20893.branch-2.0.002.patch, > HBASE-20893.branch-2.0.003.patch, HBASE-20893.branch-2.0.004.patch, > HBASE-20893.branch-2.0.005.patch > > > Similar case as HBASE-20878. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20886) [Auth] Support keytab login in hbase client
[
https://issues.apache.org/jira/browse/HBASE-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20886:
--
Issue Type: New Feature (was: Improvement)
> [Auth] Support keytab login in hbase client
> ---
>
> Key: HBASE-20886
> URL: https://issues.apache.org/jira/browse/HBASE-20886
> Project: HBase
> Issue Type: New Feature
> Components: asyncclient, Client, security
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20886.master.001.patch,
> HBASE-20886.master.002.patch, HBASE-20886.master.003.patch,
> HBASE-20886.master.004.patch, HBASE-20886.master.005.patch,
> HBASE-20886.master.006.patch, HBASE-20886.master.007.patch,
> HBASE-20886.master.008.patch
>
>
> There're lots of questions about how to connect to kerberized hbase cluster
> through hbase-client api from user-mail and slack channel.
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> already existed in code base, but they are only used in {{Canary}}.
> This issue is to make use of two configs to support client-side keytab based
> login, after this issue resolved, hbase-client should directly connect to
> kerberized cluster without changing any code as long as
> {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}} are
> specified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20968) list_procedures_test fails due to no matching regex
Ted Yu created HBASE-20968:
--
Summary: list_procedures_test fails due to no matching regex
Key: HBASE-20968
URL: https://issues.apache.org/jira/browse/HBASE-20968
Project: HBase
Issue Type: Test
Reporter: Ted Yu
>From test output against hadoop3:
{code}
2018-07-28 12:04:24,838 DEBUG [Time-limited test]
procedure2.ProcedureExecutor(948): Stored pid=12, state=RUNNABLE,
hasLock=false; org.apache.hadoop.hbase.client.procedure. ShellTestProcedure
2018-07-28 12:04:24,864 INFO [RS-EventLoopGroup-1-3]
ipc.ServerRpcConnection(556): Connection from 172.18.128.12:46918,
version=3.0.0-SNAPSHOT, sasl=false, ugi=hbase (auth: SIMPLE),
service=MasterService
2018-07-28 12:04:24,900 DEBUG [Thread-114] master.MasterRpcServices(1157):
Checking to see if procedure is done pid=11
^[[38;5;196mF^[[0m
===
Failure:
^[[48;5;124;38;5;231;1mtest_list_procedures(Hbase::ListProceduresTest)^[[0m
src/test/ruby/shell/list_procedures_test.rb:65:in `block in
test_list_procedures'
62: end
63: end
64:
^[[48;5;124;38;5;231;1m => 65: assert_equal(1, matching_lines)^[[0m
66: end
67: end
68: end
<^[[48;5;34;38;5;231;1m1^[[0m> expected but was
<^[[48;5;124;38;5;231;1m0^[[0m>
===
...
2018-07-28 12:04:25,374 INFO [PEWorker-9] procedure2.ProcedureExecutor(1316):
Finished pid=12, state=SUCCESS, hasLock=false;
org.apache.hadoop.hbase.client.procedure. ShellTestProcedure in
336msec
{code}
The completion of the ShellTestProcedure was after the assertion was raised.
{code}
def create_procedure_regexp(table_name)
regexp_string = '[0-9]+ .*ShellTestProcedure SUCCESS.*' \
{code}
The regex used by the test isn't found in test output either.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20939) There will be race when we call suspendIfNotReady and then throw ProcedureSuspendedException
[
https://issues.apache.org/jira/browse/HBASE-20939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560747#comment-16560747
]
Hudson commented on HBASE-20939:
Results for branch master
[build #411 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/411/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/411//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/411//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/411//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> There will be race when we call suspendIfNotReady and then throw
> ProcedureSuspendedException
>
>
> Key: HBASE-20939
> URL: https://issues.apache.org/jira/browse/HBASE-20939
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.1, 2.2.0, 2.1.1
>
> Attachments: HBASE-20939.patch, HBASE-20939.patch
>
>
> This is very typical usage in our procedure implementation, for example, in
> AssignProcedure, we will call AM.queueAssign and then suspend ourselves to
> wait until the AM finish processing our assign request.
> But there could be races. Think of this:
> 1. We call suspendIfNotReady on a event, and it returns true so we need to
> wait.
> 2. The event has been waked up, and the procedure will be added back to the
> scheduler.
> 3. A worker picks up the procedure and finishes it.
> 4. We finally throw ProcedureSuspendException and the ProcedureExecutor
> suspend us and store the state in procedure store.
> So we have a half done procedure in the procedure store for ever... This may
> cause assertion when loading procedures. And maybe the worker can not finish
> the procedure as when suspending we need to restore some state, for example,
> add something to RootProcedureState. But anyway, it will still lead to
> assertion or other unexpected errors.
> And this can not be done by simply adding a lock in the procedure, as most
> works are done in the ProcedureExecutor after we throw
> ProcedureSuspendException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20966) RestoreTool#getTableInfoPath should look for completed snapshot only
[
https://issues.apache.org/jira/browse/HBASE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560748#comment-16560748
]
Hudson commented on HBASE-20966:
Results for branch master
[build #411 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/411/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/411//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/411//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/411//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> RestoreTool#getTableInfoPath should look for completed snapshot only
>
>
> Key: HBASE-20966
> URL: https://issues.apache.org/jira/browse/HBASE-20966
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 20966.v1.txt
>
>
> [~gubjanos] reported seeing the following error when running backup / restore
> test on Azure:
> {code}
> 2018-07-25 17:03:56,661|INFO|MainThread|machine.py:167 -
> run()||GUID=e7de7672-ebfd-402d-8f1f-68e7e8444cb1|org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
> Couldn't read snapshot info
> from:wasb://hbase3-m30wub1711kond-115...@humbtesting8wua.blob.core.windows.net/user/hbase/backup_loc/backup_1532538064246/default/table_fnfawii1za/.hbase-snapshot/.tmp/.
> snapshotinfo
> 2018-07-25 17:03:56,661|INFO|MainThread|machine.py:167 -
> run()||GUID=e7de7672-ebfd-402d-8f1f-68e7e8444cb1|at
> org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.readSnapshotInfo(SnapshotDescriptionUtils.java:328)
> 2018-07-25 17:03:56,661|INFO|MainThread|machine.py:167 -
> run()||GUID=e7de7672-ebfd-402d-8f1f-68e7e8444cb1|at
> org.apache.hadoop.hbase.backup.util.RestoreServerUtil.getTableDesc(RestoreServerUtil.java:237)
> 2018-07-25 17:03:56,662|INFO|MainThread|machine.py:167 -
> run()||GUID=e7de7672-ebfd-402d-8f1f-68e7e8444cb1|at
> org.apache.hadoop.hbase.backup.util.RestoreServerUtil.restoreTableAndCreate(RestoreServerUtil.java:351)
> 2018-07-25 17:03:56,662|INFO|MainThread|machine.py:167 -
> run()||GUID=e7de7672-ebfd-402d-8f1f-68e7e8444cb1|at
> org.apache.hadoop.hbase.backup.util.RestoreServerUtil.fullRestoreTable(RestoreServerUtil.java:186)
> {code}
> Here is related code in master branch:
> {code}
> Path getTableInfoPath(TableName tableName) throws IOException {
> Path tableSnapShotPath = getTableSnapshotPath(backupRootPath, tableName,
> backupId);
> Path tableInfoPath = null;
> // can't build the path directly as the timestamp values are different
> FileStatus[] snapshots = fs.listStatus(tableSnapShotPath);
> {code}
> In the above code, we don't exclude incomplete snapshot, leading to exception
> later when reading snapshot info.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19369) HBase Should use Builder Pattern to Create Log Files while using WAL on Erasure Coding
[
https://issues.apache.org/jira/browse/HBASE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560749#comment-16560749
]
Hudson commented on HBASE-19369:
Results for branch master
[build #411 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/411/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/411//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/411//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/411//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase Should use Builder Pattern to Create Log Files while using WAL on
> Erasure Coding
> --
>
> Key: HBASE-19369
> URL: https://issues.apache.org/jira/browse/HBASE-19369
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Alex Leblang
>Assignee: Mike Drob
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-19369.master.001.patch,
> HBASE-19369.master.002.patch, HBASE-19369.master.003.patch,
> HBASE-19369.master.004.patch, HBASE-19369.v10.patch, HBASE-19369.v11.patch,
> HBASE-19369.v12.patch, HBASE-19369.v13.patch, HBASE-19369.v5.patch,
> HBASE-19369.v6.patch, HBASE-19369.v7.patch, HBASE-19369.v8.patch,
> HBASE-19369.v9.patch
>
>
> Right now an HBase instance using the WAL won't function properly in an
> Erasure Coded environment. We should change the following line to use the
> hdfs.DistributedFileSystem builder pattern
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java#L92
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20967) TestFromClientSide3 fails with NPE
[
https://issues.apache.org/jira/browse/HBASE-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560744#comment-16560744
]
Duo Zhang commented on HBASE-20967:
---
At least for this NPE in testLockLeakWithDelta, the problem seems to be that,
the balancer is working and wants to move the region which is injected, and
then cause all the RSes abort...
Maybe we should disable balancer in this test?
> TestFromClientSide3 fails with NPE
> --
>
> Key: HBASE-20967
> URL: https://issues.apache.org/jira/browse/HBASE-20967
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Priority: Major
> Attachments:
> org.apache.hadoop.hbase.client.TestFromClientSide3-output.txt
>
>
> https://builds.apache.org/job/HBASE-Flaky-Tests/35375/testReport/junit/org.apache.hadoop.hbase.client/TestFromClientSide3/testLockLeakWithDelta/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.find(TestFromClientSide3.java:995)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.find(TestFromClientSide3.java:1002)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.testLockLeakWithDelta(TestFromClientSide3.java:783)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20967) TestFromClientSide3 fails with NPE
Duo Zhang created HBASE-20967:
-
Summary: TestFromClientSide3 fails with NPE
Key: HBASE-20967
URL: https://issues.apache.org/jira/browse/HBASE-20967
Project: HBase
Issue Type: Bug
Components: test
Reporter: Duo Zhang
Attachments:
org.apache.hadoop.hbase.client.TestFromClientSide3-output.txt
https://builds.apache.org/job/HBASE-Flaky-Tests/35375/testReport/junit/org.apache.hadoop.hbase.client/TestFromClientSide3/testLockLeakWithDelta/
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.TestFromClientSide3.find(TestFromClientSide3.java:995)
at
org.apache.hadoop.hbase.client.TestFromClientSide3.find(TestFromClientSide3.java:1002)
at
org.apache.hadoop.hbase.client.TestFromClientSide3.testLockLeakWithDelta(TestFromClientSide3.java:783)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20967) TestFromClientSide3 fails with NPE
[
https://issues.apache.org/jira/browse/HBASE-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20967:
--
Attachment: org.apache.hadoop.hbase.client.TestFromClientSide3-output.txt
> TestFromClientSide3 fails with NPE
> --
>
> Key: HBASE-20967
> URL: https://issues.apache.org/jira/browse/HBASE-20967
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Priority: Major
> Attachments:
> org.apache.hadoop.hbase.client.TestFromClientSide3-output.txt
>
>
> https://builds.apache.org/job/HBASE-Flaky-Tests/35375/testReport/junit/org.apache.hadoop.hbase.client/TestFromClientSide3/testLockLeakWithDelta/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.find(TestFromClientSide3.java:995)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.find(TestFromClientSide3.java:1002)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.testLockLeakWithDelta(TestFromClientSide3.java:783)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19369) HBase Should use Builder Pattern to Create Log Files while using WAL on Erasure Coding
[
https://issues.apache.org/jira/browse/HBASE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560727#comment-16560727
]
Hudson commented on HBASE-19369:
Results for branch branch-2
[build #1036 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1036/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1036//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1036//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1036//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase Should use Builder Pattern to Create Log Files while using WAL on
> Erasure Coding
> --
>
> Key: HBASE-19369
> URL: https://issues.apache.org/jira/browse/HBASE-19369
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Alex Leblang
>Assignee: Mike Drob
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-19369.master.001.patch,
> HBASE-19369.master.002.patch, HBASE-19369.master.003.patch,
> HBASE-19369.master.004.patch, HBASE-19369.v10.patch, HBASE-19369.v11.patch,
> HBASE-19369.v12.patch, HBASE-19369.v13.patch, HBASE-19369.v5.patch,
> HBASE-19369.v6.patch, HBASE-19369.v7.patch, HBASE-19369.v8.patch,
> HBASE-19369.v9.patch
>
>
> Right now an HBase instance using the WAL won't function properly in an
> Erasure Coded environment. We should change the following line to use the
> hdfs.DistributedFileSystem builder pattern
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java#L92
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19369) HBase Should use Builder Pattern to Create Log Files while using WAL on Erasure Coding
[
https://issues.apache.org/jira/browse/HBASE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560726#comment-16560726
]
Hudson commented on HBASE-19369:
Results for branch branch-2.1
[build #114 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/114/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/114//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/114//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/114//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase Should use Builder Pattern to Create Log Files while using WAL on
> Erasure Coding
> --
>
> Key: HBASE-19369
> URL: https://issues.apache.org/jira/browse/HBASE-19369
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Alex Leblang
>Assignee: Mike Drob
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-19369.master.001.patch,
> HBASE-19369.master.002.patch, HBASE-19369.master.003.patch,
> HBASE-19369.master.004.patch, HBASE-19369.v10.patch, HBASE-19369.v11.patch,
> HBASE-19369.v12.patch, HBASE-19369.v13.patch, HBASE-19369.v5.patch,
> HBASE-19369.v6.patch, HBASE-19369.v7.patch, HBASE-19369.v8.patch,
> HBASE-19369.v9.patch
>
>
> Right now an HBase instance using the WAL won't function properly in an
> Erasure Coded environment. We should change the following line to use the
> hdfs.DistributedFileSystem builder pattern
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java#L92
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20895) NPE in RpcServer#readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560710#comment-16560710
]
Hudson commented on HBASE-20895:
Results for branch branch-1
[build #395 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/395/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/395//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/395//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/395//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> NPE in RpcServer#readAndProcess
> ---
>
> Key: HBASE-20895
> URL: https://issues.apache.org/jira/browse/HBASE-20895
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 1.3.2
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Major
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.7
>
> Attachments: HBASE-20895-branch-1.patch, HBASE-20895-branch-1.patch
>
>
> {noformat}
> 2018-07-10 16:25:55,005 DEBUG [.sfdc.net,port=60020] ipc.RpcServer -
> RpcServer.listener,port=60020: Caught exception while reading:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1761)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:949)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:730)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:706)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This looks like it could be a use after close problem if there is concurrent
> access to a Connection.
> In process() we might store a null back to the 'data' field.
> Meanwhile in readAndProcess() we have a case where we might be blocked on a
> channel read and then after coming back from the read we go to use 'data'
> after a null has been written back, leading to a NPE.
> {quote}count = channelRead(channel, data);
> 1761 ---> if (count >= 0 && *data.remaining()* == 0)
> \{ process(); }{quote}
> Whether a NPE happens or not is going to depend on the timing of the store
> back to 'data' in another thread and use of 'data' in this thread and whether
> or not the JVM has optimized away a reload of 'data' (it's not declared
> volatile)
> We should do a null check here just to be defensive. We should also look at
> whether concurrent access to the Connection is happening and intended.The
> above is just a theory. We should also look at other execution sequences that
> could lead to 'data' being null in this location. At a glance I didn't find
> one but the store to 'data' happens behind conditionals so it is possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20925) Canary test to expose table availability rate
[
https://issues.apache.org/jira/browse/HBASE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560709#comment-16560709
]
Hadoop QA commented on HBASE-20925:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
17s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
2s{color} | {color:red} hbase-server: The patch generated 2 new + 13 unchanged
- 36 fixed = 15 total (was 49) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
19s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 44s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}161m
3s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}201m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20925 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933463/HBASE-20925.master.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7a338fd91379 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e963694259 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13839/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13839/testReport/ |
| Max. process+thread count | 4862 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Updated] (HBASE-20925) Canary test to expose table availability rate
[ https://issues.apache.org/jira/browse/HBASE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-20925: Attachment: HBASE-20925.master.003.patch > Canary test to expose table availability rate > -- > > Key: HBASE-20925 > URL: https://issues.apache.org/jira/browse/HBASE-20925 > Project: HBase > Issue Type: Improvement > Components: canary >Affects Versions: 3.0.0, 2.0.0, 1.4.6 >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Major > Labels: Canary > Attachments: HBASE-20925.master.001.patch, > HBASE-20925.master.002.patch, HBASE-20925.master.003.patch > > > Canary test to expose table availability rate. > > It will print table availability rate such as below. > > > *2018-07-27 17:11:06,823 INFO [CanaryMonitor-1532736665083] tool.Canary: > * > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: > === Summary: ===* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : MyTable is: 1.0 .* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : mytable3 is: 0.9* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : mytable2 is: 0.8* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: Read > success rate for table : mytable4 is: 1.0* > *2018-07-27 17:11:06,824 INFO [CanaryMonitor-1532736665083] tool.Canary: > ===END==* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20925) Canary test to expose table availability rate
[
https://issues.apache.org/jira/browse/HBASE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560650#comment-16560650
]
Hadoop QA commented on HBASE-20925:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
9s{color} | {color:red} hbase-server: The patch generated 2 new + 13 unchanged
- 36 fixed = 15 total (was 49) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 56s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}109m
47s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20925 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933460/HBASE-20925.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 3783ea35e6b8 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e963694259 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13838/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13838/testReport/ |
| Max. process+thread count | 4728 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
