[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514686#comment-16514686
]
Hadoop QA commented on HBASE-20723:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
1s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
12s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
11s{color} | {color:red} hbase-server: The patch generated 1 new + 73 unchanged
- 2 fixed = 74 total (was 75) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
6s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 42s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 54s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-20723 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928064/20723.branch-2.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ca67c8470b80 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 4e9f5e8d64 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
[jira] [Commented] (HBASE-20331) clean up shaded packaging for 2.1
[
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514676#comment-16514676
]
Hudson commented on HBASE-20331:
Results for branch HBASE-20331
[build #48 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/48/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/48//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/48//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/48//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
> Issue Type: Umbrella
> Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more
> contexts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514666#comment-16514666
]
Hudson commented on HBASE-20733:
Results for branch branch-1.4
[build #355 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/355/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/355//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/355//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/355//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514642#comment-16514642
]
Ted Yu commented on HBASE-20723:
Integrated to master and branch-1 , waiting for Jenkins result before
integrating to other branches.
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.branch-1.txt, 20723.branch-2.txt, 20723.v1.txt,
> 20723.v10.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt,
> 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514641#comment-16514641
]
Ted Yu commented on HBASE-20723:
I ran TestLoadIncrementalHFiles with patch in branch-1 which passed.
Test failure was not related - the test is for bulk load.
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.branch-1.txt, 20723.branch-2.txt, 20723.v1.txt,
> 20723.v10.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt,
> 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Attachment: 20723.branch-2.txt
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.branch-1.txt, 20723.branch-2.txt, 20723.v1.txt,
> 20723.v10.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt,
> 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20706) [hack] Don't add known not-OPEN regions in reopen phase of MTP
[
https://issues.apache.org/jira/browse/HBASE-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514633#comment-16514633
]
stack commented on HBASE-20706:
---
Perhaps fix this on commit.. It is imprecise:
574* @return Return OPEN regions of the table
Ditto for javadoc on getOpenRegionInfoList
+1 from me.
[~Apache9] ?
> [hack] Don't add known not-OPEN regions in reopen phase of MTP
> --
>
> Key: HBASE-20706
> URL: https://issues.apache.org/jira/browse/HBASE-20706
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20706.001.branch-2.0.patch,
> HBASE-20706.002.branch-2.0.patch, HBASE-20706.003.branch-2.0.patch
>
>
> Shake-down of ModifyTableProcedure, talked this one out with Stack – "proper"
> fix is likely pending in HBASE-20682. Using MoveRegionProcedure is likely the
> wrong construct, we would want something specific to reopen (e.g. a
> ReopenProcedure).
> However, we're in a really bad state right now. If there are non-open regions
> for a table which has a modify submitted against it, the entire system locks
> up in a fast-spin while holding the table's lock. This fills up HDFS with PV2
> wals, and prevents you from doing anything in the hbase shell to try to fix
> those unassigned regions. You'll see spam in the master log like:
> {noformat}
> 2018-06-07 03:21:29,448 WARN [PEWorker-1] procedure.ModifyTableProcedure:
> Retriable error trying to modify table=METRIC_RECORD_HOURLY_UUID (in
> state=MODIFY_TABLE_REOPEN_ALL_REGIONS)
> org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> a3dc333606d38aeb6e2ab4b94233cfbc is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:193)
> at
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:67)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:767)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createReopenProcedures(AssignmentManager.java:705)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:128)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:50)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1472)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
> {noformat}
> We unstuck out internal test cluster giving the following change on top of
> Sergey's HBASE-20657. When choosing the regions to reopen, if we filter out a
> table's regions to only be those which are currently OPEN. There may be some
> transient failures here as well, but a subsequent retry of the reopen step
> should filter out that change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514627#comment-16514627
]
Hadoop QA commented on HBASE-20723:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
2s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
17s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
12s{color} | {color:red} hbase-server: The patch generated 3 new + 189
unchanged - 3 fixed = 192 total (was 192) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
19s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 26s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} hbase-server-jdk1.8.0_172 with JDK v1.8.0_172
generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
34s{color} | {color:red} hbase-server-jdk1.7.0_181 with JDK v1.7.0_181
generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 52s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles |
\\
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514612#comment-16514612
]
Hadoop QA commented on HBASE-20723:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
1s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
13s{color} | {color:red} hbase-server: The patch generated 1 new + 76 unchanged
- 2 fixed = 77 total (was 78) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
2s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 19s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}109m
29s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20723 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928053/20723.v10.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux f313d0ad447b 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0825462436 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514534#comment-16514534
]
Ted Yu commented on HBASE-20723:
Patch for branch-1 adds correction to TestWALFactory in the following form:
{code}
-HRegion region = HRegion.openHRegion(this.conf, this.fs, hbaseWALRootDir,
hri, htd, wal);
+HRegion region = HRegion.openHRegion(this.conf, this.fs, hbaseRootDir,
hri, htd, wal);
{code}
i.e. region should be created under rootdir, not under wal.dir
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.branch-1.txt, 20723.v1.txt, 20723.v10.txt,
> 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt, 20723.v5.txt,
> 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Attachment: 20723.branch-1.txt
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.branch-1.txt, 20723.v1.txt, 20723.v10.txt,
> 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt, 20723.v5.txt,
> 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20681) IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
[
https://issues.apache.org/jira/browse/HBASE-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514507#comment-16514507
]
Hudson commented on HBASE-20681:
Results for branch branch-2.0
[build #432 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/432/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/432//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/432//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/432//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
> ---
>
> Key: HBASE-20681
> URL: https://issues.apache.org/jira/browse/HBASE-20681
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: HBASE-20681.001.patch, HBASE-20681.002.patch,
> HBASE-20681.003.patch, HBASE-20681.004.patch, HBASE-20681.004.patch
>
>
> HADOOP-15406 marked mockito and junit as test-only dependencies which, I
> believe, has stopped them from being included in a stock Hadoop classpath.
> Prior, you'd get hamcrest at {{share/hadoop/common/lib/hamcrest-core-1.3.jar}}
> However, we depend on it being there for our junit in hbase-it:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.1:tree (default-cli) @ hbase-it ---
> [INFO] org.apache.hbase:hbase-it:jar:2.0.1-SNAPSHOT
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> {noformat}
> We need to make sure we include it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20681) IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
[
https://issues.apache.org/jira/browse/HBASE-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514479#comment-16514479
]
Hudson commented on HBASE-20681:
Results for branch branch-2
[build #866 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/866/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/866//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/866//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/866//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
> ---
>
> Key: HBASE-20681
> URL: https://issues.apache.org/jira/browse/HBASE-20681
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: HBASE-20681.001.patch, HBASE-20681.002.patch,
> HBASE-20681.003.patch, HBASE-20681.004.patch, HBASE-20681.004.patch
>
>
> HADOOP-15406 marked mockito and junit as test-only dependencies which, I
> believe, has stopped them from being included in a stock Hadoop classpath.
> Prior, you'd get hamcrest at {{share/hadoop/common/lib/hamcrest-core-1.3.jar}}
> However, we depend on it being there for our junit in hbase-it:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.1:tree (default-cli) @ hbase-it ---
> [INFO] org.apache.hbase:hbase-it:jar:2.0.1-SNAPSHOT
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> {noformat}
> We need to make sure we include it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20705) Having RPC Quota on a table prevents Space quota to be recreated/removed
[
https://issues.apache.org/jira/browse/HBASE-20705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi reassigned HBASE-20705:
--
Assignee: Sakthi
> Having RPC Quota on a table prevents Space quota to be recreated/removed
>
>
> Key: HBASE-20705
> URL: https://issues.apache.org/jira/browse/HBASE-20705
> Project: HBase
> Issue Type: Bug
>Reporter: Biju Nair
>Assignee: Sakthi
>Priority: Major
>
> * Property {{hbase.quota.remove.on.table.delete}} is set to {{true}} by
> default
> * Create a table and set RPC and Space quota
> {noformat}
> hbase(main):022:0> create 't2','cf1'
> Created table t2
> Took 0.7420 seconds
> => Hbase::Table - t2
> hbase(main):023:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '1G',
> POLICY => NO_WRITES
> Took 0.0105 seconds
> hbase(main):024:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.0186 seconds
> hbase(main):025:0> list_quotas
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> TABLE => t2 TYPE => SPACE, TABLE => t2, LIMIT => 1073741824, VIOLATION_POLICY
> => NO_WRITES{noformat}
> * Drop the table and the Space quota is set to {{REMOVE => true}}
> {noformat}
> hbase(main):026:0> disable 't2'
> Took 0.4363 seconds
> hbase(main):027:0> drop 't2'
> Took 0.2344 seconds
> hbase(main):028:0> list_quotas
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE => true
> USER => u1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10M/sec,
> SCOPE => MACHINE{noformat}
> * Recreate the table and set Space quota back. The Space quota on the table
> is still set to {{REMOVE => true}}
> {noformat}
> hbase(main):029:0> create 't2','cf1'
> Created table t2
> Took 0.7348 seconds
> => Hbase::Table - t2
> hbase(main):031:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '1G',
> POLICY => NO_WRITES
> Took 0.0088 seconds
> hbase(main):032:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE => true{noformat}
> * Remove RPC quota and drop the table, the Space Quota is not removed
> {noformat}
> hbase(main):033:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT => NONE
> Took 0.0193 seconds
> hbase(main):036:0> disable 't2'
> Took 0.4305 seconds
> hbase(main):037:0> drop 't2'
> Took 0.2353 seconds
> hbase(main):038:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE
> => true{noformat}
> * Deleting the quota entry from {{hbase:quota}} seems to be the option to
> reset it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[ https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20730: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master branch. Thanks [~mdrob] for reviews. > Add pv2 and amv2 chapters to refguide > - > > Key: HBASE-20730 > URL: https://issues.apache.org/jira/browse/HBASE-20730 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20730.master.001.patch, > HBASE-20730.master.002.patch, HBASE-20730.master.003.patch, > HBASE-20730.master.004.patch, HBASE-20730.master.005.patch > > > Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for > dev'y audience. No harm having chapters in the refguide, our bucket of > all-things doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20728) Failure and recovery of all RSes in a RSgroup requires master restart for region assignments
[
https://issues.apache.org/jira/browse/HBASE-20728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi reassigned HBASE-20728:
--
Assignee: Sakthi
> Failure and recovery of all RSes in a RSgroup requires master restart for
> region assignments
>
>
> Key: HBASE-20728
> URL: https://issues.apache.org/jira/browse/HBASE-20728
> Project: HBase
> Issue Type: Bug
> Components: master, rsgroup
>Reporter: Biju Nair
>Assignee: Sakthi
>Priority: Minor
>
> If all the RSes in a RSgroup hosting user tables fail and recover, master
> still looks for old RSes (with old timestamp in the RS identifier) to assign
> regions. i.e. Regions are left in transition making the tables in the RSGroup
> unavailable. User need to restart {{master}} or manually assign the regions
> to make the tables available. Steps to recreate the scenario in a local
> cluster
> - Add required properties to {{site.xml}} to enable {{rsgroup}} and start
> hbase
> - Bring up multiple region servers using {{local-regionservers.sh start}}
> - Create a {{rsgroup}} and move a subset of {{regionservers}} to the group
> - Create a table, move it to the group and put some data
> - Stop the {{regionservers}} in the group and restart them
> - From the {{master UI}}, we can see that the region for the table in
> transition and the RS name in the {{RIT}} message has the old timestamp.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514464#comment-16514464
]
Hadoop QA commented on HBASE-20332:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
44s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
27s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
51s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 43s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}206m
2s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 8m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}276m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20332 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928015/HBASE-20332.6.patch |
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile checkstyle shellcheck shelldocs |
| uname | Linux 60da0614d993 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 30772eef5d |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| shellcheck | v0.4.4 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13276/testReport/ |
| Max. process+thread count |
[jira] [Updated] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Summary: Custom hbase.wal.dir results in data loss because we write
recovered edits into a different place than where the recovering region server
looks for them (was: Custom hbase.wal.dir results in dataloss because we write
recovered edits into a different place than where the recovering region server
looks for them.)
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v10.txt, 20723.v2.txt, 20723.v3.txt,
> 20723.v4.txt, 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt,
> 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) Custom hbase.wal.dir results in dataloss because we write recovered edits into a different place than where the recovering region server looks for them.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Attachment: 20723.v10.txt
> Custom hbase.wal.dir results in dataloss because we write recovered edits
> into a different place than where the recovering region server looks for them.
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v10.txt, 20723.v2.txt, 20723.v3.txt,
> 20723.v4.txt, 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt,
> 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in dataloss because we write recovered edits into a different place than where the recovering region server looks for them.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514438#comment-16514438
]
Ted Yu commented on HBASE-20723:
Rohan and I did more testing in Azure cluster (with config as shown in the
description).
Killing region server no longer incurs data loss with patch v10.
> Custom hbase.wal.dir results in dataloss because we write recovered edits
> into a different place than where the recovering region server looks for them.
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v10.txt, 20723.v2.txt, 20723.v3.txt,
> 20723.v4.txt, 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt,
> 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[
https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514432#comment-16514432
]
Hadoop QA commented on HBASE-20730:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 3m
59s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 3m
58s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20730 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928050/HBASE-20730.master.005.patch
|
| Optional Tests | asflicense refguide |
| uname | Linux e2817957baf3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0e43abc78a |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/13280/artifact/patchprocess/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/13280/artifact/patchprocess/patch-site/book.html
|
| Max. process+thread count | 93 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13280/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Add pv2 and amv2 chapters to refguide
> -
>
> Key: HBASE-20730
> URL: https://issues.apache.org/jira/browse/HBASE-20730
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20730.master.001.patch,
> HBASE-20730.master.002.patch, HBASE-20730.master.003.patch,
> HBASE-20730.master.004.patch, HBASE-20730.master.005.patch
>
>
> Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for
> dev'y audience. No harm having chapters in the refguide, our bucket of
> all-things doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20674) clean up short circuit read logic and docs
[ https://issues.apache.org/jira/browse/HBASE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514418#comment-16514418 ] stack commented on HBASE-20674: --- Patch looks good. Nice clean-up. Any checking done to ensure we still doing checksumming properly? Thanks. > clean up short circuit read logic and docs > -- > > Key: HBASE-20674 > URL: https://issues.apache.org/jira/browse/HBASE-20674 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-20674.patch, HBASE-20674.v2.patch, > HBASE-20674.v3.patch, HBASE-20674.v4.patch, HBASE-20674.v5.patch > > > Mailing list discussion at > https://lists.apache.org/thread.html/f6f73df0ceae29f762f9b9088e3ffd0bf8f109d3dd692df100bf4fd6@%3Cdev.hbase.apache.org%3E > There are several inconsistencies between how our docs claim we do things and > how we actually do things. > There are two docs sections that attempt to address how SCR should work. > dfs.client.read.shortcircuit.skip.checksum is advised to set to true, but our > code in separate places ignores it and then later sets it to true anyway. > CommonFSUtils and FSUtils duplicate code related to SCR setup. > There is a workaround in HFileSystem for a bug that's been fixed in all > versions of hadoop that we support. (HADOOP-9307) > We suggest setting dfs.client.read.shortcircuit.buffer.size to a value that > is very close to what we'd set it to anyway, without clearly explaining why > this is important. > There are other properties that we claim are important, but we don't offer > any suggestions or explanations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20728) Failure and recovery of all RSes in a RSgroup requires master restart for region assignments
[
https://issues.apache.org/jira/browse/HBASE-20728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514416#comment-16514416
]
Sakthi commented on HBASE-20728:
Hey [~gsbiju] , do you mind if I work on this?
> Failure and recovery of all RSes in a RSgroup requires master restart for
> region assignments
>
>
> Key: HBASE-20728
> URL: https://issues.apache.org/jira/browse/HBASE-20728
> Project: HBase
> Issue Type: Bug
> Components: master, rsgroup
>Reporter: Biju Nair
>Priority: Minor
>
> If all the RSes in a RSgroup hosting user tables fail and recover, master
> still looks for old RSes (with old timestamp in the RS identifier) to assign
> regions. i.e. Regions are left in transition making the tables in the RSGroup
> unavailable. User need to restart {{master}} or manually assign the regions
> to make the tables available. Steps to recreate the scenario in a local
> cluster
> - Add required properties to {{site.xml}} to enable {{rsgroup}} and start
> hbase
> - Bring up multiple region servers using {{local-regionservers.sh start}}
> - Create a {{rsgroup}} and move a subset of {{regionservers}} to the group
> - Create a table, move it to the group and put some data
> - Stop the {{regionservers}} in the group and restart them
> - From the {{master UI}}, we can see that the region for the table in
> transition and the RS name in the {{RIT}} message has the old timestamp.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[ https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514400#comment-16514400 ] stack commented on HBASE-20730: --- .005 addresses formatting issues seens post-build. > Add pv2 and amv2 chapters to refguide > - > > Key: HBASE-20730 > URL: https://issues.apache.org/jira/browse/HBASE-20730 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20730.master.001.patch, > HBASE-20730.master.002.patch, HBASE-20730.master.003.patch, > HBASE-20730.master.004.patch, HBASE-20730.master.005.patch > > > Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for > dev'y audience. No harm having chapters in the refguide, our bucket of > all-things doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[ https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20730: -- Attachment: HBASE-20730.master.005.patch > Add pv2 and amv2 chapters to refguide > - > > Key: HBASE-20730 > URL: https://issues.apache.org/jira/browse/HBASE-20730 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20730.master.001.patch, > HBASE-20730.master.002.patch, HBASE-20730.master.003.patch, > HBASE-20730.master.004.patch, HBASE-20730.master.005.patch > > > Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for > dev'y audience. No harm having chapters in the refguide, our bucket of > all-things doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514389#comment-16514389 ] stack commented on HBASE-20716: --- bq. Making the booleans in UnsafeAvailChecker as final will help? Java can optimize the code path based on the final static boolean value? Perhaps. I don't know enough. Better if we could avoid them in first place? > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Priority: Minor > Labels: beginner > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20674) clean up short circuit read logic and docs
[ https://issues.apache.org/jira/browse/HBASE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514387#comment-16514387 ] Mike Drob commented on HBASE-20674: --- [~stack] - any further thoughts on this or tests that you'd request? I think this is in good shape to commit. > clean up short circuit read logic and docs > -- > > Key: HBASE-20674 > URL: https://issues.apache.org/jira/browse/HBASE-20674 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-20674.patch, HBASE-20674.v2.patch, > HBASE-20674.v3.patch, HBASE-20674.v4.patch, HBASE-20674.v5.patch > > > Mailing list discussion at > https://lists.apache.org/thread.html/f6f73df0ceae29f762f9b9088e3ffd0bf8f109d3dd692df100bf4fd6@%3Cdev.hbase.apache.org%3E > There are several inconsistencies between how our docs claim we do things and > how we actually do things. > There are two docs sections that attempt to address how SCR should work. > dfs.client.read.shortcircuit.skip.checksum is advised to set to true, but our > code in separate places ignores it and then later sets it to true anyway. > CommonFSUtils and FSUtils duplicate code related to SCR setup. > There is a workaround in HFileSystem for a bug that's been fixed in all > versions of hadoop that we support. (HADOOP-9307) > We suggest setting dfs.client.read.shortcircuit.buffer.size to a value that > is very close to what we'd set it to anyway, without clearly explaining why > this is important. > There are other properties that we claim are important, but we don't offer > any suggestions or explanations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[ https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514382#comment-16514382 ] stack commented on HBASE-20730: --- bq.. Left more notes for you... Thanks [~mdrob] Addressed in .004 (actually gave it a read-through this time). > Add pv2 and amv2 chapters to refguide > - > > Key: HBASE-20730 > URL: https://issues.apache.org/jira/browse/HBASE-20730 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20730.master.001.patch, > HBASE-20730.master.002.patch, HBASE-20730.master.003.patch, > HBASE-20730.master.004.patch > > > Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for > dev'y audience. No harm having chapters in the refguide, our bucket of > all-things doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[ https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20730: -- Attachment: HBASE-20730.master.004.patch > Add pv2 and amv2 chapters to refguide > - > > Key: HBASE-20730 > URL: https://issues.apache.org/jira/browse/HBASE-20730 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20730.master.001.patch, > HBASE-20730.master.002.patch, HBASE-20730.master.003.patch, > HBASE-20730.master.004.patch > > > Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for > dev'y audience. No harm having chapters in the refguide, our bucket of > all-things doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514371#comment-16514371
]
Hudson commented on HBASE-20733:
Results for branch master
[build #366 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/366/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/366//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/366//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/366//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20695) Implement table level RegionServer replication metrics
[
https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514372#comment-16514372
]
Hudson commented on HBASE-20695:
Results for branch master
[build #366 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/366/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/366//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/366//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/366//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Implement table level RegionServer replication metrics
> ---
>
> Key: HBASE-20695
> URL: https://issues.apache.org/jira/browse/HBASE-20695
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: HBASE-20695.master.001.patch,
> HBASE-20695.master.002.patch, HBASE-20695.master.003.patch,
> HBASE-20695.master.004.patch, HBASE-20695.master.005.patch,
> HBASE-20695.master.006.patch, HBASE-20695.master.007.patch,
> HBASE-20695.master.008.patch, HBASE-20695.master.009.patch,
> HBASE-20695.master.010.patch
>
>
> Region server metrics now are mainly global metrics. It would be nice to have
> table level metrics such as table level source.AgeOfLastShippedOp to indicate
> operators which table's replication is lagging behind.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[ https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514363#comment-16514363 ] Mike Drob commented on HBASE-20730: --- Left more notes for you, Stack :) > Add pv2 and amv2 chapters to refguide > - > > Key: HBASE-20730 > URL: https://issues.apache.org/jira/browse/HBASE-20730 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20730.master.001.patch, > HBASE-20730.master.002.patch, HBASE-20730.master.003.patch > > > Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for > dev'y audience. No harm having chapters in the refguide, our bucket of > all-things doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[
https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514337#comment-16514337
]
Hadoop QA commented on HBASE-20730:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
32s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
40s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20730 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928038/HBASE-20730.master.003.patch
|
| Optional Tests | asflicense refguide |
| uname | Linux bf766a27662e 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 0e43abc78a |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/13279/artifact/patchprocess/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/13279/artifact/patchprocess/patch-site/book.html
|
| Max. process+thread count | 83 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13279/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Add pv2 and amv2 chapters to refguide
> -
>
> Key: HBASE-20730
> URL: https://issues.apache.org/jira/browse/HBASE-20730
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20730.master.001.patch,
> HBASE-20730.master.002.patch, HBASE-20730.master.003.patch
>
>
> Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for
> dev'y audience. No harm having chapters in the refguide, our bucket of
> all-things doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20679) Add the ability to compile JSP dynamically in Jetty
[
https://issues.apache.org/jira/browse/HBASE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514334#comment-16514334
]
Mike Drob commented on HBASE-20679:
---
[~allan163] - do you have cycles to continue to iterate on this patch, or would
you prefer somebody else try to pick it up based on the precommit issues seen?
> Add the ability to compile JSP dynamically in Jetty
> ---
>
> Key: HBASE-20679
> URL: https://issues.apache.org/jira/browse/HBASE-20679
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20679.002.patch, HBASE-20679.003.patch,
> HBASE-20679.patch
>
>
> As discussed in HBASE-20617, adding the ability to dynamically compile jsp
> enable us to do some hot fix.
> For example, several days ago, in our testing HBase-2.0 cluster,
> procedureWals were corrupted due to some unknown reasons. After restarting
> the cluster, since some procedures(AssignProcedure for example) were
> corrupted and couldn't be replayed. Some regions were stuck in RIT forever.
> We couldn't use HBCK since it haven't support AssignmentV2 yet. As a matter
> of fact, the namespace region was not online, so the master was not inited,
> we even couldn't use shell command like assign/move. But, we wrote a jsp and
> fix this issue easily. The jsp file is like this:
> {code:java}
> <%
> String action = request.getParameter("action");
> HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
> List offlineRegionsToAssign = new ArrayList<>();
> List regionRITs =
> master.getAssignmentManager()
> .getRegionStates().getRegionsInTransition();
> for (RegionStates.RegionStateNode regionStateNode : regionRITs) {
> // if regionStateNode don't have a procedure attached, but meta state
> shows
> // this region is in RIT, that means the previous procedure may be
> corrupted
> // we need to create a new assignProcedure to assign them
> if (!regionStateNode.isInTransition()) {
> offlineRegionsToAssign.add(regionStateNode.getRegionInfo());
> out.println("RIT region:" + regionStateNode);
> }
> }
> // Assign offline regions. Uses round-robin.
> if ("fix".equals(action) && offlineRegionsToAssign.size() > 0) {
>
> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().
> createRoundRobinAssignProcedures(offlineRegionsToAssign));
> } else {
> out.println("use ?action=fix to fix RIT regions");
> }
> %>
> {code}
> Above it is only one example we can do if we have the ability to compile jsp
> dynamically. We think it is very useful.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20705) Having RPC Quota on a table prevents Space quota to be recreated/removed
[
https://issues.apache.org/jira/browse/HBASE-20705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514319#comment-16514319
]
Sakthi commented on HBASE-20705:
Hey [~gsbiju] , Do you mind if I take up this?
> Having RPC Quota on a table prevents Space quota to be recreated/removed
>
>
> Key: HBASE-20705
> URL: https://issues.apache.org/jira/browse/HBASE-20705
> Project: HBase
> Issue Type: Bug
>Reporter: Biju Nair
>Priority: Major
>
> * Property {{hbase.quota.remove.on.table.delete}} is set to {{true}} by
> default
> * Create a table and set RPC and Space quota
> {noformat}
> hbase(main):022:0> create 't2','cf1'
> Created table t2
> Took 0.7420 seconds
> => Hbase::Table - t2
> hbase(main):023:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '1G',
> POLICY => NO_WRITES
> Took 0.0105 seconds
> hbase(main):024:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.0186 seconds
> hbase(main):025:0> list_quotas
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> TABLE => t2 TYPE => SPACE, TABLE => t2, LIMIT => 1073741824, VIOLATION_POLICY
> => NO_WRITES{noformat}
> * Drop the table and the Space quota is set to {{REMOVE => true}}
> {noformat}
> hbase(main):026:0> disable 't2'
> Took 0.4363 seconds
> hbase(main):027:0> drop 't2'
> Took 0.2344 seconds
> hbase(main):028:0> list_quotas
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE => true
> USER => u1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10M/sec,
> SCOPE => MACHINE{noformat}
> * Recreate the table and set Space quota back. The Space quota on the table
> is still set to {{REMOVE => true}}
> {noformat}
> hbase(main):029:0> create 't2','cf1'
> Created table t2
> Took 0.7348 seconds
> => Hbase::Table - t2
> hbase(main):031:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '1G',
> POLICY => NO_WRITES
> Took 0.0088 seconds
> hbase(main):032:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE => true{noformat}
> * Remove RPC quota and drop the table, the Space Quota is not removed
> {noformat}
> hbase(main):033:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT => NONE
> Took 0.0193 seconds
> hbase(main):036:0> disable 't2'
> Took 0.4305 seconds
> hbase(main):037:0> drop 't2'
> Took 0.2353 seconds
> hbase(main):038:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE
> => true{noformat}
> * Deleting the quota entry from {{hbase:quota}} seems to be the option to
> reset it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[
https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514320#comment-16514320
]
Hadoop QA commented on HBASE-20730:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
23s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
24s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
10s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20730 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928035/HBASE-20730.master.002.patch
|
| Optional Tests | asflicense refguide |
| uname | Linux 7976ea4c65d5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0e43abc78a |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/13278/artifact/patchprocess/branch-site/book.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/13278/artifact/patchprocess/whitespace-tabs.txt
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/13278/artifact/patchprocess/patch-site/book.html
|
| asflicense |
https://builds.apache.org/job/PreCommit-HBASE-Build/13278/artifact/patchprocess/patch-asflicense-problems.txt
|
| Max. process+thread count | 93 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13278/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Add pv2 and amv2 chapters to refguide
> -
>
> Key: HBASE-20730
> URL: https://issues.apache.org/jira/browse/HBASE-20730
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20730.master.001.patch,
> HBASE-20730.master.002.patch, HBASE-20730.master.003.patch
>
>
> Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for
> dev'y audience. No harm having chapters in the refguide, our bucket of
> all-things doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[ https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514309#comment-16514309 ] stack commented on HBASE-20730: --- .003 Address helpful review by [~md...@cloudera.com] up on rb; also address white space and license complaint. > Add pv2 and amv2 chapters to refguide > - > > Key: HBASE-20730 > URL: https://issues.apache.org/jira/browse/HBASE-20730 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20730.master.001.patch, > HBASE-20730.master.002.patch, HBASE-20730.master.003.patch > > > Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for > dev'y audience. No harm having chapters in the refguide, our bucket of > all-things doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[ https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20730: -- Attachment: HBASE-20730.master.003.patch > Add pv2 and amv2 chapters to refguide > - > > Key: HBASE-20730 > URL: https://issues.apache.org/jira/browse/HBASE-20730 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20730.master.001.patch, > HBASE-20730.master.002.patch, HBASE-20730.master.003.patch > > > Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for > dev'y audience. No harm having chapters in the refguide, our bucket of > all-things doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[ https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20730: -- Attachment: HBASE-20730.master.002.patch > Add pv2 and amv2 chapters to refguide > - > > Key: HBASE-20730 > URL: https://issues.apache.org/jira/browse/HBASE-20730 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20730.master.001.patch, > HBASE-20730.master.002.patch > > > Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for > dev'y audience. No harm having chapters in the refguide, our bucket of > all-things doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20737) put collection into ArrayList instead of addAll function
[ https://issues.apache.org/jira/browse/HBASE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514272#comment-16514272 ] Mike Drob commented on HBASE-20737: --- shrugs, we'll catch it later. it's not like the current case is broken. > put collection into ArrayList instead of addAll function > > > Key: HBASE-20737 > URL: https://issues.apache.org/jira/browse/HBASE-20737 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: taiyinglee >Assignee: taiyinglee >Priority: Trivial > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20737.v0.patch, HBASE-20737.v0.patch > > > [https://docs.oracle.com/javase/7/docs/api/java/util/Collection.html] > [https://docs.oracle.com/javase/7/docs/api/java/util/ArrayList.html] > [https://docs.oracle.com/javase/7/docs/api/java/util/Set.html] > > /hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterStatusPublisher.java > change > List> entries = new ArrayList<>(); > entries.addAll(lastSent.entrySet()); > to > List> entries = new > ArrayList<>(lastSent.entrySet()); -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20738) failed with testGetPassword on Windows
[
https://issues.apache.org/jira/browse/HBASE-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514267#comment-16514267
]
Hadoop QA commented on HBASE-20738:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
1s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
54s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 16s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
37s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20738 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928022/HBASE-20738.0.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9a32749954aa 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 30772eef5d |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13277/testReport/ |
| Max. process+thread count | 295 (vs. ulimit of 1) |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13277/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> failed with testGetPassword on Windows
[jira] [Commented] (HBASE-20737) put collection into ArrayList instead of addAll function
[
https://issues.apache.org/jira/browse/HBASE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514266#comment-16514266
]
Chia-Ping Tsai commented on HBASE-20737:
{quote}Same pattern can be applied in RetriesExhaustedWithDetailsException
{quote}
I don't grep the code base and the patch has been committed to master branch.
:( [~taiyinglee] Could you file a addendum to address [~mdrob]'s comment?
> put collection into ArrayList instead of addAll function
>
>
> Key: HBASE-20737
> URL: https://issues.apache.org/jira/browse/HBASE-20737
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: taiyinglee
>Assignee: taiyinglee
>Priority: Trivial
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20737.v0.patch, HBASE-20737.v0.patch
>
>
> [https://docs.oracle.com/javase/7/docs/api/java/util/Collection.html]
> [https://docs.oracle.com/javase/7/docs/api/java/util/ArrayList.html]
> [https://docs.oracle.com/javase/7/docs/api/java/util/Set.html]
>
> /hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterStatusPublisher.java
> change
> List> entries = new ArrayList<>();
> entries.addAll(lastSent.entrySet());
> to
> List> entries = new
> ArrayList<>(lastSent.entrySet());
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514265#comment-16514265 ] stack commented on HBASE-20708: --- Left comments up on rb. Put the design doc in description because it makes clear what is going on in here. bq. ...It does made the master start process more clear Agree. This is very nice work. Simplification where it is badly needed. Thanks [~Apache9]. Yeah, its 2.1.0 though... needs a bit of heavy-testing I'd say > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708-v5.patch, > HBASE-20708-v6.patch, HBASE-20708-v7.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. > Please see [#[accompanying design document > |https://docs.google.com/document/d/1_872oHzrhJq4ck7f6zmp1J--zMhsIFvXSZyX1Mxg5MA/edit#heading=h.xy1z4alsq7uy] > ]where we call out the problem being addressed by this issue in more detail > and in which we describe our new approach to Master startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514263#comment-16514263
]
Mike Drob commented on HBASE-20332:
---
+1
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch,
> HBASE-20332.5.patch, HBASE-20332.6.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20737) put collection into ArrayList instead of addAll function
[ https://issues.apache.org/jira/browse/HBASE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514260#comment-16514260 ] Mike Drob commented on HBASE-20737: --- Same pattern can be applied in RetriesExhaustedWithDetailsException > put collection into ArrayList instead of addAll function > > > Key: HBASE-20737 > URL: https://issues.apache.org/jira/browse/HBASE-20737 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: taiyinglee >Assignee: taiyinglee >Priority: Trivial > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20737.v0.patch, HBASE-20737.v0.patch > > > [https://docs.oracle.com/javase/7/docs/api/java/util/Collection.html] > [https://docs.oracle.com/javase/7/docs/api/java/util/ArrayList.html] > [https://docs.oracle.com/javase/7/docs/api/java/util/Set.html] > > /hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterStatusPublisher.java > change > List> entries = new ArrayList<>(); > entries.addAll(lastSent.entrySet()); > to > List> entries = new > ArrayList<>(lastSent.entrySet()); -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20737) put collection into ArrayList instead of addAll function
[ https://issues.apache.org/jira/browse/HBASE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-20737: --- Fix Version/s: 2.1.0 3.0.0 > put collection into ArrayList instead of addAll function > > > Key: HBASE-20737 > URL: https://issues.apache.org/jira/browse/HBASE-20737 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: taiyinglee >Assignee: taiyinglee >Priority: Trivial > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20737.v0.patch, HBASE-20737.v0.patch > > > [https://docs.oracle.com/javase/7/docs/api/java/util/Collection.html] > [https://docs.oracle.com/javase/7/docs/api/java/util/ArrayList.html] > [https://docs.oracle.com/javase/7/docs/api/java/util/Set.html] > > /hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterStatusPublisher.java > change > List> entries = new ArrayList<>(); > entries.addAll(lastSent.entrySet()); > to > List> entries = new > ArrayList<>(lastSent.entrySet()); -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20738) failed with testGetPassword on Windows
[ https://issues.apache.org/jira/browse/HBASE-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514230#comment-16514230 ] star commented on HBASE-20738: -- [~busbey], thanks for your suggestion. I've reuploaded the patch with authorship information created by the command recommended. > failed with testGetPassword on Windows > --- > > Key: HBASE-20738 > URL: https://issues.apache.org/jira/browse/HBASE-20738 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: star >Assignee: star >Priority: Minor > Attachments: HBASE-20738.0.patch, test-hbase-configuration.patch > > > When running unit test on Windows, testGetPassword from > TestHBaseConfiguration.java failed. The original code produces a uri path > like "jceks://file/root/hbase/others" on unix system, but produces a invalid > path like "jceks://fileD:\wordspace\hbase\others"，which will cause a > URISyntaxException latter. > To solve the problem, just add a "/" prefix to windows local path and > replace all "\" with "/". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20738) failed with testGetPassword on Windows
[ https://issues.apache.org/jira/browse/HBASE-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] star updated HBASE-20738: - Attachment: HBASE-20738.0.patch > failed with testGetPassword on Windows > --- > > Key: HBASE-20738 > URL: https://issues.apache.org/jira/browse/HBASE-20738 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: star >Assignee: star >Priority: Minor > Attachments: HBASE-20738.0.patch, test-hbase-configuration.patch > > > When running unit test on Windows, testGetPassword from > TestHBaseConfiguration.java failed. The original code produces a uri path > like "jceks://file/root/hbase/others" on unix system, but produces a invalid > path like "jceks://fileD:\wordspace\hbase\others"，which will cause a > URISyntaxException latter. > To solve the problem, just add a "/" prefix to windows local path and > replace all "\" with "/". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20708: -- Description: In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only used by RMP to avoid dead lock with MoveRegionProcedure. But we will always schedule a RMP when master starting up, so we still need to make sure that there is no race between this RMP and other RMPs and SCPs scheduled before the master restarts. Please see [#[accompanying design document |https://docs.google.com/document/d/1_872oHzrhJq4ck7f6zmp1J--zMhsIFvXSZyX1Mxg5MA/edit#heading=h.xy1z4alsq7uy] ]where we call out the problem being addressed by this issue in more detail and in which we describe our new approach to Master startup. was:In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only used by RMP to avoid dead lock with MoveRegionProcedure. But we will always schedule a RMP when master starting up, so we still need to make sure that there is no race between this RMP and other RMPs and SCPs scheduled before the master restarts. > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708-v5.patch, > HBASE-20708-v6.patch, HBASE-20708-v7.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. > Please see [#[accompanying design document > |https://docs.google.com/document/d/1_872oHzrhJq4ck7f6zmp1J--zMhsIFvXSZyX1Mxg5MA/edit#heading=h.xy1z4alsq7uy] > ]where we call out the problem being addressed by this issue in more detail > and in which we describe our new approach to Master startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20705) Having RPC Quota on a table prevents Space quota to be recreated/removed
[
https://issues.apache.org/jira/browse/HBASE-20705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514220#comment-16514220
]
Biju Nair commented on HBASE-20705:
---
Hi [~elserj], {{TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE => true}} is the
output from {{list_quota}} after dropping the table on which RPC and space
quota is set.
> Having RPC Quota on a table prevents Space quota to be recreated/removed
>
>
> Key: HBASE-20705
> URL: https://issues.apache.org/jira/browse/HBASE-20705
> Project: HBase
> Issue Type: Bug
>Reporter: Biju Nair
>Priority: Major
>
> * Property {{hbase.quota.remove.on.table.delete}} is set to {{true}} by
> default
> * Create a table and set RPC and Space quota
> {noformat}
> hbase(main):022:0> create 't2','cf1'
> Created table t2
> Took 0.7420 seconds
> => Hbase::Table - t2
> hbase(main):023:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '1G',
> POLICY => NO_WRITES
> Took 0.0105 seconds
> hbase(main):024:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.0186 seconds
> hbase(main):025:0> list_quotas
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> TABLE => t2 TYPE => SPACE, TABLE => t2, LIMIT => 1073741824, VIOLATION_POLICY
> => NO_WRITES{noformat}
> * Drop the table and the Space quota is set to {{REMOVE => true}}
> {noformat}
> hbase(main):026:0> disable 't2'
> Took 0.4363 seconds
> hbase(main):027:0> drop 't2'
> Took 0.2344 seconds
> hbase(main):028:0> list_quotas
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE => true
> USER => u1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10M/sec,
> SCOPE => MACHINE{noformat}
> * Recreate the table and set Space quota back. The Space quota on the table
> is still set to {{REMOVE => true}}
> {noformat}
> hbase(main):029:0> create 't2','cf1'
> Created table t2
> Took 0.7348 seconds
> => Hbase::Table - t2
> hbase(main):031:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '1G',
> POLICY => NO_WRITES
> Took 0.0088 seconds
> hbase(main):032:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE => true{noformat}
> * Remove RPC quota and drop the table, the Space Quota is not removed
> {noformat}
> hbase(main):033:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT => NONE
> Took 0.0193 seconds
> hbase(main):036:0> disable 't2'
> Took 0.4305 seconds
> hbase(main):037:0> drop 't2'
> Took 0.2353 seconds
> hbase(main):038:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE
> => true{noformat}
> * Deleting the quota entry from {{hbase:quota}} seems to be the option to
> reset it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20705) Having RPC Quota on a table prevents Space quota to be recreated/removed
[
https://issues.apache.org/jira/browse/HBASE-20705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514209#comment-16514209
]
Josh Elser commented on HBASE-20705:
{quote}Space quota is set to {{REMOVE => true}}
{quote}
[~gsbiju], where are you getting this syntax from? If you're trying to remove a
space quota, you would provide the {{LIMIT => NONE}}
{code:java}
Space quotas can also be removed via this command. To remove a space quota,
provide NONE
for the limit.
For example:
hbase> set_quota TYPE => SPACE, TABLE => 't1', LIMIT => NONE
hbase> set_quota TYPE => SPACE, NAMESPACE => 'ns1', LIMIT => NONE{code}
> Having RPC Quota on a table prevents Space quota to be recreated/removed
>
>
> Key: HBASE-20705
> URL: https://issues.apache.org/jira/browse/HBASE-20705
> Project: HBase
> Issue Type: Bug
>Reporter: Biju Nair
>Priority: Major
>
> * Property {{hbase.quota.remove.on.table.delete}} is set to {{true}} by
> default
> * Create a table and set RPC and Space quota
> {noformat}
> hbase(main):022:0> create 't2','cf1'
> Created table t2
> Took 0.7420 seconds
> => Hbase::Table - t2
> hbase(main):023:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '1G',
> POLICY => NO_WRITES
> Took 0.0105 seconds
> hbase(main):024:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.0186 seconds
> hbase(main):025:0> list_quotas
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> TABLE => t2 TYPE => SPACE, TABLE => t2, LIMIT => 1073741824, VIOLATION_POLICY
> => NO_WRITES{noformat}
> * Drop the table and the Space quota is set to {{REMOVE => true}}
> {noformat}
> hbase(main):026:0> disable 't2'
> Took 0.4363 seconds
> hbase(main):027:0> drop 't2'
> Took 0.2344 seconds
> hbase(main):028:0> list_quotas
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE => true
> USER => u1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10M/sec,
> SCOPE => MACHINE{noformat}
> * Recreate the table and set Space quota back. The Space quota on the table
> is still set to {{REMOVE => true}}
> {noformat}
> hbase(main):029:0> create 't2','cf1'
> Created table t2
> Took 0.7348 seconds
> => Hbase::Table - t2
> hbase(main):031:0> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '1G',
> POLICY => NO_WRITES
> Took 0.0088 seconds
> hbase(main):032:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE => true{noformat}
> * Remove RPC quota and drop the table, the Space Quota is not removed
> {noformat}
> hbase(main):033:0> set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT => NONE
> Took 0.0193 seconds
> hbase(main):036:0> disable 't2'
> Took 0.4305 seconds
> hbase(main):037:0> drop 't2'
> Took 0.2353 seconds
> hbase(main):038:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => SPACE, TABLE => t2, REMOVE
> => true{noformat}
> * Deleting the quota entry from {{hbase:quota}} seems to be the option to
> reset it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514190#comment-16514190 ] Zach York commented on HBASE-20734: --- I looked into the code for this and the challenge is that Region has no concept of walFS and the regionDir is determined from the HRegionFileSystem... I'll continue to look into how we can do this. Hopefully without changing the Region contract. > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Priority: Major > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in dataloss because we write recovered edits into a different place than where the recovering region server looks for them.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514187#comment-16514187
]
Zach York commented on HBASE-20723:
---
+1 on the patch once the checkstyle is fixed. I'll push tonight if nobody
objects.
> Custom hbase.wal.dir results in dataloss because we write recovered edits
> into a different place than where the recovering region server looks for them.
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514177#comment-16514177
]
Hudson commented on HBASE-20733:
Results for branch branch-1
[build #349 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/349/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/349//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/349//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/349//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) Custom hbase.wal.dir results in dataloss because we write recovered edits into a different place than where the recovering region server looks for them.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zach York updated HBASE-20723:
--
Description:
Description:
When custom hbase.wal.dir is configured the recovery system uses it in place of
the HBase root dir and thus constructs an incorrect path for recovered edits
when splitting WALs. This causes the recovery code in Region Servers to believe
there are no recovered edits to replay, which causes a loss of writes that had
not flushed prior to loss of a server.
Reproduction:
This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
1.1.2.2.6.3.2-14
By default the underlying data is going to wasb://x@y/hbase
I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
/mnt.
hbase.wal.dir= hdfs://mycluster/walontest
hbase.wal.dir.perms=700
hbase.rootdir.perms=700
hbase.rootdir=
wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
Procedure to reproduce this issue:
1. create a table in hbase shell
2. insert a row in hbase shell
3. reboot the VM which hosts that region
4. scan the table in hbase shell and it is empty
Looking at the region server logs:
{code:java}
2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
wal.WALSplitter: This region's directory doesn't exist:
hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
It is very likely that it was already split so it's safe to discard those edits.
{code}
The log split/replay ignored actual WAL due to WALSplitter is looking for the
region directory in the hbase.wal.dir we specified rather than the
hbase.rootdir.
Looking at the source code,
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
it uses the rootDir, which is walDir, as the tableDir root path.
So if we use HBASE-17437, waldir and hbase rootdir are in different path or
even in different filesystem, then the #5 uses walDir as tableDir is apparently
wrong.
CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
was:
This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
1.1.2.2.6.3.2-14
By default the underlying data is going to wasb://x@y/hbase
I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
/mnt.
hbase.wal.dir= hdfs://mycluster/walontest
hbase.wal.dir.perms=700
hbase.rootdir.perms=700
hbase.rootdir=
wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
Procedure to reproduce this issue:
1. create a table in hbase shell
2. insert a row in hbase shell
3. reboot the VM which hosts that region
4. scan the table in hbase shell and it is empty
Looking at the region server logs:
{code:java}
2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
wal.WALSplitter: This region's directory doesn't exist:
hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
It is very likely that it was already split so it's safe to discard those edits.
{code}
The log split/replay ignored actual WAL due to WALSplitter is looking for the
region directory in the hbase.wal.dir we specified rather than the
hbase.rootdir.
Looking at the source code,
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
it uses the rootDir, which is walDir, as the tableDir root path.
So if we use HBASE-17437, waldir and hbase rootdir are in different path or
even in different filesystem, then the #5 uses walDir as tableDir is apparently
wrong.
CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
> Custom hbase.wal.dir results in dataloss because we write recovered edits
> into a different place than where the recovering region server looks for them.
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes
[jira] [Updated] (HBASE-20723) Custom hbase.wal.dir results in dataloss because we write recovered edits into a different place than where the recovering region server looks for them.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zach York updated HBASE-20723:
--
Summary: Custom hbase.wal.dir results in dataloss because we write
recovered edits into a different place than where the recovering region server
looks for them. (was: WALSplitter uses the rootDir, which is walDir, as the
recovered edits root path)
> Custom hbase.wal.dir results in dataloss because we write recovered edits
> into a different place than where the recovering region server looks for them.
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the recovered edits root path
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514172#comment-16514172
]
Zach York commented on HBASE-20723:
---
Thanks for the summary [~busbey], one minor tweak:
{quote}which causes a loss writes that had not flushed prior to loss of a
server.
{quote}
which causes a loss of writes that had not flushed prior to loss of a server.
[~elserj] I'll make a comment on the vote thread, but I do agree with your
sentiment. Andrew has been doing good work with keeping the releases regular :)
> WALSplitter uses the rootDir, which is walDir, as the recovered edits root
> path
> ---
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20738) failed with testGetPassword on Windows
[
https://issues.apache.org/jira/browse/HBASE-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514174#comment-16514174
]
Sean Busbey commented on HBASE-20738:
-
thanks for the patch [~starphin]. could you please generate it using {{git
--format-patch}} so you can provide authorship information?
e.g. {{git format-patch --stdout origin/master
>/some/path/to/maybe/Downloads/HBASE-20738.0.patch}}
> failed with testGetPassword on Windows
> ---
>
> Key: HBASE-20738
> URL: https://issues.apache.org/jira/browse/HBASE-20738
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0
>Reporter: star
>Assignee: star
>Priority: Minor
> Attachments: test-hbase-configuration.patch
>
>
> When running unit test on Windows, testGetPassword from
> TestHBaseConfiguration.java failed. The original code produces a uri path
> like "jceks://file/root/hbase/others" on unix system, but produces a invalid
> path like "jceks://fileD:\wordspace\hbase\others"，which will cause a
> URISyntaxException latter.
> To solve the problem, just add a "/" prefix to windows local path and
> replace all "\" with "/".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20738) failed with testGetPassword on Windows
[ https://issues.apache.org/jira/browse/HBASE-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-20738: --- Assignee: star > failed with testGetPassword on Windows > --- > > Key: HBASE-20738 > URL: https://issues.apache.org/jira/browse/HBASE-20738 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: star >Assignee: star >Priority: Minor > Attachments: test-hbase-configuration.patch > > > When running unit test on Windows, testGetPassword from > TestHBaseConfiguration.java failed. The original code produces a uri path > like "jceks://file/root/hbase/others" on unix system, but produces a invalid > path like "jceks://fileD:\wordspace\hbase\others"，which will cause a > URISyntaxException latter. > To solve the problem, just add a "/" prefix to windows local path and > replace all "\" with "/". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514152#comment-16514152
]
Sean Busbey commented on HBASE-20332:
-
-v6
- suppresses IllegalImport complaints when they involve
{{org.apache.htrace.core}}, which is the package for htrace v4 classes we use
With this change I get no new checkstyle errors locally comparing before and
after the patch as described earlier.
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch,
> HBASE-20332.5.patch, HBASE-20332.6.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20332:
Attachment: HBASE-20332.6.patch
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch,
> HBASE-20332.5.patch, HBASE-20332.6.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20738) failed with testGetPassword on Windows
[
https://issues.apache.org/jira/browse/HBASE-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514145#comment-16514145
]
Hadoop QA commented on HBASE-20738:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
22s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 58s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
35s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20738 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927951/test-hbase-configuration.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1d05e05fda12 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 30772eef5d |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13275/testReport/ |
| Max. process+thread count | 289 (vs. ulimit of 1) |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13275/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> failed with testGetPassword on
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[
https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514123#comment-16514123
]
Hadoop QA commented on HBASE-20708:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 21 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
4s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} The patch hbase-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch hbase-procedure passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} hbase-server: The patch generated 0 new + 363
unchanged - 16 fixed = 363 total (was 379) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 41s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
2m 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
20s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
55s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 20s{color}
| {color:red} hbase-server in the
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514122#comment-16514122
]
Hadoop QA commented on HBASE-20332:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
1s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
47s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
27s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
57s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 3s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}196m
51s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 8m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}270m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20332 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927914/HBASE-20332.5.patch |
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile checkstyle shellcheck shelldocs |
| uname | Linux 3ba3973b484a 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 86653c708f |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| shellcheck | v0.4.4 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13273/testReport/ |
| Max. process+thread count |
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the recovered edits root path
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514086#comment-16514086
]
Sean Busbey commented on HBASE-20723:
-
Can we make the subject better convey the problem for downstream users?
bq. WALSplitter uses the rootDir, which is walDir, as the recovered edits root
path
would this rephrasing be accurate?
bq. Custom hbase.wal.dir results in dataloss because we write recovered edits
into a different place than where the recovering region server looks for them.
And then this for the start of the description:
bq. When custom hbase.wal.dir is configured the recovery system uses it in
place of the HBase root dir and thus constructs an incorrect path for recovered
edits when splitting WALs. This causes the recovery code in Region Servers to
believe there are no recovered edits to replay, which causes a loss writes that
had not flushed prior to loss of a server.
> WALSplitter uses the rootDir, which is walDir, as the recovered edits root
> path
> ---
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the recovered edits root path
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514082#comment-16514082
]
Josh Elser edited comment on HBASE-20723 at 6/15/18 4:49 PM:
-
{quote}Do you think this is enough to reroll a 1.4.5 RC? It isn't a default
config, but still quite serious for those that set this config.
{quote}
[~zyork] I'd _like_ to defer it to 1.4.6 (Andrew keeps a good cadence on these
releases), I'd suggest leaving a note on the VOTE thread so it's not just me
making the call. There are some other things in flight that make us want to get
1.4.5 out now.
was (Author: elserj):
{quote}Do you think this is enough to reroll a 1.4.5 RC? It isn't a default
config, but still quite serious for those that set this config.
{quote}
I'd _like_ to defer it to 1.4.6 (Andrew keeps a good cadence on these
releases), I'd suggest leaving a note on the VOTE thread so it's not just me
making the call. There are some other things in flight that make us want to get
1.4.5 out now.
> WALSplitter uses the rootDir, which is walDir, as the recovered edits root
> path
> ---
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the recovered edits root path
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514082#comment-16514082
]
Josh Elser commented on HBASE-20723:
{quote}Do you think this is enough to reroll a 1.4.5 RC? It isn't a default
config, but still quite serious for those that set this config.
{quote}
I'd _like_ to defer it to 1.4.6 (Andrew keeps a good cadence on these
releases), I'd suggest leaving a note on the VOTE thread so it's not just me
making the call. There are some other things in flight that make us want to get
1.4.5 out now.
> WALSplitter uses the rootDir, which is walDir, as the recovered edits root
> path
> ---
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the recovered edits root path
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-20723:
---
Component/s: Recovery
> WALSplitter uses the rootDir, which is walDir, as the recovered edits root
> path
> ---
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20681) IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
[
https://issues.apache.org/jira/browse/HBASE-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514053#comment-16514053
]
Josh Elser commented on HBASE-20681:
Aw, thanks, sir!
> IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
> ---
>
> Key: HBASE-20681
> URL: https://issues.apache.org/jira/browse/HBASE-20681
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: HBASE-20681.001.patch, HBASE-20681.002.patch,
> HBASE-20681.003.patch, HBASE-20681.004.patch, HBASE-20681.004.patch
>
>
> HADOOP-15406 marked mockito and junit as test-only dependencies which, I
> believe, has stopped them from being included in a stock Hadoop classpath.
> Prior, you'd get hamcrest at {{share/hadoop/common/lib/hamcrest-core-1.3.jar}}
> However, we depend on it being there for our junit in hbase-it:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.1:tree (default-cli) @ hbase-it ---
> [INFO] org.apache.hbase:hbase-it:jar:2.0.1-SNAPSHOT
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> {noformat}
> We need to make sure we include it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20621) Unclear error for deleting from not existing table.
[
https://issues.apache.org/jira/browse/HBASE-20621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514051#comment-16514051
]
Josh Elser commented on HBASE-20621:
[~mingdaoy], cool! [~sergey.soldatov], were you working on this already?
Just want to make sure we don't have two folks trying to do the same thing.
> Unclear error for deleting from not existing table.
>
>
> Key: HBASE-20621
> URL: https://issues.apache.org/jira/browse/HBASE-20621
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Sergey Soldatov
>Assignee: Mingdao Yang
>Priority: Major
>
> When I try to delete a row from a not existing table, the error is quite
> confusing. Instead of getting a table not found exception I got
> {noformat}
> ERROR [main] client.AsyncRequestFutureImpl: Cannot get replica 0 location for
> {"totalColumns":1,"row":"r1","families":{"c1":[{"qualifier":"","vlen":0,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807}
> ERROR: Failed 1 action: t1: 1 time, servers with issues: null
> {noformat}
> That happens, because delete is using AsyncRequestFuture which wraps all
> region location errors into 'Cannot get replica' error. I expect that others
> actions like batch, mutateRow, checkAndDelete behave in the same way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20738) failed with testGetPassword on Windows
[ https://issues.apache.org/jira/browse/HBASE-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-20738: --- Status: Patch Available (was: Open) > failed with testGetPassword on Windows > --- > > Key: HBASE-20738 > URL: https://issues.apache.org/jira/browse/HBASE-20738 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: star >Priority: Minor > Attachments: test-hbase-configuration.patch > > > When running unit test on Windows, testGetPassword from > TestHBaseConfiguration.java failed. The original code produces a uri path > like "jceks://file/root/hbase/others" on unix system, but produces a invalid > path like "jceks://fileD:\wordspace\hbase\others"，which will cause a > URISyntaxException latter. > To solve the problem, just add a "/" prefix to windows local path and > replace all "\" with "/". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20681) IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
[
https://issues.apache.org/jira/browse/HBASE-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20681:
Fix Version/s: (was: 2.0.1)
2.0.2
> IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
> ---
>
> Key: HBASE-20681
> URL: https://issues.apache.org/jira/browse/HBASE-20681
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: HBASE-20681.001.patch, HBASE-20681.002.patch,
> HBASE-20681.003.patch, HBASE-20681.004.patch, HBASE-20681.004.patch
>
>
> HADOOP-15406 marked mockito and junit as test-only dependencies which, I
> believe, has stopped them from being included in a stock Hadoop classpath.
> Prior, you'd get hamcrest at {{share/hadoop/common/lib/hamcrest-core-1.3.jar}}
> However, we depend on it being there for our junit in hbase-it:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.1:tree (default-cli) @ hbase-it ---
> [INFO] org.apache.hbase:hbase-it:jar:2.0.1-SNAPSHOT
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> {noformat}
> We need to make sure we include it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20681) IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
[
https://issues.apache.org/jira/browse/HBASE-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20681:
Resolution: Fixed
Release Note:
Users of our integration tests on Hadoop 3 can now add all needed dependencies
by pointing at jars included in our binary convenience artifact.
Prior to this fix, downstream users on Hadoop 3 would need to get a copy of the
Hamcrest v1.3 jar from elsewhere.
Status: Resolved (was: Patch Available)
+1. will push to listed fix versions after I finish confirming builds. took a
crack at an appropriate release note; others should feel free to update it.
> IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
> ---
>
> Key: HBASE-20681
> URL: https://issues.apache.org/jira/browse/HBASE-20681
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20681.001.patch, HBASE-20681.002.patch,
> HBASE-20681.003.patch, HBASE-20681.004.patch, HBASE-20681.004.patch
>
>
> HADOOP-15406 marked mockito and junit as test-only dependencies which, I
> believe, has stopped them from being included in a stock Hadoop classpath.
> Prior, you'd get hamcrest at {{share/hadoop/common/lib/hamcrest-core-1.3.jar}}
> However, we depend on it being there for our junit in hbase-it:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.1:tree (default-cli) @ hbase-it ---
> [INFO] org.apache.hbase:hbase-it:jar:2.0.1-SNAPSHOT
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> {noformat}
> We need to make sure we include it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514024#comment-16514024
]
Sean Busbey commented on HBASE-20332:
-
{quote}
The reason I'm concerned about the checkstyle issues is because I know we have
some modules configured to failOnViolation=true and I don't want the build to
start failing due to these changes.{quote}
Oh sweet. I wasn't aware we had reached that point anywhere. Lemme figure
something out.
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch,
> HBASE-20332.5.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513993#comment-16513993
]
Mike Drob commented on HBASE-20332:
---
The reason I'm concerned about the checkstyle issues is because I know we have
some modules configured to {{failOnViolation=true}} and I don't want the build
to start failing due to these changes.
bq. I don't know of a way to flag that use of org.apache.htrace.core
specifically is fine
I was thinking about this more and I _think_ it's possible via the suppressions
file, using the {{message}} property.
http://checkstyle.sourceforge.net/config_filters.html#SuppressionFilter
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch,
> HBASE-20332.5.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513970#comment-16513970
]
Sean Busbey commented on HBASE-20332:
-
okay I got it.
the checkstyle output itself is not zero:
*
https://builds.apache.org/job/PreCommit-HBASE-Build/13265/artifact/patchprocess/maven-branch-checkstyle-root.txt
*
https://builds.apache.org/job/PreCommit-HBASE-Build/13265/artifact/patchprocess/maven-patch-checkstyle-root.txt
The patch version definitely shows a bunch of new warnings for htrace stuff.
The files I was looking at are the filtered versions for "checkstyle warnings
we care about"
Yetus' checkstyle plugin has an optimizations
([ref|https://github.com/apache/yetus/blob/rel/0.7.0/precommit/test-patch.d/checkstyle.sh#L213])
where it cuts the output down to just the files changed by the patch. We'll
need a way to flag that we care about all differences currently. I suspect
that'll mean either a change to yetus or keeping our own checkstyle plugin.
I'll link a jira once I figure out which. In either case I think the scope of
doing that is going to be too much to wait on.
Here's a manual diff of the two, there are 61 instances (for context, we have
16,189 existing checkstyle errors on master).
{code}
375a376,381
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:21:1:
> Illegal import - org.apache.htrace.core.HTraceConfiguration. [IllegalImport]
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:22:1:
> Illegal import - org.apache.htrace.core.Sampler. [IllegalImport]
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:23:1:
> Illegal import - org.apache.htrace.core.Span. [IllegalImport]
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:24:1:
> Illegal import - org.apache.htrace.core.SpanReceiver. [IllegalImport]
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:25:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:26:1:
> Illegal import - org.apache.htrace.core.Tracer. [IllegalImport]
376a383
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/SpanReceiverHost.java:25:1:
> Illegal import - org.apache.htrace.core.SpanReceiver. [IllegalImport]
377a385
> [ERROR]
> hbase-common/src/main/java/org/apache/hadoop/hbase/trace/HBaseHTraceConfiguration.java:22:1:
> Illegal import - org.apache.htrace.core.HTraceConfiguration. [IllegalImport]
1326a1335
> [ERROR]
> hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java:66:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
1337a1347,1348
> [ERROR]
> hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/Call.java:33:1:
> Illegal import - org.apache.htrace.core.Span. [IllegalImport]
> [ERROR]
> hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/Call.java:34:1:
> Illegal import - org.apache.htrace.core.Tracer. [IllegalImport]
3534a3546
> [ERROR]
> hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java:51:1:
> Illegal import - org.apache.htrace.core.Tracer. [IllegalImport]
4395a4408
> [ERROR]
> hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:32:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
5312a5326,5328
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java:26:1:
> Illegal import - org.apache.htrace.core.Span. [IllegalImport]
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java:27:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java:28:1:
> Illegal import - org.apache.htrace.core.Tracer. [IllegalImport]
5955a5972
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java:63:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
7516a7534
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:177:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
8518a8537
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java:54:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
8586a8606
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java:80:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
8603a8624
> [ERROR]
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java:56:1:
> Illegal import - org.apache.htrace.core.TraceScope. [IllegalImport]
8635a8657
>
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513891#comment-16513891
]
Sean Busbey commented on HBASE-20332:
-
the branch and patch checkstyle output files are both size 0. sigh.
rerunning in debug so I can try to see what's up.
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch,
> HBASE-20332.5.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20742) Always create WAL directory for region server
Duo Zhang created HBASE-20742: - Summary: Always create WAL directory for region server Key: HBASE-20742 URL: https://issues.apache.org/jira/browse/HBASE-20742 Project: HBase Issue Type: Bug Reporter: Duo Zhang After HBASE-20708, when master restart, we will scan the wal directory to find out the live servers. In most cases this is OK, as when we create a HRegion instance at RS side, we will create a WAL for it, and the directory which contains the server name will be there, even if user always use SKIP_WAL. But there could still be problem as the directory is created in the implementation of WAL, not in the initialization of region server, so if user uses DisabledWALProvider then we will be in trouble. So let's fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513818#comment-16513818 ] Duo Zhang commented on HBASE-20708: --- Add more comments. > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708-v5.patch, > HBASE-20708-v6.patch, HBASE-20708-v7.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20708: -- Attachment: HBASE-20708-v7.patch > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708-v5.patch, > HBASE-20708-v6.patch, HBASE-20708-v7.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19064) Synchronous replication for HBase
[
https://issues.apache.org/jira/browse/HBASE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513814#comment-16513814
]
Hudson commented on HBASE-19064:
Results for branch HBASE-19064
[build #162 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/162/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/162//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/162//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/162//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Synchronous replication for HBase
> -
>
> Key: HBASE-19064
> URL: https://issues.apache.org/jira/browse/HBASE-19064
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
>
> The guys from Alibaba made a presentation on HBaseCon Asia about the
> synchronous replication for HBase. We(Xiaomi) think this is a very useful
> feature for HBase so we want to bring it into the community version.
> This is a big feature so we plan to do it in a feature branch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20740) StochasticLoadBalancer should consider Coprocessor request count when computing cost
[
https://issues.apache.org/jira/browse/HBASE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513813#comment-16513813
]
Hadoop QA commented on HBASE-20740:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
50s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
14s{color} | {color:red} hbase-server: The patch generated 2 new + 423
unchanged - 1 fixed = 425 total (was 424) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} hbase-rest: The patch generated 2 new + 15 unchanged -
0 fixed = 17 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 58s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
2m 32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
57s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}106m
39s{color} | {color:green}
[jira] [Commented] (HBASE-20737) put collection into ArrayList instead of addAll function
[ https://issues.apache.org/jira/browse/HBASE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513810#comment-16513810 ] Chia-Ping Tsai commented on HBASE-20737: LGTM. Will commit it later. > put collection into ArrayList instead of addAll function > > > Key: HBASE-20737 > URL: https://issues.apache.org/jira/browse/HBASE-20737 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: taiyinglee >Assignee: taiyinglee >Priority: Trivial > Attachments: HBASE-20737.v0.patch, HBASE-20737.v0.patch > > > [https://docs.oracle.com/javase/7/docs/api/java/util/Collection.html] > [https://docs.oracle.com/javase/7/docs/api/java/util/ArrayList.html] > [https://docs.oracle.com/javase/7/docs/api/java/util/Set.html] > > /hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterStatusPublisher.java > change > List> entries = new ArrayList<>(); > entries.addAll(lastSent.entrySet()); > to > List> entries = new > ArrayList<>(lastSent.entrySet()); -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513808#comment-16513808 ] Duo Zhang commented on HBASE-20708: --- All green, good. Thanks [~allan163] for reviewing. Any comments sir [~stack]? > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708-v5.patch, > HBASE-20708-v6.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20737) put collection into ArrayList instead of addAll function
[
https://issues.apache.org/jira/browse/HBASE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513779#comment-16513779
]
Hadoop QA commented on HBASE-20737:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
4s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
24s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 53s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}122m
24s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20737 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927962/HBASE-20737.v0.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 5a6db8f4f9bc 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 86653c708f |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13271/testReport/ |
| Max. process+thread count | 4363 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13271/console |
| Powered by | Apache
[jira] [Commented] (HBASE-20727) Persist FlushedSequenceId to speed up WAL split after cluster restart
[
https://issues.apache.org/jira/browse/HBASE-20727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513763#comment-16513763
]
Sean Busbey commented on HBASE-20727:
-
{code}
+import org.apache.hbase.thirdparty.com.google.protobuf.ByteString;
{code}
This import in ServerManager is in the wrong order. It should be down by the
other thirdparty imports.
Once that's fixed I'm +1.
> Persist FlushedSequenceId to speed up WAL split after cluster restart
> -
>
> Key: HBASE-20727
> URL: https://issues.apache.org/jira/browse/HBASE-20727
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20727.002.patch, HBASE-20727.003.patch,
> HBASE-20727.004.patch, HBASE-20727.patch
>
>
> We use flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion in
> ServerManager to record the latest flushed seqids of regions and stores. So
> during log split, we can use seqids stored in those maps to filter out the
> edits which do not need to be replayed. But, those maps are not persisted.
> After cluster restart or master restart, info of flushed seqids are all lost.
> Here I offer a way to persist those info to HDFS, even if master restart, we
> can still use those info to filter WAL edits and then to speed up replay.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20727) Persist FlushedSequenceId to speed up WAL split after cluster restart
[ https://issues.apache.org/jira/browse/HBASE-20727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513752#comment-16513752 ] Allan Yang commented on HBASE-20727: [~busbey], any more advice, sir? > Persist FlushedSequenceId to speed up WAL split after cluster restart > - > > Key: HBASE-20727 > URL: https://issues.apache.org/jira/browse/HBASE-20727 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20727.002.patch, HBASE-20727.003.patch, > HBASE-20727.004.patch, HBASE-20727.patch > > > We use flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion in > ServerManager to record the latest flushed seqids of regions and stores. So > during log split, we can use seqids stored in those maps to filter out the > edits which do not need to be replayed. But, those maps are not persisted. > After cluster restart or master restart, info of flushed seqids are all lost. > Here I offer a way to persist those info to HDFS, even if master restart, we > can still use those info to filter WAL edits and then to speed up replay. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513743#comment-16513743 ] Allan Yang commented on HBASE-20708: Just reviewed the patch, I like the idea of removing the 'serverCrashProcessingEnabled' state, and queuing of the dead server. It does made the master start process more clear (y) > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708-v5.patch, > HBASE-20708-v6.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[
https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513739#comment-16513739
]
Hadoop QA commented on HBASE-20708:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 21 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
46s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} The patch hbase-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch hbase-procedure passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} hbase-server: The patch generated 0 new + 363
unchanged - 16 fixed = 363 total (was 379) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
52s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 49s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
57s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
44s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}123m
21s{color} | {color:green}
[jira] [Commented] (HBASE-20695) Implement table level RegionServer replication metrics
[
https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513733#comment-16513733
]
Hudson commented on HBASE-20695:
Results for branch branch-2
[build #864 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/864/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/864//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/864//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/864//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Implement table level RegionServer replication metrics
> ---
>
> Key: HBASE-20695
> URL: https://issues.apache.org/jira/browse/HBASE-20695
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: HBASE-20695.master.001.patch,
> HBASE-20695.master.002.patch, HBASE-20695.master.003.patch,
> HBASE-20695.master.004.patch, HBASE-20695.master.005.patch,
> HBASE-20695.master.006.patch, HBASE-20695.master.007.patch,
> HBASE-20695.master.008.patch, HBASE-20695.master.009.patch,
> HBASE-20695.master.010.patch
>
>
> Region server metrics now are mainly global metrics. It would be nice to have
> table level metrics such as table level source.AgeOfLastShippedOp to indicate
> operators which table's replication is lagging behind.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20741) Split of a region with replicas creates all daughter regions and its replica in same server
[ https://issues.apache.org/jira/browse/HBASE-20741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-20741: --- Component/s: read replicas > Split of a region with replicas creates all daughter regions and its replica > in same server > --- > > Key: HBASE-20741 > URL: https://issues.apache.org/jira/browse/HBASE-20741 > Project: HBase > Issue Type: Bug > Components: read replicas >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.1.0 > > > Generally it is better that the parent region when split creates the daughter > region in the same target server. > But for replicas also we do the same and all the replica regions are created > in the same target server. We should ideally be doing a round robin and only > the primary daughter region should be opened in the intended target server > (where the parent was previously opened). > [~huaxiang] FYI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20741) Split of a region with replicas creates all daughter regions and its replica in same server
ramkrishna.s.vasudevan created HBASE-20741: -- Summary: Split of a region with replicas creates all daughter regions and its replica in same server Key: HBASE-20741 URL: https://issues.apache.org/jira/browse/HBASE-20741 Project: HBase Issue Type: Bug Affects Versions: 2.0.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.1.0 Generally it is better that the parent region when split creates the daughter region in the same target server. But for replicas also we do the same and all the replica regions are created in the same target server. We should ideally be doing a round robin and only the primary daughter region should be opened in the intended target server (where the parent was previously opened). [~huaxiang] FYI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20740) StochasticLoadBalancer should consider Coprocessor request count when computing cost
[ https://issues.apache.org/jira/browse/HBASE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-20740: --- Summary: StochasticLoadBalancer should consider Coprocessor request count when computing cost (was: StochasticLoadBalancer should consider Coprocessor request count when compute cost) > StochasticLoadBalancer should consider Coprocessor request count when > computing cost > > > Key: HBASE-20740 > URL: https://issues.apache.org/jira/browse/HBASE-20740 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: chenxu >Assignee: chenxu >Priority: Major > Attachments: HBASE-20740-master-v1.patch > > > When compute region load cost, ReadRequest, WriteRequest, MemStoreSize and > StoreFileSize are considered, But CoprocessorRequests are not include. In our > kylin cluster, there only have coprocessor requests, and the cluster > sometimes unbalanced. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20740) StochasticLoadBalancer should consider Coprocessor request count when computing cost
[
https://issues.apache.org/jira/browse/HBASE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513667#comment-16513667
]
Ted Yu commented on HBASE-20740:
bq. only have coprocessor requests
Please describe your use case in more detail: how does coprocessor generate
requests (if there is no request from clients).
{code}
+ * @return the number of write requests made to region
+ */
+ public long getCpRequestCount();
{code}
javadoc says write requests but the method name doesn't reflect such (implying
read+write). Please go over the patch and make the method name(s) match
semantics you want.
Please upload next patch to review board.
> StochasticLoadBalancer should consider Coprocessor request count when
> computing cost
>
>
> Key: HBASE-20740
> URL: https://issues.apache.org/jira/browse/HBASE-20740
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
> Attachments: HBASE-20740-master-v1.patch
>
>
> When compute region load cost, ReadRequest, WriteRequest, MemStoreSize and
> StoreFileSize are considered, But CoprocessorRequests are not include. In our
> kylin cluster, there only have coprocessor requests, and the cluster
> sometimes unbalanced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20740) StochasticLoadBalancer should consider Coprocessor request count when compute cost
[ https://issues.apache.org/jira/browse/HBASE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-20740: --- Status: Patch Available (was: Open) > StochasticLoadBalancer should consider Coprocessor request count when compute > cost > -- > > Key: HBASE-20740 > URL: https://issues.apache.org/jira/browse/HBASE-20740 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: chenxu >Assignee: chenxu >Priority: Major > Attachments: HBASE-20740-master-v1.patch > > > When compute region load cost, ReadRequest, WriteRequest, MemStoreSize and > StoreFileSize are considered, But CoprocessorRequests are not include. In our > kylin cluster, there only have coprocessor requests, and the cluster > sometimes unbalanced. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20740) StochasticLoadBalancer should consider Coprocessor request count when compute cost
[ https://issues.apache.org/jira/browse/HBASE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513632#comment-16513632 ] chenxu commented on HBASE-20740: a patch for QA > StochasticLoadBalancer should consider Coprocessor request count when compute > cost > -- > > Key: HBASE-20740 > URL: https://issues.apache.org/jira/browse/HBASE-20740 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: chenxu >Assignee: chenxu >Priority: Major > Attachments: HBASE-20740-master-v1.patch > > > When compute region load cost, ReadRequest, WriteRequest, MemStoreSize and > StoreFileSize are considered, But CoprocessorRequests are not include. In our > kylin cluster, there only have coprocessor requests, and the cluster > sometimes unbalanced. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20740) StochasticLoadBalancer should consider Coprocessor request count when compute cost
[ https://issues.apache.org/jira/browse/HBASE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenxu updated HBASE-20740: --- Attachment: HBASE-20740-master-v1.patch > StochasticLoadBalancer should consider Coprocessor request count when compute > cost > -- > > Key: HBASE-20740 > URL: https://issues.apache.org/jira/browse/HBASE-20740 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: chenxu >Assignee: chenxu >Priority: Major > Attachments: HBASE-20740-master-v1.patch > > > When compute region load cost, ReadRequest, WriteRequest, MemStoreSize and > StoreFileSize are considered, But CoprocessorRequests are not include. In our > kylin cluster, there only have coprocessor requests, and the cluster > sometimes unbalanced. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20740) StochasticLoadBalancer should consider Coprocessor request count when compute cost
[ https://issues.apache.org/jira/browse/HBASE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenxu updated HBASE-20740: --- Description: When compute region load cost, ReadRequest, WriteRequest, MemStoreSize and StoreFileSize are considered, But CoprocessorRequests are not include. In our kylin cluster, there only have coprocessor requests, and the cluster sometimes unbalanced. > StochasticLoadBalancer should consider Coprocessor request count when compute > cost > -- > > Key: HBASE-20740 > URL: https://issues.apache.org/jira/browse/HBASE-20740 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: chenxu >Assignee: chenxu >Priority: Major > > When compute region load cost, ReadRequest, WriteRequest, MemStoreSize and > StoreFileSize are considered, But CoprocessorRequests are not include. In our > kylin cluster, there only have coprocessor requests, and the cluster > sometimes unbalanced. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20740) StochasticLoadBalancer should consider Coprocessor request count when compute cost
[ https://issues.apache.org/jira/browse/HBASE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenxu updated HBASE-20740: --- Environment: (was: When compute region load cost, ReadRequest, WriteRequest, MemStoreSize and StoreFileSize are considered, But CoprocessorRequests are not include. In our kylin cluster, there only have coprocessor requests, and the cluster sometimes unbalanced.) > StochasticLoadBalancer should consider Coprocessor request count when compute > cost > -- > > Key: HBASE-20740 > URL: https://issues.apache.org/jira/browse/HBASE-20740 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: chenxu >Assignee: chenxu >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20740) StochasticLoadBalancer should consider Coprocessor request count when compute cost
chenxu created HBASE-20740: -- Summary: StochasticLoadBalancer should consider Coprocessor request count when compute cost Key: HBASE-20740 URL: https://issues.apache.org/jira/browse/HBASE-20740 Project: HBase Issue Type: Improvement Components: Balancer Environment: When compute region load cost, ReadRequest, WriteRequest, MemStoreSize and StoreFileSize are considered, But CoprocessorRequests are not include. In our kylin cluster, there only have coprocessor requests, and the cluster sometimes unbalanced. Reporter: chenxu Assignee: chenxu -- This message was sent by Atlassian JIRA (v7.6.3#76005)
