[jira] [Commented] (HBASE-19275) TestSnapshotFileCache never worked properly
[
https://issues.apache.org/jira/browse/HBASE-19275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836898#comment-16836898
]
Hudson commented on HBASE-19275:
Results for branch branch-1
[build #815 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> TestSnapshotFileCache never worked properly
> ---
>
> Key: HBASE-19275
> URL: https://issues.apache.org/jira/browse/HBASE-19275
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-19275-branch-1.patch,
> HBASE-19275-master.001.patch, HBASE-19275-master.001.patch
>
>
> Error-prone noticed we were asking Iterables.contains() questions with the
> wrong type in TestSnapshotFileCache. I've attached a fixed version of the
> test. The results suggest the cache is not evicting entries properly.
> {noformat}
> java.lang.AssertionError: Cache found
> 'hdfs://localhost:52867/user/apurtell/test-data/8ce04c85-ce4b-4844-b454-5303482ade95/data/default/snapshot1/9e49edd0ab41657fb0c6ebb4d9dfad15/cf/f132e5b06f66443f8003363ed1535aac',
> but it shouldn't have.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.createAndTestSnapshot(TestSnapshotFileCache.java:260)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.createAndTestSnapshotV1(TestSnapshotFileCache.java:206)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.testReloadModifiedDirectory(TestSnapshotFileCache.java:102)
> {noformat}
> {noformat}
> java.lang.AssertionError: Cache found
> 'hdfs://localhost:52867/user/apurtell/test-data/8ce04c85-ce4b-4844-b454-5303482ade95/data/default/snapshot1a/2e81adb9212c98cff970eafa006fc40b/cf/a2ec478d850e4e348359699c53b732c4',
> but it shouldn't have.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.createAndTestSnapshot(TestSnapshotFileCache.java:260)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.createAndTestSnapshotV1(TestSnapshotFileCache.java:206)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.testLoadAndDelete(TestSnapshotFileCache.java:88)
> {noformat}
> These changes are part of HBASE-19239
> I've disabled the offending test cases with @Ignore in that patch, but they
> should be reenabled and fixed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20724) Sometimes some compacted storefiles are still opened after region failover
[
https://issues.apache.org/jira/browse/HBASE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836894#comment-16836894
]
Hudson commented on HBASE-20724:
Results for branch branch-1
[build #815 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Sometimes some compacted storefiles are still opened after region failover
> --
>
> Key: HBASE-20724
> URL: https://issues.apache.org/jira/browse/HBASE-20724
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.3.0
>
> Attachments: HBASE-20724.branch-2.001.patch,
> HBASE-20724.master.001.patch, HBASE-20724.master.002.patch,
> HBASE-20724.master.003.patch, HBASE-20724.master.004.patch,
> HBASE-20724.master.005.patch, HBASE-20724.master.006.patch,
> HBASE-20724.master.007.patch, HBASE-20724.master.008.patch,
> HBASE-20724.master.009.patch, HBASE-20724.master.010.patch,
> HBASE-20724.master.011.patch, HBASE-20724.master.012.patch,
> HBASE-20724.master.013.patch, HBASE-20724.master.013.patch,
> HBASE-20724.master.014.patch
>
>
> It is important that compacted storefiles of a given compaction execution are
> wholly opened or archived to insure data consistency. ie a storefile
> containing delete tombstones can be archived while older storefiles
> containing cells that were supposed to be deleted are left unarchived thereby
> undeleting those cells.
> When a server fails compaction markers (in the wal edit) are used to
> determine which storefiles are compacted and should be excluded during region
> open (during failover). But the WALs containing compaction markers can be
> prematurely archived even though there are still compacted storefiles for
> that particular compaction event that hasn't been archived yet. Thus losing
> compaction information that needs to be replayed in the event of an RS crash.
> This is because hlog archiving logic only keeps track of flushed storefiles
> and not compacted ones.
> https://issues.apache.org/jira/browse/HBASE-20704?focusedCommentId=16507680=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16507680
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22389) Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
[
https://issues.apache.org/jira/browse/HBASE-22389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836897#comment-16836897
]
Hudson commented on HBASE-22389:
Results for branch branch-1
[build #815 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
> --
>
> Key: HBASE-22389
> URL: https://issues.apache.org/jira/browse/HBASE-22389
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Fix For: 1.5.0, 1.4.11
>
>
> This commit seems broken on branch-1.
> https://github.com/apache/hbase/commit/d80d3fa45487fedf203c7144e8267753e4d53529#diff-ae046012dd310fc116a729bf36c7df81
> seems to have fixed the issue that the other HBASE-19275 patches fixed, but
> when HBASE-19275 was applied to branch-1 it appears to have reverted this
> code:
> https://github.com/apache/hbase/commit/e8e4beacb039299a9e56cd355e3ada08784a7acb#diff-ae046012dd310fc116a729bf36c7df81
> meaning the tests always pass again.
> [~apurtell] [~xucang]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836893#comment-16836893
]
Hudson commented on HBASE-22330:
Results for branch branch-1
[build #815 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Backport HBASE-20724 (Sometimes some compacted storefiles are still opened
> after region failover) to branch-1
> -
>
> Key: HBASE-22330
> URL: https://issues.apache.org/jira/browse/HBASE-22330
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction, regionserver
>Affects Versions: 1.5.0, 1.4.9, 1.3.4
>Reporter: Andrew Purtell
>Assignee: Abhishek Singh Chouhan
>Priority: Major
> Fix For: 1.5.0, 1.3.5, 1.4.11
>
> Attachments: HBASE-22330-addendum.branch-1.patch,
> HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch,
> HBASE-22330.branch-1.3.001.patch
>
>
> There appears to be a race condition between close and split which when
> combined with a side effect of HBASE-20704, leads to the parent region store
> files getting archived and cleared while daughter regions still have
> references to those parent region store files.
> Here is the timeline of events observed for an affected region:
> # RS1 faces ZooKeeper connectivity issue for master node and starts shutting
> itself down. As part of this it starts to close the store and clean up the
> compacted files (File A)
> # Master starts bulk assigning regions and assign parent region to RS2
> # Region opens on RS2 and ends up opening compacted store file(s) (suspect
> this is due to HBASE-20724)
> # Now split happens and daughter regions open on RS2 and try to run a
> compaction as part of post open
> # Split request at this point is complete. However now archiving proceeds on
> RS1 and ends up archiving the store file that is referenced by the daughter.
> Compaction fails due to FileNotFoundException and all subsequent attempts to
> open the region will fail until manual resolution.
> We think having HBASE-20724 would help in such situations since we won't end
> up loading compacted store files in the first place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22390) backport HBASE-22190 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836895#comment-16836895
]
Hudson commented on HBASE-22390:
Results for branch branch-1
[build #815 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> backport HBASE-22190 to branch-1
>
>
> Key: HBASE-22390
> URL: https://issues.apache.org/jira/browse/HBASE-22390
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.4.10
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Fix For: 1.5.0, 1.4.11
>
>
> HBASE-22190 was only applied to branch-2+ while the bug exists in branch-1 as
> well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22190) SnapshotFileCache may fail to load the correct snapshot file list when there is an on-going snapshot operation
[
https://issues.apache.org/jira/browse/HBASE-22190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836896#comment-16836896
]
Hudson commented on HBASE-22190:
Results for branch branch-1
[build #815 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/815//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> SnapshotFileCache may fail to load the correct snapshot file list when there
> is an on-going snapshot operation
> --
>
> Key: HBASE-22190
> URL: https://issues.apache.org/jira/browse/HBASE-22190
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.1.5
>
>
> And it seems that it is not only a test issue, we do delete the files under
> the archive directory, which is incorrect.
> Need to find out why, this maybe a serious bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22389) Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
[
https://issues.apache.org/jira/browse/HBASE-22389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836890#comment-16836890
]
Hudson commented on HBASE-22389:
Results for branch branch-1.4
[build #787 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
> --
>
> Key: HBASE-22389
> URL: https://issues.apache.org/jira/browse/HBASE-22389
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Fix For: 1.5.0, 1.4.11
>
>
> This commit seems broken on branch-1.
> https://github.com/apache/hbase/commit/d80d3fa45487fedf203c7144e8267753e4d53529#diff-ae046012dd310fc116a729bf36c7df81
> seems to have fixed the issue that the other HBASE-19275 patches fixed, but
> when HBASE-19275 was applied to branch-1 it appears to have reverted this
> code:
> https://github.com/apache/hbase/commit/e8e4beacb039299a9e56cd355e3ada08784a7acb#diff-ae046012dd310fc116a729bf36c7df81
> meaning the tests always pass again.
> [~apurtell] [~xucang]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22390) backport HBASE-22190 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836888#comment-16836888
]
Hudson commented on HBASE-22390:
Results for branch branch-1.4
[build #787 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> backport HBASE-22190 to branch-1
>
>
> Key: HBASE-22390
> URL: https://issues.apache.org/jira/browse/HBASE-22390
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.4.10
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Fix For: 1.5.0, 1.4.11
>
>
> HBASE-22190 was only applied to branch-2+ while the bug exists in branch-1 as
> well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22190) SnapshotFileCache may fail to load the correct snapshot file list when there is an on-going snapshot operation
[
https://issues.apache.org/jira/browse/HBASE-22190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836889#comment-16836889
]
Hudson commented on HBASE-22190:
Results for branch branch-1.4
[build #787 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> SnapshotFileCache may fail to load the correct snapshot file list when there
> is an on-going snapshot operation
> --
>
> Key: HBASE-22190
> URL: https://issues.apache.org/jira/browse/HBASE-22190
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.1.5
>
>
> And it seems that it is not only a test issue, we do delete the files under
> the archive directory, which is incorrect.
> Need to find out why, this maybe a serious bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20724) Sometimes some compacted storefiles are still opened after region failover
[
https://issues.apache.org/jira/browse/HBASE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836887#comment-16836887
]
Hudson commented on HBASE-20724:
Results for branch branch-1.4
[build #787 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Sometimes some compacted storefiles are still opened after region failover
> --
>
> Key: HBASE-20724
> URL: https://issues.apache.org/jira/browse/HBASE-20724
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.3.0
>
> Attachments: HBASE-20724.branch-2.001.patch,
> HBASE-20724.master.001.patch, HBASE-20724.master.002.patch,
> HBASE-20724.master.003.patch, HBASE-20724.master.004.patch,
> HBASE-20724.master.005.patch, HBASE-20724.master.006.patch,
> HBASE-20724.master.007.patch, HBASE-20724.master.008.patch,
> HBASE-20724.master.009.patch, HBASE-20724.master.010.patch,
> HBASE-20724.master.011.patch, HBASE-20724.master.012.patch,
> HBASE-20724.master.013.patch, HBASE-20724.master.013.patch,
> HBASE-20724.master.014.patch
>
>
> It is important that compacted storefiles of a given compaction execution are
> wholly opened or archived to insure data consistency. ie a storefile
> containing delete tombstones can be archived while older storefiles
> containing cells that were supposed to be deleted are left unarchived thereby
> undeleting those cells.
> When a server fails compaction markers (in the wal edit) are used to
> determine which storefiles are compacted and should be excluded during region
> open (during failover). But the WALs containing compaction markers can be
> prematurely archived even though there are still compacted storefiles for
> that particular compaction event that hasn't been archived yet. Thus losing
> compaction information that needs to be replayed in the event of an RS crash.
> This is because hlog archiving logic only keeps track of flushed storefiles
> and not compacted ones.
> https://issues.apache.org/jira/browse/HBASE-20704?focusedCommentId=16507680=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16507680
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19275) TestSnapshotFileCache never worked properly
[
https://issues.apache.org/jira/browse/HBASE-19275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836891#comment-16836891
]
Hudson commented on HBASE-19275:
Results for branch branch-1.4
[build #787 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> TestSnapshotFileCache never worked properly
> ---
>
> Key: HBASE-19275
> URL: https://issues.apache.org/jira/browse/HBASE-19275
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-19275-branch-1.patch,
> HBASE-19275-master.001.patch, HBASE-19275-master.001.patch
>
>
> Error-prone noticed we were asking Iterables.contains() questions with the
> wrong type in TestSnapshotFileCache. I've attached a fixed version of the
> test. The results suggest the cache is not evicting entries properly.
> {noformat}
> java.lang.AssertionError: Cache found
> 'hdfs://localhost:52867/user/apurtell/test-data/8ce04c85-ce4b-4844-b454-5303482ade95/data/default/snapshot1/9e49edd0ab41657fb0c6ebb4d9dfad15/cf/f132e5b06f66443f8003363ed1535aac',
> but it shouldn't have.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.createAndTestSnapshot(TestSnapshotFileCache.java:260)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.createAndTestSnapshotV1(TestSnapshotFileCache.java:206)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.testReloadModifiedDirectory(TestSnapshotFileCache.java:102)
> {noformat}
> {noformat}
> java.lang.AssertionError: Cache found
> 'hdfs://localhost:52867/user/apurtell/test-data/8ce04c85-ce4b-4844-b454-5303482ade95/data/default/snapshot1a/2e81adb9212c98cff970eafa006fc40b/cf/a2ec478d850e4e348359699c53b732c4',
> but it shouldn't have.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.createAndTestSnapshot(TestSnapshotFileCache.java:260)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.createAndTestSnapshotV1(TestSnapshotFileCache.java:206)
> at
> org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache.testLoadAndDelete(TestSnapshotFileCache.java:88)
> {noformat}
> These changes are part of HBASE-19239
> I've disabled the offending test cases with @Ignore in that patch, but they
> should be reenabled and fixed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836886#comment-16836886
]
Hudson commented on HBASE-22330:
Results for branch branch-1.4
[build #787 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/787//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Backport HBASE-20724 (Sometimes some compacted storefiles are still opened
> after region failover) to branch-1
> -
>
> Key: HBASE-22330
> URL: https://issues.apache.org/jira/browse/HBASE-22330
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction, regionserver
>Affects Versions: 1.5.0, 1.4.9, 1.3.4
>Reporter: Andrew Purtell
>Assignee: Abhishek Singh Chouhan
>Priority: Major
> Fix For: 1.5.0, 1.3.5, 1.4.11
>
> Attachments: HBASE-22330-addendum.branch-1.patch,
> HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch,
> HBASE-22330.branch-1.3.001.patch
>
>
> There appears to be a race condition between close and split which when
> combined with a side effect of HBASE-20704, leads to the parent region store
> files getting archived and cleared while daughter regions still have
> references to those parent region store files.
> Here is the timeline of events observed for an affected region:
> # RS1 faces ZooKeeper connectivity issue for master node and starts shutting
> itself down. As part of this it starts to close the store and clean up the
> compacted files (File A)
> # Master starts bulk assigning regions and assign parent region to RS2
> # Region opens on RS2 and ends up opening compacted store file(s) (suspect
> this is due to HBASE-20724)
> # Now split happens and daughter regions open on RS2 and try to run a
> compaction as part of post open
> # Split request at this point is complete. However now archiving proceeds on
> RS1 and ends up archiving the store file that is referenced by the daughter.
> Compaction fails due to FileNotFoundException and all subsequent attempts to
> open the region will fail until manual resolution.
> We think having HBASE-20724 would help in such situations since we won't end
> up loading compacted store files in the first place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20724) Sometimes some compacted storefiles are still opened after region failover
[
https://issues.apache.org/jira/browse/HBASE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836881#comment-16836881
]
Hudson commented on HBASE-20724:
Results for branch branch-1.3
[build #761 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/761/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/761//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/761//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/761//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Sometimes some compacted storefiles are still opened after region failover
> --
>
> Key: HBASE-20724
> URL: https://issues.apache.org/jira/browse/HBASE-20724
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.3.0
>
> Attachments: HBASE-20724.branch-2.001.patch,
> HBASE-20724.master.001.patch, HBASE-20724.master.002.patch,
> HBASE-20724.master.003.patch, HBASE-20724.master.004.patch,
> HBASE-20724.master.005.patch, HBASE-20724.master.006.patch,
> HBASE-20724.master.007.patch, HBASE-20724.master.008.patch,
> HBASE-20724.master.009.patch, HBASE-20724.master.010.patch,
> HBASE-20724.master.011.patch, HBASE-20724.master.012.patch,
> HBASE-20724.master.013.patch, HBASE-20724.master.013.patch,
> HBASE-20724.master.014.patch
>
>
> It is important that compacted storefiles of a given compaction execution are
> wholly opened or archived to insure data consistency. ie a storefile
> containing delete tombstones can be archived while older storefiles
> containing cells that were supposed to be deleted are left unarchived thereby
> undeleting those cells.
> When a server fails compaction markers (in the wal edit) are used to
> determine which storefiles are compacted and should be excluded during region
> open (during failover). But the WALs containing compaction markers can be
> prematurely archived even though there are still compacted storefiles for
> that particular compaction event that hasn't been archived yet. Thus losing
> compaction information that needs to be replayed in the event of an RS crash.
> This is because hlog archiving logic only keeps track of flushed storefiles
> and not compacted ones.
> https://issues.apache.org/jira/browse/HBASE-20704?focusedCommentId=16507680=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16507680
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836880#comment-16836880
]
Hudson commented on HBASE-22330:
Results for branch branch-1.3
[build #761 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/761/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/761//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/761//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/761//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Backport HBASE-20724 (Sometimes some compacted storefiles are still opened
> after region failover) to branch-1
> -
>
> Key: HBASE-22330
> URL: https://issues.apache.org/jira/browse/HBASE-22330
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction, regionserver
>Affects Versions: 1.5.0, 1.4.9, 1.3.4
>Reporter: Andrew Purtell
>Assignee: Abhishek Singh Chouhan
>Priority: Major
> Fix For: 1.5.0, 1.3.5, 1.4.11
>
> Attachments: HBASE-22330-addendum.branch-1.patch,
> HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch,
> HBASE-22330.branch-1.3.001.patch
>
>
> There appears to be a race condition between close and split which when
> combined with a side effect of HBASE-20704, leads to the parent region store
> files getting archived and cleared while daughter regions still have
> references to those parent region store files.
> Here is the timeline of events observed for an affected region:
> # RS1 faces ZooKeeper connectivity issue for master node and starts shutting
> itself down. As part of this it starts to close the store and clean up the
> compacted files (File A)
> # Master starts bulk assigning regions and assign parent region to RS2
> # Region opens on RS2 and ends up opening compacted store file(s) (suspect
> this is due to HBASE-20724)
> # Now split happens and daughter regions open on RS2 and try to run a
> compaction as part of post open
> # Split request at this point is complete. However now archiving proceeds on
> RS1 and ends up archiving the store file that is referenced by the daughter.
> Compaction fails due to FileNotFoundException and all subsequent attempts to
> open the region will fail until manual resolution.
> We think having HBASE-20724 would help in such situations since we won't end
> up loading compacted store files in the first place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836874#comment-16836874
]
Xu Cang commented on HBASE-22274:
-
pushed to branch-1.4
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-branch-1.005.patch, HBASE-22274-master.001.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836868#comment-16836868
]
Xu Cang commented on HBASE-22274:
-
https://issues.apache.org/jira/browse/HBASE-18043 was the one implemented cell
size limitation.
It wasn't applied to branch-1.3
So I will not push my fix to branch-1.3
(for internal use we can directly apply the patch from branch-1)
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-branch-1.005.patch, HBASE-22274-master.001.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836860#comment-16836860
]
Xu Cang commented on HBASE-22274:
-
pushed patch to branch-1.
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-branch-1.005.patch, HBASE-22274-master.001.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836864#comment-16836864
]
Xu Cang commented on HBASE-22274:
-
pushed to master branch
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-branch-1.005.patch, HBASE-22274-master.001.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20724) Sometimes some compacted storefiles are still opened after region failover
[ https://issues.apache.org/jira/browse/HBASE-20724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836850#comment-16836850 ] Hudson commented on HBASE-20724: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #554 (See [https://builds.apache.org/job/HBase-1.3-IT/554/]) Addendum HBASE-22330 Backport HBASE-20724 (Sometimes some compacted (abhishek.chouhan: [https://github.com/apache/hbase/commit/10f00547627076d79d77cf58dd2deaece2287084]) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCleanupCompactedFileAfterFailover.java > Sometimes some compacted storefiles are still opened after region failover > -- > > Key: HBASE-20724 > URL: https://issues.apache.org/jira/browse/HBASE-20724 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Guanghao Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.3.0 > > Attachments: HBASE-20724.branch-2.001.patch, > HBASE-20724.master.001.patch, HBASE-20724.master.002.patch, > HBASE-20724.master.003.patch, HBASE-20724.master.004.patch, > HBASE-20724.master.005.patch, HBASE-20724.master.006.patch, > HBASE-20724.master.007.patch, HBASE-20724.master.008.patch, > HBASE-20724.master.009.patch, HBASE-20724.master.010.patch, > HBASE-20724.master.011.patch, HBASE-20724.master.012.patch, > HBASE-20724.master.013.patch, HBASE-20724.master.013.patch, > HBASE-20724.master.014.patch > > > It is important that compacted storefiles of a given compaction execution are > wholly opened or archived to insure data consistency. ie a storefile > containing delete tombstones can be archived while older storefiles > containing cells that were supposed to be deleted are left unarchived thereby > undeleting those cells. > When a server fails compaction markers (in the wal edit) are used to > determine which storefiles are compacted and should be excluded during region > open (during failover). But the WALs containing compaction markers can be > prematurely archived even though there are still compacted storefiles for > that particular compaction event that hasn't been archived yet. Thus losing > compaction information that needs to be replayed in the event of an RS crash. > This is because hlog archiving logic only keeps track of flushed storefiles > and not compacted ones. > https://issues.apache.org/jira/browse/HBASE-20704?focusedCommentId=16507680=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16507680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836849#comment-16836849 ] Hudson commented on HBASE-22330: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #554 (See [https://builds.apache.org/job/HBase-1.3-IT/554/]) Addendum HBASE-22330 Backport HBASE-20724 (Sometimes some compacted (abhishek.chouhan: [https://github.com/apache/hbase/commit/10f00547627076d79d77cf58dd2deaece2287084]) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCleanupCompactedFileAfterFailover.java > Backport HBASE-20724 (Sometimes some compacted storefiles are still opened > after region failover) to branch-1 > - > > Key: HBASE-22330 > URL: https://issues.apache.org/jira/browse/HBASE-22330 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 1.5.0, 1.4.9, 1.3.4 >Reporter: Andrew Purtell >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.5.0, 1.3.5, 1.4.11 > > Attachments: HBASE-22330-addendum.branch-1.patch, > HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch, > HBASE-22330.branch-1.3.001.patch > > > There appears to be a race condition between close and split which when > combined with a side effect of HBASE-20704, leads to the parent region store > files getting archived and cleared while daughter regions still have > references to those parent region store files. > Here is the timeline of events observed for an affected region: > # RS1 faces ZooKeeper connectivity issue for master node and starts shutting > itself down. As part of this it starts to close the store and clean up the > compacted files (File A) > # Master starts bulk assigning regions and assign parent region to RS2 > # Region opens on RS2 and ends up opening compacted store file(s) (suspect > this is due to HBASE-20724) > # Now split happens and daughter regions open on RS2 and try to run a > compaction as part of post open > # Split request at this point is complete. However now archiving proceeds on > RS1 and ends up archiving the store file that is referenced by the daughter. > Compaction fails due to FileNotFoundException and all subsequent attempts to > open the region will fail until manual resolution. > We think having HBASE-20724 would help in such situations since we won't end > up loading compacted store files in the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22391) Fix flaky tests from TestFromClientSide
[
https://issues.apache.org/jira/browse/HBASE-22391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836838#comment-16836838
]
HBase QA commented on HBASE-22391:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
24s{color} | {color:red} hbase-server: The patch generated 2 new + 221
unchanged - 1 fixed = 223 total (was 222) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
56s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 43s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}130m
39s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HBASE-Build/281/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22391 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12968338/HBASE-22391-branch-1.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 8e8ba9a61157
[jira] [Commented] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836835#comment-16836835
]
HBase QA commented on HBASE-22274:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
47s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
45s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 42s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}129m
45s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HBASE-Build/280/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22274 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12968337/HBASE-22274-branch-1.005.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 8abef0aa581d 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb
13 15:00:41 UTC
[jira] [Commented] (HBASE-18373) fstat unimplemented error on Rasbian Jessie
[ https://issues.apache.org/jira/browse/HBASE-18373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836830#comment-16836830 ] Toshihiro Suzuki commented on HBASE-18373: -- Looks this issue is not HBase side problem, but an issue with JRuby and Oracle's JVM. See the following for details: https://github.com/jruby/jruby/issues/2913 https://github.com/elastic/logstash/issues/3127#issuecomment-101068714 https://github.com/elastic/logstash/issues/2962 Closing as not a problem. > fstat unimplemented error on Rasbian Jessie > --- > > Key: HBASE-18373 > URL: https://issues.apache.org/jira/browse/HBASE-18373 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0-alpha-1 > Environment: Raspberry Pi 3 > Rasbian Jessie (latest version) >Reporter: Nathan Green >Priority: Blocker > > I have a Hbase distributed cluster setup on my Raspberry PI cluster (1 x > Namenode & 4 x Datanode) > The stable version of Hbase shell works fine... unfortunately I had hadoop > issues (mismatch of jar files gr) so I have installed latest Hbase & > installed matching version of Hadoop (2.7.1). > HBase itself is up and running and all good. Hbase shell however gives this > error: > pi@pidoop1:/opt/hbase/lib/ruby/irb $ hbase shell > 2017-07-13 12:25:11,722 WARN [main] util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/hbase/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > HBase Shell > Use "help" to get list of supported commands. > Use "exit" to quit this interactive shell. > Version 2.0.0-alpha-1, rc830a0f47f58d4892dd3300032c8244d6278aecc, Wed Jun 7 > 22:05:17 PDT 2017 > Took 0.0300 seconds > NotImplementedError: fstat unimplemented unsupported or native support failed > to load; see http://wiki.jruby.org/Native-Libraries > initialize at org/jruby/RubyIO.java:1013 > open at org/jruby/RubyIO.java:1154 > initialize at > uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/input-method.rb:141 > initialize at > uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/context.rb:70 > initialize at > uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:426 > initialize at /opt/hbase/lib/ruby/irb/hirb.rb:43 >start at /opt/hbase/bin/hirb.rb:194 >at /opt/hbase/bin/hirb.rb:206 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] infraio commented on a change in pull request #221: HBASE-22365 Region may be opened on two RegionServers
infraio commented on a change in pull request #221: HBASE-22365 Region may be
opened on two RegionServers
URL: https://github.com/apache/hbase/pull/221#discussion_r282722269
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -1575,18 +1562,36 @@ void regionClosing(RegionStateNode regionNode) throws
IOException {
metrics.incrementOperationCounter();
}
- // should be called within the synchronized block of RegionStateNode
- // The parameter 'normally' means whether we are closed cleanly, if it is
true, then it means that
- // we are closed due to a RS crash.
- void regionClosed(RegionStateNode regionNode, boolean normally) throws
IOException {
-RegionState.State state = regionNode.getState();
+ // for open and close, they will first be persist to the procedure store in
+ // RegionRemoteProcedureBase. So here we will first change the in memory
state as it is considered
+ // as succeeded if the persistence to procedure store is succeeded, and then
when the
+ // RegionRemoteProcedureBase is woken up, we will persist the
RegionStateNode to hbase:meta.
+
+ // should be called under the RegionStateNode lock
+ void regionOpenedNoPersistence(RegionStateNode regionNode) throws
IOException {
+regionNode.transitionState(State.OPEN, STATES_EXPECTED_ON_OPEN);
+RegionInfo regionInfo = regionNode.getRegionInfo();
+regionStates.addRegionToServer(regionNode);
+regionStates.removeFromFailedOpen(regionInfo);
+ }
+
+ // should be called under the RegionStateNode lock
+ void regionClosedNoPersistence(RegionStateNode regionNode) throws
IOException {
Review comment:
ditto.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] infraio commented on a change in pull request #221: HBASE-22365 Region may be opened on two RegionServers
infraio commented on a change in pull request #221: HBASE-22365 Region may be
opened on two RegionServers
URL: https://github.com/apache/hbase/pull/221#discussion_r282724012
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionRemoteProcedureBase.java
##
@@ -175,7 +176,7 @@ void reportTransition(MasterProcedureEnv env,
RegionStateNode regionNode, Server
throw new UnexpectedStateException("Received report from " + serverName
+ ", expected " +
targetServer + ", " + regionNode + ", proc=" + this);
}
-reportTransition(regionNode, transitionCode, seqId);
+reportTransition(env, regionNode, transitionCode, seqId);
// this state means we have received the report from RS, does not mean the
result is fine, as we
Review comment:
Here we will first change the memory state and then persist to procedure
store. If persist to procedure failed, need to change the memory state back?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (HBASE-18373) fstat unimplemented error on Rasbian Jessie
[ https://issues.apache.org/jira/browse/HBASE-18373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki resolved HBASE-18373. -- Resolution: Not A Problem > fstat unimplemented error on Rasbian Jessie > --- > > Key: HBASE-18373 > URL: https://issues.apache.org/jira/browse/HBASE-18373 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0-alpha-1 > Environment: Raspberry Pi 3 > Rasbian Jessie (latest version) >Reporter: Nathan Green >Priority: Blocker > > I have a Hbase distributed cluster setup on my Raspberry PI cluster (1 x > Namenode & 4 x Datanode) > The stable version of Hbase shell works fine... unfortunately I had hadoop > issues (mismatch of jar files gr) so I have installed latest Hbase & > installed matching version of Hadoop (2.7.1). > HBase itself is up and running and all good. Hbase shell however gives this > error: > pi@pidoop1:/opt/hbase/lib/ruby/irb $ hbase shell > 2017-07-13 12:25:11,722 WARN [main] util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/hbase/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > HBase Shell > Use "help" to get list of supported commands. > Use "exit" to quit this interactive shell. > Version 2.0.0-alpha-1, rc830a0f47f58d4892dd3300032c8244d6278aecc, Wed Jun 7 > 22:05:17 PDT 2017 > Took 0.0300 seconds > NotImplementedError: fstat unimplemented unsupported or native support failed > to load; see http://wiki.jruby.org/Native-Libraries > initialize at org/jruby/RubyIO.java:1013 > open at org/jruby/RubyIO.java:1154 > initialize at > uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/input-method.rb:141 > initialize at > uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/context.rb:70 > initialize at > uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:426 > initialize at /opt/hbase/lib/ruby/irb/hirb.rb:43 >start at /opt/hbase/bin/hirb.rb:194 >at /opt/hbase/bin/hirb.rb:206 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] infraio commented on a change in pull request #221: HBASE-22365 Region may be opened on two RegionServers
infraio commented on a change in pull request #221: HBASE-22365 Region may be
opened on two RegionServers
URL: https://github.com/apache/hbase/pull/221#discussion_r282722122
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -1575,18 +1562,36 @@ void regionClosing(RegionStateNode regionNode) throws
IOException {
metrics.incrementOperationCounter();
}
- // should be called within the synchronized block of RegionStateNode
- // The parameter 'normally' means whether we are closed cleanly, if it is
true, then it means that
- // we are closed due to a RS crash.
- void regionClosed(RegionStateNode regionNode, boolean normally) throws
IOException {
-RegionState.State state = regionNode.getState();
+ // for open and close, they will first be persist to the procedure store in
+ // RegionRemoteProcedureBase. So here we will first change the in memory
state as it is considered
+ // as succeeded if the persistence to procedure store is succeeded, and then
when the
+ // RegionRemoteProcedureBase is woken up, we will persist the
RegionStateNode to hbase:meta.
+
+ // should be called under the RegionStateNode lock
+ void regionOpenedNoPersistence(RegionStateNode regionNode) throws
IOException {
Review comment:
regionOpenedWithoutPersistToMeta or regionOpenedNoPersistToMeta? It will be
better to point "persist to meta".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-21991) Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
[ https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836826#comment-16836826 ] Sean Busbey commented on HBASE-21991: - addendum looks good to me. I posted to dev@ here: https://s.apache.org/oQVF If someone has a pressing need for this in a branch for an RC please let me know. Otherwise I'll wait a day or so for feedback on the dev posting and then take care of pushing the addendum and release noting things. > Fix MetaMetrics issues - [Race condition, Faulty remove logic], few > improvements > > > Key: HBASE-21991 > URL: https://issues.apache.org/jira/browse/HBASE-21991 > Project: HBase > Issue Type: Bug > Components: Coprocessors, metrics >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0 > > Attachments: hbase-21991.addendum.patch, > hbase-21991.branch-1.001.patch, hbase-21991.branch-1.002.patch, > hbase-21991.master.001.patch, hbase-21991.master.002.patch, > hbase-21991.master.003.patch, hbase-21991.master.004.patch, > hbase-21991.master.005.patch, hbase-21991.master.006.patch > > > Here is a list of the issues related to the MetaMetrics implementation: > +*Bugs*+: > # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*: > Under certain conditions, we might end up storing/exposing all the meters > rather than top-k-ish > # MetaMetrics can throw NPE resulting in aborting of the RS because of a > *Race Condition*. > +*Improvements*+: > # With high number of regions in the cluster, exposure of metrics for each > region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of > regions. It's better to use *lossy counting to maintain top-k for region > metrics* as well. > # As the lossy meters do not represent actual counts, I think, it'll be > better to *rename the meters to include "lossy" in the name*. It would be > more informative while monitoring the metrics and there would be less > confusion regarding actual counts to lossy counts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] Apache-HBase commented on issue #120: HBASE-20494 Updated the version of metrics-core to 3.2.6
Apache-HBase commented on issue #120: HBASE-20494 Updated the version of metrics-core to 3.2.6 URL: https://github.com/apache/hbase/pull/120#issuecomment-491126822 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 36 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -0 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 | mvninstall | 254 | master passed | | +1 | compile | 159 | master passed | | +1 | shadedjars | 258 | branch has no errors when building our shaded downstream artifacts. | | +1 | javadoc | 157 | master passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 237 | the patch passed | | +1 | compile | 159 | the patch passed | | +1 | javac | 159 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedjars | 258 | patch has no errors when building our shaded downstream artifacts. | | +1 | hadoopcheck | 505 | Patch does not cause any errors with Hadoop 2.7.4 or 3.0.0. | | +1 | javadoc | 157 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 18973 | root in the patch passed. | | +1 | asflicense | 39 | The patch does not generate ASF License warnings. | | | | 21260 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-120/13/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/120 | | Optional Tests | dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile | | uname | Linux 438ffa5e40b8 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | master / 4d64dd2e82 | | maven | version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-120/13/testReport/ | | Max. process+thread count | 4994 (vs. ulimit of 1) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-120/13/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22365) Region may be opened on two RegionServers
[ https://issues.apache.org/jira/browse/HBASE-22365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836822#comment-16836822 ] Guanghao Zhang commented on HBASE-22365: Try the patch on our internal branch and the ITBLL which have 3 billions rows passed. :) > Region may be opened on two RegionServers > - > > Key: HBASE-22365 > URL: https://issues.apache.org/jira/browse/HBASE-22365 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0, 2.3.0 >Reporter: Guanghao Zhang >Assignee: Duo Zhang >Priority: Blocker > Attachments: HBASE-22365-UT.patch > > > Found this problem when run ITBLL with our internal branch which is based on > branch-2.2. So mark this as a blocker for 2.2.0. A region > 7ebdca9cd09e26074749b546586e2156 is moved from RS-st99 to RS-st98 and the > TRSP succeed. Meanwhile, RS-st99 crashed and schedule a new SCP for RS-st99. > So SCP initialized subprocedures for 7ebdca9cd09e26074749b546586e2156, too. > Then the 7ebdca9cd09e26074749b546586e2156 was assigned to two RegionServers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21536) Fix completebulkload usage instructions
[
https://issues.apache.org/jira/browse/HBASE-21536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836821#comment-16836821
]
HBase QA commented on HBASE-21536:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
1s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 11m
37s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
2s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 7m
22s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
20s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 3s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}210m
15s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} |
[jira] [Commented] (HBASE-22254) refactor and improve decommissioning logic
[ https://issues.apache.org/jira/browse/HBASE-22254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836817#comment-16836817 ] Sergey Shelukhin commented on HBASE-22254: -- Btw, these APIs were added in HBASE-17370 but the book wasn't updated, it still relies on znode creation there... I wonder if the book should be updated w/this patch > refactor and improve decommissioning logic > -- > > Key: HBASE-22254 > URL: https://issues.apache.org/jira/browse/HBASE-22254 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-22254.01.patch, HBASE-22254.02.patch, > HBASE-22254.patch > > > Making some changes needed to support better decommissioning on large > clusters and with container mode; to test those and add clarify I moved parts > of decommissioning logic from HMaster, Draining tracker, and ServerManager > into a separate class. > Features added/improvements: > 1) More resilient off-loading; right now off-loading fails for a subset of > regions in case of a single region failure; is never done on master restart, > etc. > 2) Option to kill RS after off-loading (good for container mode HBase, e.g. > on YARN). > 3) Option to specify machine names only to decommission, for the API to be > usable for an external system that doesn't care about HBase server names, or > e.g. multiple RS in containers on the same node. > 4) Option to replace existing decommissioning list instead of adding to it > (the same; to avoid additionally remembering what was previously sent to > HBase). > 5) Tests, comments ;) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22254) refactor and improve decommissioning logic
[ https://issues.apache.org/jira/browse/HBASE-22254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-22254: - Status: Patch Available (was: Open) Fixed the test, also made the client changes to make new API features usable not just directly via a request. > refactor and improve decommissioning logic > -- > > Key: HBASE-22254 > URL: https://issues.apache.org/jira/browse/HBASE-22254 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-22254.01.patch, HBASE-22254.02.patch, > HBASE-22254.patch > > > Making some changes needed to support better decommissioning on large > clusters and with container mode; to test those and add clarify I moved parts > of decommissioning logic from HMaster, Draining tracker, and ServerManager > into a separate class. > Features added/improvements: > 1) More resilient off-loading; right now off-loading fails for a subset of > regions in case of a single region failure; is never done on master restart, > etc. > 2) Option to kill RS after off-loading (good for container mode HBase, e.g. > on YARN). > 3) Option to specify machine names only to decommission, for the API to be > usable for an external system that doesn't care about HBase server names, or > e.g. multiple RS in containers on the same node. > 4) Option to replace existing decommissioning list instead of adding to it > (the same; to avoid additionally remembering what was previously sent to > HBase). > 5) Tests, comments ;) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22254) refactor and improve decommissioning logic
[ https://issues.apache.org/jira/browse/HBASE-22254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-22254: - Attachment: HBASE-22254.02.patch > refactor and improve decommissioning logic > -- > > Key: HBASE-22254 > URL: https://issues.apache.org/jira/browse/HBASE-22254 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-22254.01.patch, HBASE-22254.02.patch, > HBASE-22254.patch > > > Making some changes needed to support better decommissioning on large > clusters and with container mode; to test those and add clarify I moved parts > of decommissioning logic from HMaster, Draining tracker, and ServerManager > into a separate class. > Features added/improvements: > 1) More resilient off-loading; right now off-loading fails for a subset of > regions in case of a single region failure; is never done on master restart, > etc. > 2) Option to kill RS after off-loading (good for container mode HBase, e.g. > on YARN). > 3) Option to specify machine names only to decommission, for the API to be > usable for an external system that doesn't care about HBase server names, or > e.g. multiple RS in containers on the same node. > 4) Option to replace existing decommissioning list instead of adding to it > (the same; to avoid additionally remembering what was previously sent to > HBase). > 5) Tests, comments ;) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22389) Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
[ https://issues.apache.org/jira/browse/HBASE-22389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836807#comment-16836807 ] Zach York commented on HBASE-22389: --- Thanks for the reviews [~apurtell]! > Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1 > -- > > Key: HBASE-22389 > URL: https://issues.apache.org/jira/browse/HBASE-22389 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 1.5.0, 1.4.11 > > > This commit seems broken on branch-1. > https://github.com/apache/hbase/commit/d80d3fa45487fedf203c7144e8267753e4d53529#diff-ae046012dd310fc116a729bf36c7df81 > seems to have fixed the issue that the other HBASE-19275 patches fixed, but > when HBASE-19275 was applied to branch-1 it appears to have reverted this > code: > https://github.com/apache/hbase/commit/e8e4beacb039299a9e56cd355e3ada08784a7acb#diff-ae046012dd310fc116a729bf36c7df81 > meaning the tests always pass again. > [~apurtell] [~xucang] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] Apache-HBase commented on issue #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr…
Apache-HBase commented on issue #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr… URL: https://github.com/apache/hbase/pull/229#issuecomment-491112303 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 9 | https://github.com/apache/hbase/pull/229 does not apply to branch-1. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/229 | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-229/2/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-22389) Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
[ https://issues.apache.org/jira/browse/HBASE-22389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York resolved HBASE-22389. --- Resolution: Fixed Fix Version/s: 1.4.11 1.5.0 > Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1 > -- > > Key: HBASE-22389 > URL: https://issues.apache.org/jira/browse/HBASE-22389 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 1.5.0, 1.4.11 > > > This commit seems broken on branch-1. > https://github.com/apache/hbase/commit/d80d3fa45487fedf203c7144e8267753e4d53529#diff-ae046012dd310fc116a729bf36c7df81 > seems to have fixed the issue that the other HBASE-19275 patches fixed, but > when HBASE-19275 was applied to branch-1 it appears to have reverted this > code: > https://github.com/apache/hbase/commit/e8e4beacb039299a9e56cd355e3ada08784a7acb#diff-ae046012dd310fc116a729bf36c7df81 > meaning the tests always pass again. > [~apurtell] [~xucang] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] z-york commented on issue #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr…
z-york commented on issue #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr… URL: https://github.com/apache/hbase/pull/229#issuecomment-491110739 Reran TestSnapshotFileCache after #230 and confirmed it passes now. Fixed checkstyle. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] z-york merged pull request #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr…
z-york merged pull request #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr… URL: https://github.com/apache/hbase/pull/229 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] z-york commented on issue #230: HBASE-22390 Backport HBASE-22190 to branch-1
z-york commented on issue #230: HBASE-22390 Backport HBASE-22190 to branch-1 URL: https://github.com/apache/hbase/pull/230#issuecomment-491108669 pushed to branch-1 and branch-1.4. closing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-22391) Fix flaky tests from TestFromClientSide
[ https://issues.apache.org/jira/browse/HBASE-22391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-22391: Issue Type: Bug (was: New Feature) > Fix flaky tests from TestFromClientSide > --- > > Key: HBASE-22391 > URL: https://issues.apache.org/jira/browse/HBASE-22391 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.0.5, 1.5.1 >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.3.0, 2.1.5, 2.2.1, 1.3.5, 1.4.11 > > Attachments: HBASE-22391-branch-1.001.patch > > > tests in TestFromClientSide.java in general are flaky due to the reason that > after createTable, they did not wait for table to be ready before adding data > into table. > Found this issue when working on HBASE-22274 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] z-york closed pull request #230: HBASE-22390 Backport HBASE-22190 to branch-1
z-york closed pull request #230: HBASE-22390 Backport HBASE-22190 to branch-1 URL: https://github.com/apache/hbase/pull/230 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-22391) Fix flaky tests from TestFromClientSide
[ https://issues.apache.org/jira/browse/HBASE-22391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-22391: Attachment: HBASE-22391-branch-1.001.patch Status: Patch Available (was: Open) > Fix flaky tests from TestFromClientSide > --- > > Key: HBASE-22391 > URL: https://issues.apache.org/jira/browse/HBASE-22391 > Project: HBase > Issue Type: New Feature > Components: test >Affects Versions: 2.0.5, 3.0.0, 1.5.1 >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.3.0, 2.1.5, 2.2.1, 1.3.5, 1.4.11 > > Attachments: HBASE-22391-branch-1.001.patch > > > tests in TestFromClientSide.java in general are flaky due to the reason that > after createTable, they did not wait for table to be ready before adding data > into table. > Found this issue when working on HBASE-22274 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22390) backport HBASE-22190 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-22390: -- Resolution: Fixed Fix Version/s: 1.4.11 1.5.0 Status: Resolved (was: Patch Available) > backport HBASE-22190 to branch-1 > > > Key: HBASE-22390 > URL: https://issues.apache.org/jira/browse/HBASE-22390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0, 1.4.10 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 1.5.0, 1.4.11 > > > HBASE-22190 was only applied to branch-2+ while the bug exists in branch-1 as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated HBASE-22274:
Attachment: HBASE-22274-branch-1.005.patch
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-branch-1.005.patch, HBASE-22274-master.001.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17370) Fix or provide shell scripts to drain and decommission region server
[ https://issues.apache.org/jira/browse/HBASE-17370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836783#comment-16836783 ] Sergey Shelukhin commented on HBASE-17370: -- Should the book be updated for this? Looks like it still suggests creating znodes to decommission region servers. > Fix or provide shell scripts to drain and decommission region server > > > Key: HBASE-17370 > URL: https://issues.apache.org/jira/browse/HBASE-17370 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Nihal Jain >Priority: Major > Labels: operability > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-17370.branch-2.001.patch, > HBASE-17370.master.001.patch, HBASE-17370.master.002.patch > > > 1. Update the existing shell scripts to use the new drain related API. > 2 Or provide new shell scripts. > 3. Provide a 'decommission' shell tool that puts the server in drain mode and > offload the server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] z-york commented on issue #230: HBASE-22390 Backport HBASE-22190 to branch-1
z-york commented on issue #230: HBASE-22390 Backport HBASE-22190 to branch-1 URL: https://github.com/apache/hbase/pull/230#issuecomment-491104124 all these tests pass locally. Fixed checkstyle as well. Committing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836780#comment-16836780
]
Andrew Purtell commented on HBASE-22274:
bq. Reading code in this class "RSRpcServices", I can see the pattern people
followed before was : we do not log client originated error or informational
messages by default.
I accept this argument, thank you, sounds good to me.
+1, with the test changed moved out
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-master.001.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836777#comment-16836777
]
Xu Cang edited comment on HBASE-22274 at 5/9/19 11:30 PM:
--
When cell size exceeds the limit, we always throw exception like this
{code:java}
throw new DoNotRetryIOException(msg);{code}
And this will be highly likely end up in client's log.
Reading code in this class "RSRpcServices", I can see the pattern people
followed before was : we do not log client originated error or informational
messages by default. Such as this one
{code:java}
if (LOG.isDebugEnabled()) {
LOG.debug(
"Server shutting down and client tried to access missing scanner " +
scannerName);
}{code}
So, I am OK with keeping the current logging strategy for cell limit one. As
long as server rejects it, client will handle this case from his end. Open to
discuss more about this if you think otherwise, [~apurtell]
was (Author: xucang):
When cell size exceeds the limit, we always throw exception like this
{code:java}
throw new DoNotRetryIOException(msg);{code}
And this will be highly likely end up in client's log.
Reading doe in this class "RSRpcServices", I can see the pattern people
followed before was : we do not log client originated error or informational
messages by default. Such as this one
{code:java}
if (LOG.isDebugEnabled()) {
LOG.debug(
"Server shutting down and client tried to access missing scanner " +
scannerName);
}{code}
So, I am OK with keeping the current logging strategy for cell limit one. As
long as server rejects it, client will handle this case from his end. Open to
discuss more about this if you think otherwise, [~apurtell]
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-master.001.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836777#comment-16836777
]
Xu Cang commented on HBASE-22274:
-
When cell size exceeds the limit, we always throw exception like this
{code:java}
throw new DoNotRetryIOException(msg);{code}
And this will be highly likely end up in client's log.
Reading doe in this class "RSRpcServices", I can see the pattern people
followed before was : we do not log client originated error or informational
messages by default. Such as this one
{code:java}
if (LOG.isDebugEnabled()) {
LOG.debug(
"Server shutting down and client tried to access missing scanner " +
scannerName);
}{code}
So, I am OK with keeping the current logging strategy for cell limit one. As
long as server rejects it, client will handle this case from his end. Open to
discuss more about this if you think otherwise, [~apurtell]
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-master.001.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-22330: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Backport HBASE-20724 (Sometimes some compacted storefiles are still opened > after region failover) to branch-1 > - > > Key: HBASE-22330 > URL: https://issues.apache.org/jira/browse/HBASE-22330 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 1.5.0, 1.4.9, 1.3.4 >Reporter: Andrew Purtell >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.5.0, 1.3.5, 1.4.11 > > Attachments: HBASE-22330-addendum.branch-1.patch, > HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch, > HBASE-22330.branch-1.3.001.patch > > > There appears to be a race condition between close and split which when > combined with a side effect of HBASE-20704, leads to the parent region store > files getting archived and cleared while daughter regions still have > references to those parent region store files. > Here is the timeline of events observed for an affected region: > # RS1 faces ZooKeeper connectivity issue for master node and starts shutting > itself down. As part of this it starts to close the store and clean up the > compacted files (File A) > # Master starts bulk assigning regions and assign parent region to RS2 > # Region opens on RS2 and ends up opening compacted store file(s) (suspect > this is due to HBASE-20724) > # Now split happens and daughter regions open on RS2 and try to run a > compaction as part of post open > # Split request at this point is complete. However now archiving proceeds on > RS1 and ends up archiving the store file that is referenced by the daughter. > Compaction fails due to FileNotFoundException and all subsequent attempts to > open the region will fail until manual resolution. > We think having HBASE-20724 would help in such situations since we won't end > up loading compacted store files in the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22391) Fix flaky tests from TestFromClientSide
Xu Cang created HBASE-22391: --- Summary: Fix flaky tests from TestFromClientSide Key: HBASE-22391 URL: https://issues.apache.org/jira/browse/HBASE-22391 Project: HBase Issue Type: New Feature Components: test Affects Versions: 2.0.5, 3.0.0, 1.5.1 Reporter: Xu Cang tests in TestFromClientSide.java in general are flaky due to the reason that after createTable, they did not wait for table to be ready before adding data into table. Found this issue when working on HBASE-22274 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-22391) Fix flaky tests from TestFromClientSide
[ https://issues.apache.org/jira/browse/HBASE-22391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-22391: -- Assignee: Xu Cang > Fix flaky tests from TestFromClientSide > --- > > Key: HBASE-22391 > URL: https://issues.apache.org/jira/browse/HBASE-22391 > Project: HBase > Issue Type: New Feature > Components: test >Affects Versions: 3.0.0, 2.0.5, 1.5.1 >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.3.0, 2.1.5, 2.2.1, 1.3.5, 1.4.11 > > > tests in TestFromClientSide.java in general are flaky due to the reason that > after createTable, they did not wait for table to be ready before adding data > into table. > Found this issue when working on HBASE-22274 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22391) Fix flaky tests from TestFromClientSide
[ https://issues.apache.org/jira/browse/HBASE-22391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-22391: --- Fix Version/s: 1.4.11 1.3.5 2.2.1 2.1.5 2.3.0 1.5.0 3.0.0 > Fix flaky tests from TestFromClientSide > --- > > Key: HBASE-22391 > URL: https://issues.apache.org/jira/browse/HBASE-22391 > Project: HBase > Issue Type: New Feature > Components: test >Affects Versions: 3.0.0, 2.0.5, 1.5.1 >Reporter: Xu Cang >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.3.0, 2.1.5, 2.2.1, 1.3.5, 1.4.11 > > > tests in TestFromClientSide.java in general are flaky due to the reason that > after createTable, they did not wait for table to be ready before adding data > into table. > Found this issue when working on HBASE-22274 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836766#comment-16836766
]
Xu Cang commented on HBASE-22274:
-
Moving unit test fix to https://issues.apache.org/jira/browse/HBASE-22391
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-master.001.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836764#comment-16836764 ] Abhishek Singh Chouhan commented on HBASE-22330: Pushed to relevant branches. Thanks [~xucang] [~apurtell] > Backport HBASE-20724 (Sometimes some compacted storefiles are still opened > after region failover) to branch-1 > - > > Key: HBASE-22330 > URL: https://issues.apache.org/jira/browse/HBASE-22330 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 1.5.0, 1.4.9, 1.3.4 >Reporter: Andrew Purtell >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.5.0, 1.3.5, 1.4.11 > > Attachments: HBASE-22330-addendum.branch-1.patch, > HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch, > HBASE-22330.branch-1.3.001.patch > > > There appears to be a race condition between close and split which when > combined with a side effect of HBASE-20704, leads to the parent region store > files getting archived and cleared while daughter regions still have > references to those parent region store files. > Here is the timeline of events observed for an affected region: > # RS1 faces ZooKeeper connectivity issue for master node and starts shutting > itself down. As part of this it starts to close the store and clean up the > compacted files (File A) > # Master starts bulk assigning regions and assign parent region to RS2 > # Region opens on RS2 and ends up opening compacted store file(s) (suspect > this is due to HBASE-20724) > # Now split happens and daughter regions open on RS2 and try to run a > compaction as part of post open > # Split request at this point is complete. However now archiving proceeds on > RS1 and ends up archiving the store file that is referenced by the daughter. > Compaction fails due to FileNotFoundException and all subsequent attempts to > open the region will fail until manual resolution. > We think having HBASE-20724 would help in such situations since we won't end > up loading compacted store files in the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] Apache-HBase commented on issue #230: HBASE-22390 Backport HBASE-22190 to branch-1
Apache-HBase commented on issue #230: HBASE-22390 Backport HBASE-22190 to branch-1 URL: https://github.com/apache/hbase/pull/230#issuecomment-491093578 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1278 | Docker mode activated. | ||| _ Prechecks _ | | 0 | findbugs | 0 | Findbugs executables are not available. | | +1 | hbaseanti | 0 | Patch does not have any anti-patterns. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -0 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | +1 | mvninstall | 150 | branch-1 passed | | +1 | compile | 38 | branch-1 passed with JDK v1.8.0_212 | | +1 | compile | 48 | branch-1 passed with JDK v1.7.0_222 | | +1 | checkstyle | 80 | branch-1 passed | | +1 | shadedjars | 159 | branch has no errors when building our shaded downstream artifacts. | | +1 | javadoc | 57 | branch-1 passed with JDK v1.8.0_212 | | +1 | javadoc | 62 | branch-1 passed with JDK v1.7.0_222 | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 151 | the patch passed | | +1 | compile | 62 | the patch passed with JDK v1.8.0_212 | | +1 | javac | 62 | the patch passed | | +1 | compile | 50 | the patch passed with JDK v1.7.0_222 | | +1 | javac | 50 | the patch passed | | -1 | checkstyle | 79 | hbase-server: The patch generated 4 new + 2 unchanged - 2 fixed = 6 total (was 4) | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedjars | 155 | patch has no errors when building our shaded downstream artifacts. | | +1 | hadoopcheck | 109 | Patch does not cause any errors with Hadoop 2.7.4. | | +1 | javadoc | 28 | the patch passed with JDK v1.8.0_212 | | +1 | javadoc | 38 | the patch passed with JDK v1.7.0_222 | ||| _ Other Tests _ | | -1 | unit | 8889 | hbase-server in the patch failed. | | +1 | asflicense | 87 | The patch does not generate ASF License warnings. | | | | 11603 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.regionserver.wal.TestLogRolling | | | hadoop.hbase.regionserver.TestPerColumnFamilyFlush | | | hadoop.hbase.regionserver.TestEncryptionKeyRotation | | | hadoop.hbase.replication.TestReplicationSmallTests | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-230/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/230 | | Optional Tests | dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux b54248fbc097 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | branch-1 / 56548fc | | maven | version: Apache Maven 3.0.5 | | Default Java | 1.7.0_222 | | Multi-JDK versions | /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_212 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_222 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-230/1/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-230/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-230/1/testReport/ | | Max. process+thread count | 3812 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-230/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836731#comment-16836731
]
Andrew Purtell commented on HBASE-22274:
bq. There are 88 'createTable' calls in TestFromClientSide.java none of them
calls 'waitTableAvailable'. It's incorrect to do 'Put' before table is
available in my opinion. That's why it's causing flaky testing result.
Let's break this part of the patch out into a separate JIRA so we have a
clearer history for the test fix. We can ignore the TestFromClientSide result
in the context of this issue.
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-master.001.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836729#comment-16836729
]
Andrew Purtell edited comment on HBASE-22274 at 5/9/19 9:49 PM:
v4 patch lgtm
One minor question, on this change:
{code}
+int newCellSize = CellUtil.estimatedSerializedSizeOf(newCell);
+if (newCellSize > this.maxCellSize) {
+ String msg = "Cell with size " + newCellSize + " exceeds
limit of " +
+ this.maxCellSize + " bytes";
+ if (LOG.isDebugEnabled()) {
+LOG.debug(msg);
+ }
+ throw new DoNotRetryIOException(msg);
+}
{code}
Should we be logging this at a higher log level, like INFO, or even WARN?
Whatever we decide here should be consistent with other warnings issued at the
other sites where we do this size limit test. (I.e. if DEBUG there and we
decide WARN, update to patch both warnings to WARN, or vice versa.)
was (Author: apurtell):
v4 patch lgtm
The test issue is clearer now.
One minor question, on this change:
{code}
+int newCellSize = CellUtil.estimatedSerializedSizeOf(newCell);
+if (newCellSize > this.maxCellSize) {
+ String msg = "Cell with size " + newCellSize + " exceeds
limit of " +
+ this.maxCellSize + " bytes";
+ if (LOG.isDebugEnabled()) {
+LOG.debug(msg);
+ }
+ throw new DoNotRetryIOException(msg);
+}
{code}
Should we be logging this at a higher log level, like INFO, or even WARN?
Whatever we decide here should be consistent with other warnings issued at the
other sites where we do this size limit test. (I.e. if DEBUG there and we
decide WARN, update to patch both warnings to WARN, or vice versa.)
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-master.001.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836729#comment-16836729
]
Andrew Purtell commented on HBASE-22274:
v4 patch lgtm
The test issue is clearer now.
One minor question, on this change:
{code}
+int newCellSize = CellUtil.estimatedSerializedSizeOf(newCell);
+if (newCellSize > this.maxCellSize) {
+ String msg = "Cell with size " + newCellSize + " exceeds
limit of " +
+ this.maxCellSize + " bytes";
+ if (LOG.isDebugEnabled()) {
+LOG.debug(msg);
+ }
+ throw new DoNotRetryIOException(msg);
+}
{code}
Should we be logging this at a higher log level, like INFO, or even WARN?
Whatever we decide here should be consistent with other warnings issued at the
other sites where we do this size limit test. (I.e. if DEBUG there and we
decide WARN, update to patch both warnings to WARN, or vice versa.)
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-master.001.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22389) Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
[ https://issues.apache.org/jira/browse/HBASE-22389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836728#comment-16836728 ] Andrew Purtell commented on HBASE-22389: Approved PRs #229 and #230. > Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1 > -- > > Key: HBASE-22389 > URL: https://issues.apache.org/jira/browse/HBASE-22389 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Zach York >Assignee: Zach York >Priority: Major > > This commit seems broken on branch-1. > https://github.com/apache/hbase/commit/d80d3fa45487fedf203c7144e8267753e4d53529#diff-ae046012dd310fc116a729bf36c7df81 > seems to have fixed the issue that the other HBASE-19275 patches fixed, but > when HBASE-19275 was applied to branch-1 it appears to have reverted this > code: > https://github.com/apache/hbase/commit/e8e4beacb039299a9e56cd355e3ada08784a7acb#diff-ae046012dd310fc116a729bf36c7df81 > meaning the tests always pass again. > [~apurtell] [~xucang] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21991) Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
[ https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi updated HBASE-21991: --- Attachment: hbase-21991.addendum.patch > Fix MetaMetrics issues - [Race condition, Faulty remove logic], few > improvements > > > Key: HBASE-21991 > URL: https://issues.apache.org/jira/browse/HBASE-21991 > Project: HBase > Issue Type: Bug > Components: Coprocessors, metrics >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0 > > Attachments: hbase-21991.addendum.patch, > hbase-21991.branch-1.001.patch, hbase-21991.branch-1.002.patch, > hbase-21991.master.001.patch, hbase-21991.master.002.patch, > hbase-21991.master.003.patch, hbase-21991.master.004.patch, > hbase-21991.master.005.patch, hbase-21991.master.006.patch > > > Here is a list of the issues related to the MetaMetrics implementation: > +*Bugs*+: > # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*: > Under certain conditions, we might end up storing/exposing all the meters > rather than top-k-ish > # MetaMetrics can throw NPE resulting in aborting of the RS because of a > *Race Condition*. > +*Improvements*+: > # With high number of regions in the cluster, exposure of metrics for each > region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of > regions. It's better to use *lossy counting to maintain top-k for region > metrics* as well. > # As the lossy meters do not represent actual counts, I think, it'll be > better to *rename the meters to include "lossy" in the name*. It would be > more informative while monitoring the metrics and there would be less > confusion regarding actual counts to lossy counts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22389) Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
[ https://issues.apache.org/jira/browse/HBASE-22389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836726#comment-16836726 ] Andrew Purtell commented on HBASE-22389: Do not remember how it happened at this point. Let's fix it. +1 > Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1 > -- > > Key: HBASE-22389 > URL: https://issues.apache.org/jira/browse/HBASE-22389 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Zach York >Assignee: Zach York >Priority: Major > > This commit seems broken on branch-1. > https://github.com/apache/hbase/commit/d80d3fa45487fedf203c7144e8267753e4d53529#diff-ae046012dd310fc116a729bf36c7df81 > seems to have fixed the issue that the other HBASE-19275 patches fixed, but > when HBASE-19275 was applied to branch-1 it appears to have reverted this > code: > https://github.com/apache/hbase/commit/e8e4beacb039299a9e56cd355e3ada08784a7acb#diff-ae046012dd310fc116a729bf36c7df81 > meaning the tests always pass again. > [~apurtell] [~xucang] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21991) Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
[
https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836722#comment-16836722
]
Andrew Purtell edited comment on HBASE-21991 at 5/9/19 9:35 PM:
When resolving please consider copying the below release note text proposal
into this issue's release note field. In case you want to raise this on dev@
first [~busbey] I think we will defer to you to re-resolve this:
{quote}
The class LossyCounting was unintentionally marked Public but was never
intended to be part of our public API. This oversight has been corrected and
LossyCounting is now marked as Private and going forward may be subject to
additional breaking changes or removal without notice. If you have taken a
dependency on this class we recommend cloning it locally into your project
before upgrading to this release.
{quote}
was (Author: apurtell):
When resolving please consider copying the below release note text proposal
into this issue's release note field. In case you want to raise this on dev@
first [~busbey] I think we will defer to you to re-resolve this:
{quote}
The class LossyCounting was unintentionally marked Public but was never
intended to be part of our public API. This oversight has been corrected and
LossyCounting is now marked as Private and going forward may be subject to
breaking changes or removal without notice. If you have taken a dependency on
this class we recommend cloning it locally into your project before further
changes are made.
{quote}
> Fix MetaMetrics issues - [Race condition, Faulty remove logic], few
> improvements
>
>
> Key: HBASE-21991
> URL: https://issues.apache.org/jira/browse/HBASE-21991
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, metrics
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0
>
> Attachments: hbase-21991.branch-1.001.patch,
> hbase-21991.branch-1.002.patch, hbase-21991.master.001.patch,
> hbase-21991.master.002.patch, hbase-21991.master.003.patch,
> hbase-21991.master.004.patch, hbase-21991.master.005.patch,
> hbase-21991.master.006.patch
>
>
> Here is a list of the issues related to the MetaMetrics implementation:
> +*Bugs*+:
> # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*:
> Under certain conditions, we might end up storing/exposing all the meters
> rather than top-k-ish
> # MetaMetrics can throw NPE resulting in aborting of the RS because of a
> *Race Condition*.
> +*Improvements*+:
> # With high number of regions in the cluster, exposure of metrics for each
> region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of
> regions. It's better to use *lossy counting to maintain top-k for region
> metrics* as well.
> # As the lossy meters do not represent actual counts, I think, it'll be
> better to *rename the meters to include "lossy" in the name*. It would be
> more informative while monitoring the metrics and there would be less
> confusion regarding actual counts to lossy counts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21991) Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
[
https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836722#comment-16836722
]
Andrew Purtell commented on HBASE-21991:
When resolving please consider copying the below release note text proposal
into this issue's release note field. In case you want to raise this on dev@
first [~busbey] I think we will defer to you to re-resolve this:
{quote}
The class LossyCounting was unintentionally marked Public but was never
intended to be part of our public API. This oversight has been corrected and
LossyCounting is now marked as Private and going forward may be subject to
breaking changes or removal without notice. If you have taken a dependency on
this class we recommend cloning it locally into your project before further
changes are made.
{quote}
> Fix MetaMetrics issues - [Race condition, Faulty remove logic], few
> improvements
>
>
> Key: HBASE-21991
> URL: https://issues.apache.org/jira/browse/HBASE-21991
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, metrics
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0
>
> Attachments: hbase-21991.branch-1.001.patch,
> hbase-21991.branch-1.002.patch, hbase-21991.master.001.patch,
> hbase-21991.master.002.patch, hbase-21991.master.003.patch,
> hbase-21991.master.004.patch, hbase-21991.master.005.patch,
> hbase-21991.master.006.patch
>
>
> Here is a list of the issues related to the MetaMetrics implementation:
> +*Bugs*+:
> # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*:
> Under certain conditions, we might end up storing/exposing all the meters
> rather than top-k-ish
> # MetaMetrics can throw NPE resulting in aborting of the RS because of a
> *Race Condition*.
> +*Improvements*+:
> # With high number of regions in the cluster, exposure of metrics for each
> region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of
> regions. It's better to use *lossy counting to maintain top-k for region
> metrics* as well.
> # As the lossy meters do not represent actual counts, I think, it'll be
> better to *rename the meters to include "lossy" in the name*. It would be
> more informative while monitoring the metrics and there would be less
> confusion regarding actual counts to lossy counts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [hbase] Apache-HBase commented on issue #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr…
Apache-HBase commented on issue #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr… URL: https://github.com/apache/hbase/pull/229#issuecomment-491076582 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 85 | Docker mode activated. | ||| _ Prechecks _ | | 0 | findbugs | 5 | Findbugs executables are not available. | | +1 | hbaseanti | 0 | Patch does not have any anti-patterns. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 | mvninstall | 241 | branch-1 passed | | +1 | compile | 48 | branch-1 passed with JDK v1.8.0_212 | | +1 | compile | 44 | branch-1 passed with JDK v1.7.0_222 | | +1 | checkstyle | 88 | branch-1 passed | | +1 | shadedjars | 221 | branch has no errors when building our shaded downstream artifacts. | | +1 | javadoc | 47 | branch-1 passed with JDK v1.8.0_212 | | +1 | javadoc | 45 | branch-1 passed with JDK v1.7.0_222 | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 109 | the patch passed | | +1 | compile | 40 | the patch passed with JDK v1.8.0_212 | | +1 | javac | 40 | the patch passed | | +1 | compile | 44 | the patch passed with JDK v1.7.0_222 | | +1 | javac | 44 | the patch passed | | -1 | checkstyle | 82 | hbase-server: The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedjars | 167 | patch has no errors when building our shaded downstream artifacts. | | +1 | hadoopcheck | 98 | Patch does not cause any errors with Hadoop 2.7.4. | | +1 | javadoc | 29 | the patch passed with JDK v1.8.0_212 | | +1 | javadoc | 38 | the patch passed with JDK v1.7.0_222 | ||| _ Other Tests _ | | -1 | unit | 7184 | hbase-server in the patch failed. | | +1 | asflicense | 25 | The patch does not generate ASF License warnings. | | | | 8746 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.master.snapshot.TestSnapshotFileCache | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-229/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/229 | | Optional Tests | dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4a8c9b6ca0cd 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | branch-1 / 56548fc | | maven | version: Apache Maven 3.0.5 | | Default Java | 1.7.0_222 | | Multi-JDK versions | /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_212 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_222 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-229/1/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-229/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-229/1/testReport/ | | Max. process+thread count | 3672 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-229/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836714#comment-16836714
]
HBase QA commented on HBASE-22330:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
48s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} branch-1 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 40s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m
30s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HBASE-Build/278/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22330 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12968320/HBASE-22330-addendum.branch-1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9953d0ee7b05 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb
13 15:00:41
[jira] [Commented] (HBASE-21991) Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
[
https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836715#comment-16836715
]
Sakthi commented on HBASE-21991:
Will upload an addendum patch with
{code:java}
+@InterfaceAudience.Private
-@InterfaceAudience.Public
public class LossyCounting {{code}
> Fix MetaMetrics issues - [Race condition, Faulty remove logic], few
> improvements
>
>
> Key: HBASE-21991
> URL: https://issues.apache.org/jira/browse/HBASE-21991
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, metrics
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0
>
> Attachments: hbase-21991.branch-1.001.patch,
> hbase-21991.branch-1.002.patch, hbase-21991.master.001.patch,
> hbase-21991.master.002.patch, hbase-21991.master.003.patch,
> hbase-21991.master.004.patch, hbase-21991.master.005.patch,
> hbase-21991.master.006.patch
>
>
> Here is a list of the issues related to the MetaMetrics implementation:
> +*Bugs*+:
> # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*:
> Under certain conditions, we might end up storing/exposing all the meters
> rather than top-k-ish
> # MetaMetrics can throw NPE resulting in aborting of the RS because of a
> *Race Condition*.
> +*Improvements*+:
> # With high number of regions in the cluster, exposure of metrics for each
> region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of
> regions. It's better to use *lossy counting to maintain top-k for region
> metrics* as well.
> # As the lossy meters do not represent actual counts, I think, it'll be
> better to *rename the meters to include "lossy" in the name*. It would be
> more informative while monitoring the metrics and there would be less
> confusion regarding actual counts to lossy counts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21991) Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
[ https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836712#comment-16836712 ] Andrew Purtell commented on HBASE-21991: Let's mark it Private immediate. We can make a release note. It was never intended to be part of our public API, that seems pretty clear, and is trivial to clone if someone decided to depend on it. > Fix MetaMetrics issues - [Race condition, Faulty remove logic], few > improvements > > > Key: HBASE-21991 > URL: https://issues.apache.org/jira/browse/HBASE-21991 > Project: HBase > Issue Type: Bug > Components: Coprocessors, metrics >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0 > > Attachments: hbase-21991.branch-1.001.patch, > hbase-21991.branch-1.002.patch, hbase-21991.master.001.patch, > hbase-21991.master.002.patch, hbase-21991.master.003.patch, > hbase-21991.master.004.patch, hbase-21991.master.005.patch, > hbase-21991.master.006.patch > > > Here is a list of the issues related to the MetaMetrics implementation: > +*Bugs*+: > # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*: > Under certain conditions, we might end up storing/exposing all the meters > rather than top-k-ish > # MetaMetrics can throw NPE resulting in aborting of the RS because of a > *Race Condition*. > +*Improvements*+: > # With high number of regions in the cluster, exposure of metrics for each > region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of > regions. It's better to use *lossy counting to maintain top-k for region > metrics* as well. > # As the lossy meters do not represent actual counts, I think, it'll be > better to *rename the meters to include "lossy" in the name*. It would be > more informative while monitoring the metrics and there would be less > confusion regarding actual counts to lossy counts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21800) RegionServer aborted due to NPE from MetaTableMetrics coprocessor
[
https://issues.apache.org/jira/browse/HBASE-21800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836713#comment-16836713
]
Andrew Purtell commented on HBASE-21800:
Seems we are going to address the issues with LossyCounting in the context of
HBASE-21991 so will resolve this one.
> RegionServer aborted due to NPE from MetaTableMetrics coprocessor
> -
>
> Key: HBASE-21800
> URL: https://issues.apache.org/jira/browse/HBASE-21800
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, meta, metrics, Operability
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Critical
> Labels: Meta
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0
>
> Attachments: hbase-21800.branch-1.001.patch,
> hbase-21800.branch-1.002.patch, hbase-21800.branch-1.003.patch,
> hbase-21800.branch-1.004.patch, hbase-21800.master.001.patch,
> hbase-21800.master.002.patch, hbase-21800.master.003.patch
>
>
> I was just playing around the code, trying to capture "Top k" table metrics
> from MetaMetrics, when I bumped into this issue. Though currently we are not
> capturing "Top K" table metrics, but we can encounter this issue because of
> the "Top k Clients" that is implemented using the LossyAlgo.
>
> RegionServer gets aborted due to a NPE from MetaTableMetrics coprocessor. The
> log looks somewhat like this:
> {code:java}
> 2019-01-28 23:31:10,311 ERROR
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> coprocessor.CoprocessorHost: The coprocessor
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.markMeterIfPresent(MetaTableMetrics.java:123)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.tableMetricRegisterAndMark2(MetaTableMetrics.java:233)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.preGetOp(MetaTableMetrics.java:82)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:840)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:551)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:625)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2608)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2547)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> 2019-01-28 23:31:10,314 ERROR
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.HRegionServer: * ABORTING region server
> 10.0.0.24,16020,1548747043814: The coprocessor
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw
> java.lang.NullPointerException *
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.markMeterIfPresent(MetaTableMetrics.java:123)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.tableMetricRegisterAndMark2(MetaTableMetrics.java:233)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.preGetOp(MetaTableMetrics.java:82)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:840)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:551)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:625)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2608)
> at
>
[jira] [Commented] (HBASE-21991) Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
[ https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836711#comment-16836711 ] Sakthi commented on HBASE-21991: [~busbey] sounds good to me. > Fix MetaMetrics issues - [Race condition, Faulty remove logic], few > improvements > > > Key: HBASE-21991 > URL: https://issues.apache.org/jira/browse/HBASE-21991 > Project: HBase > Issue Type: Bug > Components: Coprocessors, metrics >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0 > > Attachments: hbase-21991.branch-1.001.patch, > hbase-21991.branch-1.002.patch, hbase-21991.master.001.patch, > hbase-21991.master.002.patch, hbase-21991.master.003.patch, > hbase-21991.master.004.patch, hbase-21991.master.005.patch, > hbase-21991.master.006.patch > > > Here is a list of the issues related to the MetaMetrics implementation: > +*Bugs*+: > # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*: > Under certain conditions, we might end up storing/exposing all the meters > rather than top-k-ish > # MetaMetrics can throw NPE resulting in aborting of the RS because of a > *Race Condition*. > +*Improvements*+: > # With high number of regions in the cluster, exposure of metrics for each > region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of > regions. It's better to use *lossy counting to maintain top-k for region > metrics* as well. > # As the lossy meters do not represent actual counts, I think, it'll be > better to *rename the meters to include "lossy" in the name*. It would be > more informative while monitoring the metrics and there would be less > confusion regarding actual counts to lossy counts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836718#comment-16836718 ] Andrew Purtell commented on HBASE-22330: +1 Please commit addendum. > Backport HBASE-20724 (Sometimes some compacted storefiles are still opened > after region failover) to branch-1 > - > > Key: HBASE-22330 > URL: https://issues.apache.org/jira/browse/HBASE-22330 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 1.5.0, 1.4.9, 1.3.4 >Reporter: Andrew Purtell >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.5.0, 1.3.5, 1.4.11 > > Attachments: HBASE-22330-addendum.branch-1.patch, > HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch, > HBASE-22330.branch-1.3.001.patch > > > There appears to be a race condition between close and split which when > combined with a side effect of HBASE-20704, leads to the parent region store > files getting archived and cleared while daughter regions still have > references to those parent region store files. > Here is the timeline of events observed for an affected region: > # RS1 faces ZooKeeper connectivity issue for master node and starts shutting > itself down. As part of this it starts to close the store and clean up the > compacted files (File A) > # Master starts bulk assigning regions and assign parent region to RS2 > # Region opens on RS2 and ends up opening compacted store file(s) (suspect > this is due to HBASE-20724) > # Now split happens and daughter regions open on RS2 and try to run a > compaction as part of post open > # Split request at this point is complete. However now archiving proceeds on > RS1 and ends up archiving the store file that is referenced by the daughter. > Compaction fails due to FileNotFoundException and all subsequent attempts to > open the region will fail until manual resolution. > We think having HBASE-20724 would help in such situations since we won't end > up loading compacted store files in the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21800) RegionServer aborted due to NPE from MetaTableMetrics coprocessor
[
https://issues.apache.org/jira/browse/HBASE-21800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-21800.
Resolution: Fixed
> RegionServer aborted due to NPE from MetaTableMetrics coprocessor
> -
>
> Key: HBASE-21800
> URL: https://issues.apache.org/jira/browse/HBASE-21800
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, meta, metrics, Operability
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Critical
> Labels: Meta
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.3.0, 1.4.10
>
> Attachments: hbase-21800.branch-1.001.patch,
> hbase-21800.branch-1.002.patch, hbase-21800.branch-1.003.patch,
> hbase-21800.branch-1.004.patch, hbase-21800.master.001.patch,
> hbase-21800.master.002.patch, hbase-21800.master.003.patch
>
>
> I was just playing around the code, trying to capture "Top k" table metrics
> from MetaMetrics, when I bumped into this issue. Though currently we are not
> capturing "Top K" table metrics, but we can encounter this issue because of
> the "Top k Clients" that is implemented using the LossyAlgo.
>
> RegionServer gets aborted due to a NPE from MetaTableMetrics coprocessor. The
> log looks somewhat like this:
> {code:java}
> 2019-01-28 23:31:10,311 ERROR
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> coprocessor.CoprocessorHost: The coprocessor
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.markMeterIfPresent(MetaTableMetrics.java:123)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.tableMetricRegisterAndMark2(MetaTableMetrics.java:233)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.preGetOp(MetaTableMetrics.java:82)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:840)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:551)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:625)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2608)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2547)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> 2019-01-28 23:31:10,314 ERROR
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.HRegionServer: * ABORTING region server
> 10.0.0.24,16020,1548747043814: The coprocessor
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw
> java.lang.NullPointerException *
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.markMeterIfPresent(MetaTableMetrics.java:123)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.tableMetricRegisterAndMark2(MetaTableMetrics.java:233)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.preGetOp(MetaTableMetrics.java:82)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:840)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:551)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:625)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2608)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2547)
> at
>
[jira] [Commented] (HBASE-21536) Fix completebulkload usage instructions
[
https://issues.apache.org/jira/browse/HBASE-21536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836691#comment-16836691
]
Artem Ervits commented on HBASE-21536:
--
patch v2 applies to current master, additionally clarified invocation for both
legacy and hbase 3.0. [~psomogyi]
> Fix completebulkload usage instructions
> ---
>
> Key: HBASE-21536
> URL: https://issues.apache.org/jira/browse/HBASE-21536
> Project: HBase
> Issue Type: Task
> Components: documentation, mapreduce
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
> Attachments: HBASE-21536.v01.patch, HBASE-21536.v02.patch
>
>
> Usage information upon invoking LoadIncrementalHFiles is misleading and
> error-prone.
> {code:java}
> usage: completebulkload /path/to/hfileoutputformat-output tablename -loadTable
> -Dcreate.table=no - can be used to avoid creation of table by this tool
> Note: if you set this to 'no', then the target table must already exist in
> HBase
> -loadTable implies your baseDirectory to store file has a depth of 3 ,you
> must have an existing table
> -Dignore.unmatched.families=yes - can be used to ignore unmatched column
> families{code}
> in case of invoking the class via hbase command, completebulkload argument is
> unnecessary and only required via hadoop jar invocation. This is also an
> attempt to clarify where <-loadTable> and <-Dargs> arguments go on the
> command line.
> Furthermore, since LoadIncrementalHFiles was recently moved out of
> hbase-server.jar to hbase-mapreduce, updating ref guide to demonstrate as
> such.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21991) Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
[ https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836686#comment-16836686 ] Sean Busbey commented on HBASE-21991: - Agreed LossyCounting shouldn't be public. Maybe a quick DISCUSS on dev@ about cutting our losses and just release noting making it Private without any deprecation? > Fix MetaMetrics issues - [Race condition, Faulty remove logic], few > improvements > > > Key: HBASE-21991 > URL: https://issues.apache.org/jira/browse/HBASE-21991 > Project: HBase > Issue Type: Bug > Components: Coprocessors, metrics >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0 > > Attachments: hbase-21991.branch-1.001.patch, > hbase-21991.branch-1.002.patch, hbase-21991.master.001.patch, > hbase-21991.master.002.patch, hbase-21991.master.003.patch, > hbase-21991.master.004.patch, hbase-21991.master.005.patch, > hbase-21991.master.006.patch > > > Here is a list of the issues related to the MetaMetrics implementation: > +*Bugs*+: > # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*: > Under certain conditions, we might end up storing/exposing all the meters > rather than top-k-ish > # MetaMetrics can throw NPE resulting in aborting of the RS because of a > *Race Condition*. > +*Improvements*+: > # With high number of regions in the cluster, exposure of metrics for each > region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of > regions. It's better to use *lossy counting to maintain top-k for region > metrics* as well. > # As the lossy meters do not represent actual counts, I think, it'll be > better to *rename the meters to include "lossy" in the name*. It would be > more informative while monitoring the metrics and there would be less > confusion regarding actual counts to lossy counts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21536) Fix completebulkload usage instructions
[
https://issues.apache.org/jira/browse/HBASE-21536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artem Ervits updated HBASE-21536:
-
Attachment: HBASE-21536.v02.patch
> Fix completebulkload usage instructions
> ---
>
> Key: HBASE-21536
> URL: https://issues.apache.org/jira/browse/HBASE-21536
> Project: HBase
> Issue Type: Task
> Components: documentation, mapreduce
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
> Attachments: HBASE-21536.v01.patch, HBASE-21536.v02.patch
>
>
> Usage information upon invoking LoadIncrementalHFiles is misleading and
> error-prone.
> {code:java}
> usage: completebulkload /path/to/hfileoutputformat-output tablename -loadTable
> -Dcreate.table=no - can be used to avoid creation of table by this tool
> Note: if you set this to 'no', then the target table must already exist in
> HBase
> -loadTable implies your baseDirectory to store file has a depth of 3 ,you
> must have an existing table
> -Dignore.unmatched.families=yes - can be used to ignore unmatched column
> families{code}
> in case of invoking the class via hbase command, completebulkload argument is
> unnecessary and only required via hadoop jar invocation. This is also an
> attempt to clarify where <-loadTable> and <-Dargs> arguments go on the
> command line.
> Furthermore, since LoadIncrementalHFiles was recently moved out of
> hbase-server.jar to hbase-mapreduce, updating ref guide to demonstrate as
> such.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21536) Fix completebulkload usage instructions
[
https://issues.apache.org/jira/browse/HBASE-21536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artem Ervits updated HBASE-21536:
-
Status: Patch Available (was: Open)
> Fix completebulkload usage instructions
> ---
>
> Key: HBASE-21536
> URL: https://issues.apache.org/jira/browse/HBASE-21536
> Project: HBase
> Issue Type: Task
> Components: documentation, mapreduce
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
> Attachments: HBASE-21536.v01.patch, HBASE-21536.v02.patch
>
>
> Usage information upon invoking LoadIncrementalHFiles is misleading and
> error-prone.
> {code:java}
> usage: completebulkload /path/to/hfileoutputformat-output tablename -loadTable
> -Dcreate.table=no - can be used to avoid creation of table by this tool
> Note: if you set this to 'no', then the target table must already exist in
> HBase
> -loadTable implies your baseDirectory to store file has a depth of 3 ,you
> must have an existing table
> -Dignore.unmatched.families=yes - can be used to ignore unmatched column
> families{code}
> in case of invoking the class via hbase command, completebulkload argument is
> unnecessary and only required via hadoop jar invocation. This is also an
> attempt to clarify where <-loadTable> and <-Dargs> arguments go on the
> command line.
> Furthermore, since LoadIncrementalHFiles was recently moved out of
> hbase-server.jar to hbase-mapreduce, updating ref guide to demonstrate as
> such.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21536) Fix completebulkload usage instructions
[
https://issues.apache.org/jira/browse/HBASE-21536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artem Ervits updated HBASE-21536:
-
Status: Open (was: Patch Available)
> Fix completebulkload usage instructions
> ---
>
> Key: HBASE-21536
> URL: https://issues.apache.org/jira/browse/HBASE-21536
> Project: HBase
> Issue Type: Task
> Components: documentation, mapreduce
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
> Attachments: HBASE-21536.v01.patch, HBASE-21536.v02.patch
>
>
> Usage information upon invoking LoadIncrementalHFiles is misleading and
> error-prone.
> {code:java}
> usage: completebulkload /path/to/hfileoutputformat-output tablename -loadTable
> -Dcreate.table=no - can be used to avoid creation of table by this tool
> Note: if you set this to 'no', then the target table must already exist in
> HBase
> -loadTable implies your baseDirectory to store file has a depth of 3 ,you
> must have an existing table
> -Dignore.unmatched.families=yes - can be used to ignore unmatched column
> families{code}
> in case of invoking the class via hbase command, completebulkload argument is
> unnecessary and only required via hadoop jar invocation. This is also an
> attempt to clarify where <-loadTable> and <-Dargs> arguments go on the
> command line.
> Furthermore, since LoadIncrementalHFiles was recently moved out of
> hbase-server.jar to hbase-mapreduce, updating ref guide to demonstrate as
> such.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21991) Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
[ https://issues.apache.org/jira/browse/HBASE-21991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836689#comment-16836689 ] Sean Busbey commented on HBASE-21991: - sorry for the unclear phrasing. to be clear, I plan to do the above unless someone complains here. > Fix MetaMetrics issues - [Race condition, Faulty remove logic], few > improvements > > > Key: HBASE-21991 > URL: https://issues.apache.org/jira/browse/HBASE-21991 > Project: HBase > Issue Type: Bug > Components: Coprocessors, metrics >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.3.0 > > Attachments: hbase-21991.branch-1.001.patch, > hbase-21991.branch-1.002.patch, hbase-21991.master.001.patch, > hbase-21991.master.002.patch, hbase-21991.master.003.patch, > hbase-21991.master.004.patch, hbase-21991.master.005.patch, > hbase-21991.master.006.patch > > > Here is a list of the issues related to the MetaMetrics implementation: > +*Bugs*+: > # [_Lossy counting for top-k_] *Faulty remove logic of non-eligible meters*: > Under certain conditions, we might end up storing/exposing all the meters > rather than top-k-ish > # MetaMetrics can throw NPE resulting in aborting of the RS because of a > *Race Condition*. > +*Improvements*+: > # With high number of regions in the cluster, exposure of metrics for each > region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of > regions. It's better to use *lossy counting to maintain top-k for region > metrics* as well. > # As the lossy meters do not represent actual counts, I think, it'll be > better to *rename the meters to include "lossy" in the name*. It would be > more informative while monitoring the metrics and there would be less > confusion regarding actual counts to lossy counts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22358) Change rubocop configuration for method length
[
https://issues.apache.org/jira/browse/HBASE-22358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836684#comment-16836684
]
Hudson commented on HBASE-22358:
Results for branch branch-2.2
[build #247 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/247/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/247//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/247//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/247//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change rubocop configuration for method length
> --
>
> Key: HBASE-22358
> URL: https://issues.apache.org/jira/browse/HBASE-22358
> Project: HBase
> Issue Type: Improvement
> Components: community, shell
>Reporter: Jan Hentschel
>Assignee: Murtaza Hassan
>Priority: Minor
> Labels: beginner, beginners
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.1.5, 2.2.1, 1.3.5, 1.4.11
>
>
> rubocop currently uses a maximum method length for the Ruby code of 10, which
> is way too restrictive. In Checkstyle we're using 150 lines per method. Don't
> know if it needs to be that much, but something between 50 and 75 seems to be
> more realistic, especially for test cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22324) loss a mass of data when the sequenceId of cells greater than Integer.Max, because MemStoreMergerSegmentsIterator can not merge segments
[
https://issues.apache.org/jira/browse/HBASE-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836683#comment-16836683
]
Hudson commented on HBASE-22324:
Results for branch branch-2.2
[build #247 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/247/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/247//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/247//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/247//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> loss a mass of data when the sequenceId of cells greater than Integer.Max,
> because MemStoreMergerSegmentsIterator can not merge segments
> --
>
> Key: HBASE-22324
> URL: https://issues.apache.org/jira/browse/HBASE-22324
> Project: HBase
> Issue Type: Bug
> Components: in-memory-compaction
>Affects Versions: 2.1.0, 2.2.0
>Reporter: chenyang
>Assignee: ChenYang
>Priority: Blocker
> Labels: patch
> Fix For: 3.0.0, 2.1.0, 2.2.0, 2.3.0, 2.0.6
>
> Attachments: HBASE-22324.branch-2.1.0005.patch,
> HBASE-22324.branch-2.1.0006.patch
>
>
> if your memstore type is CompactingMemStore，MemStoreMergerSegmentsIterator
> can not merge memstore segments when the seqId of cells greater than
> Integer.Max, as a result, lossing a mass of data. the reason is that
> MemStoreMergerSegmentsIterator use Integer.Max as readPt when create Scanner,
> but the seqId of cell may be greater than Integer.MAX_VALUE, it`s type is
> long. code as below:
> {code:java}
> public MemStoreMergerSegmentsIterator(List segments,
> CellComparator comparator,
> int compactionKVMax) throws IOException {
> super(compactionKVMax);
> // create the list of scanners to traverse over all the data
> // no dirty reads here as these are immutable segments
> AbstractMemStore.addToScanners(segments, Integer.MAX_VALUE, scanners);
> //bug, should use Long.MAX_VALUE
> heap = new KeyValueHeap(scanners, comparator);
> }
> SegmentScanner.java code as below
> protected void updateCurrent() {
> Cell startKV = current;
> Cell next = null;
> try {
> while (iter.hasNext()) {
> next = iter.next();
> // here, if seqId>readPoint(Integer.MAX_VALUE), never read cell, as a
> result, lossing lots of cells
> if (next.getSequenceId() <= this.readPoint) {
> current = next;
> return;// skip irrelevant versions
> }
> if (stopSkippingKVsIfNextRow && // for backwardSeek() stay in the
> startKV != null &&// boundaries of a single row
> segment.compareRows(next, startKV) > 0) {
> current = null;
> return;
> }
> } // end of while
> current = null; // nothing found
> } finally {
> if (next != null) {
> // in all cases, remember the last KV we iterated to, needed for
> reseek()
> last = next;
> }
> }
> }
> MemStoreCompactorSegmentsIterator has the same bug
> public MemStoreCompactorSegmentsIterator(List segments,
> CellComparator comparator, int compactionKVMax, HStore store) throws
> IOException {
> super(compactionKVMax);
> List scanners = new ArrayList();
> AbstractMemStore.addToScanners(segments, Integer.MAX_VALUE, scanners);
> //bug, should use Long.MAX_VALUE
> // build the scanner based on Query Matcher
> // reinitialize the compacting scanner for each instance of iterator
> compactingScanner = createScanner(store, scanners);
> refillKVS();
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22389) Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
[ https://issues.apache.org/jira/browse/HBASE-22389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836645#comment-16836645 ] Zach York commented on HBASE-22389: --- looks to also be an issue with branch 1.4 > Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1 > -- > > Key: HBASE-22389 > URL: https://issues.apache.org/jira/browse/HBASE-22389 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Zach York >Assignee: Zach York >Priority: Major > > This commit seems broken on branch-1. > https://github.com/apache/hbase/commit/d80d3fa45487fedf203c7144e8267753e4d53529#diff-ae046012dd310fc116a729bf36c7df81 > seems to have fixed the issue that the other HBASE-19275 patches fixed, but > when HBASE-19275 was applied to branch-1 it appears to have reverted this > code: > https://github.com/apache/hbase/commit/e8e4beacb039299a9e56cd355e3ada08784a7acb#diff-ae046012dd310fc116a729bf36c7df81 > meaning the tests always pass again. > [~apurtell] [~xucang] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22390) backport HBASE-22190 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-22390: -- Status: Patch Available (was: Open) > backport HBASE-22190 to branch-1 > > > Key: HBASE-22390 > URL: https://issues.apache.org/jira/browse/HBASE-22390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.10, 1.5.0 >Reporter: Zach York >Assignee: Zach York >Priority: Major > > HBASE-22190 was only applied to branch-2+ while the bug exists in branch-1 as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] z-york commented on issue #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr…
z-york commented on issue #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr… URL: https://github.com/apache/hbase/pull/229#issuecomment-491035742 https://github.com/apache/hbase/pull/230 should be merged first since it will cause test failures otherwise (as mentioned previously) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] z-york commented on issue #230: HBASE-22390 Backport HBASE-22190 to branch-1
z-york commented on issue #230: HBASE-22390 Backport HBASE-22190 to branch-1 URL: https://github.com/apache/hbase/pull/230#issuecomment-491035414 Also please advise if I attributed this incorrectly, it's a little wonky with a PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] z-york commented on issue #230: HBASE-22390 Backport HBASE-22190 to branch-1
z-york commented on issue #230: HBASE-22390 Backport HBASE-22190 to branch-1 URL: https://github.com/apache/hbase/pull/230#issuecomment-491035174 @Apache9 @apurtell (for visibility since this fixes HBASE-21070) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] z-york opened a new pull request #230: HBASE-22390 Backport HBASE-22190 to branch-1
z-york opened a new pull request #230: HBASE-22390 Backport HBASE-22190 to branch-1 URL: https://github.com/apache/hbase/pull/230 HBASE-22190 SnapshotFileCache may fail to load the correct snapshot file list when there is an on-going snapshot operation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22220) Release hbase-connectors-1.0.0
[
https://issues.apache.org/jira/browse/HBASE-0?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836643#comment-16836643
]
Hudson commented on HBASE-0:
Results for branch master
[build #995 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/995/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/995//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/995//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/995//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Release hbase-connectors-1.0.0
> --
>
> Key: HBASE-0
> URL: https://issues.apache.org/jira/browse/HBASE-0
> Project: HBase
> Issue Type: Task
> Components: hbase-connectors
>Affects Versions: connector-1.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: connector-1.0.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22358) Change rubocop configuration for method length
[
https://issues.apache.org/jira/browse/HBASE-22358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836642#comment-16836642
]
Hudson commented on HBASE-22358:
Results for branch master
[build #995 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/995/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/995//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/995//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/995//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change rubocop configuration for method length
> --
>
> Key: HBASE-22358
> URL: https://issues.apache.org/jira/browse/HBASE-22358
> Project: HBase
> Issue Type: Improvement
> Components: community, shell
>Reporter: Jan Hentschel
>Assignee: Murtaza Hassan
>Priority: Minor
> Labels: beginner, beginners
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.1.5, 2.2.1, 1.3.5, 1.4.11
>
>
> rubocop currently uses a maximum method length for the Ruby code of 10, which
> is way too restrictive. In Checkstyle we're using 150 lines per method. Don't
> know if it needs to be that much, but something between 50 and 75 seems to be
> more realistic, especially for test cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-22390) backport HBASE-22190 to branch-1
Zach York created HBASE-22390: - Summary: backport HBASE-22190 to branch-1 Key: HBASE-22390 URL: https://issues.apache.org/jira/browse/HBASE-22390 Project: HBase Issue Type: Bug Affects Versions: 1.4.10, 1.5.0 Reporter: Zach York Assignee: Zach York HBASE-22190 was only applied to branch-2+ while the bug exists in branch-1 as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836640#comment-16836640 ] Xu Cang commented on HBASE-22330: - Thanks! I ran tests after applying the new addendum. 5 out of 5 times passed. +1 > Backport HBASE-20724 (Sometimes some compacted storefiles are still opened > after region failover) to branch-1 > - > > Key: HBASE-22330 > URL: https://issues.apache.org/jira/browse/HBASE-22330 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 1.5.0, 1.4.9, 1.3.4 >Reporter: Andrew Purtell >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.5.0, 1.3.5, 1.4.11 > > Attachments: HBASE-22330-addendum.branch-1.patch, > HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch, > HBASE-22330.branch-1.3.001.patch > > > There appears to be a race condition between close and split which when > combined with a side effect of HBASE-20704, leads to the parent region store > files getting archived and cleared while daughter regions still have > references to those parent region store files. > Here is the timeline of events observed for an affected region: > # RS1 faces ZooKeeper connectivity issue for master node and starts shutting > itself down. As part of this it starts to close the store and clean up the > compacted files (File A) > # Master starts bulk assigning regions and assign parent region to RS2 > # Region opens on RS2 and ends up opening compacted store file(s) (suspect > this is due to HBASE-20724) > # Now split happens and daughter regions open on RS2 and try to run a > compaction as part of post open > # Split request at this point is complete. However now archiving proceeds on > RS1 and ends up archiving the store file that is referenced by the daughter. > Compaction fails due to FileNotFoundException and all subsequent attempts to > open the region will fail until manual resolution. > We think having HBASE-20724 would help in such situations since we won't end > up loading compacted store files in the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-22330: --- Attachment: HBASE-22330-addendum.branch-1.patch > Backport HBASE-20724 (Sometimes some compacted storefiles are still opened > after region failover) to branch-1 > - > > Key: HBASE-22330 > URL: https://issues.apache.org/jira/browse/HBASE-22330 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 1.5.0, 1.4.9, 1.3.4 >Reporter: Andrew Purtell >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.5.0, 1.3.5, 1.4.11 > > Attachments: HBASE-22330-addendum.branch-1.patch, > HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch, > HBASE-22330.branch-1.3.001.patch > > > There appears to be a race condition between close and split which when > combined with a side effect of HBASE-20704, leads to the parent region store > files getting archived and cleared while daughter regions still have > references to those parent region store files. > Here is the timeline of events observed for an affected region: > # RS1 faces ZooKeeper connectivity issue for master node and starts shutting > itself down. As part of this it starts to close the store and clean up the > compacted files (File A) > # Master starts bulk assigning regions and assign parent region to RS2 > # Region opens on RS2 and ends up opening compacted store file(s) (suspect > this is due to HBASE-20724) > # Now split happens and daughter regions open on RS2 and try to run a > compaction as part of post open > # Split request at this point is complete. However now archiving proceeds on > RS1 and ends up archiving the store file that is referenced by the daughter. > Compaction fails due to FileNotFoundException and all subsequent attempts to > open the region will fail until manual resolution. > We think having HBASE-20724 would help in such situations since we won't end > up loading compacted store files in the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-22330: --- Hadoop Flags: (was: Reviewed) Status: Patch Available (was: Reopened) > Backport HBASE-20724 (Sometimes some compacted storefiles are still opened > after region failover) to branch-1 > - > > Key: HBASE-22330 > URL: https://issues.apache.org/jira/browse/HBASE-22330 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 1.3.4, 1.4.9, 1.5.0 >Reporter: Andrew Purtell >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.5.0, 1.3.5, 1.4.11 > > Attachments: HBASE-22330-addendum.branch-1.patch, > HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch, > HBASE-22330.branch-1.3.001.patch > > > There appears to be a race condition between close and split which when > combined with a side effect of HBASE-20704, leads to the parent region store > files getting archived and cleared while daughter regions still have > references to those parent region store files. > Here is the timeline of events observed for an affected region: > # RS1 faces ZooKeeper connectivity issue for master node and starts shutting > itself down. As part of this it starts to close the store and clean up the > compacted files (File A) > # Master starts bulk assigning regions and assign parent region to RS2 > # Region opens on RS2 and ends up opening compacted store file(s) (suspect > this is due to HBASE-20724) > # Now split happens and daughter regions open on RS2 and try to run a > compaction as part of post open > # Split request at this point is complete. However now archiving proceeds on > RS1 and ends up archiving the store file that is referenced by the daughter. > Compaction fails due to FileNotFoundException and all subsequent attempts to > open the region will fail until manual resolution. > We think having HBASE-20724 would help in such situations since we won't end > up loading compacted store files in the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan reopened HBASE-22330: Hardening the test > Backport HBASE-20724 (Sometimes some compacted storefiles are still opened > after region failover) to branch-1 > - > > Key: HBASE-22330 > URL: https://issues.apache.org/jira/browse/HBASE-22330 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 1.5.0, 1.4.9, 1.3.4 >Reporter: Andrew Purtell >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.5.0, 1.3.5, 1.4.11 > > Attachments: HBASE-22330.branch-1.001.patch, > HBASE-22330.branch-1.002.patch, HBASE-22330.branch-1.3.001.patch > > > There appears to be a race condition between close and split which when > combined with a side effect of HBASE-20704, leads to the parent region store > files getting archived and cleared while daughter regions still have > references to those parent region store files. > Here is the timeline of events observed for an affected region: > # RS1 faces ZooKeeper connectivity issue for master node and starts shutting > itself down. As part of this it starts to close the store and clean up the > compacted files (File A) > # Master starts bulk assigning regions and assign parent region to RS2 > # Region opens on RS2 and ends up opening compacted store file(s) (suspect > this is due to HBASE-20724) > # Now split happens and daughter regions open on RS2 and try to run a > compaction as part of post open > # Split request at this point is complete. However now archiving proceeds on > RS1 and ends up archiving the store file that is referenced by the daughter. > Compaction fails due to FileNotFoundException and all subsequent attempts to > open the region will fail until manual resolution. > We think having HBASE-20724 would help in such situations since we won't end > up loading compacted store files in the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] z-york commented on issue #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr…
z-york commented on issue #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr… URL: https://github.com/apache/hbase/pull/229#issuecomment-491029923 Note this will actually cause the new test added in HBASE-21070 in TestSnapshotFileCache to fail until HBASE-22190 is backported (I'm opening a different PR for that). I originally missed that it wasn't backported to branch-1, then when I realized it wasn't, I was confused why the test I added was passing which led me to this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836633#comment-16836633 ] Abhishek Singh Chouhan commented on HBASE-22330: 1 in 10 runs fail for me locally. Looks to be a test only issue due to differences in what testUtil.waitTableAvailable(..) does between master and branch-1. Let me put up an addendum for the test which removes flakiness. > Backport HBASE-20724 (Sometimes some compacted storefiles are still opened > after region failover) to branch-1 > - > > Key: HBASE-22330 > URL: https://issues.apache.org/jira/browse/HBASE-22330 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 1.5.0, 1.4.9, 1.3.4 >Reporter: Andrew Purtell >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.5.0, 1.3.5, 1.4.11 > > Attachments: HBASE-22330.branch-1.001.patch, > HBASE-22330.branch-1.002.patch, HBASE-22330.branch-1.3.001.patch > > > There appears to be a race condition between close and split which when > combined with a side effect of HBASE-20704, leads to the parent region store > files getting archived and cleared while daughter regions still have > references to those parent region store files. > Here is the timeline of events observed for an affected region: > # RS1 faces ZooKeeper connectivity issue for master node and starts shutting > itself down. As part of this it starts to close the store and clean up the > compacted files (File A) > # Master starts bulk assigning regions and assign parent region to RS2 > # Region opens on RS2 and ends up opening compacted store file(s) (suspect > this is due to HBASE-20724) > # Now split happens and daughter regions open on RS2 and try to run a > compaction as part of post open > # Split request at this point is complete. However now archiving proceeds on > RS1 and ends up archiving the store file that is referenced by the daughter. > Compaction fails due to FileNotFoundException and all subsequent attempts to > open the region will fail until manual resolution. > We think having HBASE-20724 would help in such situations since we won't end > up loading compacted store files in the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22389) Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1
Zach York created HBASE-22389: - Summary: Revert HBASE-19275 TestSnapshotFileCache never worked properly on branch-1 Key: HBASE-22389 URL: https://issues.apache.org/jira/browse/HBASE-22389 Project: HBase Issue Type: Bug Affects Versions: 1.5.0 Reporter: Zach York Assignee: Zach York This commit seems broken on branch-1. https://github.com/apache/hbase/commit/d80d3fa45487fedf203c7144e8267753e4d53529#diff-ae046012dd310fc116a729bf36c7df81 seems to have fixed the issue that the other HBASE-19275 patches fixed, but when HBASE-19275 was applied to branch-1 it appears to have reverted this code: https://github.com/apache/hbase/commit/e8e4beacb039299a9e56cd355e3ada08784a7acb#diff-ae046012dd310fc116a729bf36c7df81 meaning the tests always pass again. [~apurtell] [~xucang] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] z-york opened a new pull request #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr…
z-york opened a new pull request #229: HBASE-22389 Revert "HBASE-19275 TestSnapshotFileCache never worked pr… URL: https://github.com/apache/hbase/pull/229 …operly" This reverts commit e8e4beacb039299a9e56cd355e3ada08784a7acb. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22358) Change rubocop configuration for method length
[
https://issues.apache.org/jira/browse/HBASE-22358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836632#comment-16836632
]
Hudson commented on HBASE-22358:
Results for branch branch-2
[build #1879 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1879/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1879//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1879//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1879//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change rubocop configuration for method length
> --
>
> Key: HBASE-22358
> URL: https://issues.apache.org/jira/browse/HBASE-22358
> Project: HBase
> Issue Type: Improvement
> Components: community, shell
>Reporter: Jan Hentschel
>Assignee: Murtaza Hassan
>Priority: Minor
> Labels: beginner, beginners
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.0.6, 2.1.5, 2.2.1, 1.3.5, 1.4.11
>
>
> rubocop currently uses a maximum method length for the Ruby code of 10, which
> is way too restrictive. In Checkstyle we're using 150 lines per method. Don't
> know if it needs to be that much, but something between 50 and 75 seems to be
> more realistic, especially for test cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22324) loss a mass of data when the sequenceId of cells greater than Integer.Max, because MemStoreMergerSegmentsIterator can not merge segments
[
https://issues.apache.org/jira/browse/HBASE-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836631#comment-16836631
]
Hudson commented on HBASE-22324:
Results for branch branch-2
[build #1879 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1879/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1879//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1879//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1879//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> loss a mass of data when the sequenceId of cells greater than Integer.Max,
> because MemStoreMergerSegmentsIterator can not merge segments
> --
>
> Key: HBASE-22324
> URL: https://issues.apache.org/jira/browse/HBASE-22324
> Project: HBase
> Issue Type: Bug
> Components: in-memory-compaction
>Affects Versions: 2.1.0, 2.2.0
>Reporter: chenyang
>Assignee: ChenYang
>Priority: Blocker
> Labels: patch
> Fix For: 3.0.0, 2.1.0, 2.2.0, 2.3.0, 2.0.6
>
> Attachments: HBASE-22324.branch-2.1.0005.patch,
> HBASE-22324.branch-2.1.0006.patch
>
>
> if your memstore type is CompactingMemStore，MemStoreMergerSegmentsIterator
> can not merge memstore segments when the seqId of cells greater than
> Integer.Max, as a result, lossing a mass of data. the reason is that
> MemStoreMergerSegmentsIterator use Integer.Max as readPt when create Scanner,
> but the seqId of cell may be greater than Integer.MAX_VALUE, it`s type is
> long. code as below:
> {code:java}
> public MemStoreMergerSegmentsIterator(List segments,
> CellComparator comparator,
> int compactionKVMax) throws IOException {
> super(compactionKVMax);
> // create the list of scanners to traverse over all the data
> // no dirty reads here as these are immutable segments
> AbstractMemStore.addToScanners(segments, Integer.MAX_VALUE, scanners);
> //bug, should use Long.MAX_VALUE
> heap = new KeyValueHeap(scanners, comparator);
> }
> SegmentScanner.java code as below
> protected void updateCurrent() {
> Cell startKV = current;
> Cell next = null;
> try {
> while (iter.hasNext()) {
> next = iter.next();
> // here, if seqId>readPoint(Integer.MAX_VALUE), never read cell, as a
> result, lossing lots of cells
> if (next.getSequenceId() <= this.readPoint) {
> current = next;
> return;// skip irrelevant versions
> }
> if (stopSkippingKVsIfNextRow && // for backwardSeek() stay in the
> startKV != null &&// boundaries of a single row
> segment.compareRows(next, startKV) > 0) {
> current = null;
> return;
> }
> } // end of while
> current = null; // nothing found
> } finally {
> if (next != null) {
> // in all cases, remember the last KV we iterated to, needed for
> reseek()
> last = next;
> }
> }
> }
> MemStoreCompactorSegmentsIterator has the same bug
> public MemStoreCompactorSegmentsIterator(List segments,
> CellComparator comparator, int compactionKVMax, HStore store) throws
> IOException {
> super(compactionKVMax);
> List scanners = new ArrayList();
> AbstractMemStore.addToScanners(segments, Integer.MAX_VALUE, scanners);
> //bug, should use Long.MAX_VALUE
> // build the scanner based on Query Matcher
> // reinitialize the compacting scanner for each instance of iterator
> compactingScanner = createScanner(store, scanners);
> refillKVS();
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
