[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587864#comment-15587864 ] Anastasia Braginsky commented on HBASE-16608: - bq. Doing the same load test with CompactingMemstore I can see so many times we miss a getChunk call on MSLABPool.. Seems the bug fix that was added in latest patch is not releasing chunks correctly. [~anoop.hbase], there indeed might be a problem there. I can debug this path, but unfortunately it can not be done just now as I have no access to my workplace. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16864) Procedure v2 - Fix StateMachineProcedure support for child procs at last step
[ https://issues.apache.org/jira/browse/HBASE-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587842#comment-15587842 ] Appy commented on HBASE-16864: -- +1 > Procedure v2 - Fix StateMachineProcedure support for child procs at last step > - > > Key: HBASE-16864 > URL: https://issues.apache.org/jira/browse/HBASE-16864 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16864-v0.patch, HBASE-16864-v1.patch > > > There is a bug in the StateMachineProcedure when we add child procs in the > last step. On recovery we end up spinning on the last step without ever > completing. the fix is to introduce an eof step so recovery knows that we are > already done once the all the children are terminated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587838#comment-15587838 ] Hudson commented on HBASE-16824: SUCCESS: Integrated in Jenkins build HBase-1.4 #477 (See [https://builds.apache.org/job/HBase-1.4/477/]) HBASE-16824 Writer.flush() can be called on already closed streams in (enis: rev 019c7f9303a7242b7c5d6713bed414b180b5c84a) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587829#comment-15587829 ] Hudson commented on HBASE-16824: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1813 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1813/]) HBASE-16824 Writer.flush() can be called on already closed streams in (enis: rev ef8c65e54201b37edfb9a8f4f4d24137544b8ec1) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16880) Correct the javadoc/behaviour of the APIs in ByteBufferUtils
ramkrishna.s.vasudevan created HBASE-16880:
--
Summary: Correct the javadoc/behaviour of the APIs in
ByteBufferUtils
Key: HBASE-16880
URL: https://issues.apache.org/jira/browse/HBASE-16880
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
There are some issues either with the javadoc or the actual behaviour of some
APIs in BBUtils.
For eg,
BBUtil#copyFromBufferToBuffer() says
{code}
/**
* Copy one buffer's whole data to another. Write starts at the current
position of 'out' buffer.
* Note : This will advance the position marker of {@code out} but not change
the position maker
* for {@code in}. The position and limit of the {@code in} buffer to be set
properly by caller.
* @param in source buffer
* @param out destination buffer
{code}
But this is true in case of UNSAFE.
{code}
if (UNSAFE_AVAIL) {
int length = in.remaining();
UnsafeAccess.copy(in, in.position(), out, out.position(), length);
out.position(out.position() + length);
} else {
out.put(in);
}
{code}
But in other case where we move the else - the 'in' is also advanced. So we
need to either correct the behaviour or change the doc and see all the used
places. This JIRA can be used to correct all the APIs in this util class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16578) Mob data loss after mob compaction and normal compaction
[
https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-16578:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
Status: Resolved (was: Patch Available)
> Mob data loss after mob compaction and normal compaction
>
>
> Key: HBASE-16578
> URL: https://issues.apache.org/jira/browse/HBASE-16578
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-16578-V2.patch, HBASE-16578-V3.patch,
> HBASE-16578.patch, TestMobCompaction.java, TestMobCompaction.java
>
>
> StoreFileScanners on MOB cells rely on the scannerOrder to find the latest
> cells after mob compaction. The value of scannerOrder is assigned by the
> order of maxSeqId of StoreFile, and this maxSeqId is valued only after the
> reader of the StoreFile is created.
> In {{Compactor.compact}}, the compacted store files are cloned and their
> readers are not created. And in {{StoreFileScanner.getScannersForStoreFiles}}
> the StoreFiles are sorted before the readers are created and at that time the
> maxSeqId for each file is -1 (the default value). This will lead to a chaos
> in scanners in the following normal compaction. Some older cells might be
> chosen during the normal compaction.
> We need to create readers either before the sorting in the method
> {{StoreFileScanner.getScannersForStoreFiles}}, or create readers just after
> the store files are cloned in {{Compactor.compact}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587797#comment-15587797 ] Hudson commented on HBASE-16824: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #48 (See [https://builds.apache.org/job/HBase-1.2-JDK7/48/]) HBASE-16824 Writer.flush() can be called on already closed streams in (enis: rev 57181442577c36689114334b011a6e72de4ae785) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587795#comment-15587795
]
Hadoop QA commented on HBASE-16874:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 27s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 16s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
| Timed out junit tests |
org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
| | org.apache.hadoop.hbase.master.TestSplitLogManager |
| | org.apache.hadoop.hbase.master.procedure.TestWALProcedureStoreOnHDFS |
| |
org.apache.hadoop.hbase.master.procedure.TestDeleteColumnFamilyProcedureFromClient
|
| | org.apache.hadoop.hbase.master.TestRegionPlacement2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834042/HBASE-16874-v0.patch |
| JIRA Issue | HBASE-16874 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 46bb5e1afaa8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / ef8c65e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4096/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4096/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCom
[jira] [Commented] (HBASE-16578) Mob data loss after mob compaction and normal compaction
[
https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587783#comment-15587783
]
Jingcheng Du commented on HBASE-16578:
--
Thanks for the reviews [~tedyu], [~ram_krish], and thanks for the reviews and
the finding [~huaxiang].
Committing this patch to master.
> Mob data loss after mob compaction and normal compaction
>
>
> Key: HBASE-16578
> URL: https://issues.apache.org/jira/browse/HBASE-16578
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Jingcheng Du
> Attachments: HBASE-16578-V2.patch, HBASE-16578-V3.patch,
> HBASE-16578.patch, TestMobCompaction.java, TestMobCompaction.java
>
>
> StoreFileScanners on MOB cells rely on the scannerOrder to find the latest
> cells after mob compaction. The value of scannerOrder is assigned by the
> order of maxSeqId of StoreFile, and this maxSeqId is valued only after the
> reader of the StoreFile is created.
> In {{Compactor.compact}}, the compacted store files are cloned and their
> readers are not created. And in {{StoreFileScanner.getScannersForStoreFiles}}
> the StoreFiles are sorted before the readers are created and at that time the
> maxSeqId for each file is -1 (the default value). This will lead to a chaos
> in scanners in the following normal compaction. Some older cells might be
> chosen during the normal compaction.
> We need to create readers either before the sorting in the method
> {{StoreFileScanner.getScannersForStoreFiles}}, or create readers just after
> the store files are cloned in {{Compactor.compact}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15532) core favored nodes enhancements
[
https://issues.apache.org/jira/browse/HBASE-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587776#comment-15587776
]
Hadoop QA commented on HBASE-15532:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 36s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
16s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s
{color} | {color:red} hbase-protocol-shaded in master has 22 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 0s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 90 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 3s
{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 51s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s
{color} | {color:red} hbase-server generated 19 new + 1 unchanged - 0 fixed =
20 total (was 1) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server generated 1 new + 1 unchanged - 0 fixed = 2
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 32s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 37s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {colo
[jira] [Updated] (HBASE-16578) Mob data loss after mob compaction and normal compaction
[
https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-16578:
-
Summary: Mob data loss after mob compaction and normal compaction (was:
Mob data loss after mob compaction and normal compcation)
> Mob data loss after mob compaction and normal compaction
>
>
> Key: HBASE-16578
> URL: https://issues.apache.org/jira/browse/HBASE-16578
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Jingcheng Du
> Attachments: HBASE-16578-V2.patch, HBASE-16578-V3.patch,
> HBASE-16578.patch, TestMobCompaction.java, TestMobCompaction.java
>
>
> StoreFileScanners on MOB cells rely on the scannerOrder to find the latest
> cells after mob compaction. The value of scannerOrder is assigned by the
> order of maxSeqId of StoreFile, and this maxSeqId is valued only after the
> reader of the StoreFile is created.
> In {{Compactor.compact}}, the compacted store files are cloned and their
> readers are not created. And in {{StoreFileScanner.getScannersForStoreFiles}}
> the StoreFiles are sorted before the readers are created and at that time the
> maxSeqId for each file is -1 (the default value). This will lead to a chaos
> in scanners in the following normal compaction. Some older cells might be
> chosen during the normal compaction.
> We need to create readers either before the sorting in the method
> {{StoreFileScanner.getScannersForStoreFiles}}, or create readers just after
> the store files are cloned in {{Compactor.compact}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[ https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587767#comment-15587767 ] Hiroshi Ikeda commented on HBASE-10656: --- - The Counter is not intended to be frequently created and discarded into garbage. There seems many abuses :P - The javadoc of sun.misc.Contended in Java 8 says: "The effects of this annotation will nearly always add significant space overhead to objects." It seems VM implementations will automatically add pads for you and abusing LongAdder will still cause the same memory issue. - As I wrote in HBASE-16616, it would be good to change indexHolderThreadLocal to be static, not only to avoid frequently calling ThreadLocalMap.expungeStaleEntry, also to avoid frequently calling ThreadLocalMap.getEntryAfterMiss, which seems the cause of HBASE-16146 according to comments, but still counters require memory overhead and should not be abused. - Revert HBASE-16146 or it would be better to replace all counters with AtomicLong. Cache line contention is difficult to be detected on the language level and we should stretch a net to catch the sign by chance through many cache line contentions occur, even though the degradation is visible, and it wastes to use the sign just once, throw away, and continue to suffer further contentions. - I don't like to remove pads and pray :P > high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug > > > Key: HBASE-10656 > URL: https://issues.apache.org/jira/browse/HBASE-10656 > Project: HBase > Issue Type: Bug >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt, > HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch, > MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java, > MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java, > output.pdf, output.txt, output2.pdf, output2.txt > > > Cliff's high-scale-lib's Counter is used in important classes (for example, > HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation > detail of the Java standard library and belongs to Oracle (Sun). That > consequently makes HBase depend on the specific JRE Implementation. > To make matters worse, Counter has a bug and you may get wrong result if you > mix a reading method into your logic calling writing methods. > In more detail, I think the bug is caused by reading an internal array field > without resolving memory caching, which is intentional the comment says, but > storing the read result into a volatile field. That field may be not changed > after you can see the true values of the array field, and also may be not > changed after updating the "next" CAT instance's values in some race > condition when extending CAT instance chain. > Anyway, it is possible that you create a new alternative class which only > depends on the standard library. I know Java8 provides its alternative, but > HBase should support Java6 and Java7 for some time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587765#comment-15587765 ] Hudson commented on HBASE-16824: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #51 (See [https://builds.apache.org/job/HBase-1.3-JDK8/51/]) HBASE-16824 Writer.flush() can be called on already closed streams in (enis: rev c51722629418b8b5e3a6e688219ee7d806f251c7) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587763#comment-15587763 ] Hudson commented on HBASE-16824: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #45 (See [https://builds.apache.org/job/HBase-1.3-JDK7/45/]) HBASE-16824 Writer.flush() can be called on already closed streams in (enis: rev c51722629418b8b5e3a6e688219ee7d806f251c7) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16872) Implement mutateRow and checkAndMutate
[ https://issues.apache.org/jira/browse/HBASE-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587749#comment-15587749 ] Duo Zhang commented on HBASE-16872: --- Ah there is a typo in the comment. "We need to the MultiRequest" => "We need the MultiRequest" Will fix on commit. > Implement mutateRow and checkAndMutate > -- > > Key: HBASE-16872 > URL: https://issues.apache.org/jira/browse/HBASE-16872 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16872.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16872) Implement mutateRow and checkAndMutate
[ https://issues.apache.org/jira/browse/HBASE-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16872: -- Attachment: HBASE-16872.patch > Implement mutateRow and checkAndMutate > -- > > Key: HBASE-16872 > URL: https://issues.apache.org/jira/browse/HBASE-16872 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16872.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16872) Implement mutateRow and checkAndMutate
[ https://issues.apache.org/jira/browse/HBASE-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16872: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > Implement mutateRow and checkAndMutate > -- > > Key: HBASE-16872 > URL: https://issues.apache.org/jira/browse/HBASE-16872 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16872.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16837) Implement checkAndPut and checkAndDelete
[ https://issues.apache.org/jira/browse/HBASE-16837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16837: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~carp84] for reviewing. > Implement checkAndPut and checkAndDelete > > > Key: HBASE-16837 > URL: https://issues.apache.org/jira/browse/HBASE-16837 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16837.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16837) Implement checkAndPut and checkAndDelete
[ https://issues.apache.org/jira/browse/HBASE-16837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587728#comment-15587728 ] Yu Li commented on HBASE-16837: --- +1, patch lgtm. Also checked failed UT HadoopQA reported and confirmed all could pass w/ patch locally. > Implement checkAndPut and checkAndDelete > > > Key: HBASE-16837 > URL: https://issues.apache.org/jira/browse/HBASE-16837 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16837.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587724#comment-15587724 ] Hudson commented on HBASE-16824: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #43 (See [https://builds.apache.org/job/HBase-1.2-JDK8/43/]) HBASE-16824 Writer.flush() can be called on already closed streams in (enis: rev 57181442577c36689114334b011a6e72de4ae785) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16864) Procedure v2 - Fix StateMachineProcedure support for child procs at last step
[
https://issues.apache.org/jira/browse/HBASE-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587620#comment-15587620
]
Hadoop QA commented on HBASE-16864:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 56s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 17s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 48s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834092/HBASE-16864-v1.patch |
| JIRA Issue | HBASE-16864 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 49a0ffea4e59 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ef8c65e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4095/testReport/ |
| modules | C: hbase-procedure U: hbase-procedure |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4095/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Procedure v2 - Fix StateMachineProcedure support for child procs at last step
> -
>
> Key: HBASE-16864
> URL: https://issues.apache.org/jira/browse/HBASE-16864
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>
[jira] [Commented] (HBASE-16864) Procedure v2 - Fix StateMachineProcedure support for child procs at last step
[ https://issues.apache.org/jira/browse/HBASE-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587564#comment-15587564 ] Matteo Bertozzi commented on HBASE-16864: - sure, open a jira and provide a patch. > Procedure v2 - Fix StateMachineProcedure support for child procs at last step > - > > Key: HBASE-16864 > URL: https://issues.apache.org/jira/browse/HBASE-16864 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16864-v0.patch, HBASE-16864-v1.patch > > > There is a bug in the StateMachineProcedure when we add child procs in the > last step. On recovery we end up spinning on the last step without ever > completing. the fix is to introduce an eof step so recovery knows that we are > already done once the all the children are terminated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16864) Procedure v2 - Fix StateMachineProcedure support for child procs at last step
[ https://issues.apache.org/jira/browse/HBASE-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587558#comment-15587558 ] Matteo Bertozzi commented on HBASE-16864: - MasterProcedureTestingUtility.testRecoveryAndDoubleExecution() should stay because Master procedures will need a different restart operation as soon we have the new AM, that's why I added the Runnable customRestart argument. so we can use this as base and have the custom restart in the master. there will be another patch for that. > Procedure v2 - Fix StateMachineProcedure support for child procs at last step > - > > Key: HBASE-16864 > URL: https://issues.apache.org/jira/browse/HBASE-16864 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16864-v0.patch, HBASE-16864-v1.patch > > > There is a bug in the StateMachineProcedure when we add child procs in the > last step. On recovery we end up spinning on the last step without ever > completing. the fix is to introduce an eof step so recovery knows that we are > already done once the all the children are terminated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16864) Procedure v2 - Fix StateMachineProcedure support for child procs at last step
[ https://issues.apache.org/jira/browse/HBASE-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16864: Attachment: HBASE-16864-v1.patch > Procedure v2 - Fix StateMachineProcedure support for child procs at last step > - > > Key: HBASE-16864 > URL: https://issues.apache.org/jira/browse/HBASE-16864 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16864-v0.patch, HBASE-16864-v1.patch > > > There is a bug in the StateMachineProcedure when we add child procs in the > last step. On recovery we end up spinning on the last step without ever > completing. the fix is to introduce an eof step so recovery knows that we are > already done once the all the children are terminated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587546#comment-15587546 ] Phil Yang commented on HBASE-15302: --- What about TestWALSplitCompressed, will you also pull it back? > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.1 > > Attachments: HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt, > HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587538#comment-15587538 ] Gary Helmling commented on HBASE-15302: --- Thanks, but please exclude TestWALSplit. I'm pulling that back in as part of HBASE-16754. > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.1 > > Attachments: HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt, > HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587527#comment-15587527 ] Phil Yang commented on HBASE-15302: --- Oh, I missed the reverting. Let me put these three tests back. > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.1 > > Attachments: HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt, > HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587510#comment-15587510 ] Phil Yang commented on HBASE-16388: --- Yes, this is a client-only change covering all rpc requests while AP's limit only covering part of operations in HTable. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16388-branch-1-v1.patch, > HBASE-16388-branch-1-v2.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v3.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15532) core favored nodes enhancements
[ https://issues.apache.org/jira/browse/HBASE-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-15532: - Fix Version/s: 2.0.0 Status: Patch Available (was: Open) Submitting patch for precommit tests to run. > core favored nodes enhancements > --- > > Key: HBASE-15532 > URL: https://issues.apache.org/jira/browse/HBASE-15532 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-15532.master.000.patch, > HBASE-15532.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-16824: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to 1.1+. > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15532) core favored nodes enhancements
[ https://issues.apache.org/jira/browse/HBASE-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-15532: - Attachment: HBASE-15532.master.001.patch Attaching rebased patch. > core favored nodes enhancements > --- > > Key: HBASE-15532 > URL: https://issues.apache.org/jira/browse/HBASE-15532 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Thiruvel Thirumoolan > Attachments: HBASE-15532.master.000.patch, > HBASE-15532.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587406#comment-15587406 ] Ted Yu commented on HBASE-16845: [~apurtell]: Can you give the full command involving -DskipITs ? > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt, 16845.v2.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16876) RatioBasedCompactionPolicy ignores mayBeStuck
[
https://issues.apache.org/jira/browse/HBASE-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587347#comment-15587347
]
Allan Yang commented on HBASE-16876:
It seems like you are right. But we use the derived class
{{ExploringCompactionPolicy}} for default, so no one has found out that.
> RatioBasedCompactionPolicy ignores mayBeStuck
> -
>
> Key: HBASE-16876
> URL: https://issues.apache.org/jira/browse/HBASE-16876
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Neal Young
> Labels: newbie
>
> I'm a newbie so may not be reporting this bug correctly. The bug currently
> shows in lines 181 - 190 here :
> http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.html#181
> .
> {code:title=Bar.java|borderStyle=solid}
> 176 while (countOfFiles - start >= comConf.getMinFilesToCompact() &&
> 177 fileSizes[start] > Math.max(comConf.getMinCompactSize(),
> 178 (long) (sumSize[start + 1] * ratio))) {
> 179 ++start;
> 180 }
> 181 if (start < countOfFiles) {
> 182 LOG.info("Default compaction algorithm has selected " +
> (countOfFiles - start)
> 183 + " files from " + countOfFiles + " candidates");
> 184 } else if (mayBeStuck) {
> 185 // We may be stuck. Compact the latest files if we can.
> 186 int filesToLeave = candidates.size() -
> comConf.getMinFilesToCompact();
> 187 if (filesToLeave >= 0) {
> 188 start = filesToLeave;
> 189 }
> 190 }
> {code}
> On reaching line 176, start = 0. When comConf.getMinFilesToCompact() is at
> least 2 (as occurs in the default), the while loop is guaranteed to terminate
> with start < countOfFiles. Hence, the else clause starting at line 184 never
> executes, regardless of the value of mayBeStuck.
> Perhaps the following code would be better, but I'm not sure:
> {code:title=Bar.java|borderStyle=solid}
> while (start < countOfFiles
>&& fileSizes[start] > Math.max(comConf.getMinCompactSize(),
> (long) (sumSize[start + 1] *
> ratio))) {
> ++start;
> }
> if (countOfFiles - start < comConf.getMinFilesToCompact()) {
> if (mayBeStuck && countOfFiles >= comConf.getMinFilesToCompact()) {
> start = countOfFiles - comConf.getMinFilesToCompact();
> } else {
> start = countOfFiles;
> }
> }
> if (countOfFiles - start > 0) {
> LOG.info("Default compaction algorithm has selected " + (countOfFiles
> - start)
> + " files from " + countOfFiles + " candidates");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16864) Procedure v2 - Fix StateMachineProcedure support for child procs at last step
[
https://issues.apache.org/jira/browse/HBASE-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587329#comment-15587329
]
Appy commented on HBASE-16864:
--
- Can we remove
{{MasterProcedureTestingUtility.testRecoveryAndDoubleExecution(procExec,
procId)}} in favor of
{{ProcedureTestingUtillity.testRecoveryAndDoubleExecution(procExec, procId)}}
since they are same.
- Since {{i}} is not used in stopping condition {{for (int i = 0;
!procExec.isFinished(procId); ++i)}}, while loop might be more appropriate
{{int counter = 0; while(!procExec.isFinished(procId)) \{ counter++;
\}}}
- {{if (!(this instanceof Exception)) return false;}} this-->other
At first i thought i thought was wasn't execCount 4 for double execution, then
looking into code I realized the toggle thing and how it fails after every
step. That's a pretty nifty testing hook.
Overall looks fine.
> Procedure v2 - Fix StateMachineProcedure support for child procs at last step
> -
>
> Key: HBASE-16864
> URL: https://issues.apache.org/jira/browse/HBASE-16864
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-16864-v0.patch
>
>
> There is a bug in the StateMachineProcedure when we add child procs in the
> last step. On recovery we end up spinning on the last step without ever
> completing. the fix is to introduce an eof step so recovery knows that we are
> already done once the all the children are terminated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16837) Implement checkAndPut and checkAndDelete
[ https://issues.apache.org/jira/browse/HBASE-16837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587298#comment-15587298 ] Duo Zhang commented on HBASE-16837: --- [~carp84] [~stack] PTAL. Thanks. > Implement checkAndPut and checkAndDelete > > > Key: HBASE-16837 > URL: https://issues.apache.org/jira/browse/HBASE-16837 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16837.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16733) add hadoop 3.0.0-alpha1 to precommit checks
[ https://issues.apache.org/jira/browse/HBASE-16733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587295#comment-15587295 ] Duo Zhang commented on HBASE-16733: --- Fine. Let me setup a discussion on the dev list. > add hadoop 3.0.0-alpha1 to precommit checks > --- > > Key: HBASE-16733 > URL: https://issues.apache.org/jira/browse/HBASE-16733 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-16733.v0.patch > > > Been working on getting hadoop3 related build up and running and woudl ike to > add a precommit check so that new commits don't break the mvn compile/install. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587287#comment-15587287
]
Guanghao Zhang commented on HBASE-16815:
[~mbertozzi] Any ideas? Thanks.
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Attachments: HBASE-16815-branch-1.2.patch, HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one read
> handler/queue to begin with. Otherwise the scan handler/queue should be zero
> and share the one read handler/queue.
> Makes sense?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15816) Provide client with ability to set priority on Operations
[
https://issues.apache.org/jira/browse/HBASE-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587235#comment-15587235
]
Hadoop QA commented on HBASE-15816:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-15816 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12803956/HBASE-15816-v1.patch |
| JIRA Issue | HBASE-15816 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4093/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Provide client with ability to set priority on Operations
> --
>
> Key: HBASE-15816
> URL: https://issues.apache.org/jira/browse/HBASE-15816
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: churro morales
>Assignee: churro morales
> Attachments: HBASE-15816-v1.patch, HBASE-15816.patch
>
>
> First round will just be to expose the ability to set priorities for client
> operations. For more background:
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201604.mbox/%3CCA+RK=_BG_o=q8HMptcP2WauAinmEsL+15f3YEJuz=qbpcya...@mail.gmail.com%3E
> Next step would be to remove AnnotationReadingPriorityFunction and have the
> client send priorities explicitly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15816) Provide client with ability to set priority on Operations
[ https://issues.apache.org/jira/browse/HBASE-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587211#comment-15587211 ] Enis Soztutar commented on HBASE-15816: --- [~churromorales] are you still interested in pushing for this? The master has changed a bit and the patch needs a rebase. I can pick up the work if you have no cycles. > Provide client with ability to set priority on Operations > -- > > Key: HBASE-15816 > URL: https://issues.apache.org/jira/browse/HBASE-15816 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: churro morales >Assignee: churro morales > Attachments: HBASE-15816-v1.patch, HBASE-15816.patch > > > First round will just be to expose the ability to set priorities for client > operations. For more background: > http://mail-archives.apache.org/mod_mbox/hbase-dev/201604.mbox/%3CCA+RK=_BG_o=q8HMptcP2WauAinmEsL+15f3YEJuz=qbpcya...@mail.gmail.com%3E > Next step would be to remove AnnotationReadingPriorityFunction and have the > client send priorities explicitly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[ https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587208#comment-15587208 ] Misha Dmitriev commented on HBASE-10656: Correct - no user activity. Though you may be right that the monitoring tool does something inside HBase. > high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug > > > Key: HBASE-10656 > URL: https://issues.apache.org/jira/browse/HBASE-10656 > Project: HBase > Issue Type: Bug >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt, > HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch, > MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java, > MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java, > output.pdf, output.txt, output2.pdf, output2.txt > > > Cliff's high-scale-lib's Counter is used in important classes (for example, > HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation > detail of the Java standard library and belongs to Oracle (Sun). That > consequently makes HBase depend on the specific JRE Implementation. > To make matters worse, Counter has a bug and you may get wrong result if you > mix a reading method into your logic calling writing methods. > In more detail, I think the bug is caused by reading an internal array field > without resolving memory caching, which is intentional the comment says, but > storing the read result into a volatile field. That field may be not changed > after you can see the true values of the array field, and also may be not > changed after updating the "next" CAT instance's values in some race > condition when extending CAT instance chain. > Anyway, it is possible that you create a new alternative class which only > depends on the standard library. I know Java8 provides its alternative, but > HBase should support Java6 and Java7 for some time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587206#comment-15587206 ] Vladimir Rodionov commented on HBASE-16827: --- Yes > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch > > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[ https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587191#comment-15587191 ] stack commented on HBASE-10656: --- [~mi...@cloudera.com] There was no load on the cluster, right? > high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug > > > Key: HBASE-10656 > URL: https://issues.apache.org/jira/browse/HBASE-10656 > Project: HBase > Issue Type: Bug >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt, > HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch, > MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java, > MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java, > output.pdf, output.txt, output2.pdf, output2.txt > > > Cliff's high-scale-lib's Counter is used in important classes (for example, > HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation > detail of the Java standard library and belongs to Oracle (Sun). That > consequently makes HBase depend on the specific JRE Implementation. > To make matters worse, Counter has a bug and you may get wrong result if you > mix a reading method into your logic calling writing methods. > In more detail, I think the bug is caused by reading an internal array field > without resolving memory caching, which is intentional the comment says, but > storing the read result into a volatile field. That field may be not changed > after you can see the true values of the array field, and also may be not > changed after updating the "next" CAT instance's values in some race > condition when extending CAT instance chain. > Anyway, it is possible that you create a new alternative class which only > depends on the standard library. I know Java8 provides its alternative, but > HBase should support Java6 and Java7 for some time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587153#comment-15587153 ] stack commented on HBASE-16824: --- +1 from me. > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[
https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587146#comment-15587146
]
Misha Dmitriev edited comment on HBASE-10656 at 10/19/16 12:23 AM:
---
To be more specific, our RegionServers end up with millions of Counter$Cell
objects in memory:
{code}
#instances Shallow size #instancesShallow size Class name
garbage garbage live live
2,985,951 396,571K (32.8%) 766,919101,856K (8.4%)
org.apache.hadoop.hbase.util.Counter$Cell
2,985,949 69,983K (5.8%) 766,918 17,974K (1.5%)
org.apache.hadoop.hbase.util.Counter$Cell[]
{code}
I think there is no point in talking about optimizations where we forcefully
prevent two Cell objects from sharing a single cache line where there are so
many of them that they just cause memory blowup.
A simple way of solving this problem would be to just remove the extra padding
long fields. However, I am totally new to HBase and don't know whether a large
number of these objects is always the case. Maybe in some setups there are very
few of them. In that case, maybe it would make sense to have two alternative
implementations of Cell:
- one that assumes a small number of objects and optimized for cache speed, as
now
- another that's just as compact as possible.
was (Author: mi...@cloudera.com):
To be more specific, our RegionServers end up with millions of Counter$Cell
objects in memory:
{code}
#instances Shallow size #instancesShallow size Class name
garbage garbage live live
2,985,951 396,571K (32.8%) 766,919101,856K (8.4%)
org.apache.hadoop.hbase.util.Counter$Cell
2,985,949 69,983K (5.8%) 766,918 17,974K (1.5%)
org.apache.hadoop.hbase.util.Counter$Cell[]
{/code}
I think there is no point in talking about optimizations where we forcefully
prevent two Cell objects from sharing a single cache line where there are so
many of them that they just cause memory blowup.
A simple way of solving this problem would be to just remove the extra padding
long fields. However, I am totally new to HBase and don't know whether a large
number of these objects is always the case. Maybe in some setups there are very
few of them. In that case, maybe it would make sense to have two alternative
implementations of Cell:
- one that assumes a small number of objects and optimized for cache speed, as
now
- another that's just as compact as possible.
> high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
>
>
> Key: HBASE-10656
> URL: https://issues.apache.org/jira/browse/HBASE-10656
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Fix For: 0.96.2, 0.98.1, 0.99.0
>
> Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt,
> HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch,
> MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java,
> MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java,
> output.pdf, output.txt, output2.pdf, output2.txt
>
>
> Cliff's high-scale-lib's Counter is used in important classes (for example,
> HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation
> detail of the Java standard library and belongs to Oracle (Sun). That
> consequently makes HBase depend on the specific JRE Implementation.
> To make matters worse, Counter has a bug and you may get wrong result if you
> mix a reading method into your logic calling writing methods.
> In more detail, I think the bug is caused by reading an internal array field
> without resolving memory caching, which is intentional the comment says, but
> storing the read result into a volatile field. That field may be not changed
> after you can see the true values of the array field, and also may be not
> changed after updating the "next" CAT instance's values in some race
> condition when extending CAT instance chain.
> Anyway, it is possible that you create a new alternative class which only
> depends on the standard library. I know Java8 provides its alternative, but
> HBase should support Java6 and Java7 for some time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[
https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587146#comment-15587146
]
Misha Dmitriev commented on HBASE-10656:
To be more specific, our RegionServers end up with millions of Counter$Cell
objects in memory:
{code}
#instances Shallow size #instancesShallow size Class name
garbage garbage live live
2,985,951 396,571K (32.8%) 766,919101,856K (8.4%)
org.apache.hadoop.hbase.util.Counter$Cell
2,985,949 69,983K (5.8%) 766,918 17,974K (1.5%)
org.apache.hadoop.hbase.util.Counter$Cell[]
{/code}
I think there is no point in talking about optimizations where we forcefully
prevent two Cell objects from sharing a single cache line where there are so
many of them that they just cause memory blowup.
A simple way of solving this problem would be to just remove the extra padding
long fields. However, I am totally new to HBase and don't know whether a large
number of these objects is always the case. Maybe in some setups there are very
few of them. In that case, maybe it would make sense to have two alternative
implementations of Cell:
- one that assumes a small number of objects and optimized for cache speed, as
now
- another that's just as compact as possible.
> high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
>
>
> Key: HBASE-10656
> URL: https://issues.apache.org/jira/browse/HBASE-10656
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Fix For: 0.96.2, 0.98.1, 0.99.0
>
> Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt,
> HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch,
> MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java,
> MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java,
> output.pdf, output.txt, output2.pdf, output2.txt
>
>
> Cliff's high-scale-lib's Counter is used in important classes (for example,
> HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation
> detail of the Java standard library and belongs to Oracle (Sun). That
> consequently makes HBase depend on the specific JRE Implementation.
> To make matters worse, Counter has a bug and you may get wrong result if you
> mix a reading method into your logic calling writing methods.
> In more detail, I think the bug is caused by reading an internal array field
> without resolving memory caching, which is intentional the comment says, but
> storing the read result into a volatile field. That field may be not changed
> after you can see the true values of the array field, and also may be not
> changed after updating the "next" CAT instance's values in some race
> condition when extending CAT instance chain.
> Anyway, it is possible that you create a new alternative class which only
> depends on the standard library. I know Java8 provides its alternative, but
> HBase should support Java6 and Java7 for some time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-16879) Add ThreadPool in Legacy implementations of MasterStorage/ RegionStorage
[ https://issues.apache.org/jira/browse/HBASE-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe reassigned HBASE-16879: Assignee: Umesh Agashe > Add ThreadPool in Legacy implementations of MasterStorage/ RegionStorage > > > Key: HBASE-16879 > URL: https://issues.apache.org/jira/browse/HBASE-16879 > Project: HBase > Issue Type: Sub-task >Reporter: Umesh Agashe >Assignee: Umesh Agashe > > Certain bulk operations can be and needs to be optimized by doing them in > parallel with ThreadPool (e.g. ModifyRegionUtils.createRegions()). This is FS > dependent and may not be required for other implementations. Instead of > instantiating a ThreadPool per request we can have single instance of > StorageThreadPool and support bulk operations. > Also add APIs for bulk operations: createRegions(), deleteRegions(), archive > regions() etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587128#comment-15587128 ] Enis Soztutar commented on HBASE-16824: --- I'll commit this assuming the +1 stands. The test failures does not seem to be related. > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587119#comment-15587119 ] Ted Yu commented on HBASE-16827: Have you run all backup / restore tests ? > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch > > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[ https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587121#comment-15587121 ] Enis Soztutar commented on HBASE-10656: --- Great. Looked at the code for Striped64, it uses Contended annotation though which is Java-8 only AFAIK. So we have have to do the same trickery with our own padding, thus suffering from the same GC problem above. > high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug > > > Key: HBASE-10656 > URL: https://issues.apache.org/jira/browse/HBASE-10656 > Project: HBase > Issue Type: Bug >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt, > HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch, > MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java, > MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java, > output.pdf, output.txt, output2.pdf, output2.txt > > > Cliff's high-scale-lib's Counter is used in important classes (for example, > HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation > detail of the Java standard library and belongs to Oracle (Sun). That > consequently makes HBase depend on the specific JRE Implementation. > To make matters worse, Counter has a bug and you may get wrong result if you > mix a reading method into your logic calling writing methods. > In more detail, I think the bug is caused by reading an internal array field > without resolving memory caching, which is intentional the comment says, but > storing the read result into a volatile field. That field may be not changed > after you can see the true values of the array field, and also may be not > changed after updating the "next" CAT instance's values in some race > condition when extending CAT instance chain. > Anyway, it is possible that you create a new alternative class which only > depends on the standard library. I know Java8 provides its alternative, but > HBase should support Java6 and Java7 for some time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15905) Makefile build env incorrectly links in tests
[ https://issues.apache.org/jira/browse/HBASE-15905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587120#comment-15587120 ] Sudeep Sunthankar commented on HBASE-15905: --- Hi [~xiaobingo], Forgot to add a point here. I will submit an addon patch to address point 1 raised by Elliott. I still didn't get point 2. -- Thanks > Makefile build env incorrectly links in tests > - > > Key: HBASE-15905 > URL: https://issues.apache.org/jira/browse/HBASE-15905 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Priyadharshini karthikeyan > Attachments: HBASE-15905.HBASE-14850.v1.patch > > > Right now the makefile build system doesn't seem to do so well. > * Tests are included in the lib > * Documentation includes the protobuf dir > * just running make on a clean check out fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[
https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587115#comment-15587115
]
Hadoop QA commented on HBASE-16827:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-16827 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834067/HBASE-16827-v1.patch |
| JIRA Issue | HBASE-16827 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4092/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Backup.proto refactoring
>
>
> Key: HBASE-16827
> URL: https://issues.apache.org/jira/browse/HBASE-16827
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16827-v1.patch
>
>
> Make server_name type ServerName (from HBase.proto)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15905) Makefile build env incorrectly links in tests
[ https://issues.apache.org/jira/browse/HBASE-15905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587111#comment-15587111 ] Sudeep Sunthankar commented on HBASE-15905: --- Hi [~xiaobingo], This patch addresses 2 issues:- 1) Protobuf object files were not included in the release version of the library. 2) As per point 3 running make on a clean check out works properly. -- Thanks > Makefile build env incorrectly links in tests > - > > Key: HBASE-15905 > URL: https://issues.apache.org/jira/browse/HBASE-15905 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Priyadharshini karthikeyan > Attachments: HBASE-15905.HBASE-14850.v1.patch > > > Right now the makefile build system doesn't seem to do so well. > * Tests are included in the lib > * Documentation includes the protobuf dir > * just running make on a clean check out fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16827: -- Status: Patch Available (was: Open) > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch > > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16879) Add ThreadPool in Legacy implementations of MasterStorage/ RegionStorage
Umesh Agashe created HBASE-16879: Summary: Add ThreadPool in Legacy implementations of MasterStorage/ RegionStorage Key: HBASE-16879 URL: https://issues.apache.org/jira/browse/HBASE-16879 Project: HBase Issue Type: Sub-task Reporter: Umesh Agashe Certain bulk operations can be and needs to be optimized by doing them in parallel with ThreadPool (e.g. ModifyRegionUtils.createRegions()). This is FS dependent and may not be required for other implementations. Instead of instantiating a ThreadPool per request we can have single instance of StorageThreadPool and support bulk operations. Also add APIs for bulk operations: createRegions(), deleteRegions(), archive regions() etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16827: -- Attachment: HBASE-16827-v1.patch v1. cc: [~tedyu]. > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch > > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16827: -- Summary: Backup.proto refactoring (was: Backup.proto small refactoring) > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587055#comment-15587055
]
Matteo Bertozzi commented on HBASE-16874:
-
some of the tests in hbase-procedure are running testing concurrency, and all
the hbase tests calling admin.xyz() have a multi-threaded executor. The only
ones with threads=1 are the ones where we force kill/restart every step
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16878) Call Queue length enforced not accounting for the queue handler factor
[ https://issues.apache.org/jira/browse/HBASE-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deepankar updated HBASE-16878: -- Attachment: HBASE-16878.patch > Call Queue length enforced not accounting for the queue handler factor > -- > > Key: HBASE-16878 > URL: https://issues.apache.org/jira/browse/HBASE-16878 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.0.0, 0.98.4 >Reporter: deepankar >Assignee: deepankar >Priority: Minor > Attachments: HBASE-16878.patch > > > After HBASE-11355 currently we are not accounting for the handler factor when > deciding callQueue length, this leading to some pretty large queue sizes if > the handler factor is high enough. > Patch attached, change is oneline -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[
https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587043#comment-15587043
]
Hadoop QA commented on HBASE-16345:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 37s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 39s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 19s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 11s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.TestMultiVersions |
| | org.apache.hadoop.hbase.constraint.TestConstraint |
| | org.apache.hadoop.hbase.TestNamespace |
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834038/HBASE-16345.master.006.patch
|
| JIRA Issue | HBASE-16345 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f5f3724bea20 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6c89c62 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Buil
[jira] [Created] (HBASE-16878) Call Queue length enforced not accounting for the queue handler factor
deepankar created HBASE-16878: - Summary: Call Queue length enforced not accounting for the queue handler factor Key: HBASE-16878 URL: https://issues.apache.org/jira/browse/HBASE-16878 Project: HBase Issue Type: Bug Components: rpc Affects Versions: 0.98.4, 1.0.0 Reporter: deepankar Assignee: deepankar Priority: Minor After HBASE-11355 currently we are not accounting for the handler factor when deciding callQueue length, this leading to some pretty large queue sizes if the handler factor is high enough. Patch attached, change is oneline -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16866) Avoid NPE in AsyncRequestFutureImpl#updateStats
[
https://issues.apache.org/jira/browse/HBASE-16866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587031#comment-15587031
]
Hudson commented on HBASE-16866:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1811 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1811/])
HBASE-16866 Avoid NPE in AsyncRequestFutureImpl#updateStats (ChiaPing (tedyu:
rev 6c89c6251ff611c2f10bfd6f8c9f8a2d717dc71b)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java
> Avoid NPE in AsyncRequestFutureImpl#updateStats
> ---
>
> Key: HBASE-16866
> URL: https://issues.apache.org/jira/browse/HBASE-16866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16866.v0.patch, HBASE-16866.v1.patch
>
>
> If region disables the stats, it won’t response any
> ClientProtos.RegionLoadStats to client. So the NPE will happen in
> AsyncRequestFutureImpl#updateStats.
> We should use relevant log instead of NPE because the data manipulation
> shouldn’t be broken by statistics.
> {noformat}
> Â protected void updateStats(ServerName server, Map MultiResponse.RegionResult> results) {
>    …
> Â Â Â ClientProtos.RegionLoadStats stat = regionStats.getValue().getStat();
> Â Â Â RegionLoadStats regionLoadstats =
> ProtobufUtil.createRegionLoadStats(stat);
>    …
> Â }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587024#comment-15587024 ] Andrew Purtell commented on HBASE-16845: I have been using -skipITs to skip integration tests forever, and it works. > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt, 16845.v2.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16862) Remove directory layout/ filesystem references form the code in master/procedure directory
[
https://issues.apache.org/jira/browse/HBASE-16862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587017#comment-15587017
]
Hadoop QA commented on HBASE-16862:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
9s {color} | {color:green} hbase-14439 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} hbase-14439 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s {color} | {color:green} hbase-14439 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} hbase-14439 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s
{color} | {color:red} hbase-server in hbase-14439 has 5 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} hbase-14439 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 13s
{color} | {color:red} hbase-server generated 4 new + 5 unchanged - 0 fixed = 9
total (was 5) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} hbase-server generated 0 new + 11 unchanged - 1 fixed =
11 total (was 12) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 34s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Dead store to ms in
org.apache.hadoop.hbase.master.procedure.CloneSnapshotProcedure$1.createRegionsOnStorage(MasterProcedureEnv,
TableName, List) At
CloneSnapshotProcedure.java:org.apache.hadoop.hbase.master.procedure.CloneSnapshotProcedure$1.createRegionsOnStorage(MasterProcedureEnv,
TableName, List) At CloneSnapshotProcedure.java:[line 350] |
| | Dead store to ms in
org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.prepareRestore(MasterProcedureEnv)
At
RestoreSnapshotProcedure.java:org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.prepareRestore(MasterProcedureEnv)
At RestoreSnapshotProcedure.java:[line 335] |
| | Dead store to ms in
org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(MasterProcedureEnv)
At
RestoreSnapshotProcedure.java:org.apache.hadoop.hbase.master.procedure.RestoreSnapshotProcedure.restoreSnapshot(MasterProcedureEnv)
At RestoreSnapshotProcedure.java:[line 363] |
| | Dead store to confForWAL in
org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(Configuration,
HTableDescriptor, HRegionInfo, ModifyRegionUtils$RegionFillTask) At
ModifyRegionUtils.java:org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(Configuration,
HTableDescriptor, HRegionInfo, ModifyRegionUtils$Regi
[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587014#comment-15587014 ] Ted Yu commented on HBASE-16845: Is skipITs shorthand for skipIntegrationTests ? I didn't find skipITs in code base. Support for skipIntegrationTests is added in patch v2. > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt, 16845.v2.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587002#comment-15587002 ] Mikhail Antonov commented on HBASE-16388: - [~yangzhe1991] I good one, I missed that. Is it fair to say that the primary motivation for that is that global, per-region and per-server limits in AP are flawed since they only ever enforced on the write path (going through AP#submit() / buffered mutator)? With that being client-only change I'd consider backporting it to 1.3.. Anything that reduced blast radius from bad RS is an important reliability fix IMO. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16388-branch-1-v1.patch, > HBASE-16388-branch-1-v2.patch, HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v3.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[
https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586960#comment-15586960
]
Andrew Purtell commented on HBASE-10656:
Public domain, category A, provided we use the JSR166 sources not what was
donated to OpenJDK and relicensed as GPL 2 (category X)
http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co
{noformat}
/*
* Written by Doug Lea with assistance from members of JCP JSR-166
* Expert Group and released to the public domain, as explained at
* http://creativecommons.org/publicdomain/zero/1.0/
*/
{noformat}
> high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
>
>
> Key: HBASE-10656
> URL: https://issues.apache.org/jira/browse/HBASE-10656
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Fix For: 0.96.2, 0.98.1, 0.99.0
>
> Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt,
> HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch,
> MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java,
> MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java,
> output.pdf, output.txt, output2.pdf, output2.txt
>
>
> Cliff's high-scale-lib's Counter is used in important classes (for example,
> HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation
> detail of the Java standard library and belongs to Oracle (Sun). That
> consequently makes HBase depend on the specific JRE Implementation.
> To make matters worse, Counter has a bug and you may get wrong result if you
> mix a reading method into your logic calling writing methods.
> In more detail, I think the bug is caused by reading an internal array field
> without resolving memory caching, which is intentional the comment says, but
> storing the read result into a volatile field. That field may be not changed
> after you can see the true values of the array field, and also may be not
> changed after updating the "next" CAT instance's values in some race
> condition when extending CAT instance chain.
> Anyway, it is possible that you create a new alternative class which only
> depends on the standard library. I know Java8 provides its alternative, but
> HBase should support Java6 and Java7 for some time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15905) Makefile build env incorrectly links in tests
[ https://issues.apache.org/jira/browse/HBASE-15905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586950#comment-15586950 ] Xiaobing Zhou commented on HBASE-15905: --- I have the same question, can you comment how this patch is related to point 1 and 2? Thanks. > Makefile build env incorrectly links in tests > - > > Key: HBASE-15905 > URL: https://issues.apache.org/jira/browse/HBASE-15905 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Priyadharshini karthikeyan > Attachments: HBASE-15905.HBASE-14850.v1.patch > > > Right now the makefile build system doesn't seem to do so well. > * Tests are included in the lib > * Documentation includes the protobuf dir > * just running make on a clean check out fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase
[
https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586953#comment-15586953
]
Andrew Purtell commented on HBASE-16845:
Tested with {{-DskipTests}} - tests are skipped.
Can you also add support for the alias {{-DskipITs}} to skip integration tests?
This works elsewhere.
> Run tests under hbase-spark module in test phase
>
>
> Key: HBASE-16845
> URL: https://issues.apache.org/jira/browse/HBASE-16845
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16845.v1.txt, 16845.v2.txt
>
>
> [~appy] reported over in the discussion thread titled 'Testing and CI' that
> tests under hbase-spark module are not run by QA bot.
> This issue is to run the unit tests in test phase so that we detect build
> breakage early.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[ https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586946#comment-15586946 ] Enis Soztutar commented on HBASE-10656: --- bq. Master / 2.0 can use LongAdder, available in JRE 8 We already do so. bq. I don't have a ready answer for 1.x or 0.98 I was suggesting in HBASE-16146 that we can do a backport of LongAdder in HBase. I did not check the license implications though. > high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug > > > Key: HBASE-10656 > URL: https://issues.apache.org/jira/browse/HBASE-10656 > Project: HBase > Issue Type: Bug >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt, > HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch, > MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java, > MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java, > output.pdf, output.txt, output2.pdf, output2.txt > > > Cliff's high-scale-lib's Counter is used in important classes (for example, > HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation > detail of the Java standard library and belongs to Oracle (Sun). That > consequently makes HBase depend on the specific JRE Implementation. > To make matters worse, Counter has a bug and you may get wrong result if you > mix a reading method into your logic calling writing methods. > In more detail, I think the bug is caused by reading an internal array field > without resolving memory caching, which is intentional the comment says, but > storing the read result into a volatile field. That field may be not changed > after you can see the true values of the array field, and also may be not > changed after updating the "next" CAT instance's values in some race > condition when extending CAT instance chain. > Anyway, it is possible that you create a new alternative class which only > depends on the standard library. I know Java8 provides its alternative, but > HBase should support Java6 and Java7 for some time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12894) Upgrade Jetty to 9.2.6
[
https://issues.apache.org/jira/browse/HBASE-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586924#comment-15586924
]
Hadoop QA commented on HBASE-12894:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
10s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hbase-resource-bundle . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 39s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 7s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 6s
{color} | {color:red} hbase-thrift in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 5s
{color} | {color:red} hbase-rest in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 9s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m
52s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hbase-resource-bundle . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 1s
{color} | {color:red} hbase-server generated 1 new + 1 unchanged - 0 fixed = 2
total (was 1) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 45s
{color} | {color:red} hbase-rest generated 3 new + 0 unchanged - 0 fixed = 3
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 6s
{color} | {color:green} hbase-resource-bundle in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 16s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 37s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{co
[jira] [Updated] (HBASE-16876) RatioBasedCompactionPolicy ignores mayBeStuck
[
https://issues.apache.org/jira/browse/HBASE-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neal Young updated HBASE-16876:
---
Description:
I'm a newbie so may not be reporting this bug correctly. The bug currently
shows in lines 181 - 190 here :
http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.html#181
.
{code:title=Bar.java|borderStyle=solid}
176 while (countOfFiles - start >= comConf.getMinFilesToCompact() &&
177 fileSizes[start] > Math.max(comConf.getMinCompactSize(),
178 (long) (sumSize[start + 1] * ratio))) {
179 ++start;
180 }
181 if (start < countOfFiles) {
182 LOG.info("Default compaction algorithm has selected " + (countOfFiles
- start)
183 + " files from " + countOfFiles + " candidates");
184 } else if (mayBeStuck) {
185 // We may be stuck. Compact the latest files if we can.
186 int filesToLeave = candidates.size() - comConf.getMinFilesToCompact();
187 if (filesToLeave >= 0) {
188 start = filesToLeave;
189 }
190 }
{code}
On reaching line 176, start = 0. When comConf.getMinFilesToCompact() is at
least 2 (as occurs in the default), the while loop is guaranteed to terminate
with start < countOfFiles. Hence, the else clause starting at line 184 never
executes, regardless of the value of mayBeStuck.
Perhaps the following code would be better, but I'm not sure:
{code:title=Bar.java|borderStyle=solid}
while (start < countOfFiles
&& fileSizes[start] > Math.max(comConf.getMinCompactSize(),
(long) (sumSize[start + 1] * ratio)))
{
++start;
}
if (countOfFiles - start < comConf.getMinFilesToCompact()) {
if (mayBeStuck && countOfFiles >= comConf.getMinFilesToCompact()) {
start = countOfFiles - comConf.getMinFilesToCompact();
} else {
start = countOfFiles;
}
}
if (countOfFiles - start > 0) {
LOG.info("Default compaction algorithm has selected " + (countOfFiles -
start)
+ " files from " + countOfFiles + " candidates");
}
{code}
was:
I'm a newbie so may not be reporting this bug correctly. The bug currently
shows in lines 181 - 190 here :
http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.html#181
.
{code:title=Bar.java|borderStyle=solid}
176 while (countOfFiles - start >= comConf.getMinFilesToCompact() &&
177 fileSizes[start] > Math.max(comConf.getMinCompactSize(),
178 (long) (sumSize[start + 1] * ratio))) {
179 ++start;
180 }
181 if (start < countOfFiles) {
182 LOG.info("Default compaction algorithm has selected " + (countOfFiles
- start)
183 + " files from " + countOfFiles + " candidates");
184 } else if (mayBeStuck) {
185 // We may be stuck. Compact the latest files if we can.
186 int filesToLeave = candidates.size() - comConf.getMinFilesToCompact();
187 if (filesToLeave >= 0) {
188 start = filesToLeave;
189 }
190 }
{code}
On reaching line 176, start = 0. When comConf.getMinFilesToCompact() is at
least 2 (as occurs in the default), the while loop is guaranteed to terminate
with start < countOfFiles. Hence, the else clause starting at line 184 never
executes, regardless of the value of mayBeStuck.
Perhaps the following code would be better, but I'm not sure:
{code:title=Bar.java|borderStyle=solid}
while (start < countOfFiles
&& fileSizes[start] > Math.max(comConf.getMinCompactSize(),
(long) (sumSize[start + 1] * ratio)))
{
++start;
}
if (start < countOfFiles) {
if (countOfFiles - start >= comConf.getMinFilesToCompact()) {
LOG.info("Default compaction algorithm has selected " +
(countOfFiles - start)
+ " files from " + countOfFiles + " candidates");
} else {
start = countOfFiles;
}
} else if (mayBeStuck) {
// We may be stuck. Compact the latest files if we can.
int filesToLeave = candidates.size() - comConf.getMinFilesToCompact();
if (filesToLeave >= 0) {
start = filesToLeave;
}
}
{code}
> RatioBasedCompactionPolicy ignores mayBeStuck
> -
>
> Key: HBASE-16876
> URL: https://issues.apache.org/jira/browse/HBASE-16876
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Neal Young
> Labels: newbie
>
> I'm a newbie so may not be reporting this bug correctly. The bug currently
> shows in lines 181 - 190 here :
> http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.html#181
> .
> {c
[jira] [Commented] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[ https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586863#comment-15586863 ] Andrew Purtell commented on HBASE-10656: Master / 2.0 can use LongAdder, available in JRE 8: https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/LongAdder.html . I don't have a ready answer for 1.x or 0.98 (although the latter may be retired soon). > high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug > > > Key: HBASE-10656 > URL: https://issues.apache.org/jira/browse/HBASE-10656 > Project: HBase > Issue Type: Bug >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt, > HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch, > MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java, > MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java, > output.pdf, output.txt, output2.pdf, output2.txt > > > Cliff's high-scale-lib's Counter is used in important classes (for example, > HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation > detail of the Java standard library and belongs to Oracle (Sun). That > consequently makes HBase depend on the specific JRE Implementation. > To make matters worse, Counter has a bug and you may get wrong result if you > mix a reading method into your logic calling writing methods. > In more detail, I think the bug is caused by reading an internal array field > without resolving memory caching, which is intentional the comment says, but > storing the read result into a volatile field. That field may be not changed > after you can see the true values of the array field, and also may be not > changed after updating the "next" CAT instance's values in some race > condition when extending CAT instance chain. > Anyway, it is possible that you create a new alternative class which only > depends on the standard library. I know Java8 provides its alternative, but > HBase should support Java6 and Java7 for some time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16877) Remove directory layout/ filesystem references form Master
Umesh Agashe created HBASE-16877: Summary: Remove directory layout/ filesystem references form Master Key: HBASE-16877 URL: https://issues.apache.org/jira/browse/HBASE-16877 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Umesh Agashe Assignee: Umesh Agashe Remove directory layout/ filesystem references form Master and add required APIs to MasterStorage/ RegionStorage -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16876) RatioBasedCompactionPolicy ignores mayBeStuck
Neal Young created HBASE-16876:
--
Summary: RatioBasedCompactionPolicy ignores mayBeStuck
Key: HBASE-16876
URL: https://issues.apache.org/jira/browse/HBASE-16876
Project: HBase
Issue Type: Bug
Components: Compaction
Reporter: Neal Young
I'm a newbie so may not be reporting this bug correctly. The bug currently
shows in lines 181 - 190 here :
http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.html#181
.
{code:title=Bar.java|borderStyle=solid}
176 while (countOfFiles - start >= comConf.getMinFilesToCompact() &&
177 fileSizes[start] > Math.max(comConf.getMinCompactSize(),
178 (long) (sumSize[start + 1] * ratio))) {
179 ++start;
180 }
181 if (start < countOfFiles) {
182 LOG.info("Default compaction algorithm has selected " + (countOfFiles
- start)
183 + " files from " + countOfFiles + " candidates");
184 } else if (mayBeStuck) {
185 // We may be stuck. Compact the latest files if we can.
186 int filesToLeave = candidates.size() - comConf.getMinFilesToCompact();
187 if (filesToLeave >= 0) {
188 start = filesToLeave;
189 }
190 }
{code}
On reaching line 176, start = 0. When comConf.getMinFilesToCompact() is at
least 2 (as occurs in the default), the while loop is guaranteed to terminate
with start < countOfFiles. Hence, the else clause starting at line 184 never
executes, regardless of the value of mayBeStuck.
Perhaps the following code would be better, but I'm not sure:
{code:title=Bar.java|borderStyle=solid}
while (start < countOfFiles
&& fileSizes[start] > Math.max(comConf.getMinCompactSize(),
(long) (sumSize[start + 1] * ratio)))
{
++start;
}
if (start < countOfFiles) {
if (countOfFiles - start >= comConf.getMinFilesToCompact()) {
LOG.info("Default compaction algorithm has selected " +
(countOfFiles - start)
+ " files from " + countOfFiles + " candidates");
} else {
start = countOfFiles;
}
} else if (mayBeStuck) {
// We may be stuck. Compact the latest files if we can.
int filesToLeave = candidates.size() - comConf.getMinFilesToCompact();
if (filesToLeave >= 0) {
start = filesToLeave;
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586840#comment-15586840
]
stack commented on HBASE-16698:
---
[~carp84] points out above that WALPE doesn't exercise this patch AT ALL so
above runs were useless (would account for why the different so small). Ignore
the above. Dumb on my part.
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16775) Flakey test with TestExportSnapshot#testExportRetry and TestMobExportSnapshot#testExportRetry
[
https://issues.apache.org/jira/browse/HBASE-16775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586809#comment-15586809
]
Hadoop QA commented on HBASE-16775:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 4s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 37s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 115m 42s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 167m 4s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService
|
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes |
| | org.apache.hadoop.hbase.quotas.TestQuotaThrottle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.1 Server=1.12.1 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834016/HBASE-16775.master.003.patch
|
| JIRA Issue | HBASE-16775 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f5f4a5e36e73 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6c89c62 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4085/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4085/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4085/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4085/testReport/ |
| modules
[jira] [Updated] (HBASE-16862) Remove directory layout/ filesystem references form the code in master/procedure directory
[ https://issues.apache.org/jira/browse/HBASE-16862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-16862: - Attachment: HBASE-16862-hbase-14439.v1.patch Removed filesystem/ directory layout references from the code in master/procedure directory. Added required APIs in MasterStorage/ RegionStorage. > Remove directory layout/ filesystem references form the code in > master/procedure directory > -- > > Key: HBASE-16862 > URL: https://issues.apache.org/jira/browse/HBASE-16862 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Attachments: HBASE-16862-hbase-14439.v1.patch > > > Remove directory layout/ filesystem references form the code in > master/procedure directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16862) Remove directory layout/ filesystem references form the code in master/procedure directory
[ https://issues.apache.org/jira/browse/HBASE-16862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-16862: - Status: Patch Available (was: In Progress) > Remove directory layout/ filesystem references form the code in > master/procedure directory > -- > > Key: HBASE-16862 > URL: https://issues.apache.org/jira/browse/HBASE-16862 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Attachments: HBASE-16862-hbase-14439.v1.patch > > > Remove directory layout/ filesystem references form the code in > master/procedure directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586789#comment-15586789
]
Ted Yu commented on HBASE-16874:
Which test covers multi-executor thread case ?
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16775) Flakey test with TestExportSnapshot#testExportRetry and TestMobExportSnapshot#testExportRetry
[
https://issues.apache.org/jira/browse/HBASE-16775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586760#comment-15586760
]
Hadoop QA commented on HBASE-16775:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 39s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 32s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 8s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestFromClientSide3 |
| | org.apache.hadoop.hbase.client.TestAdmin2 |
| | org.apache.hadoop.hbase.client.TestHCM |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834016/HBASE-16775.master.003.patch
|
| JIRA Issue | HBASE-16775 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9106c86fbd85 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6c89c62 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4086/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4086/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4086/artifact/patchprocess/
[jira] [Work started] (HBASE-16862) Remove directory layout/ filesystem references form the code in master/procedure directory
[ https://issues.apache.org/jira/browse/HBASE-16862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-16862 started by Umesh Agashe. > Remove directory layout/ filesystem references form the code in > master/procedure directory > -- > > Key: HBASE-16862 > URL: https://issues.apache.org/jira/browse/HBASE-16862 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Reporter: Umesh Agashe >Assignee: Umesh Agashe > > Remove directory layout/ filesystem references form the code in > master/procedure directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586767#comment-15586767 ] Anastasia Braginsky commented on HBASE-16608: - Hi [~anoop.hbase] and [~ram_krish], I first want to thank you for your unstoppable interest and support for this big CompactingMemStore issue! You are getting everything very fast and see it all in the deep details. Really pleasure to work with such smart people as you are. So you are saying that you want not to use any merge in order to have the things simpler. Am I getting you right? I do not see how merges (especially if done once in a while) can affect GC. The size of CellArrayMap is negligible relative to the data size. It is reasonable to think that GC is busy with releasing the chunks that are not accessible after flush to disk. When we hold more data in the memory either due to index or data compaction we flush bigger files to the disk and bigger chunks of memory are need to be freed upon each flush to disk. This sounds like a reason for making GC to work harder. If you see how merges directly affect garbage collection please explain. All this makes me think that your suggestion (not to merge) will result in the same GC behavior. We already had an implementation with flattening only, and there (under stress) we have seen tens of segments in the compaction pipeline. I wonder the performance of the gets and scans in this structure. The process of flushing multiple segments to disk should also prolong the flushing to disk, which is undesirable. When we had flushes waiting for merge Ram had seen lots of blocking writes till the system became unresponsive. So I do not clearly see why not-to-merge-at-all is a better solution. I am not attached to anything, but really trying to understand why one way is better then the other. I mean can we measure the better quality of not merging relative to intermediate merging? Regarding the issue of N-MSLABs merge, raised in the RB, it looks irrelevant now, till we decide whether to merge or not to merge. Thus let's discuss it later. I also believe this is not such a big issue and we can arrange it this way or another. Best, Anastasia > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-16874:
Fix Version/s: 2.0.0
Status: Patch Available (was: Open)
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-16874:
Component/s: proc-v2
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[
https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586749#comment-15586749
]
stack commented on HBASE-10656:
---
[~ikeda] Here is an interesting observation by a coworker
[~mi...@cloudera.com]. I can open new issue to discuss but posting here for
moment:
{quote}
To induce high load on MONITORING TOOL in my small 8-machine cluster, V
suggested to create 10 hbase tables with 1K regions each - in this way,
MONITORING TOOL gets 10K new entities to monitor. I've done that and it worked
for MONITORING TOOL as expected. However, one thing that we noticed is that
HBase Region Servers in my cluster are now constantly running GC
I decided to take a quick look, took a heap dump from one of the region servers
and analyzed it with the same tool (http://www.jxray.com) that I use in the
MONITORING TOOL work. The output is attached.
One finding is that 41% of memory is occupied by instances of
org.apache.hadoop.hbase.util.Counter$Cell class, and they seem to be actively
"churned" by GC all the time. I looked at the code of this class, and one thing
that immediately caught my eye is this:
private static class Cell {
// Pads are added around the value to avoid cache-line contention with
// another cell's value. The cache-line size is expected to be equal to or
// less than about 128 Bytes (= 64 Bits * 16).
@SuppressWarnings("unused")
volatile long p0, p1, p2, p3, p4, p5, p6;
volatile long value;
@SuppressWarnings("unused")
volatile long q0, q1, q2, q3, q4, q5, q6;
So, as far as I understand, the only meaningful data field in this class,
'value', is deliberately "padded" with empty fields just to make an instance of
this class big enough to fit the entire 128-byte cache line.
This looks like a very extreme optimization that would work if there were very
few objects in memory, or at least very few of Counter$Cell instances, so that
they were kept in the cache all the time. But clearly in our case making these
objects artificially large greatly increases the GC pressure and ultimately
makes everything much slower.
Can somebody shed some light on this? In particular:
- Why do so many Counter instances are created and destroyed all the time
despite the fact that there is no HBase activity going on?
- I don't think the setup with 10K regions is very unconventional. If so many
Cell objects need to be maintained, then probably it's worth providing e.g.
another implementation that's simply optimized for size rather than for memory
cache performance?
{quote}
On Question #1, it is probably our metrics accounting that is going on. On #2,
you might have input.
> high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
>
>
> Key: HBASE-10656
> URL: https://issues.apache.org/jira/browse/HBASE-10656
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Fix For: 0.96.2, 0.98.1, 0.99.0
>
> Attachments: 10656-098.v2.txt, 10656-trunk.v2.patch, 10656.096v2.txt,
> HBASE-10656-0.96.patch, HBASE-10656-addition.patch, HBASE-10656-trunk.patch,
> MyCounter.java, MyCounter2.java, MyCounter3.java, MyCounterTest.java,
> MyCounterTest.java, PerformanceTestApp.java, PerformanceTestApp2.java,
> output.pdf, output.txt, output2.pdf, output2.txt
>
>
> Cliff's high-scale-lib's Counter is used in important classes (for example,
> HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation
> detail of the Java standard library and belongs to Oracle (Sun). That
> consequently makes HBase depend on the specific JRE Implementation.
> To make matters worse, Counter has a bug and you may get wrong result if you
> mix a reading method into your logic calling writing methods.
> In more detail, I think the bug is caused by reading an internal array field
> without resolving memory caching, which is intentional the comment says, but
> storing the read result into a volatile field. That field may be not changed
> after you can see the true values of the array field, and also may be not
> changed after updating the "next" CAT instance's values in some race
> condition when extending CAT instance chain.
> Anyway, it is possible that you create a new alternative class which only
> depends on the standard library. I know Java8 provides its alternative, but
> HBase should support Java6 and Java7 for some time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-16874:
Attachment: HBASE-16874-v0.patch
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586746#comment-15586746
]
Matteo Bertozzi commented on HBASE-16874:
-
Looks like in this test we forgot to set the number of executors thread to 1.
When we inject failures with the setToggleKillBeforeStoreUpdate() we have the
assumption that there is only one executor. In this case we have multiple
executor running and toggling the flag and killing the executor when we are
restarting it on the other side.
{noformat}
2016-10-18 18:55:30,857 INFO [ProcedureExecutorWorker-5]
procedure.ServerCrashProcedure(204): Start processing crashed
priapus.apache.org,38407,1476816438880
2016-10-18 18:55:30,857 WARN [ProcedureExecutorWorker-5]
procedure2.ProcedureExecutor$Testing(92): Toggle Kill before store update to:
true
Exception in thread "ProcedureExecutorWorker-5" java.lang.RuntimeException: the
store must be running before inserting data
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:542)
{noformat}
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Attachments: 16874.v1.txt, HBASE-16874-v0.patch
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16874:
---
Status: Open (was: Patch Available)
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Attachments: 16874.v1.txt
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16874:
---
Status: Patch Available (was: Open)
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Attachments: 16874.v1.txt
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi reassigned HBASE-16874:
---
Assignee: Matteo Bertozzi
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Attachments: 16874.v1.txt
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586706#comment-15586706
]
Matteo Bertozzi commented on HBASE-16874:
-
let me look into it, but patch is not good. stop should not be triggered after
a join.. and if it is called after a join it means the executor was started, so
the timeoutExecutor should not be null
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16874.v1.txt
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[
https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586699#comment-15586699
]
Hadoop QA commented on HBASE-16824:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 50s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 152m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestReplicaWithCluster |
| | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834008/hbase-16824_v2.patch |
| JIRA Issue | HBASE-16824 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5aad3b9eb8c9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 6c89c62 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4084/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4084/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4084/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4084/testReport/ |
| modules |
[jira] [Commented] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[ https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586696#comment-15586696 ] huaxiang sun commented on HBASE-16345: -- master-006.patch is the rebased one, thanks. > RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer > Exceptions > -- > > Key: HBASE-16345 > URL: https://issues.apache.org/jira/browse/HBASE-16345 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16345-v001.patch, HBASE-16345.branch-1.001.patch, > HBASE-16345.branch-1.001.patch, HBASE-16345.master.001.patch, > HBASE-16345.master.002.patch, HBASE-16345.master.003.patch, > HBASE-16345.master.004.patch, HBASE-16345.master.005.patch, > HBASE-16345.master.005.patch, HBASE-16345.master.006.patch > > > Update for the description. Debugged more at this front based on the comments > from Enis. > The cause is that for the primary replica, if its retry is exhausted too > fast, f.get() [1] returns ExecutionException. This Exception needs to be > ignored and continue with the replicas. > The other issue is that after adding calls for the replicas, if the first > completed task gets ExecutionException (due to the retry exhausted), it > throws the exception to the client[2]. > In this case, it needs to loop through these tasks, waiting for the success > one. If no one succeeds, throw exception. > Similar for the scan as well > [1] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197 > [2] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[ https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-16345: - Attachment: HBASE-16345.master.006.patch > RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer > Exceptions > -- > > Key: HBASE-16345 > URL: https://issues.apache.org/jira/browse/HBASE-16345 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16345-v001.patch, HBASE-16345.branch-1.001.patch, > HBASE-16345.branch-1.001.patch, HBASE-16345.master.001.patch, > HBASE-16345.master.002.patch, HBASE-16345.master.003.patch, > HBASE-16345.master.004.patch, HBASE-16345.master.005.patch, > HBASE-16345.master.005.patch, HBASE-16345.master.006.patch > > > Update for the description. Debugged more at this front based on the comments > from Enis. > The cause is that for the primary replica, if its retry is exhausted too > fast, f.get() [1] returns ExecutionException. This Exception needs to be > ignored and continue with the replicas. > The other issue is that after adding calls for the replicas, if the first > completed task gets ExecutionException (due to the retry exhausted), it > throws the exception to the client[2]. > In this case, it needs to loop through these tasks, waiting for the success > one. If no one succeeds, throw exception. > Similar for the scan as well > [1] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197 > [2] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16863) Move Backup constants from HConstants to BackupRestoreConstants
[ https://issues.apache.org/jira/browse/HBASE-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16863: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: (was: HBASE-7912) 2.0.0 Status: Resolved (was: Patch Available) > Move Backup constants from HConstants to BackupRestoreConstants > --- > > Key: HBASE-16863 > URL: https://issues.apache.org/jira/browse/HBASE-16863 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16863-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16863) Move Backup constants from HConstants to BackupRestoreConstants
[
https://issues.apache.org/jira/browse/HBASE-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586667#comment-15586667
]
Hadoop QA commented on HBASE-16863:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-16863 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834034/HBASE-16863-v1.patch |
| JIRA Issue | HBASE-16863 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4088/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Move Backup constants from HConstants to BackupRestoreConstants
> ---
>
> Key: HBASE-16863
> URL: https://issues.apache.org/jira/browse/HBASE-16863
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: HBASE-7912
>
> Attachments: HBASE-16863-v1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16875) Cleanup docs' use of try-with-resources
Dima Spivak created HBASE-16875: --- Summary: Cleanup docs' use of try-with-resources Key: HBASE-16875 URL: https://issues.apache.org/jira/browse/HBASE-16875 Project: HBase Issue Type: Bug Components: documentation Reporter: Dima Spivak Priority: Trivial In a [number|https://github.com/apache/hbase/blame/bb3d9ccd489fb64e3cb2020583935a393382a678/src/main/asciidoc/_chapters/security.adoc#L205-L206] [of|https://github.com/apache/hbase/blame/bb3d9ccd489fb64e3cb2020583935a393382a678/src/main/asciidoc/_chapters/security.adoc#L1019-L1020] [places|https://github.com/apache/hbase/blame/bb3d9ccd489fb64e3cb2020583935a393382a678/src/main/asciidoc/_chapters/architecture.adoc#L222-L223], we show examples that lend themselves to using Java 7's try-with-resources statement, but we use the statement in a less-than-ideal nested way. Let's change our docs throughout to do it [the recommended way|https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
[
https://issues.apache.org/jira/browse/HBASE-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16874:
---
Attachment: 16874.v1.txt
> Potential NPE from ProcedureExecutor#stop()
> ---
>
> Key: HBASE-16874
> URL: https://issues.apache.org/jira/browse/HBASE-16874
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16874.v1.txt
>
>
> When examining failed test :
> https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
> I noticed the following:
> {code}
> 2016-10-18 18:47:39,313 INFO [Time-limited test]
> procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
> TRUNCATE_TABLE_CLEAR_FS_LAYOUT
> Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
> {code}
> This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16874) Potential NPE from ProcedureExecutor#stop()
Ted Yu created HBASE-16874:
--
Summary: Potential NPE from ProcedureExecutor#stop()
Key: HBASE-16874
URL: https://issues.apache.org/jira/browse/HBASE-16874
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
When examining failed test :
https://builds.apache.org/job/HBase-TRUNK_matrix/lastCompletedBuild/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterFailoverWithProcedures/org_apache_hadoop_hbase_master_procedure_TestMasterFailoverWithProcedures/
I noticed the following:
{code}
2016-10-18 18:47:39,313 INFO [Time-limited test]
procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state:
TRUNCATE_TABLE_CLEAR_FS_LAYOUT
Exception in thread "ProcedureExecutorWorker-1" java.lang.NullPointerException
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.stop(ProcedureExecutor.java:533)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1197)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:959)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1405)
{code}
This seems to be the result of race between stop() and join() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16863) Move Backup constants from HConstants to BackupRestoreConstants
[ https://issues.apache.org/jira/browse/HBASE-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586651#comment-15586651 ] Vladimir Rodionov commented on HBASE-16863: --- [~tedyu], can you commit this first? > Move Backup constants from HConstants to BackupRestoreConstants > --- > > Key: HBASE-16863 > URL: https://issues.apache.org/jira/browse/HBASE-16863 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: HBASE-7912 > > Attachments: HBASE-16863-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
