[jira] [Resolved] (HBASE-7326) SortedCopyOnWriteSet is not thread safe due to leaked TreeSet implementations
[ https://issues.apache.org/jira/browse/HBASE-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved HBASE-7326. --- Resolution: Duplicate > SortedCopyOnWriteSet is not thread safe due to leaked TreeSet implementations > - > > Key: HBASE-7326 > URL: https://issues.apache.org/jira/browse/HBASE-7326 > Project: HBase > Issue Type: Bug > Components: util >Reporter: Gary Helmling > Fix For: 2.0.0 > > > The SortedCopyOnWriteSet implementation uses an internal TreeSet that is > copied and replaced on mutation operations. However, in a few areas, > SortedCopyOnWriteSet leaks references to the underlying TreeSet > implementations, allowing for unsafe usage: > * iterator() > * subSet() > * headSet() > * tailSet() > For Iterator.remove(), we can wrap in an implementation that throws > UnsupportedOperationException. For the sub set methods, we could return new > SortedCopyOnWriteSet instances (which would not modify the parent set), or > wrap with a new sub set implementation that safely allows modification of the > parent set. > To be clear, the current usage of SortedCopyOnWriteSet does not make use of > any of these non-thread-safe methods, but the implementation should be fixed > to be completely thread safe and prevent any new issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-7326) SortedCopyOnWriteSet is not thread safe due to leaked TreeSet implementations
[ https://issues.apache.org/jira/browse/HBASE-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-7326: -- Fix Version/s: 2.0.0 > SortedCopyOnWriteSet is not thread safe due to leaked TreeSet implementations > - > > Key: HBASE-7326 > URL: https://issues.apache.org/jira/browse/HBASE-7326 > Project: HBase > Issue Type: Bug > Components: util >Reporter: Gary Helmling > Fix For: 2.0.0 > > > The SortedCopyOnWriteSet implementation uses an internal TreeSet that is > copied and replaced on mutation operations. However, in a few areas, > SortedCopyOnWriteSet leaks references to the underlying TreeSet > implementations, allowing for unsafe usage: > * iterator() > * subSet() > * headSet() > * tailSet() > For Iterator.remove(), we can wrap in an implementation that throws > UnsupportedOperationException. For the sub set methods, we could return new > SortedCopyOnWriteSet instances (which would not modify the parent set), or > wrap with a new sub set implementation that safely allows modification of the > parent set. > To be clear, the current usage of SortedCopyOnWriteSet does not make use of > any of these non-thread-safe methods, but the implementation should be fixed > to be completely thread safe and prevent any new issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-7326) SortedCopyOnWriteSet is not thread safe due to leaked TreeSet implementations
[ https://issues.apache.org/jira/browse/HBASE-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-7326: -- Affects Version/s: (was: 0.94.3) (was: 0.95.2) (was: 0.92.2) > SortedCopyOnWriteSet is not thread safe due to leaked TreeSet implementations > - > > Key: HBASE-7326 > URL: https://issues.apache.org/jira/browse/HBASE-7326 > Project: HBase > Issue Type: Bug > Components: util >Reporter: Gary Helmling > Fix For: 2.0.0 > > > The SortedCopyOnWriteSet implementation uses an internal TreeSet that is > copied and replaced on mutation operations. However, in a few areas, > SortedCopyOnWriteSet leaks references to the underlying TreeSet > implementations, allowing for unsafe usage: > * iterator() > * subSet() > * headSet() > * tailSet() > For Iterator.remove(), we can wrap in an implementation that throws > UnsupportedOperationException. For the sub set methods, we could return new > SortedCopyOnWriteSet instances (which would not modify the parent set), or > wrap with a new sub set implementation that safely allows modification of the > parent set. > To be clear, the current usage of SortedCopyOnWriteSet does not make use of > any of these non-thread-safe methods, but the implementation should be fixed > to be completely thread safe and prevent any new issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-7326) SortedCopyOnWriteSet is not thread safe due to leaked TreeSet implementations
[ https://issues.apache.org/jira/browse/HBASE-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952520#comment-15952520 ] stack commented on HBASE-7326: -- Close it boss [~chia7712] > SortedCopyOnWriteSet is not thread safe due to leaked TreeSet implementations > - > > Key: HBASE-7326 > URL: https://issues.apache.org/jira/browse/HBASE-7326 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.92.2, 0.94.3, 0.95.2 >Reporter: Gary Helmling > > The SortedCopyOnWriteSet implementation uses an internal TreeSet that is > copied and replaced on mutation operations. However, in a few areas, > SortedCopyOnWriteSet leaks references to the underlying TreeSet > implementations, allowing for unsafe usage: > * iterator() > * subSet() > * headSet() > * tailSet() > For Iterator.remove(), we can wrap in an implementation that throws > UnsupportedOperationException. For the sub set methods, we could return new > SortedCopyOnWriteSet instances (which would not modify the parent set), or > wrap with a new sub set implementation that safely allows modification of the > parent set. > To be clear, the current usage of SortedCopyOnWriteSet does not make use of > any of these non-thread-safe methods, but the implementation should be fixed > to be completely thread safe and prevent any new issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-7326) SortedCopyOnWriteSet is not thread safe due to leaked TreeSet implementations
[ https://issues.apache.org/jira/browse/HBASE-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952486#comment-15952486 ] Chia-Ping Tsai commented on HBASE-7326: --- SortedCopyOnWriteSet had been removed in 2.0.0 (see HBASE-17809). We should close this jira. > SortedCopyOnWriteSet is not thread safe due to leaked TreeSet implementations > - > > Key: HBASE-7326 > URL: https://issues.apache.org/jira/browse/HBASE-7326 > Project: HBase > Issue Type: Bug > Components: util >Affects Versions: 0.92.2, 0.94.3, 0.95.2 >Reporter: Gary Helmling > > The SortedCopyOnWriteSet implementation uses an internal TreeSet that is > copied and replaced on mutation operations. However, in a few areas, > SortedCopyOnWriteSet leaks references to the underlying TreeSet > implementations, allowing for unsafe usage: > * iterator() > * subSet() > * headSet() > * tailSet() > For Iterator.remove(), we can wrap in an implementation that throws > UnsupportedOperationException. For the sub set methods, we could return new > SortedCopyOnWriteSet instances (which would not modify the parent set), or > wrap with a new sub set implementation that safely allows modification of the > parent set. > To be clear, the current usage of SortedCopyOnWriteSet does not make use of > any of these non-thread-safe methods, but the implementation should be fixed > to be completely thread safe and prevent any new issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16338) update jackson to 2.y
[ https://issues.apache.org/jira/browse/HBASE-16338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952438#comment-15952438 ] Sean Busbey commented on HBASE-16338: - If we need discussion, then dev@hbase is a better place to do it. Personally, I don't think we need to; plenty of dependencies are updating in HBase 2.0. Interested in putting up a patch [~vrushalic]? > update jackson to 2.y > - > > Key: HBASE-16338 > URL: https://issues.apache.org/jira/browse/HBASE-16338 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Sean Busbey >Priority: Minor > Fix For: 2.0.0 > > > Our jackson dependency is from ~3 years ago. Update to the jackson 2.y line, > using 2.7.0+. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17857) Remove IS annotations from IA.Public classes
[
https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952327#comment-15952327
]
Hadoop QA commented on HBASE-17857:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 48s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
17s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 2m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 9s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s
{color} | {color:green} hbase-annotations in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 52s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 104m 10s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 32s
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 34s
{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 49s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
36s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 182m 43s {color}
|
[jira] [Commented] (HBASE-16338) update jackson to 2.y
[ https://issues.apache.org/jira/browse/HBASE-16338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952306#comment-15952306 ] Vrushali C commented on HBASE-16338: Yes, it will be very good to move to jackson 2.x. Hadoop is moving to jackson 2.x in HADOOP-13332 . Timeline Service v2 in YARN uses HBase 1.2.4 with both hbase server and hbase client dependencies. These in turn depend on org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile . Hoping to get some more conversation on this. cc [~saint@gmail.com] [~jrottinghuis] [~steve_l] > update jackson to 2.y > - > > Key: HBASE-16338 > URL: https://issues.apache.org/jira/browse/HBASE-16338 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Sean Busbey >Priority: Minor > Fix For: 2.0.0 > > > Our jackson dependency is from ~3 years ago. Update to the jackson 2.y line, > using 2.7.0+. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17863) Procedure V2: Some cleanup around isFinished() and procedure executor
[ https://issues.apache.org/jira/browse/HBASE-17863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952297#comment-15952297 ] stack commented on HBASE-17863: --- Those failures are interesting [~uagashe] Looks like there is a dependency on a fuzzy semantic > Procedure V2: Some cleanup around isFinished() and procedure executor > - > > Key: HBASE-17863 > URL: https://issues.apache.org/jira/browse/HBASE-17863 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Attachments: HBASE-17863.v1.patch > > > Clean up around isFinished() and procedure executor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17867) Implement async procedure RPC API(list/exec/abort/isFinished)
[ https://issues.apache.org/jira/browse/HBASE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952277#comment-15952277 ] Stephen Yuan Jiang commented on HBASE-17867: personally, I don't think isFinished should be async. Also for list, sync is also makes sense. > Implement async procedure RPC API(list/exec/abort/isFinished) > - > > Key: HBASE-17867 > URL: https://issues.apache.org/jira/browse/HBASE-17867 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17857) Remove IS annotations from IA.Public classes
[ https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17857: -- Attachment: HBASE-17857-v2.patch Also remove the IS annotations for scala files in hbase-spark module. > Remove IS annotations from IA.Public classes > > > Key: HBASE-17857 > URL: https://issues.apache.org/jira/browse/HBASE-17857 > Project: HBase > Issue Type: Sub-task > Components: API >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17857.patch, HBASE-17857-v1.patch, > HBASE-17857-v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952191#comment-15952191
]
Hudson commented on HBASE-17859:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2779 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2779/])
HBASE-17859 ByteBufferUtils#compareTo is wrong (chia7712: rev
73e1bcd33515061be2dc2e51e6ad19d9798a8ef6)
* (edit)
hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java
* (edit)
hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch, HBASE-17859.v1.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952135#comment-15952135
]
Hadoop QA commented on HBASE-17854:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 54s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 14s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 114m 31s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 161m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | org.apache.hadoop.hbase.master.cleaner.HFileCleaner$HFileDeleteTask
defines equals and uses Object.hashCode() At
HFileCleaner.java:Object.hashCode() At HFileCleaner.java:[lines 318-325] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861581/HBASE-17854.v2.patch |
| JIRA Issue | HBASE-17854 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d93537423b65 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 73e1bcd |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6294/artifact/patchprocess/new-findbugs-hbase-server.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6294/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6294/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Use StealJobQueue in HFileCleaner after HBASE-17215
>
[jira] [Created] (HBASE-17867) Implement async procedure RPC API(list/exec/abort/isFinished)
Zheng Hu created HBASE-17867: Summary: Implement async procedure RPC API(list/exec/abort/isFinished) Key: HBASE-17867 URL: https://issues.apache.org/jira/browse/HBASE-17867 Project: HBase Issue Type: Sub-task Reporter: Zheng Hu -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17867) Implement async procedure RPC API(list/exec/abort/isFinished)
[ https://issues.apache.org/jira/browse/HBASE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu reassigned HBASE-17867: Assignee: Zheng Hu > Implement async procedure RPC API(list/exec/abort/isFinished) > - > > Key: HBASE-17867 > URL: https://issues.apache.org/jira/browse/HBASE-17867 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17866) Implement async setQuota/getQuota methods.
Zheng Hu created HBASE-17866: Summary: Implement async setQuota/getQuota methods. Key: HBASE-17866 URL: https://issues.apache.org/jira/browse/HBASE-17866 Project: HBase Issue Type: Sub-task Reporter: Zheng Hu Assignee: Zheng Hu -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17865) Implement async listSnapshot/deleteSnapshot methods.
Zheng Hu created HBASE-17865: Summary: Implement async listSnapshot/deleteSnapshot methods. Key: HBASE-17865 URL: https://issues.apache.org/jira/browse/HBASE-17865 Project: HBase Issue Type: Sub-task Reporter: Zheng Hu Assignee: Zheng Hu -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17864) Implement async snapshot/cloneSnapshot/restoreSnapshot methods
Zheng Hu created HBASE-17864: Summary: Implement async snapshot/cloneSnapshot/restoreSnapshot methods Key: HBASE-17864 URL: https://issues.apache.org/jira/browse/HBASE-17864 Project: HBase Issue Type: Sub-task Reporter: Zheng Hu Assignee: Zheng Hu -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17836) CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell
[
https://issues.apache.org/jira/browse/HBASE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952126#comment-15952126
]
Hadoop QA commented on HBASE-17836:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 104m 14s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 150m 54s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861579/HBASE-17836.v1.patch |
| JIRA Issue | HBASE-17836 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 63f6ceec56c7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 73e1bcd |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6293/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6293/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Commented] (HBASE-17215) Separate small/large file delete threads in HFileCleaner to accelerate archived hfile cleanup speed
[
https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952118#comment-15952118
]
Hudson commented on HBASE-17215:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2778 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2778/])
HBASE-17215 Separate small/large file delete threads in HFileCleaner to (liyu:
rev 9facfa550f1e7386be3a04d84f7e8013f5002965)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
> Separate small/large file delete threads in HFileCleaner to accelerate
> archived hfile cleanup speed
> ---
>
> Key: HBASE-17215
> URL: https://issues.apache.org/jira/browse/HBASE-17215
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-17215.patch, HBASE-17215.v2.patch,
> HBASE-17215.v3.patch
>
>
> When using PCIe-SSD the flush speed will be really quick, and although we
> have per CF flush, we still have the
> {{hbase.regionserver.optionalcacheflushinterval}} setting and some other
> mechanism to avoid data kept in memory for too long to flush small hfiles. In
> our online environment we found the single thread cleaner kept cleaning
> earlier flushed small files while large files got no chance, which caused
> disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the
> workload of namenode, so here we propose to separate large/small hfile
> cleaner threads just like we do for compaction, and it turned out to work
> well in our cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952106#comment-15952106
]
Anoop Sam John commented on HBASE-17861:
{quote}
hbase.rootdir = hdfs://xx/xx
hbase.wal.dir = s3a://xx//xx
{quote}
U mean root dir in S3 while WAL in HDFS? When root is in HDFS am not sure why
bulk load staging dir is getting created inside S3.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17854:
--
Attachment: HBASE-17854.v2.patch
New patch resolves fingbugs issue.
> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived)
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs
> from small queue, based on the experience that in real world there'll be much
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete
> from the larger file in the queues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
