[jira] [Updated] (HBASE-17821) The CompoundConfiguration#toString is wrong
[
https://issues.apache.org/jira/browse/HBASE-17821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17821:
---
Fix Version/s: 1.4.0
2.0.0
> The CompoundConfiguration#toString is wrong
> ---
>
> Key: HBASE-17821
> URL: https://issues.apache.org/jira/browse/HBASE-17821
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Yi Liang
>Priority: Trivial
> Labels: beginner
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBase-17821-V1.patch, HBase-17821-V1.patch
>
>
> Find this bug when reading code. We dont use the API, so it is a trivial bug.
> sb.append(this.configs); -> sb.append(m);
> {noformat}
> @Override
> public String toString() {
> StringBuffer sb = new StringBuffer();
> sb.append("CompoundConfiguration: " + this.configs.size() + " configs");
> for (ImmutableConfigMap m : this.configs) {
> sb.append(this.configs);
> }
> return sb.toString();
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17821) The CompoundConfiguration#toString is wrong
[
https://issues.apache.org/jira/browse/HBASE-17821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17821:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
push to master and branch-1. Thanks for the patch. [~easyliangjob]
> The CompoundConfiguration#toString is wrong
> ---
>
> Key: HBASE-17821
> URL: https://issues.apache.org/jira/browse/HBASE-17821
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Yi Liang
>Priority: Trivial
> Labels: beginner
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBase-17821-V1.patch, HBase-17821-V1.patch
>
>
> Find this bug when reading code. We dont use the API, so it is a trivial bug.
> sb.append(this.configs); -> sb.append(m);
> {noformat}
> @Override
> public String toString() {
> StringBuffer sb = new StringBuffer();
> sb.append("CompoundConfiguration: " + this.configs.size() + " configs");
> for (ImmutableConfigMap m : this.configs) {
> sb.append(this.configs);
> }
> return sb.toString();
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17668) Implement async assgin/offline/move/unassign methods
[ https://issues.apache.org/jira/browse/HBASE-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-17668: - Attachment: HBASE-17668.v4.patch Change timeout to 3s for testOfflineRegion ut. > Implement async assgin/offline/move/unassign methods > > > Key: HBASE-17668 > URL: https://issues.apache.org/jira/browse/HBASE-17668 > Project: HBase > Issue Type: Sub-task > Components: Admin, asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: 2.0.0 > > Attachments: HBASE-17668.v1.patch, HBASE-17668.v1.patch, > HBASE-17668.v2.patch, HBASE-17668.v3.patch, HBASE-17668.v4.patch > > > Implement following methods for async admin client: > 1. assign region; > 2. unassign region; > 3. offline region; > 4. move region; -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950463#comment-15950463
]
Hadoop QA commented on HBASE-16438:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 24 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 8s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 52s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 124m 11s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 172m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861377/HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch
|
| JIRA Issue | HBASE-16438 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f9b535d23927 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d033cbb |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6275/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6275/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically genera
[jira] [Created] (HBASE-17857) Remove IS annotations from IA.Public classes
Duo Zhang created HBASE-17857: - Summary: Remove IS annotations from IA.Public classes Key: HBASE-17857 URL: https://issues.apache.org/jira/browse/HBASE-17857 Project: HBase Issue Type: Sub-task Components: API Affects Versions: 2.0.0 Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17828) Remove IS annotations from IA.Public classes
[ https://issues.apache.org/jira/browse/HBASE-17828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950471#comment-15950471 ] Duo Zhang commented on HBASE-17828: --- There too many files being affected so let' split this into two parts. The removal will be done by a script. And we can file another issue for documentation change. > Remove IS annotations from IA.Public classes > > > Key: HBASE-17828 > URL: https://issues.apache.org/jira/browse/HBASE-17828 > Project: HBase > Issue Type: Task > Components: API >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > > As discussed in the dev mailing list, we do not mention the IS annotations > for public API in our refguide so the IS annotations for IA.Public classes > only makes people confusing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17828) Revisit the IS annotation and the documentation
[ https://issues.apache.org/jira/browse/HBASE-17828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17828: -- Summary: Revisit the IS annotation and the documentation (was: Remove IS annotations from IA.Public classes) > Revisit the IS annotation and the documentation > --- > > Key: HBASE-17828 > URL: https://issues.apache.org/jira/browse/HBASE-17828 > Project: HBase > Issue Type: Task > Components: API >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > > As discussed in the dev mailing list, we do not mention the IS annotations > for public API in our refguide so the IS annotations for IA.Public classes > only makes people confusing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17858) Update refguide about the IS annotation if necessary
Duo Zhang created HBASE-17858: - Summary: Update refguide about the IS annotation if necessary Key: HBASE-17858 URL: https://issues.apache.org/jira/browse/HBASE-17858 Project: HBase Issue Type: Sub-task Components: API, documentation Affects Versions: 2.0.0 Reporter: Duo Zhang Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17215) Separate small/large file delete threads in HFileCleaner to accelerate archived hfile cleanup speed
[
https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950474#comment-15950474
]
Yu Li commented on HBASE-17215:
---
UT looks good, please check patch v3 and let me know if more comments [~tedyu]
[~huaxiang]. Thanks.
> Separate small/large file delete threads in HFileCleaner to accelerate
> archived hfile cleanup speed
> ---
>
> Key: HBASE-17215
> URL: https://issues.apache.org/jira/browse/HBASE-17215
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17215.patch, HBASE-17215.v2.patch,
> HBASE-17215.v3.patch
>
>
> When using PCIe-SSD the flush speed will be really quick, and although we
> have per CF flush, we still have the
> {{hbase.regionserver.optionalcacheflushinterval}} setting and some other
> mechanism to avoid data kept in memory for too long to flush small hfiles. In
> our online environment we found the single thread cleaner kept cleaning
> earlier flushed small files while large files got no chance, which caused
> disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the
> workload of namenode, so here we propose to separate large/small hfile
> cleaner threads just like we do for compaction, and it turned out to work
> well in our cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17857) Remove IS annotations from IA.Public classes
[
https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950476#comment-15950476
]
Duo Zhang commented on HBASE-17857:
---
The script is like this
{noformat}
#!/bin/bash
IS_PUBLIC=`awk '{print $1}' < $1 | grep -s "^@InterfaceAudience.Public"`
if [ ! -n "$IS_PUBLIC" ]; then
# echo "Skipping $1"
exit 1
fi
echo "Processing $1"
grep -v "^[ ]*@InterfaceStability." $1 > /tmp/tmp.java
mv /tmp/tmp.java $1
{noformat}
And executed with
{noformat}
find . -type f -name *.java -exec ./rm_anno.sh {} \;
{noformat}
> Remove IS annotations from IA.Public classes
>
>
> Key: HBASE-17857
> URL: https://issues.apache.org/jira/browse/HBASE-17857
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17855) Fix typo in async client implementation
[ https://issues.apache.org/jira/browse/HBASE-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950499#comment-15950499 ] Hudson commented on HBASE-17855: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2772 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2772/]) HBASE-17855 Fix typo in async client implementation (zhangduo: rev 0ec1459467f280280630e5b597bef302ec43cedd) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/AbstractTestAsyncTableScan.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionLocator.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncMetaRegionLocator.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java > Fix typo in async client implementation > --- > > Key: HBASE-17855 > URL: https://issues.apache.org/jira/browse/HBASE-17855 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Labels: trivial > Fix For: 2.0.0 > > Attachments: HBASE-17855.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17835) Spelling mistakes in the Java source
[ https://issues.apache.org/jira/browse/HBASE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-17835: -- Attachment: HBASE-17835-001.patch > Spelling mistakes in the Java source > > > Key: HBASE-17835 > URL: https://issues.apache.org/jira/browse/HBASE-17835 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17835-001.patch > > > I found spelling mistakes in the hbase java source files viz. recieved > instead of received, and SpanReciever instead of SpanReceiver -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17835) Spelling mistakes in the Java source
[ https://issues.apache.org/jira/browse/HBASE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-17835: -- Attachment: (was: HBASE-17835-001.patch) > Spelling mistakes in the Java source > > > Key: HBASE-17835 > URL: https://issues.apache.org/jira/browse/HBASE-17835 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17835-001.patch > > > I found spelling mistakes in the hbase java source files viz. recieved > instead of received, and SpanReciever instead of SpanReceiver -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17843) JUnit test timed out in TestRegionReplicaFailover.java
[ https://issues.apache.org/jira/browse/HBASE-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-17843: -- Priority: Trivial (was: Minor) > JUnit test timed out in TestRegionReplicaFailover.java > -- > > Key: HBASE-17843 > URL: https://issues.apache.org/jira/browse/HBASE-17843 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Trivial > Attachments: HBASE-17843-v1.patch > > > I found junit test failed in TestRegionReplicaFailover.java, so I changed the > testPrimaryRegionKill method test timeout to 24ms, and add sleep 5000ms > for verify result. > error logs: > Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec > <<< FAILURE! - in > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover > testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) > Time elapsed: 125.963 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 12 > milliseconds > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:460) > at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) > at > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) > at > org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) > at > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17857) Remove IS annotations from IA.Public classes
[
https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950506#comment-15950506
]
Duo Zhang commented on HBASE-17857:
---
And this is the classes which are marked as IA.Public and also IS.Unstable.
{noformat}
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncTable.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableBuilder.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ExponentialClientBackoffPolicy.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicy.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScanResultConsumer.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableRegionLocator.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableBase.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/CompactType.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnection.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTable.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawScanResultConsumer.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
./hbase-common/src/test/java/org/apache/hadoop/hbase/HBaseCommonTestingUtility.java
./hbase-endpoint/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AsyncAggregationClient.java
{noformat}
For {{Scan}}, the unstable part is an inner class called ReadType, I think it
is OK to remove the unstable annotation for it.
And for async client related part, I suggest we mark AsyncAdmin related as
IA.Private first and change it to IA.Public when we finish the most works and
it is ready for end user.
For {{CompactType}}, I think we should remove the unstable mark as it is used
in Admin interface which is IA.Public and consisdered to be stable.
And for {{HBaseCommonTestingUtility}}, why Unstable? It is sub class
{{HBaseTestingUtility}} is evolving... Let's just remove it.
And for backoff related classess, Let's make it IA.Private first as it is not
complete and seems there is no progress on it.
Thanks.
> Remove IS annotations from IA.Public classes
>
>
> Key: HBASE-17857
> URL: https://issues.apache.org/jira/browse/HBASE-17857
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17843) JUnit test timed out in TestRegionReplicaFailover.java
[ https://issues.apache.org/jira/browse/HBASE-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qilin Cao updated HBASE-17843: -- Description: Junit test sometimes failed in TestRegionReplicaFailover.java, so I changed the testPrimaryRegionKill method test timeout to 24ms, and add sleep 5000ms for verify result. error logs: Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) Time elapsed: 125.963 sec <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:460) at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) at org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) at org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) at org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) was: I found junit test failed in TestRegionReplicaFailover.java, so I changed the testPrimaryRegionKill method test timeout to 24ms, and add sleep 5000ms for verify result. error logs: Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) Time elapsed: 125.963 sec <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:460) at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:209) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:428) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:392) at org.apache.hadoop.hbase.HBaseTestingUtility.verifyNumericRows(HBaseTestingUtility.java:2197) at org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.verifyNumericRowsWithTimeout(TestRegionReplicaFailover.java:227) at org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testPrimaryRegionKill(TestRegionReplicaFailover.java:200) > JUnit test timed out in TestRegionReplicaFailover.java > -- > > Key: HBASE-17843 > URL: https://issues.apache.org/jira/browse/HBASE-17843 > Project: HBase > Issue Type: Improvement >Reporter: Qilin Cao >Priority: Trivial > Attachments: HBASE-17843-v1.patch > > > Junit test sometimes failed in TestRegionReplicaFailover.java, so I changed > the testPrimaryRegionKill method test timeout to 24ms, and add sleep > 5000ms for verify result. > error logs: > Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 285.221 sec > <<< FAILURE! - in > org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover > testPrimaryRegionKill[0](org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover) > Time elapsed: 125.963 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 12 > milliseconds > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:460) > at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258) > at > org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.
[jira] [Commented] (HBASE-17215) Separate small/large file delete threads in HFileCleaner to accelerate archived hfile cleanup speed
[
https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950522#comment-15950522
]
Ted Yu commented on HBASE-17215:
lgtm
> Separate small/large file delete threads in HFileCleaner to accelerate
> archived hfile cleanup speed
> ---
>
> Key: HBASE-17215
> URL: https://issues.apache.org/jira/browse/HBASE-17215
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17215.patch, HBASE-17215.v2.patch,
> HBASE-17215.v3.patch
>
>
> When using PCIe-SSD the flush speed will be really quick, and although we
> have per CF flush, we still have the
> {{hbase.regionserver.optionalcacheflushinterval}} setting and some other
> mechanism to avoid data kept in memory for too long to flush small hfiles. In
> our online environment we found the single thread cleaner kept cleaning
> earlier flushed small files while large files got no chance, which caused
> disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the
> workload of namenode, so here we propose to separate large/small hfile
> cleaner threads just like we do for compaction, and it turned out to work
> well in our cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17857) Remove IS annotations from IA.Public classes
[
https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950476#comment-15950476
]
Duo Zhang edited comment on HBASE-17857 at 3/31/17 8:35 AM:
The script is like this
{noformat}
#!/bin/bash
IS_PUBLIC=`awk '{print $1}' < $1 | grep -s "^@InterfaceAudience.Public"`
if [ ! -n "$IS_PUBLIC" ]; then
# echo "Skipping $1"
exit 1
fi
echo "Processing $1"
grep -v "^[ ]*@InterfaceStability." $1 | grep -v "import
org.apache.hadoop.hbase.classification.InterfaceStability" > /tmp/tmp.java
mv /tmp/tmp.java $1
{noformat}
And executed with
{noformat}
find . -type f -name *.java -exec ./rm_anno.sh {} \;
{noformat}
was (Author: apache9):
The script is like this
{noformat}
#!/bin/bash
IS_PUBLIC=`awk '{print $1}' < $1 | grep -s "^@InterfaceAudience.Public"`
if [ ! -n "$IS_PUBLIC" ]; then
# echo "Skipping $1"
exit 1
fi
echo "Processing $1"
grep -v "^[ ]*@InterfaceStability." $1 > /tmp/tmp.java
mv /tmp/tmp.java $1
{noformat}
And executed with
{noformat}
find . -type f -name *.java -exec ./rm_anno.sh {} \;
{noformat}
> Remove IS annotations from IA.Public classes
>
>
> Key: HBASE-17857
> URL: https://issues.apache.org/jira/browse/HBASE-17857
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17857) Remove IS annotations from IA.Public classes
[
https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950506#comment-15950506
]
Duo Zhang edited comment on HBASE-17857 at 3/31/17 8:37 AM:
And this is the classes which are marked as IA.Public and also IS.Unstable.
{noformat}
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncTable.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableBuilder.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ExponentialClientBackoffPolicy.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicy.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScanResultConsumer.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableRegionLocator.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableBase.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/CompactType.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnection.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTable.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawScanResultConsumer.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
./hbase-common/src/test/java/org/apache/hadoop/hbase/HBaseCommonTestingUtility.java
./hbase-endpoint/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AsyncAggregationClient.java
{noformat}
For {{Scan}}, the unstable part is an inner class called ReadType, I think it
is OK to remove the unstable annotation for it.
And for async client related part, I suggest we mark AsyncAdmin related as
IA.Private first and change it to IA.Public when we finish the most works and
it is ready for end user.
For {{CompactType}}, I think we should remove the unstable mark as it is used
in Admin interface which is IA.Public and consisdered to be stable.
And for {{HBaseCommonTestingUtility}}, why Unstable? Its sub class
{{HBaseTestingUtility}} is evolving... Let's just remove it.
And for backoff related classess, Let's make it IA.Private first as it is not
complete and seems there is no progress on it.
Thanks.
was (Author: apache9):
And this is the classes which are marked as IA.Public and also IS.Unstable.
{noformat}
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncTable.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableBuilder.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ExponentialClientBackoffPolicy.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicy.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScanResultConsumer.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableRegionLocator.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableBase.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/CompactType.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnection.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTable.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawScanResultConsumer.java
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
./hbase-common/src/test/java/org/apache/hadoop/hbase/HBaseCommonTestingUtility.java
./hbase-endpoint/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AsyncAggregationClient.java
{noformat}
For {{Scan}}, the unstable part is an inner class called ReadType, I think it
is OK to remove the unstable annotation for it.
And for async client related part, I suggest we mark AsyncAdmin related as
IA.Private first and change it to IA.Public when we finish the most works and
it is ready for end user.
For {{CompactType}}, I think we should remove the unstable mark as it is used
in Admin interface which is IA.Public and consisdered to be stable.
And for {{HBaseCommonTestingUtility}}, why Unstable? It is sub class
{{HBaseTestingUtility}} is evolving... Let's just remove it.
And for backoff related classess, Let's make it IA.Private first as it is not
complete and seems there is no progress on it.
Thanks.
> Remove IS annotations from IA.Public classes
>
>
> Key: HBASE-17857
> URL: https://issues.apache.org/jira/browse/HBASE-17857
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17857) Remove IS annotations from IA.Public classes
[ https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17857: -- Status: Patch Available (was: Open) > Remove IS annotations from IA.Public classes > > > Key: HBASE-17857 > URL: https://issues.apache.org/jira/browse/HBASE-17857 > Project: HBase > Issue Type: Sub-task > Components: API >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17857.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17857) Remove IS annotations from IA.Public classes
[ https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17857: -- Attachment: HBASE-17857.patch > Remove IS annotations from IA.Public classes > > > Key: HBASE-17857 > URL: https://issues.apache.org/jira/browse/HBASE-17857 > Project: HBase > Issue Type: Sub-task > Components: API >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17857.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17215) Separate small/large file delete threads in HFileCleaner to accelerate archived hfile cleanup speed
[
https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950561#comment-15950561
]
Yu Li commented on HBASE-17215:
---
Thanks [~tedyu]. Will commit this tomorrow if no objections.
> Separate small/large file delete threads in HFileCleaner to accelerate
> archived hfile cleanup speed
> ---
>
> Key: HBASE-17215
> URL: https://issues.apache.org/jira/browse/HBASE-17215
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17215.patch, HBASE-17215.v2.patch,
> HBASE-17215.v3.patch
>
>
> When using PCIe-SSD the flush speed will be really quick, and although we
> have per CF flush, we still have the
> {{hbase.regionserver.optionalcacheflushinterval}} setting and some other
> mechanism to avoid data kept in memory for too long to flush small hfiles. In
> our online environment we found the single thread cleaner kept cleaning
> earlier flushed small files while large files got no chance, which caused
> disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the
> workload of namenode, so here we propose to separate large/small hfile
> cleaner threads just like we do for compaction, and it turned out to work
> well in our cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17821) The CompoundConfiguration#toString is wrong
[
https://issues.apache.org/jira/browse/HBASE-17821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950599#comment-15950599
]
Hudson commented on HBASE-17821:
SUCCESS: Integrated in Jenkins build HBase-1.4 #685 (See
[https://builds.apache.org/job/HBase-1.4/685/])
HBASE-17821: The CompoundConfiguration#toString is wrong (chia7712: rev
cc700ef4c14d6b5c1e5ef03444196ed02e5e605e)
* (edit)
hbase-common/src/main/java/org/apache/hadoop/hbase/CompoundConfiguration.java
> The CompoundConfiguration#toString is wrong
> ---
>
> Key: HBASE-17821
> URL: https://issues.apache.org/jira/browse/HBASE-17821
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Yi Liang
>Priority: Trivial
> Labels: beginner
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBase-17821-V1.patch, HBase-17821-V1.patch
>
>
> Find this bug when reading code. We dont use the API, so it is a trivial bug.
> sb.append(this.configs); -> sb.append(m);
> {noformat}
> @Override
> public String toString() {
> StringBuffer sb = new StringBuffer();
> sb.append("CompoundConfiguration: " + this.configs.size() + " configs");
> for (ImmutableConfigMap m : this.configs) {
> sb.append(this.configs);
> }
> return sb.toString();
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17668) Implement async assgin/offline/move/unassign methods
[
https://issues.apache.org/jira/browse/HBASE-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950635#comment-15950635
]
Hadoop QA commented on HBASE-17668:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 114m 22s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861395/HBASE-17668.v4.patch |
| JIRA Issue | HBASE-17668 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4547c461bb31 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a9682ca |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6276/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6276/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Implement async assgin/offline/mo
[jira] [Created] (HBASE-17859) ByteBufferUtils#compareTo is wrong
Chia-Ping Tsai created HBASE-17859:
--
Summary: ByteBufferUtils#compareTo is wrong
Key: HBASE-17859
URL: https://issues.apache.org/jira/browse/HBASE-17859
Project: HBase
Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
Fix For: 2.0.0
buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
{noformat}
public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
int o2, int l2) {
//
int end1 = o1 + l1;
int end2 = o2 + l2;
for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
int a = buf1[i] & 0xFF;
int b = buf2.get(i) & 0xFF;
if (a != b) {
return a - b;
}
}
return l1 - l2;
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17859:
---
Attachment: HBASE-17859.v0.patch
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17859:
---
Status: Patch Available (was: Open)
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950649#comment-15950649
]
Ted Yu commented on HBASE-17859:
Good find.
Add a unit test to prevent regression ?
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950651#comment-15950651
]
ramkrishna.s.vasudevan commented on HBASE-17859:
+1. So this will lead to issues when we have Bytebuffer cell and normal KV?
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17858) Update refguide about the IS annotation if necessary
[ https://issues.apache.org/jira/browse/HBASE-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17858: -- Status: Patch Available (was: Open) > Update refguide about the IS annotation if necessary > > > Key: HBASE-17858 > URL: https://issues.apache.org/jira/browse/HBASE-17858 > Project: HBase > Issue Type: Sub-task > Components: API, documentation >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17858.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17858) Update refguide about the IS annotation if necessary
[ https://issues.apache.org/jira/browse/HBASE-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17858: -- Attachment: HBASE-17858.patch > Update refguide about the IS annotation if necessary > > > Key: HBASE-17858 > URL: https://issues.apache.org/jira/browse/HBASE-17858 > Project: HBase > Issue Type: Sub-task > Components: API, documentation >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17858.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17858) Update refguide about the IS annotation if necessary
[ https://issues.apache.org/jira/browse/HBASE-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-17858: - Assignee: Duo Zhang > Update refguide about the IS annotation if necessary > > > Key: HBASE-17858 > URL: https://issues.apache.org/jira/browse/HBASE-17858 > Project: HBase > Issue Type: Sub-task > Components: API, documentation >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17858.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950658#comment-15950658
]
Chia-Ping Tsai commented on HBASE-17859:
bq. Add a unit test to prevent regression ?
I will add tests in next patch.
bq. So this will lead to issues when we have Bytebuffer cell and normal KV?
yes, it makes the wrong result if we don't use the UNSAFE.
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950664#comment-15950664
]
Anoop Sam John commented on HBASE-17859:
Thanks a ton... Great catch..
So silly copy paste problem.. Am sure we have struggled because of this in our
recent perf tests.
+1
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-17859:
---
Priority: Critical (was: Major)
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950667#comment-15950667
]
ramkrishna.s.vasudevan commented on HBASE-17859:
All our recent perf test had a perf issue. Should be this one.
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950686#comment-15950686
]
Hadoop QA commented on HBASE-17859:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
35m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861419/HBASE-17859.v0.patch |
| JIRA Issue | HBASE-17859 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1ec64f8e7f09 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a9682ca |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6279/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6279/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
>
[jira] [Commented] (HBASE-17835) Spelling mistakes in the Java source
[
https://issues.apache.org/jira/browse/HBASE-17835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950697#comment-15950697
]
Hadoop QA commented on HBASE-17835:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 4s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 119m 45s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 173m 31s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861401/HBASE-17835-001.patch
|
| JIRA Issue | HBASE-17835 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3d7835833a74 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / a9682ca |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6277/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6277/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Spelling mistakes in the Java s
[jira] [Updated] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17859:
---
Status: Open (was: Patch Available)
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17859:
---
Attachment: HBASE-17859.v1.patch
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch, HBASE-17859.v1.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17859:
---
Status: Patch Available (was: Open)
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch, HBASE-17859.v1.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17849) PE tool randomness is not totally random
[ https://issues.apache.org/jira/browse/HBASE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17849: --- Attachment: HBASE-17849.patch A simple way to allow randomReads to do randomscans. For now --size and --rows are mutually exclusive. But I think for randomReads and randomSeekScans we can allow both. With --size we can see the end range of the rows and --rows will specify per client how many rows to be scanned. I thought adding a new config was not needed. Open to suggestions here. > PE tool randomness is not totally random > > > Key: HBASE-17849 > URL: https://issues.apache.org/jira/browse/HBASE-17849 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17849.patch > > > Recently we were using the PE tool for doing some bucket cache related > performance tests. One thing that we noted was that the way the random read > works is not totally random. > Suppose we load 200G of data using --size param and then we use --rows=50 > to do the randomRead. The assumption was among the 200G of data it could > generate randomly 50 row keys to do the reads. > But it so happens that the PE tool generates random rows only on those set of > row keys which falls under the first 50 rows. > This was quite evident when we tried to use HBASE-15314 in our testing. > Suppose we split the bucket cache of size 200G into 2 files each 100G the > randomReads with --rows=50 always lands in the first file and not in the > 2nd file. Better to make PE purely random. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17849) PE tool randomness is not totally random
[ https://issues.apache.org/jira/browse/HBASE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17849: --- Status: Patch Available (was: Open) > PE tool randomness is not totally random > > > Key: HBASE-17849 > URL: https://issues.apache.org/jira/browse/HBASE-17849 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17849.patch > > > Recently we were using the PE tool for doing some bucket cache related > performance tests. One thing that we noted was that the way the random read > works is not totally random. > Suppose we load 200G of data using --size param and then we use --rows=50 > to do the randomRead. The assumption was among the 200G of data it could > generate randomly 50 row keys to do the reads. > But it so happens that the PE tool generates random rows only on those set of > row keys which falls under the first 50 rows. > This was quite evident when we tried to use HBASE-15314 in our testing. > Suppose we split the bucket cache of size 200G into 2 files each 100G the > randomReads with --rows=50 always lands in the first file and not in the > 2nd file. Better to make PE purely random. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17857) Remove IS annotations from IA.Public classes
[
https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950755#comment-15950755
]
Hadoop QA commented on HBASE-17857:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 1s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 42s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 9s
{color} | {color:green} hbase-annotations in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 58s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 7s {color} |
{color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 120m 39s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 17s
{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 197m 4s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestInterfaceAudienceAnnotations |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861402/HBASE-17857.patch |
| JIRA Issue | HBASE-17857 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux db5e0fcd76af 3.13.0-105-generic #152-Ubuntu
[jira] [Commented] (HBASE-17821) The CompoundConfiguration#toString is wrong
[
https://issues.apache.org/jira/browse/HBASE-17821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950766#comment-15950766
]
Hudson commented on HBASE-17821:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2773 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2773/])
HBASE-17821: The CompoundConfiguration#toString is wrong (chia7712: rev
a9682ca5dc46a74edca3da630560fbaf5d0cf38b)
* (edit)
hbase-common/src/main/java/org/apache/hadoop/hbase/CompoundConfiguration.java
> The CompoundConfiguration#toString is wrong
> ---
>
> Key: HBASE-17821
> URL: https://issues.apache.org/jira/browse/HBASE-17821
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Yi Liang
>Priority: Trivial
> Labels: beginner
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBase-17821-V1.patch, HBase-17821-V1.patch
>
>
> Find this bug when reading code. We dont use the API, so it is a trivial bug.
> sb.append(this.configs); -> sb.append(m);
> {noformat}
> @Override
> public String toString() {
> StringBuffer sb = new StringBuffer();
> sb.append("CompoundConfiguration: " + this.configs.size() + " configs");
> for (ImmutableConfigMap m : this.configs) {
> sb.append(this.configs);
> }
> return sb.toString();
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v16.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, > HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, > HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, > HBASE-9393.v16.patch, HBASE-9393.v1.patch, HBASE-9393.v2.patch, > HBASE-9393.v3.patch, HBASE-9393.v4.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v7.patch, > HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950785#comment-15950785
]
Ashish Singhi commented on HBASE-9393:
--
Attached patch v16 for further reviews. Once the master branch patch is
accepted I will attach patches for other branches.
Response to [~busbey] comments,
{quote}
The addition of the unbuffer call here means that we need to update the
javadocs for HFile.createReader(FileSystem, Path, FSDataInputStreamWrapper,
long, CacheConfig, Configuration) and HFile.createReaderFromStream(Path,
FSDataInputStream, long, CacheConfig, Configuration) to note that callers need
to ensure no other threads have access to the passed FSDISW instance.
We should also ensure that existing calls to those methods are safely passing
the FSDISW instance.
{quote}
No need, the new reference of FSDISW is just created and passed from this
methods.
{quote}
Just want to make sure I'm following the rationale correctly here. This won't
actually take care of unbuffering if the lock is held e.g. for reading. I think
this is fine, since it implies someone else is still using the stream and
presumably they will also attempt to unbuffer when they are done.
{quote}
Yes.
> Hbase does not closing a closed socket resulting in many CLOSE_WAIT
>
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node,
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-9393.patch, HBASE-9393.v10.patch,
> HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch,
> HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch,
> HBASE-9393.v16.patch, HBASE-9393.v1.patch, HBASE-9393.v2.patch,
> HBASE-9393.v3.patch, HBASE-9393.v4.patch, HBASE-9393.v5.patch,
> HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v6.patch,
> HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v7.patch,
> HBASE-9393.v8.patch, HBASE-9393.v9.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect
> to the datanode because too many mapped sockets from one host to another on
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart
> hbase to solve the porblem, later in time it will incease to 60-100K sockets
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219 0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -Dhbase.log.dir=/var/log/hbase
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950794#comment-15950794
]
Hadoop QA commented on HBASE-17859:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 6s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 54s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 40s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861435/HBASE-17859.v1.patch |
| JIRA Issue | HBASE-17859 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9865a64654f0 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a9682ca |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6282/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6282/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch, HBASE-17859.v1.patch
>
>
> buf2.get( i ) & 0xFF; -> buf
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Attachment: HBASE-15143-BM-0007.patch > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, HBASE-15143-BM-0004.patch, > HBASE-15143-BM-0005.patch, HBASE-15143-BM-0006.patch, > HBASE-15143-BM-0006.patch, HBASE-15143-BM-0007.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Status: Open (was: Patch Available) > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, HBASE-15143-BM-0004.patch, > HBASE-15143-BM-0005.patch, HBASE-15143-BM-0006.patch, > HBASE-15143-BM-0006.patch, HBASE-15143-BM-0007.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Status: Patch Available (was: Open) > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, HBASE-15143-BM-0004.patch, > HBASE-15143-BM-0005.patch, HBASE-15143-BM-0006.patch, > HBASE-15143-BM-0006.patch, HBASE-15143-BM-0007.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17857) Remove IS annotations from IA.Public classes
[ https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17857: -- Attachment: HBASE-17857-v1.patch Modify TestInterfaceAudienceAnnotations to assert that we do not have IS annotations for IA.Public classes. And I've also added a test to assert that we must have a IS annotation for IA.LimitedPrivate classes but it failed so I've ignored it first. > Remove IS annotations from IA.Public classes > > > Key: HBASE-17857 > URL: https://issues.apache.org/jira/browse/HBASE-17857 > Project: HBase > Issue Type: Sub-task > Components: API >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17857.patch, HBASE-17857-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17858) Update refguide about the IS annotation if necessary
[
https://issues.apache.org/jira/browse/HBASE-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950849#comment-15950849
]
Hadoop QA commented on HBASE-17858:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 53s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 127m 11s
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 172m 45s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.util.TestHBaseFsckTwoRS |
| | hadoop.hbase.quotas.TestQuotaThrottle |
| Timed out junit tests |
org.apache.hadoop.hbase.client.TestCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
| | org.apache.hadoop.hbase.util.TestHBaseFsckTwoRS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.10.1 Server=1.10.1 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861421/HBASE-17858.patch |
| JIRA Issue | HBASE-17858 |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux fe1b71ab06e4 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a9682ca |
| Default Java | 1.8.0_121 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6280/artifact/patchprocess/patch-unit-root.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6280/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6280/testReport/ |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6280/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Update refguide about the IS annotation if necessary
>
>
> Key: HBASE-17858
> URL: https://issues.apache.org/jira/browse/HBASE-17858
> Project: HBase
> Issue Type: Sub-task
> Components: API, documentation
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17858.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Status: Open (was: Patch Available) > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, HBASE-15143-BM-0004.patch, > HBASE-15143-BM-0005.patch, HBASE-15143-BM-0006.patch, > HBASE-15143-BM-0006.patch, HBASE-15143-BM-0007.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Attachment: HBASE-15143-BM-0008.patch > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, HBASE-15143-BM-0004.patch, > HBASE-15143-BM-0005.patch, HBASE-15143-BM-0006.patch, > HBASE-15143-BM-0006.patch, HBASE-15143-BM-0007.patch, > HBASE-15143-BM-0008.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Status: Patch Available (was: Open) > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, HBASE-15143-BM-0004.patch, > HBASE-15143-BM-0005.patch, HBASE-15143-BM-0006.patch, > HBASE-15143-BM-0006.patch, HBASE-15143-BM-0007.patch, > HBASE-15143-BM-0008.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17853) Link to a webpage does not work
[ https://issues.apache.org/jira/browse/HBASE-17853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950934#comment-15950934 ] Sean Busbey commented on HBASE-17853: - we could switch to the internet archive mirror: https://web-beta.archive.org/web/20140104070155/http://blog.devving.com/why-does-hbase-care-about-etchosts > Link to a webpage does not work > --- > > Key: HBASE-17853 > URL: https://issues.apache.org/jira/browse/HBASE-17853 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 1.1.8 >Reporter: manojkumargandikota > > when we go to the webpage in the link, > https://hbase.apache.org/book.html#quickstart, there is a text as below. > "Prior to HBase 0.94.x, HBase expected the loopback IP address to be > 127.0.0.1. Ubuntu and some other distributions default to 127.0.1.1 and this > will cause problems for you. See Why does HBase care about /etc/hosts? for > detail". > The web link provided for the text "Why does HBase care about /etc/hosts?" > does not work. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16820) BulkLoad mvcc visibility only works accidentally
[ https://issues.apache.org/jira/browse/HBASE-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16820: Status: In Progress (was: Patch Available) > BulkLoad mvcc visibility only works accidentally > - > > Key: HBASE-16820 > URL: https://issues.apache.org/jira/browse/HBASE-16820 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.8 >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Critical > Attachments: HBASE-16820-branch-1.1-v0.patch > > > [~sergey.soldatov] has been debugging an issue with a 1.1 code base where the > commit for HBASE-16721 broke the bulk load visibility. After bulk load, the > bulk load files is not visible because the sequence id assigned to the bulk > load is not advanced in mvcc. > Debugging further, we have noticed that bulk load behavior is wrong, but it > works "accidentally" in all code bases (but broken in 1.1 after HBASE-16721). > Let me explain: > - BL request can optionally request a flush before hand (this should be the > default) which causes the flush to happen with some sequenceId. The flush > sequence id is one past all the cells' sequenceids. This flush sequence id is > returned as a result to the flush operation. > - BL then uses this particular sequenceId to mark the files, but itself does > not get a new sequenceid of its own, or advance the mvcc number. > - BL completes WITHOUT making sure that the sequence id is visible. > - BL itself though writes entries to the WAL for the BL event, which in 1.2 > code bases goes through the whole mvcc + seqId paths, which makes sure that > earlier sequenceIds (the flush sequenceId) are visible via mvcc. > The problem with 1.1 is that the WAL entries only get sequence ids, but do > not touch mvcc. With the patch for HBASE-16721, we have made it so that the > flushedSequenceId is not used in mvcc as the highest read point (although all > the data is still visible). > BL relying on the flush sequence id is wrong for two reasons: > - BL files are loaded with the flush sequence id from the memstore. This > particular sequence id is used twice for two different things and ends up > being the sequence id for flushed file as well as BL'ed files. > - BL should make sure that it gets a new sequence id and that sequence id is > visible before returning the results. > [~ndimiduk] FYI. > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15143) Procedure v2 - Web UI displaying queues
[
https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951025#comment-15951025
]
Hadoop QA commented on HBASE-15143:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 27 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch 13 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 1s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 49s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 18s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 58s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 20s {color}
| {color:red} hbase-shell in the patch failed. {color} |
| {color:red}-1{color
[jira] [Commented] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951055#comment-15951055
]
Chia-Ping Tsai commented on HBASE-17859:
Will commit it tomorrow if no objection.
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch, HBASE-17859.v1.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17343) Make Compacting Memstore default in 2.0 with BASIC as the default type
[ https://issues.apache.org/jira/browse/HBASE-17343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951105#comment-15951105 ] stack commented on HBASE-17343: --- On my comment above about commit first and then test, a thread on 2.0 up on dev list suggests this as general approach (using this feature as example) and no pushback as yet. Lets let the mail thread hang another day or so to see if objection. If none, lets commit (next week)? Meantime, [~anoop.hbase] tests will help. > Make Compacting Memstore default in 2.0 with BASIC as the default type > -- > > Key: HBASE-17343 > URL: https://issues.apache.org/jira/browse/HBASE-17343 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17343-V01.patch, HBASE-17343-V02.patch > > > FYI [~anastas], [~eshcar] and [~ebortnik]. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951106#comment-15951106 ] Ted Yu commented on HBASE-16179: [~busbey] [~jerryhe]: How can we move this forward ? Without support for Spark 2.x, it makes little sense to include hbase-spark module for 2.0.0 > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, > 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, > 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, > 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt, > 16179.v25.txt, 16179.v26.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, > 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17857) Remove IS annotations from IA.Public classes
[ https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951173#comment-15951173 ] stack commented on HBASE-17857: --- Skimmed. +1. Excellent. Nice doc in TestIA thingy class. Thanks. > Remove IS annotations from IA.Public classes > > > Key: HBASE-17857 > URL: https://issues.apache.org/jira/browse/HBASE-17857 > Project: HBase > Issue Type: Sub-task > Components: API >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17857.patch, HBASE-17857-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17859) ByteBufferUtils#compareTo is wrong
[
https://issues.apache.org/jira/browse/HBASE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951176#comment-15951176
]
stack commented on HBASE-17859:
---
+1 Great find [~chia7712]
> ByteBufferUtils#compareTo is wrong
> --
>
> Key: HBASE-17859
> URL: https://issues.apache.org/jira/browse/HBASE-17859
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17859.v0.patch, HBASE-17859.v1.patch
>
>
> buf2.get( i ) & 0xFF; -> buf2.get(j) & 0xFF;
> {noformat}
> public static int compareTo(byte [] buf1, int o1, int l1, ByteBuffer buf2,
> int o2, int l2) {
>//
> int end1 = o1 + l1;
> int end2 = o2 + l2;
> for (int i = o1, j = o2; i < end1 && j < end2; i++, j++) {
> int a = buf1[i] & 0xFF;
> int b = buf2.get(i) & 0xFF;
> if (a != b) {
> return a - b;
> }
> }
> return l1 - l2;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17215) Separate small/large file delete threads in HFileCleaner to accelerate archived hfile cleanup speed
[
https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951178#comment-15951178
]
huaxiang sun commented on HBASE-17215:
--
{quote}
Please carefully check and confirm whether this is caused by pressure on
Namenode, and if so the change here might worsen it (more requests in parallel,
although not that much). And good luck(smile).
{quote}
Thanks [~carp84] for the great advice. We checked the name node, its workload
is not heavy. Still investigating why it takes 120 ms to delete one file as
tracelog seems to tell us it should be much faster.
Thanks for the patch! We will apply this patch and HBASE-17854 to see how much
is improved.
> Separate small/large file delete threads in HFileCleaner to accelerate
> archived hfile cleanup speed
> ---
>
> Key: HBASE-17215
> URL: https://issues.apache.org/jira/browse/HBASE-17215
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17215.patch, HBASE-17215.v2.patch,
> HBASE-17215.v3.patch
>
>
> When using PCIe-SSD the flush speed will be really quick, and although we
> have per CF flush, we still have the
> {{hbase.regionserver.optionalcacheflushinterval}} setting and some other
> mechanism to avoid data kept in memory for too long to flush small hfiles. In
> our online environment we found the single thread cleaner kept cleaning
> earlier flushed small files while large files got no chance, which caused
> disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the
> workload of namenode, so here we propose to separate large/small hfile
> cleaner threads just like we do for compaction, and it turned out to work
> well in our cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17857) Remove IS annotations from IA.Public classes
[
https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951201#comment-15951201
]
Hadoop QA commented on HBASE-17857:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 10s
{color} | {color:green} hbase-annotations in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 110m 55s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 3s
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 37s
{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 189m 37s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861445/HBASE-17857-v1.patch |
| JIRA Issue | HBASE-17857 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5891a8d73df7 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | mav
[jira] [Updated] (HBASE-16222) fix RegionServer Host Name
[ https://issues.apache.org/jira/browse/HBASE-16222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weizhan Zeng updated HBASE-16222: - Component/s: (was: Balancer) (was: Region Assignment) (was: Client) rsgroup > fix RegionServer Host Name > -- > > Key: HBASE-16222 > URL: https://issues.apache.org/jira/browse/HBASE-16222 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0, 1.1.2, 1.2.2 >Reporter: weizhan Zeng >Priority: Minor > Attachments: HBASE-16222.patch > > > RegionServer Host Name was not equa meta table'r regionserver hostName. > For example, host BeiJIN.98.100.hbase.com In zk or master is > beijin.98.100.hbase.com , but in meter table is BeiJIN.98.199.hbase.com. > Because of this , when rsgroup balance compare host , that cause problem .The > reason is when regionserver report host to master , it will make the > hostname to lowcase。 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951249#comment-15951249
]
Sean Busbey commented on HBASE-16179:
-
bq. How can we move this forward ?
I've been waiting on the dev@ discussion. You could start it.
{quote}
bq. just properly kept out of the classpath for general HBase uses
Any suggestion on how the above can be achieved ?
{quote}
We currently build a classpath for our services and utilities. Just make sure
that however those classpaths are formed we don't include the various
sparkXscala specific jars.
bq. Without support for Spark 2.x, it makes little sense to include hbase-spark
module for 2.0.0
I don't agree with this statement; there are plenty of Spark 1.6 users still. I
do agree that I'd like Spark 2 support for HBase 2.0.0.
If [~jerryhe] is also game, I'd be up for us changing gears a bit here. We
could focus here just on what _has_ to be done for correctness and do things
that would be nice in follow-ons.
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt,
> 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt,
> 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt,
> 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt,
> 16179.v25.txt, 16179.v26.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt,
> 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15143) Procedure v2 - Web UI displaying queues
[
https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951291#comment-15951291
]
Hadoop QA commented on HBASE-15143:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
8s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 7s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 25 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch 13 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 52s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 52s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 23s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 39s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 3s {color} |
{color:red} hbase-shell in the patch failed. {color} |
| {color:green}+1{colo
[jira] [Commented] (HBASE-17172) Optimize mob compaction with _del files
[
https://issues.apache.org/jira/browse/HBASE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951360#comment-15951360
]
huaxiang sun commented on HBASE-17172:
--
[~jingcheng.du], a late follow-up on this. Grouping delete files by its
first/last key is to avoid including delete files to set of
files-to-be-compacted as much as possible. If only started key is used, there
is one case which I am not sure how to handle it (maybe I am following your
idea correctly).
Let's say, for region 1, it starts with key0, ends at key2. It has one delete
file key0***_del. After that, the region may split to region1-0, region1-1, For
region1-1, key0***_del may be included for compaction as it may contain keys
for it. My understanding is that if we only use startKey to group files,
key0***_del will not be included in region1-1's mob compaction.
Maybe as you said
{quote}
Since now we have always retained the delete markers in hfiles,
{quota}
It is ok not to include the delete file with reigon1-1, data for the delete
cells will still be kept, and they will be bulkloaded after mob compaction,
since delete markers are still in hfiles, they will not show up.
Is my understanding correct? Thanks [~jingcheng.du]!
> Optimize mob compaction with _del files
> ---
>
> Key: HBASE-17172
> URL: https://issues.apache.org/jira/browse/HBASE-17172
> Project: HBase
> Issue Type: Improvement
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Fix For: 2.0.0
>
> Attachments: HBASE-17172-master-001.patch,
> HBASE-17172.master.001.patch, HBASE-17172.master.002.patch,
> HBASE-17172.master.003.patch
>
>
> Today, when there is a _del file in mobdir, with major mob compaction, every
> mob file will be recompacted, this causes lots of IO and slow down major mob
> compaction (may take months to finish). This needs to be improved. A few
> ideas are:
> 1) Do not compact all _del files into one, instead, compact them based on
> groups with startKey as the key. Then use firstKey/startKey to make each mob
> file to see if the _del file needs to be included for this partition.
> 2). Based on the timerange of the _del file, compaction for files after that
> timerange does not need to include the _del file as these are newer files.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17172) Optimize mob compaction with _del files
[
https://issues.apache.org/jira/browse/HBASE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951360#comment-15951360
]
huaxiang sun edited comment on HBASE-17172 at 3/31/17 5:35 PM:
---
[~jingcheng.du], a late follow-up on this. Grouping delete files by its
first/last key is to avoid including delete files to set of
files-to-be-compacted as much as possible. If only started key is used, there
is one case which I am not sure how to handle it (maybe I am following your
idea correctly).
Let's say, for region 1, it starts with key0, ends at key2. It has one delete
file key0***_del. After that, the region may split to region1-0, region1-1, For
region1-1, key0***_del may be included for compaction as it may contain keys
for it. My understanding is that if we only use startKey to group files,
key0***_del will not be included in region1-1's mob compaction.
Maybe as you said
{quote}
Since now we have always retained the delete markers in hfiles,
{quote}
It is ok not to include the delete file with reigon1-1, data for the delete
cells will still be kept, and they will be bulkloaded after mob compaction,
since delete markers are still in hfiles, they will not show up.
Is my understanding correct? Thanks [~jingcheng.du]!
was (Author: huaxiang):
[~jingcheng.du], a late follow-up on this. Grouping delete files by its
first/last key is to avoid including delete files to set of
files-to-be-compacted as much as possible. If only started key is used, there
is one case which I am not sure how to handle it (maybe I am following your
idea correctly).
Let's say, for region 1, it starts with key0, ends at key2. It has one delete
file key0***_del. After that, the region may split to region1-0, region1-1, For
region1-1, key0***_del may be included for compaction as it may contain keys
for it. My understanding is that if we only use startKey to group files,
key0***_del will not be included in region1-1's mob compaction.
Maybe as you said
{quote}
Since now we have always retained the delete markers in hfiles,
{quota}
It is ok not to include the delete file with reigon1-1, data for the delete
cells will still be kept, and they will be bulkloaded after mob compaction,
since delete markers are still in hfiles, they will not show up.
Is my understanding correct? Thanks [~jingcheng.du]!
> Optimize mob compaction with _del files
> ---
>
> Key: HBASE-17172
> URL: https://issues.apache.org/jira/browse/HBASE-17172
> Project: HBase
> Issue Type: Improvement
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Fix For: 2.0.0
>
> Attachments: HBASE-17172-master-001.patch,
> HBASE-17172.master.001.patch, HBASE-17172.master.002.patch,
> HBASE-17172.master.003.patch
>
>
> Today, when there is a _del file in mobdir, with major mob compaction, every
> mob file will be recompacted, this causes lots of IO and slow down major mob
> compaction (may take months to finish). This needs to be improved. A few
> ideas are:
> 1) Do not compact all _del files into one, instead, compact them based on
> groups with startKey as the key. Then use firstKey/startKey to make each mob
> file to see if the _del file needs to be included for this partition.
> 2). Based on the timerange of the _del file, compaction for files after that
> timerange does not need to include the _del file as these are newer files.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17836) CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell
[ https://issues.apache.org/jira/browse/HBASE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17836: --- Attachment: HBASE-17836.v0.patch > CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell > --- > > Key: HBASE-17836 > URL: https://issues.apache.org/jira/browse/HBASE-17836 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17836.v0.patch > > > We call CellUtil#estimatedSerializedSize to calculate the size of rows when > scanning. If the input is ByteBufferCell, the > CellUtil#estimatedSerializedSizeOf parses many length components to get the > qualifierLength stored in the backing buffer. > We should consider using the KeyValueUtil#getSerializedSize. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17836) CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell
[ https://issues.apache.org/jira/browse/HBASE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17836: --- Assignee: Chia-Ping Tsai Status: Patch Available (was: Open) > CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell > --- > > Key: HBASE-17836 > URL: https://issues.apache.org/jira/browse/HBASE-17836 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17836.v0.patch > > > We call CellUtil#estimatedSerializedSize to calculate the size of rows when > scanning. If the input is ByteBufferCell, the > CellUtil#estimatedSerializedSizeOf parses many length components to get the > qualifierLength stored in the backing buffer. > We should consider using the KeyValueUtil#getSerializedSize. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17836) CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell
[
https://issues.apache.org/jira/browse/HBASE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951411#comment-15951411
]
Anoop Sam John commented on HBASE-17836:
You will have to add Bytes.SIZEOF_INT. (See KV case)
if (cell instanceof KeyValue) {
1392 return ((KeyValue)cell).getLength() + Bytes.SIZEOF_INT;
1393}
This check can be remove now as KV is ExtendedCell type any way
> CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell
> ---
>
> Key: HBASE-17836
> URL: https://issues.apache.org/jira/browse/HBASE-17836
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17836.v0.patch
>
>
> We call CellUtil#estimatedSerializedSize to calculate the size of rows when
> scanning. If the input is ByteBufferCell, the
> CellUtil#estimatedSerializedSizeOf parses many length components to get the
> qualifierLength stored in the backing buffer.
> We should consider using the KeyValueUtil#getSerializedSize.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17836) CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell
[ https://issues.apache.org/jira/browse/HBASE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951428#comment-15951428 ] Chia-Ping Tsai commented on HBASE-17836: bq. This check can be remove now as KV is ExtendedCell type any way copy that. Thanks for the feedback. [~anoop.hbase] > CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell > --- > > Key: HBASE-17836 > URL: https://issues.apache.org/jira/browse/HBASE-17836 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17836.v0.patch > > > We call CellUtil#estimatedSerializedSize to calculate the size of rows when > scanning. If the input is ByteBufferCell, the > CellUtil#estimatedSerializedSizeOf parses many length components to get the > qualifierLength stored in the backing buffer. > We should consider using the KeyValueUtil#getSerializedSize. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17836) CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell
[
https://issues.apache.org/jira/browse/HBASE-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951467#comment-15951467
]
Hadoop QA commented on HBASE-17836:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
35m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 30s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861482/HBASE-17836.v0.patch |
| JIRA Issue | HBASE-17836 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3c17d10c73d4 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a9682ca |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6287/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6287/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell
> ---
>
> Key: HBASE-17836
> URL: https://issues.apache.org/jira/browse/HBASE-17836
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>
[jira] [Updated] (HBASE-16780) Since move to protobuf3.1, Cells are limited to 64MB where previous they had no limit
[ https://issues.apache.org/jira/browse/HBASE-16780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16780: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Committed to master. Resolving. > Since move to protobuf3.1, Cells are limited to 64MB where previous they had > no limit > - > > Key: HBASE-16780 > URL: https://issues.apache.org/jira/browse/HBASE-16780 > Project: HBase > Issue Type: Bug > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16780.master.001.patch, > HBASE-16780.master.002.patch > > > Change in protobuf behavior noticed by [~mbertozzi]. His test > TestStressWALProcedureStore#testEntrySizeLimit keeps upping size we write and > he found that now we are bound at 64MB. Digging, yeah, there is a check in > place that was not there before. Filed > https://github.com/grpc/grpc-java/issues/2324 but making issue here in > meantime in case we have to note a change-in-behavior in hbase-2.0.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17860) Implement secure native client connection
Ted Yu created HBASE-17860: -- Summary: Implement secure native client connection Key: HBASE-17860 URL: https://issues.apache.org/jira/browse/HBASE-17860 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Priority: Critical So far, the native client communicates with insecure cluster. This JIRA is to add secure connection support for native client using Cyrus library. The work is based on earlier implementation and is redone via wangle and folly frameworks. Thanks to [~devaraj] who started the initiative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17698) ReplicationEndpoint choosing sinks
[ https://issues.apache.org/jira/browse/HBASE-17698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951573#comment-15951573 ] Andrew Purtell commented on HBASE-17698: +1 will commit shortly > ReplicationEndpoint choosing sinks > -- > > Key: HBASE-17698 > URL: https://issues.apache.org/jira/browse/HBASE-17698 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: churro morales >Assignee: Karan Mehta > Attachments: HBASE-17698.patch > > > The only time we choose new sinks is when we have a ConnectException, but we > have encountered other exceptions where there is a problem contacting a > particular sink and replication gets backed up for any sources that try that > sink > HBASE-17675 occurred when there was a bad keytab refresh and the source was > stuck. > Another issue we recently had was a bad drive controller on the sink side and > replication was stuck again. > Is there any reason not to choose new sinks anytime we have a > RemoteException? I can understand TableNotFound we don't have to choose new > sinks, but for all other cases this seems like the safest approach. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17860) Implement secure native client connection
[ https://issues.apache.org/jira/browse/HBASE-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951575#comment-15951575 ] Ted Yu commented on HBASE-17860: Currently cleaning up code and replacing hardcoded principal with one retrieved from kerberos ccache. Secure mode is detected through the value for config "hbase.security.authentication" being "kerberos". Service name is retrieved from the value for config "hbase.regionserver.kerberos.principal" > Implement secure native client connection > - > > Key: HBASE-17860 > URL: https://issues.apache.org/jira/browse/HBASE-17860 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > > So far, the native client communicates with insecure cluster. > This JIRA is to add secure connection support for native client using Cyrus > library. > The work is based on earlier implementation and is redone via wangle and > folly frameworks. > Thanks to [~devaraj] who started the initiative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17861:
-
Fix Version/s: 1.4.0
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 1.4.0
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17861:
-
Affects Version/s: 1.4.0
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 1.4.0
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
Yi Liang created HBASE-17861:
Summary: Regionserver down when checking the permission of staging
dir if hbase.rootdir is on S3
Key: HBASE-17861
URL: https://issues.apache.org/jira/browse/HBASE-17861
Project: HBase
Issue Type: Bug
Reporter: Yi Liang
Assignee: Yi Liang
Found some issue, when set up HBASE-17437: Support specifying a WAL directory
outside of the root directory.
The region server are showdown when I add following config into hbase-site.xml
hbase.rootdir = hdfs://xx/xx
hbase.wal.dir = s3a://xx//xx
hbase.coprocessor.region.classes =
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
Error is below
{noformat}
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
java.lang.IllegalStateException: Directory already exists but permissions
aren't set to '-rwx--x--x'
{noformat}
The reason is that, when hbase enable securebulkload, hbase will create a
folder in s3, it can not set above permission, because in s3, all files are
listed as having full read/write permissions and all directories appear to have
full rwx permissions. See simulated permission section in
https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951589#comment-15951589
]
Yi Liang commented on HBASE-17861:
--
And in our master branch, the code in SecureBulkLoadEndpoint has been
refactored, and the statement check the folder permission is removed. so the
patch is only for branch-1.0
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 1.4.0
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17861:
-
Attachment: HBASE-17861-V1.patch
HBase-17821-V1.patch
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17861:
-
Attachment: (was: HBase-17821-V1.patch)
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951589#comment-15951589
]
Yi Liang edited comment on HBASE-17861 at 3/31/17 8:21 PM:
---
And in our master branch, the code in SecureBulkLoadEndpoint has been
refactored, and the statement check the staging folder permission is removed.
so the patch is only for branch-1.0
was (Author: easyliangjob):
And in our master branch, the code in SecureBulkLoadEndpoint has been
refactored, and the statement check the folder permission is removed. so the
patch is only for branch-1.0
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17861:
-
Labels: filesystem s3 wal (was: )
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17860) Implement secure native client connection
[
https://issues.apache.org/jira/browse/HBASE-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951593#comment-15951593
]
Ted Yu commented on HBASE-17860:
Here is brief procedure for testing:
* install cyrus-sasl-2.1.26 on docker vm
* follow this link to install kerberos packages:
https://help.ubuntu.com/lts/serverguide/kerberos.html
* follow this link to configure KDC:
https://www.rootusers.com/how-to-configure-linux-to-authenticate-using-kerberos/
* apply the patch which sets necessary config in conf/hbase-site.xml
* run bin/start-hbase.sh to start hbase server
* use hbase shell to create table and populate with:
{code}
test1 column=d:1,
timestamp=1490984371943, value=value1
test1 column=d:extra,
timestamp=1490984371949, value=value for extra
test2 column=d:2,
timestamp=1490831145321, value=value2
test2 column=d:extra,
timestamp=1490831219721, value=value for extra
{code}
* run the following command and verify that ClientTest.PutGet passes:
buck test //core:client-test --no-results-cache
> Implement secure native client connection
> -
>
> Key: HBASE-17860
> URL: https://issues.apache.org/jira/browse/HBASE-17860
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
>
> So far, the native client communicates with insecure cluster.
> This JIRA is to add secure connection support for native client using Cyrus
> library.
> The work is based on earlier implementation and is redone via wangle and
> folly frameworks.
> Thanks to [~devaraj] who started the initiative.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17862) Condition that always returns true
JC created HBASE-17862:
--
Summary: Condition that always returns true
Key: HBASE-17862
URL: https://issues.apache.org/jira/browse/HBASE-17862
Project: HBase
Issue Type: Bug
Components: Client
Reporter: JC
Priority: Trivial
Hi
In recent github mirror of hbase, I've found the following code smell.
Path:
hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
{code}
209
210 ColumnPaginationFilter other = (ColumnPaginationFilter)o;
211 if (this.columnOffset != null) {
212 return this.getLimit() == this.getLimit() &&
213 Bytes.equals(this.getColumnOffset(), other.getColumnOffset());
214 }
{code}
It should be?
{code}
212 return this.getLimit() == other.getLimit() &&
{code}
This might be just a code smell as Bytes.equals can be enough for the return
value but wanted to report just in case.
Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17860) Implement secure native client connection
[
https://issues.apache.org/jira/browse/HBASE-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951593#comment-15951593
]
Ted Yu edited comment on HBASE-17860 at 3/31/17 8:28 PM:
-
Here is brief procedure for testing:
* install cyrus-sasl-2.1.26 on docker vm
* follow this link to install kerberos packages:
https://help.ubuntu.com/lts/serverguide/kerberos.html
* follow this link to configure KDC:
https://www.rootusers.com/how-to-configure-linux-to-authenticate-using-kerberos/
* generate hbase-host.keytab for server (and optionally hbase.keytab for user)
* run kinit with the keytab
* apply the patch which sets necessary config in conf/hbase-site.xml
* run bin/start-hbase.sh to start hbase server
* use hbase shell to create table and populate with:
{code}
test1 column=d:1,
timestamp=1490984371943, value=value1
test1 column=d:extra,
timestamp=1490984371949, value=value for extra
test2 column=d:2,
timestamp=1490831145321, value=value2
test2 column=d:extra,
timestamp=1490831219721, value=value for extra
{code}
* run the following command and verify that ClientTest.PutGet passes:
buck test //core:client-test --no-results-cache
was (Author: yuzhih...@gmail.com):
Here is brief procedure for testing:
* install cyrus-sasl-2.1.26 on docker vm
* follow this link to install kerberos packages:
https://help.ubuntu.com/lts/serverguide/kerberos.html
* follow this link to configure KDC:
https://www.rootusers.com/how-to-configure-linux-to-authenticate-using-kerberos/
* apply the patch which sets necessary config in conf/hbase-site.xml
* run bin/start-hbase.sh to start hbase server
* use hbase shell to create table and populate with:
{code}
test1 column=d:1,
timestamp=1490984371943, value=value1
test1 column=d:extra,
timestamp=1490984371949, value=value for extra
test2 column=d:2,
timestamp=1490831145321, value=value2
test2 column=d:extra,
timestamp=1490831219721, value=value for extra
{code}
* run the following command and verify that ClientTest.PutGet passes:
buck test //core:client-test --no-results-cache
> Implement secure native client connection
> -
>
> Key: HBASE-17860
> URL: https://issues.apache.org/jira/browse/HBASE-17860
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
>
> So far, the native client communicates with insecure cluster.
> This JIRA is to add secure connection support for native client using Cyrus
> library.
> The work is based on earlier implementation and is redone via wangle and
> folly frameworks.
> Thanks to [~devaraj] who started the initiative.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17861:
-
Status: Patch Available (was: Open)
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16942) Add FavoredStochasticLoadBalancer and FN Candidate generators
[ https://issues.apache.org/jira/browse/HBASE-16942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951596#comment-15951596 ] Thiruvel Thirumoolan commented on HBASE-16942: -- Will rebase and upload. > Add FavoredStochasticLoadBalancer and FN Candidate generators > - > > Key: HBASE-16942 > URL: https://issues.apache.org/jira/browse/HBASE-16942 > Project: HBase > Issue Type: Sub-task > Components: FavoredNodes >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16942.master.001.patch, > HBASE-16942.master.002.patch, HBASE-16942.master.003.patch, > HBASE-16942.master.004.patch, HBASE-16942.master.005.patch, > HBASE-16942.master.006.patch, HBASE-16942.master.007.patch, > HBASE-16942.master.008.patch, HBASE_16942_rough_draft.patch > > > This deals with the balancer based enhancements to favored nodes patch as > discussed in HBASE-15532. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951605#comment-15951605
]
Ted Yu commented on HBASE-17861:
lgtm
Have you tested with s3a filesystem ?
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951606#comment-15951606
]
Hadoop QA commented on HBASE-17861:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s {color}
| {color:red} HBASE-17861 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861500/HBASE-17861-V1.patch |
| JIRA Issue | HBASE-17861 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6288/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17861:
-
Description:
Found some issue, when set up HBASE-17437: Support specifying a WAL directory
outside of the root directory.
The region server are showdown when I add following config into hbase-site.xml
hbase.rootdir = hdfs://xx/xx
hbase.wal.dir = s3a://xx//xx
hbase.coprocessor.region.classes =
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
Error is below
{noformat}
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
java.lang.IllegalStateException: Directory already exists but permissions
aren't set to '-rwx--x--x'
{noformat}
The reason is that, when hbase enable securebulkload, hbase will create a
folder in s3, it can not set above permission, because in s3, all files are
listed as having full read/write permissions and all directories appear to have
full rwx permissions. See Object stores have differerent authorization models
in
https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
was:
Found some issue, when set up HBASE-17437: Support specifying a WAL directory
outside of the root directory.
The region server are showdown when I add following config into hbase-site.xml
hbase.rootdir = hdfs://xx/xx
hbase.wal.dir = s3a://xx//xx
hbase.coprocessor.region.classes =
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
Error is below
{noformat}
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
java.lang.IllegalStateException: Directory already exists but permissions
aren't set to '-rwx--x--x'
{noformat}
The reason is that, when hbase enable securebulkload, hbase will create a
folder in s3, it can not set above permission, because in s3, all files are
listed as having full read/write permissions and all directories appear to have
full rwx permissions. See simulated permission section in
https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951607#comment-15951607
]
Yi Liang commented on HBASE-17861:
--
Hi Ted, Thanks for reviewing, I have test it on S3 with HBase 1.2.4.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951640#comment-15951640
]
Ted Yu commented on HBASE-17861:
You can name your patch HBASE-17861.branch-1.0.V1.patch
But is there going to be any more release for branch-1.0 ?
Did you mean branch-1.1 ?
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17860) Implement secure native client connection
[
https://issues.apache.org/jira/browse/HBASE-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951593#comment-15951593
]
Ted Yu edited comment on HBASE-17860 at 3/31/17 9:30 PM:
-
Here is brief procedure for testing:
* install cyrus-sasl-2.1.26 on docker vm
* follow this link to install kerberos packages:
https://help.ubuntu.com/lts/serverguide/kerberos.html
* follow this link to configure KDC:
https://www.rootusers.com/how-to-configure-linux-to-authenticate-using-kerberos/
* generate hbase-host.keytab for server (and optionally hbase.keytab for user)
* run kinit with the keytab
* apply the patch which sets necessary config in conf/hbase-site.xml
* run bin/start-hbase.sh to start hbase server
* use hbase shell to create table (test would populate the table with:)
{code}
test1 column=d:1,
timestamp=1490984371943, value=value1
test1 column=d:extra,
timestamp=1490984371949, value=value for extra
test2 column=d:2,
timestamp=1490831145321, value=value2
test2 column=d:extra,
timestamp=1490831219721, value=value for extra
{code}
* run the following command and verify that ClientTest.PutGet passes:
buck test //core:client-test --no-results-cache
was (Author: yuzhih...@gmail.com):
Here is brief procedure for testing:
* install cyrus-sasl-2.1.26 on docker vm
* follow this link to install kerberos packages:
https://help.ubuntu.com/lts/serverguide/kerberos.html
* follow this link to configure KDC:
https://www.rootusers.com/how-to-configure-linux-to-authenticate-using-kerberos/
* generate hbase-host.keytab for server (and optionally hbase.keytab for user)
* run kinit with the keytab
* apply the patch which sets necessary config in conf/hbase-site.xml
* run bin/start-hbase.sh to start hbase server
* use hbase shell to create table and populate with:
{code}
test1 column=d:1,
timestamp=1490984371943, value=value1
test1 column=d:extra,
timestamp=1490984371949, value=value for extra
test2 column=d:2,
timestamp=1490831145321, value=value2
test2 column=d:extra,
timestamp=1490831219721, value=value for extra
{code}
* run the following command and verify that ClientTest.PutGet passes:
buck test //core:client-test --no-results-cache
> Implement secure native client connection
> -
>
> Key: HBASE-17860
> URL: https://issues.apache.org/jira/browse/HBASE-17860
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
>
> So far, the native client communicates with insecure cluster.
> This JIRA is to add secure connection support for native client using Cyrus
> library.
> The work is based on earlier implementation and is redone via wangle and
> folly frameworks.
> Thanks to [~devaraj] who started the initiative.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17860) Implement secure native client connection
[
https://issues.apache.org/jira/browse/HBASE-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17860:
---
Description:
So far, the native client communicates with insecure cluster.
This JIRA is to add secure connection support for native client using Cyrus
library.
The work is based on earlier implementation and is redone via wangle and folly
frameworks.
Thanks to [~devaraj] who started the initiative.
Here is high level description of the design:
* SaslHandler is declared as:
{code}
class SaslHandler
: public wangle::HandlerAdapter>{
{code}
It would be inserted between EventBaseHandler and LengthFieldBasedFrameDecoder
in the pipeline (via ConnectionFactory::Connect())
* SaslHandler would intercept writes to server by buffering the IOBuf's and
start the handshake process (via sasl_client_XX calls provided by Cyrus)
* after handshake is complete, SaslHandler would send the buffered IOBuf's to
server and act as pass-thru from then on
was:
So far, the native client communicates with insecure cluster.
This JIRA is to add secure connection support for native client using Cyrus
library.
The work is based on earlier implementation and is redone via wangle and folly
frameworks.
Thanks to [~devaraj] who started the initiative.
> Implement secure native client connection
> -
>
> Key: HBASE-17860
> URL: https://issues.apache.org/jira/browse/HBASE-17860
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
>
> So far, the native client communicates with insecure cluster.
> This JIRA is to add secure connection support for native client using Cyrus
> library.
> The work is based on earlier implementation and is redone via wangle and
> folly frameworks.
> Thanks to [~devaraj] who started the initiative.
> Here is high level description of the design:
> * SaslHandler is declared as:
> {code}
> class SaslHandler
> : public wangle::HandlerAdapter std::unique_ptr>{
> {code}
> It would be inserted between EventBaseHandler and
> LengthFieldBasedFrameDecoder in the pipeline (via
> ConnectionFactory::Connect())
> * SaslHandler would intercept writes to server by buffering the IOBuf's and
> start the handshake process (via sasl_client_XX calls provided by Cyrus)
> * after handshake is complete, SaslHandler would send the buffered IOBuf's to
> server and act as pass-thru from then on
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
