subject:"\[jira\] \[Commented\] \(HBASE\-15536\) Make AsyncFSWAL as our default WAL"

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-15 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292745#comment-16292745
 ] 

Hudson commented on HBASE-15536:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4230 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/4230/])
HBASE-15536 Make AsyncFSWAL as our default WAL (zhangduo: rev 
11e82de8a1d782be85a776ec08e8cd6a071185bf)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java


> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.3.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-15 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292440#comment-16292440
 ] 

Duo Zhang commented on HBASE-15536:
---

TestLogRolling is used to test FSHLog, so unrelated. But it showed up several 
times in the recent pre commit result. Will open a issue to track it.

And let me commit the patch which change our default WAL to AsyncFSWAL.

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.3.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-15 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292416#comment-16292416
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
3s{color} | {color:blue} The patch file was not named according to hbase's 
naming conventions. Please see 
https://yetus.apache.org/documentation/0.6.0/precommit-patchnames for 
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
37s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
 5s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
22m 32s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0-beta1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 25s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 25s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.wal.TestLogRolling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-15536 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12902257/15536.addendum2.enable.asyncfswal.by.default.txt
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 20d4c115a84e 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / deba43b156 |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| unit |

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-15 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292176#comment-16292176
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
3s{color} | {color:blue} The patch file was not named according to hbase's 
naming conventions. Please see 
https://yetus.apache.org/documentation/0.6.0/precommit-patchnames for 
instructions. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  6s{color} 
| {color:red} HBASE-15536 does not apply to master. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.6.0/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-15536 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12902237/15536.addendum2.enable.asyncfswal.by.default.2.txt
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/10473/console |
| Powered by | Apache Yetus 0.6.0   http://yetus.apache.org |


This message was automatically generated.



> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.3.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-14 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292128#comment-16292128
 ] 

Duo Zhang commented on HBASE-15536:
---

{quote}
[ERROR] testThreeRSAbort(org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL)  
Time elapsed: 20.627 s  <<< ERROR!
org.apache.hadoop.hbase.TableNotFoundException: Region of 
'hbase:namespace,,1513320505933.451650152885a3b41d0b1110deca513c.' is expected 
in the table of 'testThreeRSAbort', but hbase:meta says it is in the table of 
'hbase:namespace'. hbase:meta might be damaged.
{quote}

The error message itself is an error I think... Let me dig...

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.3.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292124#comment-16292124
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
9s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
3s{color} | {color:blue} The patch file was not named according to hbase's 
naming conventions. Please see 
https://yetus.apache.org/documentation/0.6.0/precommit-patchnames for 
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
27s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
39s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
19m 54s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0-beta1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 15s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 11s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-15536 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12902208/15536.addendum2.enable.asyncfswal.by.default.txt
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 94b1fc1e630c 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / deba43b156 |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/10466/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-14 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292094#comment-16292094
 ] 

stack commented on HBASE-15536:
---

TestZKSecretWatcher is flakey.

There must be a failing test in TestHRegion inherited by the other two 
superclasses?

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.3.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290702#comment-16290702
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
3s{color} | {color:blue} The patch file was not named according to hbase's 
naming conventions. Please see 
https://yetus.apache.org/documentation/0.6.0/precommit-patchnames for 
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  6m 
16s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
52s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
22m 31s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0-beta1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 17s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 51s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.regionserver.TestWalAndCompactingMemStoreFlush |
|   | hadoop.hbase.regionserver.TestHRegion |
|   | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
|   | hadoop.hbase.security.token.TestZKSecretWatcher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-15536 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12902013/15536.addendum2.enable.asyncfswal.by.default.3.txt
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux da7c0b965260 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 7466e64abb |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-13 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290478#comment-16290478
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


bq.Is the TestBlockEvictionFromClient failure an asyncfs issue or just a 
general flakey? Let me put up a patch that disables it in the patch to see how 
far we get now.
You need not disable it. Pls see HBASE-19511.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-13 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290477#comment-16290477
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


bq.https://issues.apache.org/jira/browse/HBASE-19511
Sorry I missed pinging you [~Apache9].
with the patch over there the TestBlockEvictionFromClient should no longer be 
flaky. It is actually a bug over the ref counting.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-13 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290473#comment-16290473
 ] 

stack commented on HBASE-15536:
---

Is the TestBlockEvictionFromClient failure an asyncfs issue or just a general 
flakey? Let me put up a patch that disables it in the patch to see how far we 
get now.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-13 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290465#comment-16290465
 ] 

Duo Zhang commented on HBASE-15536:
---

Any progress on TestBlockEvictionFromClient? [~ramkrishna]

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-13 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289074#comment-16289074
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


I found the problem causing test case in that TestBlockEvictionFromClient. Will 
think over and get back to it. It is not always failing but it is now flaky. 

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-12 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288833#comment-16288833
 ] 

Duo Zhang commented on HBASE-15536:
---

No, I haven't started to dig this one yet. Thanks [~ram_krish]. Just go ahead.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-12 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288827#comment-16288827
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


bq.TestBlockEvictionFromClient, it also fails for me locally but do not know 
the reason yet. Will dig later.
I can check this [~Apache9] . Let me know if you have already not started 
checking this. 

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-12 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288809#comment-16288809
 ] 

Duo Zhang commented on HBASE-15536:
---

TestWALOpenAfterDNRollingStart is a problem. Let me open a issue to address it 
first.

And for TestBlockEvictionFromClient, it also fails for me locally but do not 
know the reason yet. Will dig later.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-12 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288746#comment-16288746
 ] 

Duo Zhang commented on HBASE-15536:
---

Let me check the failed UTs.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.2.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-12 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288354#comment-16288354
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
9s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
2s{color} | {color:blue} The patch file was not named according to hbase's 
naming conventions. Please see 
https://yetus.apache.org/documentation/0.6.0/precommit-patchnames for 
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
40s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
13s{color} | {color:red} hbase-server: The patch generated 1 new + 9 unchanged 
- 0 fixed = 10 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
 7s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
60m 30s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 58s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}208m 12s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.regionserver.TestWalAndCompactingMemStoreFlush |
|   | hadoop.hbase.wal.TestWALOpenAfterDNRollingStart |
|   | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
|   | hadoop.hbase.client.TestBlockEvictionFromClient |
|   | hadoop.hbase.master.TestDistributedLogSplitting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-15536 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12901733/15536.addendum2.enable.asyncfswal.by.default.2.txt
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux f83fb5cdfd0f 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-12 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288032#comment-16288032
 ] 

stack commented on HBASE-15536:
---

TestWALOpenAfterDNRollingStart seems to be only legit failure.

WALFactory#createReader at line #300 keeps catching below exception then 
retrying.

Cannot obtain block length for 
LocatedBlock{BP-564709508-192.168.1.139-1513103544167:blk_1073741830_1006; 
getBlockSize()=0; corrupt=false; offset=0; 
locs=[DatanodeInfoWithStorage[127.0.0.1:50269,DS-a382f1d6-b4a4-4a06-a96a-bc94fe7624e0,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:50326,DS-7b446e20-5e84-414e-a2cd-aae470f9ce12,DISK],
 
DatanodeInfoWithStorage[127.0.0.1:50382,DS-1fefa6d9-15fa-4d63-94f6-4dc0dd5d7d94,DISK]]}

You might have input here [~Apache9]? Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.addendum2.enable.asyncfswal.by.default.txt, 
> 15536.addendum2.enable.asyncfswal.by.default.txt, 15536.minor.addendum.patch, 
> HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch, 
> HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-12 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287841#comment-16287841
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
53s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
2s{color} | {color:blue} The patch file was not named according to hbase's 
naming conventions. Please see 
https://yetus.apache.org/documentation/0.6.0/precommit-patchnames for 
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
26s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
29s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
54m 29s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}140m 37s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}215m 26s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
|   | hadoop.hbase.wal.TestWALOpenAfterDNRollingStart |
|   | hadoop.hbase.client.TestBlockEvictionFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-15536 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12901676/15536.addendum2.enable.asyncfswal.by.default.txt
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux e915f96e4a92 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 2e813f106f |
| maven | version: Apache

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286755#comment-16286755
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
8s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
4s{color} | {color:blue} The patch file was not named according to hbase's 
naming conventions. Please see 
https://yetus.apache.org/documentation/0.6.0/precommit-patchnames for 
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
26s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
29s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
54m 17s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 24s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
11s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m  4s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.regionserver.TestWALMonotonicallyIncreasingSeqId |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-15536 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12901561/15536.addendum2.enable.asyncfswal.by.default.txt
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 0e84c69d19ba 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 856ee283fa |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
|

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-11 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286623#comment-16286623
 ] 

stack commented on HBASE-15536:
---

I pulled this into beta-1. We going to do this? Lets. Numbers are good over in  
HBASE-16890 

Here is an addendum to set below:



hbase.wal.provider
asyncfs


  hbase.wal.async.use-shared-event-loop
  
  false
  


> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 15536.minor.addendum.patch, HBASE-15536-v1.patch, 
> HBASE-15536-v2.patch, HBASE-15536-v3.patch, HBASE-15536-v4.patch, 
> HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280767#comment-16280767
 ] 

stack commented on HBASE-15536:
---

Or, sorry, pushed over on the original JIRA, on 15628

commit 52ffa68f8bf94131af18fff79a8b44618ad3515c
Author: Michael Stack 
Date:   Wed Dec 6 11:35:54 2017 -0800

HBASE-15628 Implement an AsyncOutputStream which can work with any 
FileSystem implementation; ADDENDUM -- Change Log output

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15536.minor.addendum.patch, HBASE-15536-v1.patch, 
> HBASE-15536-v2.patch, HBASE-15536-v3.patch, HBASE-15536-v4.patch, 
> HBASE-15536-v5.patch, HBASE-15536.patch, 
> latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280561#comment-16280561
 ] 

stack commented on HBASE-15536:
---

Yeah. Let me add that (Let me change the release note here; I'd made it so it 
says only hadoop-2.8.0^

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-05 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279756#comment-16279756
 ] 

Duo Zhang commented on HBASE-15536:
---

The log level is DEBUG sir, this is just a debug log to tell you that you are 
on hadoop-2.7- and there is PBHelperClient class so we will use PBHelper class 
instead. Maybe I should not print the stack trace out so the message here will 
be less confusing?

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-12-05 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279693#comment-16279693
 ] 

stack commented on HBASE-15536:
---

[~Apache9] Here is why I changed the release note to add this:

commit 95f8e93691ad79b77ae9a4a13208c0d7ca405c96 
Author: Haohui Mai  
Date: Sat Aug 22 13:30:19 2015 -0700 

HDFS-8934. Move ShortCircuitShm to hdfs-client. Contributed by Mingliang 
Liu.


... it is because I get this when I try to run asyncfswal on 
hadoop-2.7.4-SNAPSHOT


2017-12-05 21:52:48,523 DEBUG [main] 
asyncfs.FanOutOneBlockAsyncDFSOutputHelper: No PBHelperClient class found, 
should be hadoop 2.7-
java.lang.ClassNotFoundException: 
org.apache.hadoop.hdfs.protocolPB.PBHelperClient
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at 
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputHelper.java:413)
at 
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:546)
at 
org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:62)
at 
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:158)
at 
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167)
at 
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:100)
at 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:621)
at 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:131)
at 
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:751)
at 
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:489)
at 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:257)
at 
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:70)
at 
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:45)
at 
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:139)
at 
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:55)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:244)
at 
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.openRegion(WALPerformanceEvaluation.java:502)
at 
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.run(WALPerformanceEvaluation.java:336)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at 
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.innerMain(WALPerformanceEvaluation.java:597)
at 
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.main(WALPerformanceEvaluation.java:601)

Do you not get the above?

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-11-28 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270204#comment-16270204
 ] 

Duo Zhang commented on HBASE-15536:
---

2.7.4 is OK sir. In FanOutOneBlockAsyncDFSOutputHelper we will use reflection 
to decide to use PBHelper or PBHelperClient. Maybe the error message is a bit 
confusing and makes you think we can not work together with 2.7.x.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-11-28 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270157#comment-16270157
 ] 

stack commented on HBASE-15536:
---

I think its ok. We can update our hadoop np.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2017-11-28 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270150#comment-16270150
 ] 

Anoop Sam John commented on HBASE-15536:


bq.Requires Hadoop-2.8.0 at least. Depends on PBHelperClient added here:
Oh!  But we are still at 2.7.4 
officially.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-20 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592472#comment-15592472
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


bq. Let me help here.
Thanks Stack. Just seeing your ping.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-20 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592468#comment-15592468
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


I just tested by commenting out log roll. Still it performs slower. I think the 
above things that I mentioned are not the major reasons for the slowness but 
they also add on. Will dig more particularly with the sync part.


> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-20 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592441#comment-15592441
 ] 

stack commented on HBASE-15536:
---

I could do test again, np. Nice digging [~ram_krish]. Am in here myself looking 
at dfsclient at mo (which makes me wish the more for asyncfswal smile).

Looking at the release note, it is missing mention of the limitations of 
asyncwal -- i.e. that maximum size is 16M. Need to add that too. Or rather, 
lets test what happens when too-big load. Let me help here.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-18 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585417#comment-15585417
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


I would generally say that pls do not consider the profile report in terms of 
comparisons in CellComparator. It is always in the top but sometimes it tends 
to vary because
-> Not always profilers give the source of truth
-> Next is that all the comparsions are going to be same like the mutations 
added to memstore and the Scanquerymatcher that does comparsions during 
flushes/compactions.
Ya may be in default case comparisoins are more because they are much faster so 
more flushes/compactions?
As I said the profile does not immediately give the hot spot. But there is 
something other than the comparators that we need to check. I will report back 
on this tomorrow after doing some more analysis.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-18 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585406#comment-15585406
 ] 

Duo Zhang commented on HBASE-15536:
---

Seems the biggest difference comes from the CellComparator related code?

For FSHLog, the hotest method is Bytes.toShort, 28.07%, it is called from 
CellComparator. And unsafe compareTo, 4.29%, and some other stuffs but overall 
it is less than 35% I think.

And for AsyncFSWAL, the CellComparator related code is more than 45%.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch, latesttrunk_asyncWAL_50threads_10cols.jfr, 
> latesttrunk_defaultWAL_50threads_10cols.jfr
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-18 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585326#comment-15585326
 ] 

Duo Zhang commented on HBASE-15536:
---

Good. Put it here please. Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-18 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585327#comment-15585327
 ] 

Anoop Sam John commented on HBASE-15536:


In any case if there are no regression from old way it is ok. Ya in case of 1+ 
DNs we might see adv.  But in any case let us not cause regression.  Ya we need 
see the profile reports.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-18 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585316#comment-15585316
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


bq.Oh only one DN? I haven't tested this scenario before. In general, 
AsyncFSWAL will have a lower latency if you write to more than 1 DN as it sends 
the data to multiple DNs concurrently.
True. That is why I just asked if 1 DN is not the right way to check this out. 
I have profile reports. I can attach here if you want to have a look.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-18 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585289#comment-15585289
 ] 

Duo Zhang commented on HBASE-15536:
---

Oh only one DN? I haven't tested this scenario before. In general, AsyncFSWAL 
will have a lower latency if you write to more than 1 DN as it sends the data 
to multiple DNs concurrently.

And it is interesting that WALPE and PE show different result. Will check the 
checksum work you mentioned.

And I think we need to do some profiling work to see where is the hot spot.

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-18 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585269#comment-15585269
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


As said, we are doing some experiments to check how we can run with AsyncWAL 
when we have the write path in offheap structures. This JIRA creates an 
outputStream which allows to work with ByteBuffers. 
But overall we see a significant degrade just in using the current trunk with 
ASYNC_WAL and default_WAL. 
With PE tool with 10 cols and 50 threads
With Default WAL

2185.113s

With Async WAL

3758.702s

Since this is a single node system the load on the DN and NN is heavy and so 
the msg
bq.Slow sync cost: 166 ms, current pipeline: 
[DatanodeInfoWithStorage[10.224.54.65:18223,DS-bc84f329-2043-4891-b878-f368c774b246,DISK]]

The thread model between the default case and the ASYNC WALbasically tries to 
be the same. We have some contentions with the ArrayDeque that is in AsyncWAL 
now. 
But the major difference I could see is that in the default WAL case the 
checksum is being done in the append() flow because append() adds the data to 
the FSDataOS.
In the AsyncWAL we add the data to the new BAOS but it is the sync call that 
actually does the check summing. 

Interesting thing is that when we run the WALPE tool with 50 threads with key 
size 50 : val : 200 , AsyncWAL is performing better in terms of ops/sec.
Default WAL
{code}
Summary: threads=50, iterations=100, syncInterval=0 took 1475.368s 
33889.848ops/s
{code}

AsyncWAL
{code}
Summary: threads=50, iterations=100, syncInterval=0 took 1368.501s 
36536.328ops/s
{code}

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-17 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584474#comment-15584474
 ] 

Duo Zhang commented on HBASE-15536:
---

I haven't tested it recently. You can post your result here or in a new jira 
and I can help figuring out the problem.

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-17 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584446#comment-15584446
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


[~Apache9]
Do you have any latest reports on this? what do you think when running with 
AsyncWAL on a single node. I think now we need to see how well we are 
performing with AsyncWAL and with offheap write path. 
In my recent runs with PE tool and 10 cols per row with 50 threads I found 
AsyncWAL to be performing slower than the default WAL. (for the default 
memstore)..
Today I have another cluster where I can also do some profiling. But wanted to 
get your feedback.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-12 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570176#comment-15570176
 ] 

Duo Zhang commented on HBASE-15536:
---

At RS, we will fail the request if the wall edit is too large.

And a simple split does not work, it we fail at the middle, we have no 
information to know if there are more pieces for this edit so either we loss 
data or can not guarantee atomic.
We need to introduce a chunk senmatic for wal edit if we want to do things like 
this. This will also have a benefit that a larger write will not block other 
write operations a lot.

Let me dig more. Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-12 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570156#comment-15570156
 ] 

stack commented on HBASE-15536:
---

bq. We can introduce a new config named 'max.wal.edit.size' which is used to 
limit the size of one atomic operation and set the default value less than 
16MB? This could make it possible to make AsyncFSWAL as our default WAL.

Where would we do the check? On client-side?

We would have to dissassemble the WALEdit to break it into 16MB pieces? Each 
edit would have same sequenceid?





> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-02 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15540424#comment-15540424
 ] 

Duo Zhang commented on HBASE-15536:
---

Seems there is no key value size limitation at server side currently. And the 
max key-value size is only used to limit one key-value, which means you can add 
100 cells into one put object and each cell is 1MB, then the size of the put is 
100MB and breaks our {{AsyncFSWAL}}...

We can introduce a new config named 'max.wal.edit.size' which is used to limit 
the size of one atomic operation and set the default value less than 16MB? This 
could make it possible to make AsyncFSWAL as our default WAL.

BTW, I think we could add the 'max.wal.edit.size' as a new stop condition when 
grab row locks in the multi operation as multi does not guarantee cross row 
atomic. Then AsyncFSWAL could also work with batch operation larger than 16MB 
unless the size of one row is larger than 16MB or MultiRowMutationEndpoint is 
used.

What do you think? [~stack] [~anoopsamjohn]. Too many configs maybe... Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-01 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15539723#comment-15539723
 ] 

Duo Zhang commented on HBASE-15536:
---

Let me revisit all the related configs. Seems they are all hard limit which 
means it will reject the request.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-01 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15539709#comment-15539709
 ] 

stack commented on HBASE-15536:
---

2MB seems fine to get a warning at. In warning, it would say you can go up to 
16MB but if you want Cells > 16MB, you need to switch the WAL implementation? I 
think that is ok.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-01 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15539332#comment-15539332
 ] 

Duo Zhang commented on HBASE-15536:
---

Fine. Let me prepare a patch. Is 2MB a suitable size for max key value size? 
The old default value is -1 which means unlimited.

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-01 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15539183#comment-15539183
 ] 

stack commented on HBASE-15536:
---

+1 as this on by default

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-01 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15539179#comment-15539179
 ] 

stack commented on HBASE-15536:
---

I like the a anoop suggestion.  Let's not have the mob feature prevent us 
making this default.  Have the few that use mob have to set config; not the 
majority

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-01 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15538902#comment-15538902
 ] 

Anoop Sam John commented on HBASE-15536:


So seems like based on the max Cell size we can go here.  AsyncWAL can work 
with 16MB max cell size. There is a config for specifying max cell size. This 
defaults to 2 MB (correct? )  And we have MAX_CELL_SIZE_FOR_ASYNC_WAL = 16 MB. 
When the max cell size config is less than 16 MB we can default to async wal. 
But of user specifically gone with old wal, go with that. When user configured 
max cell size as >16MB go with old wal as default. And if user configured aync 
wal there by mistake, fail the start up.  In all case log clearly what WAL 
impl, we go with while RS start.  We can update this jira details accordingly 
and handle. Also the other issue of >16MB cell size, let us close as a 
limitation of the feature. We can document clearly.  Sounds fine?

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-10-01 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15538548#comment-15538548
 ] 

Duo Zhang commented on HBASE-15536:
---

I filed HBASE-15709 when debugging the problem of {{TestMobStoreScanner}}, 
where we write a cell with 25MB which exceeded the limit of max packet 
size(16MB) in DTP. It could be fixed by implementing chunked logic in 
{{AsyncOneBlockDFSOutputStream}}. But still, if you try a put larger than the 
block size, the async WAL will fail since it can only handle a one block file.

I think this means the {{AsyncFSWAL}} can not handle all the cases in the 
current HBase so I'm not sure if it is OK to make it default. And why I leave 
HBASE-15709 open is that, I do not think it is a good idea to write cells 
larger than 16MB at one time if we do not enable the mob feature. In our 
company the limitation is 2MB. And I wonder why mob still writes the whole 
entry to WAL? For a typical object storage built upon HBase and HDFS, only meta 
is stored in HBase and the actual data is written to HDFS directly which means 
it does not need a WAL, it has been persistent already. So ideally, I think we 
should enable mob by default and the threshold maybe 4MB? And if it does not 
write the big data directly to WAL, then we could make {{AsyncFSWAL}} default.

Thanks. [~anoopsamjohn]

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-07-26 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393354#comment-15393354
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


And to make things clear, the 
org.apache.hadoop.hbase.io.ByteBufferSupportOutputStream  was not there 
previously in trunk and hence we were not able to get the offheap cells to be 
written to WAL without bringing them onheap. 
In addition to that since AsyncWAL solves the perf penalty of using the 
hadoop's outputstream we are able to see good benefits wokring with AsyncWAL.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-07-25 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393252#comment-15393252
 ] 

stack commented on HBASE-15536:
---

Ugh. That'd put a stake in the offheaping effort if had to copy onheap just to 
write the WAL!

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-07-25 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393219#comment-15393219
 ] 

ramkrishna.s.vasudevan commented on HBASE-15536:


For the write path offheap work - we were able to make things work with 
AsyncFSWAL without having the need to copy the offheap cells to onheap for 
writing to WAL. This is because 
org.apache.hadoop.hbase.io.ByteBufferSupportOutputStream supports writing 
directly from offheap cells to the ByteBufferOS. One more reason to make 
AsyncWAL  as default in 2.0.
Without this the offheap cells have to be brought onheap and then flushed to 
the WAL OutputStream which generates lot of garbage.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-05-09 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277519#comment-15277519
 ] 

Duo Zhang commented on HBASE-15536:
---

Good. Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-05-09 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277389#comment-15277389
 ] 

stack commented on HBASE-15536:
---

Yeah. Seems like discussion has calmed down especially given you have addressed 
any issues folks have raised. I am sure there are other issues to be found but 
if this is the default in 2.0, we should trip over them sooner rather than 
later. Lets give it another 24 hours  and then commit. Thanks [~Apache9]

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536-v5.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-05-08 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275905#comment-15275905
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 39 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 0s {color} 
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 2 new + 4 
unchanged - 2 fixed = 6 total (was 6) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
44s {color} | {color:green} hbase-server: patch generated 0 new + 453 unchanged 
- 13 fixed = 453 total (was 466) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 8m 
37s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1 
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 11s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
11s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 36s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.ipc.TestIPC |
|   | hadoop.hbase.ipc.TestAsyncIPC |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12802911/HBASE-15536-v5.patch |
| JIRA Issue | HBASE-15536 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-05-08 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275867#comment-15275867
 ] 

Duo Zhang commented on HBASE-15536:
---

So what is the final decision? Seems no objections on mailing-list for now? 
[~stack]

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-29 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265074#comment-15265074
 ] 

Duo Zhang commented on HBASE-15536:
---

TestAccessController3 can not pass on master either. Seems the same problem 
with TestNamespaceCommands. The grant method takes a very very long time to 
finish.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-29 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265049#comment-15265049
 ] 

Duo Zhang commented on HBASE-15536:
---

Seems the javac issues are still there [~busbey]. I checked MasterRpcServices, 
no 'sun.misc.' imports.

Let me check the failed UT. Haven't seen it before.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-29 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264392#comment-15264392
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 4s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 39 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s 
{color} | {color:green} master passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 38s {color} 
| {color:red} hbase-server-jdk1.7.0_95 with JDK v1.7.0_95 generated 2 new + 4 
unchanged - 2 fixed = 6 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
14s {color} | {color:green} hbase-server: The patch generated 0 new + 472 
unchanged - 16 fixed = 472 total (was 488) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 44s {color} | {color:green} The patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 58s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
28s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 166m 27s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.access.TestAccessController3 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-04-29 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12801462/HBASE-15536-v4.patch |
| JIRA Issue | HBASE-15536 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 75ac0d5ffde6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-29 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264361#comment-15264361
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 39 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 56s {color} 
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 2 new + 4 
unchanged - 2 fixed = 6 total (was 6) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
22s {color} | {color:green} hbase-server: patch generated 0 new + 472 unchanged 
- 16 fixed = 472 total (was 488) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 8m 
27s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1 
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 115m 27s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
14s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 10s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12801462/HBASE-15536-v4.patch |
| JIRA Issue | HBASE-15536 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-29 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264138#comment-15264138
 ] 

Sean Busbey commented on HBASE-15536:
-

the Javac warnings are about JVM internal API use. Looks like an error in the 
diff calculation, since MasterRpcServices isn't touched by the patch.

Lemme relaunch a test run with the current in-dev version of Yetus.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536-v4.patch, HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-29 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264054#comment-15264054
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 39 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m 
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 51s {color} 
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 2 new + 4 
unchanged - 2 fixed = 6 total (was 6) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 5m 11s 
{color} | {color:red} hbase-server: patch generated 6 new + 459 unchanged - 15 
fixed = 465 total (was 474) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
33m 16s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 134m 23s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 199m 13s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.access.TestAccessController3 |
| Timed out junit tests | 
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
|   | org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
|   | org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12801433/HBASE-15536-v3.patch |
| JIRA Issue | HBASE-15536 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-29 Thread Yu Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263957#comment-15263957
 ] 

Yu Li commented on HBASE-15536:
---

Thanks for the encouragement sir [~stack] :-)

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536-v3.patch, HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263592#comment-15263592
 ] 

stack commented on HBASE-15536:
---

[~carp84] you did no wrong. There was a regression... just not an important one 
(smile). We also learned some stuff from the experience.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263566#comment-15263566
 ] 

Sean Busbey commented on HBASE-15536:
-

{quote}
bq. Maybe add a note on commit to the DefaultWALProvider about this 'odd' fact.
I think we could change the name of DefaultWALProvider to a more reasonable 
name, but still map 'o.a.h.h.regionserver.wal.DefaultWALProvider' to the 
renamed class. This does not break the config compatibility.
{quote}

We shouldn't need to do this. One motivating factor for having config enums, 
like the wal provider names, is so that the names of Classes aren't in our 
config compatibility. Folks relying on a classname should already know they're 
taking an unsupported route.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread Yu Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263554#comment-15263554
 ] 

Yu Li commented on HBASE-15536:
---

bq. How many WALs per RS in this test?
4WALs with PCIe-SSD, will share more details in a new JIRA later. Will do more 
confirmation that it's a generous problem instead of anything caused by our 
private backports, don't want to give misleading message like ever did in 
HBASE-15619 any more... :-)

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263527#comment-15263527
 ] 

Anoop Sam John commented on HBASE-15536:


+1 
bq.Recently we observed performance downgrade when using multiwal 
How many WALs per RS in this test?

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread Yu Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263496#comment-15263496
 ] 

Yu Li commented on HBASE-15536:
---

Ah I see, thanks for the note, then I think we also need a solution because our 
current *from* version don't have HBASE-14949 for sure...

Good enough to know that currently no backport work in progress, will get 
necessary work done and grab some perf number before filing any JIRA.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263462#comment-15263462
 ] 

Duo Zhang commented on HBASE-15536:
---

There is no plan to backport it to branch-1 currently because of the 
compatibility issues. As said in HBASE-14949, if you want to do rolling upgrade 
to use AsyncFSWAL, then the from and to versions must both have HBASE-14949 in 
it, otherwise we may lose data. So it will break the compatibility guarantee if 
we backport AsyncFSWAL to branch-1...

I think you can file a backport issue to see what others think of it.

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread Yu Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263454#comment-15263454
 ] 

Yu Li commented on HBASE-15536:
---

+1 (non-binding) to make AsyncFSWAL as default in 2.0

[~Apache9] Any plan to backport the great work here to branch-1? We're planning 
to use AsyncFSWAL online by July (well, I'd say the perf number is really 
attractive. :-D) and will start the work soon but duplicated work is no good, 
so please let me know if you already got anything in progress for the backport 
work. Thanks. :-)

OTOH, it would be great if there could be some perf number on multiwal in 
comparison to single-wal with AsyncFSWAL. Recently we observed performance 
downgrade when using multiwal (with FSHLog) in branch-1 and now trying hard to 
locate the root cause (will open another JIRA to tell the details), so I'm a 
little bit concerned about the multiwal part.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263410#comment-15263410
 ] 

Duo Zhang commented on HBASE-15536:
---

{quote}
Maybe add a note on commit to the DefaultWALProvider about this 'odd' fact.
{quote}
I think we could change the name of DefaultWALProvider to a more reasonable 
name, but still map 'o.a.h.h.regionserver.wal.DefaultWALProvider' to the 
renamed class. This does not break the config compatibility.

{quote}
Any place you know where we are missing coverage? Any secure deploy type? Or a 
deploy type that needs more testing? Could file an issue for such 'weak' areas.
{quote}
Two things I can tell right now.
First is what if the whole HDFS crashes. Of course we can say that if HDFS 
crashes then we can not guarantee much since we heavily rely on HDFS. But if 
the behavior is changed then the end users may need to change their maintain 
guide of how to deal with HDFS crash.
Second is the performance of secure output.

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262309#comment-15262309
 ] 

stack commented on HBASE-15536:
---

On the patch, this is a bit odd but probably not much you can do about it:

defaultProvider(AsyncFSWALProvider.class),

DefaultWALProvider is 'taken' it is the old FSHLog.

Maybe add a note on commit to the DefaultWALProvider about this 'odd' fact.

Otherwise, +1.

Any place you know where we are missing coverage? Any secure deploy type? Or a 
deploy type that needs more testing? Could file an issue for such 'weak' areas.



> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262295#comment-15262295
 ] 

stack commented on HBASE-15536:
---

Yes... Please flag dev that we are making this switch if only to raise the 
profile of this very nice work that you've done.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-28 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261970#comment-15261970
 ] 

Duo Zhang commented on HBASE-15536:
---

[~stack] [~anoopsamjohn] Any other concerns? Do we need to start a discussion 
on mailing-list on whether we can change the default WAL implementation to 
AsyncFSWAL?

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-27 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259624#comment-15259624
 ] 

Duo Zhang commented on HBASE-15536:
---

[~anoopsamjohn] I have changed the 'defaultProvider' from DefaultWALProvider to 
AsyncFSWALProvider. The multiwal's default delegate provider is also 
'defaultProvider', not 'filesystem', so it will also use AsyncFSWAL. Let me add 
this in release note.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-26 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259597#comment-15259597
 ] 

Anoop Sam John commented on HBASE-15536:


So if the config 'hbase.wal.provider' is given as multiwal, it will be old way 
WAL with multiple WALs.  We have added a new config to specify this multi wal 
thing any way.  Just for confirmation asked.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-26 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259421#comment-15259421
 ] 

Duo Zhang commented on HBASE-15536:
---

Will retry, we are in an infinite loop here.

Do I need to upload the patch to review board for a better review?

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-26 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258329#comment-15258329
 ] 

stack commented on HBASE-15536:
---

We are going to retry here? Or are we skipping the exception?

{code}
745 } else {
746   LOG.warn("complete file " + src + " failed, retry = " + 
retry, e);
747 }
{code}

Otherwise, the patch looks great. Needs a release note.


> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-26 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258187#comment-15258187
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 20 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 24s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 2s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 58s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12800771/HBASE-15536-v2.patch |
| JIRA Issue | HBASE-15536 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / e5d0157 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions |  /home/jenkins/tools/java/jdk1.8.0:1.8.0 
/usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79 |
| findbugs | v3.0.0 |
|

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-26 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257934#comment-15257934
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 20 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 5m 13s 
{color} | {color:red} hbase-server: patch generated 2 new + 10 unchanged - 0 
fixed = 12 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 57s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 11s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
18s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 23s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.snapshot.TestSecureExportSnapshot |
|   | hadoop.hbase.snapshot.TestExportSnapshot |
|   | hadoop.hbase.snapshot.TestMobExportSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12800739/HBASE-15536-v1.patch |
| JIRA Issue | HBASE-15536 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / e5d0157 |
| Default Java | 1.7.0_79

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-26 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257923#comment-15257923
 ] 

Duo Zhang commented on HBASE-15536:
---

I found this

https://github.com/mockito/mockito/issues/133

The last post said that it is impossible to deal with anonymous inner class in 
Mockito. But I tried to declare a Consumer inner class explicitly and it does 
not work...

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-26 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257896#comment-15257896
 ] 

stack commented on HBASE-15536:
---

+1

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-26 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257711#comment-15257711
 ] 

Duo Zhang commented on HBASE-15536:
---

Ah an annoying problem with TestHRegion.testDurability...

In this test we create a spied WAL and the Mockito,spy will create a copy of 
the object passed in. The problem is AsyncFSWAL.consumer, it is an instance of 
an inner class and hold a reference to the outside AsyncFSWAL object. The 
copied AsyncFSWAL object just the same instance of consumer with original 
AsyncFSWAL object, so it is still reference the original AsyncFSWAL, not the 
copied one. This introduce some logic errors and cause our UT hanging forever...

Will change it to use FSHLog since this is not test for WAL implementation.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257602#comment-15257602
 ] 

Hadoop QA commented on HBASE-15536:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 17 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 36s 
{color} | {color:red} hbase-server: patch generated 2 new + 9 unchanged - 0 
fixed = 11 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
24m 31s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 102m 58s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 149m 19s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| Timed out junit tests | org.apache.hadoop.hbase.TestFullLogReconstruction |
|   | org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
|   | org.apache.hadoop.hbase.mapreduce.TestSyncTable |
|   | org.apache.hadoop.hbase.mapreduce.TestHashTable |
|   | org.apache.hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12800699/HBASE-15536.patch |
| JIRA Issue | HBASE-15536 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-25 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257476#comment-15257476
 ] 

Duo Zhang commented on HBASE-15536:
---

Oh the problem here is that HDFS only accepts packets less than 16MB to prevent 
OOME and our FanOutOneBlockAsyncDFSOutput does not handle this...

Let me file a issue.

And here I will prepare a patch which sets the UT to use FSHLog first.

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-25 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256527#comment-15256527
 ] 

stack commented on HBASE-15536:
---

+1 on using FSHLog explicitly in the MOB test with comments and a follow-on 
issue to dig in when big entries.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-25 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256393#comment-15256393
 ] 

Duo Zhang commented on HBASE-15536:
---

A little progress here, with HBASE-15628, the only UT that can not pass is 
TestMobStoreScanner. I think the problem is testGetMassive which we make a 
single put with 75MB data. Now the AsyncFSWAL does not have special logic to 
handle large edits so here we may have a OOME or some other errors so the UT 
fails.

And since mob is also an experimental feature, we could just use FSHLog and add 
some comments here? And I do not think it is a good idea to write a 75MB WAL 
entry...

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-20 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249843#comment-15249843
 ] 

Duo Zhang commented on HBASE-15536:
---

Some progress here.

https://github.com/Apache9/hbase/tree/HBASE-15628

With a simple patch of HBASE-15628 and some modification on UTs, I got most of 
them pass.

Now the biggest problem is TestRegionReplicaReplicationEndpoint. Let me dig in.

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-17 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244749#comment-15244749
 ] 

stack commented on HBASE-15536:
---

bq. So we are using more CPUs with asyncwal? 

I wouldn't put much weight on our wandering across CPUs. Probably not the thing 
to focus on.

bq. What does the '2.597 GHz' 

I don't know how 2.597GHz relates to 6,814,056,657,994 cycles Poked around 
some... Should poke some more. Look at perf src?

bq. But usually these optimizations are only useful on loops and I can not find 
a heavy loop in AsyncFSWAL (maybe the append loop in consume method?)... 

Loops would be one aspect but others would be stuff like stalls while we have 
to go to main memory or other than the immediate caches because of sub optimal 
object layout and poor locality in the data, say, because the data structures 
we use work against good alignment. Messing around in code, even minor 
improvement in instructions per cycle makes for massive improvement in 
throughput.

bq. Maybe we need to find the hot spots in AsyncFSWAL first?

At the macro level, aswyncfswall is more tthan 2x better than what was there 
previous. That is more than enough to make it default.

I think that there is more speedup to be had here especially given there so few 
moving parts after the nice work you've done in here but would take study in a 
micro-benchmark context. We could set up a little jmh testbench for asyncwal? 
I'd be more interested in a jmh micro-benchmark for studying our merge sort; a 
different Cell layout and some study could get us a better scan throughput.









> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-11 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234812#comment-15234812
 ] 

Duo Zhang commented on HBASE-15536:
---

I tried to set the AsyncFSWAL as defaultProvider and found that lots of UTs are 
failed because AsyncFSWAL can only work with DistributedFileSystem.

I think we should make it work with a general FileSystem first.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-04-01 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221395#comment-15221395
 ] 

Duo Zhang commented on HBASE-15536:
---

So we are using more CPUs with asyncwal? But actually we spend less cycles with 
asyncwal(Yeah you said, less work)? What does the '2.597 GHz' mean?

And for the IPC, I remember the time when I was a graduate student who was 
learning how to do 'software pipeline' and 'instruction scheduling' in a 
compiler... But usually these optimizations are only useful on loops and I can 
not find a heavy loop in {{AsyncFSWAL}}... And it is interesting that we have a 
less branch-miss rate but a less IPC...
Maybe we need to find the hot spots in {{AsyncFSWAL}} first?

Thanks.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-03-30 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219295#comment-15219295
 ] 

stack commented on HBASE-15536:
---

Running this compare from HBASE-10156

{code}
$ for i in 1 3 5 10 25 50 100 200; do for j in 1; do perf stat ./bin/hbase 
--config /home/stack/conf_hbase 
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation -threads $i   -iterations 
100 -keySize 50 -valueSize 100  &> "/tmp/nopatch2${i}.${j}.txt"; done; done
{code}

Here is table of completion times (I mangled recording 200 threads w/ no patch):

||Threads||Defaul||Async||Diff||
|1|837|228|3.7x|
|3|647|274|2.4x|
|5|609|310|2x|
|10|916|376|2.5x|
|25|1177|556|2.1x|
|50|1463|828|1.8x|
|100|1902|1382|1.4x|
|200|-|2445|-|

Comparing perf stat for ten threads, you can see the async is doing less work. 
Here is the default WAL provider stat output:
{code}
4016  Performance counter stats for './hbase/bin/hbase --config 
/home/stack/conf_hbase org.apache.hadoop.hbase.wal.WALPerformanceEvaluation 
-threads 10 -iterations 100 -keySize 50 -valueSize 100':
4017
4018 3473402.908284 task-clock (msec) #3.791 CPUs utilized
4019 79,614,165 context-switches  #0.023 M/sec
4020  4,927,049 cpu-migrations#0.001 M/sec
4021  1,390,882 page-faults   #0.400 K/sec
4022  7,457,646,572,542 cycles#2.147 GHz
4023 stalled-cycles-frontend
4024 stalled-cycles-backend
4025  2,088,450,796,192 instructions  #0.28  insns per cycle
4026340,979,920,761 branches  #   98.169 M/sec
4027  5,330,989,389 branch-misses #1.56% of all branches
4028
4029  916.241768049 seconds time elapsed
{code}

Here is running with async enabled:
{code}
1050  Performance counter stats for './hbase/bin/hbase --config 
/home/stack/conf_hbase org.apache.hadoop.hbase.wal.WALPerformanceEvaluation 
-threads 10 -iterations 100 -keySize 50 -valueSize 100':
1051
1052 2624320.161261 task-clock (msec) #6.964 CPUs utilized
1053 13,097,786 context-switches  #0.005 M/sec
1054202,869 cpu-migrations#0.077 K/sec
1055968,708 page-faults   #0.369 K/sec
1056  6,814,056,657,994 cycles#2.597 GHz
1057 stalled-cycles-frontend
1058 stalled-cycles-backend
1059  1,577,447,250,139 instructions  #0.23  insns per cycle
1060244,194,003,573 branches  #   93.050 M/sec
1061  2,927,181,625 branch-misses #1.20% of all branches
1062
1063  376.825234068 seconds time elapsed
{code}

80M context-switches vs 13M, 7.5B cycles vs 6.8B. There is loads of work to be 
done in here -- our numbers for ins per cycle are pretty abysmal -- but seems 
like clear benefit to running async provider though it doesn't look much when 
you look at macro-level with YCSB load with 50 threads.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-03-30 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218822#comment-15218822
 ] 

stack commented on HBASE-15536:
---

I did a rough compare where I ran ycsb load against a single server instance 
running over an HDFS. Numbers were hbase 2.0 tip with asyncwal disabled/enabled:

{code}
18096 2016-03-30 13:07:21:600 4330 sec: 61688560 operations; 14370.85 current 
ops/sec; est completion in 7 hours 10 minutes [INSERT: Count=143767, 
Max=201727, Min=916, Avg=3465.51, 90=4615, 99=13639, 99.9=78207, 99.99=200703]
18097 2016-03-30 13:07:31:596 4340 sec: 61833035 operations; 14453.28 current 
ops/sec; est completion in 7 hours 10 minutes [INSERT: Count=144469, 
Max=267007, Min=923, Avg=3463.2, 90=4603, 99=6875, 99.9=152319, 99.99=264959]
18098 2016-03-30 13:07:41:596 4350 sec: 61995299 operations; 16226.4 current 
ops/sec; est completion in 7 hours 9 minutes [INSERT: Count=162271, Max=64063, 
Min=850, Avg=3076.64, 90=4535, 99=6427, 99.9=45983, 99.99=62719]
18099 2016-03-30 13:07:51:596 4360 sec: 62137855 operations; 14255.6 current 
ops/sec; est completion in 7 hours 9 minutes [INSERT: Count=142547, Max=226303, 
Min=880, Avg=3502.47, 90=4571, 99=7707, 99.9=192639, 99.99=225279]
18100 2016-03-30 13:08:01:596 4370 sec: 62290243 operations; 15238.8 current 
ops/sec; est completion in 7 hours 9 minutes [INSERT: Count=152393, Max=231423, 
Min=919, Avg=3275.69, 90=4523, 99=6831, 99.9=88447, 99.99=229503]
18101 2016-03-30 13:08:11:596 4380 sec: 62432949 operations; 14270.6 current 
ops/sec; est completion in 7 hours 9 minutes [INSERT: Count=142698, Max=205311, 
Min=910, Avg=3499.68, 90=4631, 99=17071, 99.9=146687, 99.99=202495]
18102 2016-03-30 13:08:21:596 4390 sec: 62574524 operations; 14157.5 current 
ops/sec; est completion in 7 hours 9 minutes [INSERT: Count=141581, Max=259071, 
Min=918, Avg=3526.68, 90=4443, 99=6627, 99.9=189055, 99.99=257151]
18103 2016-03-30 13:08:31:596 4400 sec: 62710962 operations; 13643.8 current 
ops/sec; est completion in 7 hours 8 minutes [INSERT: Count=136431, Max=220671, 
Min=877, Avg=3606.32, 90=4547, 99=26495, 99.9=194303, 99.99=217855]
18104 2016-03-30 13:08:41:596 4410 sec: 62859312 operations; 14835 current 
ops/sec; est completion in 7 hours 8 minutes [INSERT: Count=148350, Max=258303, 
Min=957, Avg=3413.23, 90=4543, 99=7787, 99.9=121663, 99.99=256255]
18105 2016-03-30 13:08:51:596 4420 sec: 63003208 operations; 14389.6 current 
ops/sec; est completion in 7 hours 8 minutes [INSERT: Count=143896, Max=226559, 
Min=957, Avg=3468.59, 90=4631, 99=7127, 99.9=126783, 99.99=225535]
18106 2016-03-30 13:09:01:596 4430 sec: 63147394 operations; 14418.6 current 
ops/sec; est completion in 7 hours 8 minutes [INSERT: Count=144193, Max=202751, 
Min=931, Avg=3465.31, 90=4443, 99=15271, 99.9=122879, 99.99=200959]
18107 2016-03-30 13:09:11:596 4440 sec: 63283125 operations; 13573.1 current 
ops/sec; est completion in 7 hours 8 minutes [INSERT: Count=135739, Max=527871, 
Min=919, Avg=3678.7, 90=4567, 99=9375, 99.9=159615, 99.99=525311]
{code}

{code}
2016-03-30 13:35:26:261 1130 sec: 18067105 operations; 16813.1 current ops/sec; 
est completion in 7 hours 8 minutes [INSERT: Count=168133, Max=52863, Min=437, 
Avg=2970.54, 90=5899, 99=10199, 99.9=35903, 99.99=50623]
2016-03-30 13:35:36:261 1140 sec: 18223814 operations; 15670.9 current ops/sec; 
est completion in 7 hours 8 minutes [INSERT: Count=156711, Max=175103, Min=426, 
Avg=3185.56, 90=5767, 99=19727, 99.9=80639, 99.99=173055]
2016-03-30 13:35:46:261 1150 sec: 18398146 operations; 17433.2 current ops/sec; 
est completion in 7 hours 8 minutes [INSERT: Count=174328, Max=54815, Min=452, 
Avg=2862.38, 90=5575, 99=8079, 99.9=43615, 99.99=53759]
2016-03-30 13:35:56:262 1160 sec: 18561746 operations; 16360 current ops/sec; 
est completion in 7 hours 8 minutes [INSERT: Count=163605, Max=95423, Min=434, 
Avg=3051.5, 90=5775, 99=19503, 99.9=47967, 99.99=94015]
2016-03-30 13:36:06:261 1170 sec: 18725215 operations; 16346.9 current ops/sec; 
est completion in 7 hours 7 minutes [INSERT: Count=163471, Max=83327, Min=443, 
Avg=3053.24, 90=5787, 99=14263, 99.9=47615, 99.99=80767]
2016-03-30 13:36:16:261 1180 sec: 18875810 operations; 15059.5 current ops/sec; 
est completion in 7 hours 7 minutes [INSERT: Count=150615, Max=452863, Min=434, 
Avg=3314.69, 90=5867, 99=13527, 99.9=83135, 99.99=451327]
2016-03-30 13:36:26:262 1190 sec: 19038377 operations; 16256.7 current ops/sec; 
est completion in 7 hours 7 minutes [INSERT: Count=162549, Max=155775, Min=409, 
Avg=3069.92, 90=5731, 99=13287, 99.9=79935, 99.99=153087]
2016-03-30 13:36:36:261 1200 sec: 19172700 operations; 13432.3 current ops/sec; 
est completion in 7 hours 8 minutes [INSERT: Count=134331, Max=452863, Min=395, 
Avg=3716.84, 90=5907, 99=19503, 99.9=224767, 99.99=447999]
2016-03-30 13:36:46:261 1210 sec: 19335308 operations; 16260.8 current ops/sec; 
est completion in 7 hours 7 minutes [INSERT:

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-03-30 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218703#comment-15218703
 ] 

stack commented on HBASE-15536:
---

I reviewed HBASE-15407 and HBASE-15538. They seem good to go. Before we can cut 
over, we should do a deploy that is secure to ensure all basically works 
(though the tests bundled are good).

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

2016-03-30 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218558#comment-15218558
 ] 

stack commented on HBASE-15536:
---

An overnight run passed (10B ITBLL with chaos monkeys on 8-node cluster) so 
this provider is probably as good/bad as the one we currently have (10 hours). 
An earlier test of two hours also passed. Let me run a perf compare next.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Duo Zhang
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

97 matches

Mail list logo