[jira] [Commented] (HBASE-19064) Synchronous replication for HBase
[ https://issues.apache.org/jira/browse/HBASE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465457#comment-16465457 ] Hudson commented on HBASE-19064: Results for branch HBASE-19064 [build #121 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/121/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/121//General_Nightly_Build_Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/121//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/121//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. > Synchronous replication for HBase > - > > Key: HBASE-19064 > URL: https://issues.apache.org/jira/browse/HBASE-19064 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0 > > > The guys from Alibaba made a presentation on HBaseCon Asia about the > synchronous replication for HBase. We(Xiaomi) think this is a very useful > feature for HBase so we want to bring it into the community version. > This is a big feature so we plan to do it in a feature branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20411) Ameliorate MutableSegment synchronize
[ https://issues.apache.org/jira/browse/HBASE-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465421#comment-16465421 ] stack commented on HBASE-20411: --- Rough patch that rolls @huaxiang's work into what I was at. Lets see what qa build thinks of it. Doesn't have the block signal suggestion just yet. > Ameliorate MutableSegment synchronize > - > > Key: HBASE-20411 > URL: https://issues.apache.org/jira/browse/HBASE-20411 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Attachments: 2.load.patched.17704.lock.svg, > 2.load.patched.2.17704.lock.svg, 2.more.patch.12010.lock.svg, 41901.lock.svg, > HBASE-20411-atomiclong-longadder.patch, HBASE-20411.branch-2.0.001.patch, > HBASE-20411.branch-2.0.002.patch, HBASE-20411.branch-2.0.003.patch, > HBASE-20411.branch-2.0.004.patch, HBASE-20411.branch-2.0.005.patch, > HBASE-20411.branch-2.0.006.patch, HBASE-20411.branch-2.0.007.patch, > HBASE-20411.branch-2.0.008.patch > > > This item is migrated from HBASE-20236 so it gets dedicated issue. > Let me upload evidence that has this synchronize as a stake in our write-time > perf. I'll migrate the patch I posted with updates that come of comments > posted by [~mdrob] on the HBASE-20236 issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20411) Ameliorate MutableSegment synchronize
[ https://issues.apache.org/jira/browse/HBASE-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20411: -- Attachment: HBASE-20411.branch-2.0.008.patch > Ameliorate MutableSegment synchronize > - > > Key: HBASE-20411 > URL: https://issues.apache.org/jira/browse/HBASE-20411 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Attachments: 2.load.patched.17704.lock.svg, > 2.load.patched.2.17704.lock.svg, 2.more.patch.12010.lock.svg, 41901.lock.svg, > HBASE-20411-atomiclong-longadder.patch, HBASE-20411.branch-2.0.001.patch, > HBASE-20411.branch-2.0.002.patch, HBASE-20411.branch-2.0.003.patch, > HBASE-20411.branch-2.0.004.patch, HBASE-20411.branch-2.0.005.patch, > HBASE-20411.branch-2.0.006.patch, HBASE-20411.branch-2.0.007.patch, > HBASE-20411.branch-2.0.008.patch > > > This item is migrated from HBASE-20236 so it gets dedicated issue. > Let me upload evidence that has this synchronize as a stake in our write-time > perf. I'll migrate the patch I posted with updates that come of comments > posted by [~mdrob] on the HBASE-20236 issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20508) TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test
[ https://issues.apache.org/jira/browse/HBASE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465368#comment-16465368 ] maoling commented on HBASE-20508: - [~chia7712] Thanks for your review and the suggestion about the patch rule.I will remember it next time > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test > --- > > Key: HBASE-20508 > URL: https://issues.apache.org/jira/browse/HBASE-20508 > Project: HBase > Issue Type: Test > Components: backup&restore >Reporter: Ted Yu >Assignee: maoling >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20508-master.v0.patch, HBASE-20508-master.v1.patch > > > TestIncrementalBackupWithBulkLoad currently is Parameterized with only one > value returned from data() method. > In its ctor, this value is ignored. > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20535) Check the UT TestSaslFanOutOneBlockAsyncDFSOutput which is always flaky
[ https://issues.apache.org/jira/browse/HBASE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465361#comment-16465361 ] Duo Zhang commented on HBASE-20535: --- IIRC the bug is in HDFS. The feature does not work with the newest jdk8. > Check the UT TestSaslFanOutOneBlockAsyncDFSOutput which is always flaky > --- > > Key: HBASE-20535 > URL: https://issues.apache.org/jira/browse/HBASE-20535 > Project: HBase > Issue Type: Bug >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > > 1. > https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html > 2. https://builds.apache.org/job/HBASE-Flaky-Tests//30949 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20535) Check the UT TestSaslFanOutOneBlockAsyncDFSOutput which is always flaky
Zheng Hu created HBASE-20535: Summary: Check the UT TestSaslFanOutOneBlockAsyncDFSOutput which is always flaky Key: HBASE-20535 URL: https://issues.apache.org/jira/browse/HBASE-20535 Project: HBase Issue Type: Bug Reporter: Zheng Hu Assignee: Zheng Hu 1. https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html 2. https://builds.apache.org/job/HBASE-Flaky-Tests//30949 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20508) TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test
[ https://issues.apache.org/jira/browse/HBASE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-20508: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks for the patch. [~maoling] Nit: hbaes has rule about the commit message. Please take a look at https://hbase.apache.org/book.html#submitting.patches > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test > --- > > Key: HBASE-20508 > URL: https://issues.apache.org/jira/browse/HBASE-20508 > Project: HBase > Issue Type: Test > Components: backup&restore >Reporter: Ted Yu >Assignee: maoling >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20508-master.v0.patch, HBASE-20508-master.v1.patch > > > TestIncrementalBackupWithBulkLoad currently is Parameterized with only one > value returned from data() method. > In its ctor, this value is ignored. > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20526) multithreads bulkload performance
[ https://issues.apache.org/jira/browse/HBASE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465345#comment-16465345 ] Ted Yu commented on HBASE-20526: Every contribution, once accepted, goes to master branch first. > multithreads bulkload performance > - > > Key: HBASE-20526 > URL: https://issues.apache.org/jira/browse/HBASE-20526 > Project: HBase > Issue Type: Improvement > Components: mapreduce, Zookeeper >Affects Versions: 1.2.5, 1.3.2 > Environment: hbase-server-1.2.0-cdh5.12.1 > spark version 1.6 >Reporter: Key Hutu >Assignee: Key Hutu >Priority: Minor > Labels: performance > Fix For: 1.3.2 > > Attachments: HBASE-20526-branch-1.3.V1.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > When doing bulkload , some interactive with zookeeper to getting region key > range may be cost more time. > In multithreads enviorment, the duration maybe cost 5 minute or more. > From the executor log, like 'Reading reply sessionid:0x262fb37f4a07080 , > packet:: clientPath:null server ...' contents appear many times. > > It likely to provide new method for bulkload, caching the key range outside > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20526) multithreads bulkload performance
[ https://issues.apache.org/jira/browse/HBASE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465343#comment-16465343 ] Key Hutu commented on HBASE-20526: -- Thanks , Ted Yu I see the implementation of master branch and branch-1.2/1.3 is different Can we accept submissions based on branch-1.3 ? > multithreads bulkload performance > - > > Key: HBASE-20526 > URL: https://issues.apache.org/jira/browse/HBASE-20526 > Project: HBase > Issue Type: Improvement > Components: mapreduce, Zookeeper >Affects Versions: 1.2.5, 1.3.2 > Environment: hbase-server-1.2.0-cdh5.12.1 > spark version 1.6 >Reporter: Key Hutu >Assignee: Key Hutu >Priority: Minor > Labels: performance > Fix For: 1.3.2 > > Attachments: HBASE-20526-branch-1.3.V1.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > When doing bulkload , some interactive with zookeeper to getting region key > range may be cost more time. > In multithreads enviorment, the duration maybe cost 5 minute or more. > From the executor log, like 'Reading reply sessionid:0x262fb37f4a07080 , > packet:: clientPath:null server ...' contents appear many times. > > It likely to provide new method for bulkload, caching the key range outside > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20224) Web UI is broken in standalone mode
[ https://issues.apache.org/jira/browse/HBASE-20224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465313#comment-16465313 ] Mike Drob commented on HBASE-20224: --- We need a better RN for this, particularly something that instructs downstream projects that they have to change their configurations for this. And a note in the ref guide on upgrading maybe? Filing HBASE-20534 to track that. Is it possible not to put a burden on downstream folks? Sounds like based on the back-and-forth, no? > Web UI is broken in standalone mode > --- > > Key: HBASE-20224 > URL: https://issues.apache.org/jira/browse/HBASE-20224 > Project: HBase > Issue Type: Bug > Components: UI, Usability >Affects Versions: 2.0.0-beta-2 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Blocker > Fix For: 2.0.0 > > Attachments: > 0001-HBASE-20224-Web-UI-is-broken-in-standalone-mode-ADDE.ADDENDUM.patch, > 20224-addendum.3.txt, 20224.addendum.4, 20224.addendum.5, 20224.addendum.6, > HBASE-20224.master.004.patch, hbase-20224.master.001.patch, > hbase-20224.master.002.patch, hbase-20224.master.003.patch, > hbase-20224.master.addendum.patch > > > Web UI doesn't show up in standalone mode on default port. This can be seen > on master and branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20534) need docs on downstream projects required to set random local port assignment for tests
Mike Drob created HBASE-20534: - Summary: need docs on downstream projects required to set random local port assignment for tests Key: HBASE-20534 URL: https://issues.apache.org/jira/browse/HBASE-20534 Project: HBase Issue Type: Bug Components: documentation, test Affects Versions: 2.0.0 Reporter: Mike Drob We need better user facing text to explain that they need to set hbase.localcluster.assign.random.ports to true. See HBASE-20224. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20508) TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test
[ https://issues.apache.org/jira/browse/HBASE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465204#comment-16465204 ] Hadoop QA commented on HBASE-20508: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 1s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 52s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 33s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 49s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 14m 36s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 34s{color} | {color:green} hbase-backup in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 11s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 50m 48s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f | | JIRA Issue | HBASE-20508 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12922187/HBASE-20508-master.v1.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 2a52445e7d1a 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 971f5350e8 | | maven | version: Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC3 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/12734/testReport/ | | Max. process+thread count | 4154 (vs. ulimit of 1) | | modules | C: hbase-backup U: hbase-backup | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/12734/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > TestIncrem
[jira] [Commented] (HBASE-20526) multithreads bulkload performance
[ https://issues.apache.org/jira/browse/HBASE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465191#comment-16465191 ] Hadoop QA commented on HBASE-20526: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-1.3 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 6s{color} | {color:green} branch-1.3 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_172 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_181 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 22s{color} | {color:green} branch-1.3 passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m 37s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_172 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_181 {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 20s{color} | {color:red} hbase-server: The patch generated 6 new + 25 unchanged - 0 fixed = 31 total (was 25) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m 28s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 8m 40s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1 2.5.2 2.6.5 2.7.4. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 45s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}134m 20s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery | | | hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery | | | hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:dca6535 | | JIRA Issue | HBASE-20526 | | JIRA Patch URL | https://issues.apache.org/
[jira] [Commented] (HBASE-20508) TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test
[ https://issues.apache.org/jira/browse/HBASE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465189#comment-16465189 ] Ted Yu commented on HBASE-20508: lgtm Chia-ping: Feel free to integrate once QA comes back. thanks > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test > --- > > Key: HBASE-20508 > URL: https://issues.apache.org/jira/browse/HBASE-20508 > Project: HBase > Issue Type: Test > Components: backup&restore >Reporter: Ted Yu >Assignee: maoling >Priority: Minor > Attachments: HBASE-20508-master.v0.patch, HBASE-20508-master.v1.patch > > > TestIncrementalBackupWithBulkLoad currently is Parameterized with only one > value returned from data() method. > In its ctor, this value is ignored. > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20508) TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test
[ https://issues.apache.org/jira/browse/HBASE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maoling updated HBASE-20508: Attachment: HBASE-20508-master.v1.patch > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test > --- > > Key: HBASE-20508 > URL: https://issues.apache.org/jira/browse/HBASE-20508 > Project: HBase > Issue Type: Test > Components: backup&restore >Reporter: Ted Yu >Assignee: maoling >Priority: Minor > Attachments: HBASE-20508-master.v0.patch, HBASE-20508-master.v1.patch > > > TestIncrementalBackupWithBulkLoad currently is Parameterized with only one > value returned from data() method. > In its ctor, this value is ignored. > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20526) multithreads bulkload performance
[ https://issues.apache.org/jira/browse/HBASE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465143#comment-16465143 ] Ted Yu commented on HBASE-20526: The change makes sense. {code} 382 public void doBulkLoad(Path hfofDir, final Admin admin, Table table, 383 final Pair startEndKeys) throws TableNotFoundException, IOException { {code} Please complete the javadoc for parameters. > multithreads bulkload performance > - > > Key: HBASE-20526 > URL: https://issues.apache.org/jira/browse/HBASE-20526 > Project: HBase > Issue Type: Improvement > Components: mapreduce, Zookeeper >Affects Versions: 1.2.5, 1.3.2 > Environment: hbase-server-1.2.0-cdh5.12.1 > spark version 1.6 >Reporter: Key Hutu >Assignee: Key Hutu >Priority: Minor > Labels: performance > Fix For: 1.3.2 > > Attachments: HBASE-20526-branch-1.3.V1.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > When doing bulkload , some interactive with zookeeper to getting region key > range may be cost more time. > In multithreads enviorment, the duration maybe cost 5 minute or more. > From the executor log, like 'Reading reply sessionid:0x262fb37f4a07080 , > packet:: clientPath:null server ...' contents appear many times. > > It likely to provide new method for bulkload, caching the key range outside > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20526) multithreads bulkload performance
[ https://issues.apache.org/jira/browse/HBASE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465144#comment-16465144 ] Ted Yu commented on HBASE-20526: Please base the next patch on master branch. > multithreads bulkload performance > - > > Key: HBASE-20526 > URL: https://issues.apache.org/jira/browse/HBASE-20526 > Project: HBase > Issue Type: Improvement > Components: mapreduce, Zookeeper >Affects Versions: 1.2.5, 1.3.2 > Environment: hbase-server-1.2.0-cdh5.12.1 > spark version 1.6 >Reporter: Key Hutu >Assignee: Key Hutu >Priority: Minor > Labels: performance > Fix For: 1.3.2 > > Attachments: HBASE-20526-branch-1.3.V1.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > When doing bulkload , some interactive with zookeeper to getting region key > range may be cost more time. > In multithreads enviorment, the duration maybe cost 5 minute or more. > From the executor log, like 'Reading reply sessionid:0x262fb37f4a07080 , > packet:: clientPath:null server ...' contents appear many times. > > It likely to provide new method for bulkload, caching the key range outside > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20531) RS may throw NPE when close meta regions in shutdown procedure.
[ https://issues.apache.org/jira/browse/HBASE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465115#comment-16465115 ] Hudson commented on HBASE-20531: Results for branch master [build #322 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/322/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/322//General_Nightly_Build_Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/322//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/322//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. > RS may throw NPE when close meta regions in shutdown procedure. > > > Key: HBASE-20531 > URL: https://issues.apache.org/jira/browse/HBASE-20531 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20531.v1.patch > > > See also : > https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16463322&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16463322 > The NPE stack is as following: > {code} > 2018-05-03 21:05:58,075 ERROR [RS_CLOSE_REGION-regionserver/instance-2:0-1] > helpers.MarkerIgnoringBase(159): * ABORTING region server > instance-2.c.gcp-hbase.internal,42063,1525381545380: Unrecoverable exception > while closing region tes > t,,1525381436038.66de217a470764f3b37d8faebfd8e8c8., still finishing close > * > java.io.IOException: java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1637) > at > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1466) > at > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3709) > at > org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2718) > at > org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2649) > at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:929) > at > org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1615) > at > org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1612) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ... 3 more > {code} > In HRegionServer#run(), we have the following: > {code} > @Override > public void run() { > .. > // Stop the quota manager > if (rsQuotaManager != null) { > rsQuotaManager.stop(); > } > if (rsSpaceQuotaManager != null) { > rsSpaceQuotaManager.stop(); > rsSpaceQuotaManager = null; > } > .. > // Closing the compactSplit thread before closing meta regions > if (!this.killed && containsMetaTableRegions()) { > if (!abortRequested || this.fsOk) { > if (this.compactSplitThread != null) { > this.compactSplitThread.join(); > this.compactSplitThread = null; > } > closeMetaTableRegions(abortRequested); > } > } > .. > } > {code} > We stop the rsSpaceQuotaManager firstly, and then close the meta region, but > when close meta region, we need to use rsSpaceQuotaManager to > reportFileArchivalForQuotas() , just as the stack trace said ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20527) Remove unused code in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-20527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465114#comment-16465114 ] Hudson commented on HBASE-20527: Results for branch master [build #322 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/322/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/322//General_Nightly_Build_Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/322//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/322//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. > Remove unused code in MetaTableAccessor > --- > > Key: HBASE-20527 > URL: https://issues.apache.org/jira/browse/HBASE-20527 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Mingdao Yang >Assignee: Mingdao Yang >Priority: Trivial > Fix For: 3.0.0, 2.1.0, 2.0.1 > > Attachments: HBASE-20527.v0.patch > > > META_REGION_PREFIX isn't used. I'll clean it up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20526) multithreads bulkload performance
[ https://issues.apache.org/jira/browse/HBASE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Key Hutu updated HBASE-20526: - Fix Version/s: (was: 1.2.5) 1.3.2 Affects Version/s: 1.3.2 Attachment: HBASE-20526-branch-1.3.V1.patch Status: Patch Available (was: In Progress) > multithreads bulkload performance > - > > Key: HBASE-20526 > URL: https://issues.apache.org/jira/browse/HBASE-20526 > Project: HBase > Issue Type: Improvement > Components: mapreduce, Zookeeper >Affects Versions: 1.3.2, 1.2.5 > Environment: hbase-server-1.2.0-cdh5.12.1 > spark version 1.6 >Reporter: Key Hutu >Assignee: Key Hutu >Priority: Minor > Labels: performance > Fix For: 1.3.2 > > Attachments: HBASE-20526-branch-1.3.V1.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > When doing bulkload , some interactive with zookeeper to getting region key > range may be cost more time. > In multithreads enviorment, the duration maybe cost 5 minute or more. > From the executor log, like 'Reading reply sessionid:0x262fb37f4a07080 , > packet:: clientPath:null server ...' contents appear many times. > > It likely to provide new method for bulkload, caching the key range outside > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20508) TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test
[ https://issues.apache.org/jira/browse/HBASE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned HBASE-20508: -- Assignee: maoling > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test > --- > > Key: HBASE-20508 > URL: https://issues.apache.org/jira/browse/HBASE-20508 > Project: HBase > Issue Type: Test > Components: backup&restore >Reporter: Ted Yu >Assignee: maoling >Priority: Minor > Attachments: HBASE-20508-master.v0.patch > > > TestIncrementalBackupWithBulkLoad currently is Parameterized with only one > value returned from data() method. > In its ctor, this value is ignored. > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20508) TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test
[ https://issues.apache.org/jira/browse/HBASE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465109#comment-16465109 ] Chia-Ping Tsai commented on HBASE-20508: see the git history. seems the Parameterized ctor had already been useless in its first commit. So +1 after the checkstyle warnings are fixed. > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test > --- > > Key: HBASE-20508 > URL: https://issues.apache.org/jira/browse/HBASE-20508 > Project: HBase > Issue Type: Test > Components: backup&restore >Reporter: Ted Yu >Priority: Minor > Attachments: HBASE-20508-master.v0.patch > > > TestIncrementalBackupWithBulkLoad currently is Parameterized with only one > value returned from data() method. > In its ctor, this value is ignored. > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20508) TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test
[ https://issues.apache.org/jira/browse/HBASE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465060#comment-16465060 ] Hadoop QA commented on HBASE-20508: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 35s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 52s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 35s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s{color} | {color:red} hbase-backup: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 53s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 14m 26s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 1s{color} | {color:green} hbase-backup in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 11s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 48m 36s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f | | JIRA Issue | HBASE-20508 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12922170/HBASE-20508-master.v0.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux bf6ef10c71fa 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 971f5350e8 | | maven | version: Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC3 | | checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/12732/artifact/patchprocess/diff-checkstyle-hbase-backup.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/12732/testReport/ | | Max. process+thread count | 4092 (vs. ulimit of 1) | | modules | C: hbase-backup U: hbase-backup | | Console output | h
[jira] [Updated] (HBASE-20508) TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test
[ https://issues.apache.org/jira/browse/HBASE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maoling updated HBASE-20508: Status: Patch Available (was: Open) > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test > --- > > Key: HBASE-20508 > URL: https://issues.apache.org/jira/browse/HBASE-20508 > Project: HBase > Issue Type: Test > Components: backup&restore >Reporter: Ted Yu >Priority: Minor > Attachments: HBASE-20508-master.v0.patch > > > TestIncrementalBackupWithBulkLoad currently is Parameterized with only one > value returned from data() method. > In its ctor, this value is ignored. > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20508) TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test
[ https://issues.apache.org/jira/browse/HBASE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maoling updated HBASE-20508: Attachment: HBASE-20508-master.v0.patch > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized test > --- > > Key: HBASE-20508 > URL: https://issues.apache.org/jira/browse/HBASE-20508 > Project: HBase > Issue Type: Test > Components: backup&restore >Reporter: Ted Yu >Priority: Minor > Attachments: HBASE-20508-master.v0.patch > > > TestIncrementalBackupWithBulkLoad currently is Parameterized with only one > value returned from data() method. > In its ctor, this value is ignored. > TestIncrementalBackupWithBulkLoad doesn't need to be Parameterized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)