[jira] [Updated] (HBASE-20298) Doc change in read/write/total accounting metrics
[ https://issues.apache.org/jira/browse/HBASE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20298: -- Attachment: HBASE-20298.master.003.patch > Doc change in read/write/total accounting metrics > - > > Key: HBASE-20298 > URL: https://issues.apache.org/jira/browse/HBASE-20298 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20298.master.001.patch, > HBASE-20298.master.002.patch, HBASE-20298.master.003.patch > > > Doc the change wrought by the parent issue. Get it up into the refguide as > part of the difference between old hbases and hbase2. > The change confused me and took me a while to untangle. > The read count is for reads that return a non-empty result now. In old > hbase1, we'd increment the read-count even if an empty result. This makes > reads look bad in YCSB runs when compared to hbase1 (see how > totalRequestCount in hbase2 can be way above the sum of reads+writes; it is > because it increments even if the row is not found). > Let me get this into refguide otherwise poor old operators will be baffled. > The release note on the parent is great; it just needs to be in our guide. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424243#comment-16424243 ] stack commented on HBASE-20188: --- Thanks [~ram_krish] It helps to have independent observation. I'm doing default configs. and basic YCSB loadings. When all is cached, the two come closer but still a significant difference; not as near as you see (15%). What happens if reads are not all from cache in your setup? Currently looking at shortcircuit reading. We don't seem to be doing it my hbase2 runs compared to hbase1 runs. Will be back. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20256) Document commands that do not work with 1.2 shell
[ https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424239#comment-16424239 ] Peter Somogyi commented on HBASE-20256: --- No, I’ll retrigger it later. > Document commands that do not work with 1.2 shell > - > > Key: HBASE-20256 > URL: https://issues.apache.org/jira/browse/HBASE-20256 > Project: HBase > Issue Type: Sub-task > Components: documentation, shell >Affects Versions: 2.0.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Attachments: HBASE-20256.master.001.patch, > HBASE-20256.master.002.patch > > > Some commands do not work from 1.2 shell when running against 2.0 server. Add > a section to the reference guide mentioning the incompatibilities. > Some of these are collected in this document: > https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20256) Document commands that do not work with 1.2 shell
[ https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424231#comment-16424231 ] Sean Busbey commented on HBASE-20256: - any idea why qabot hasn't shown up? > Document commands that do not work with 1.2 shell > - > > Key: HBASE-20256 > URL: https://issues.apache.org/jira/browse/HBASE-20256 > Project: HBase > Issue Type: Sub-task > Components: documentation, shell >Affects Versions: 2.0.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Attachments: HBASE-20256.master.001.patch, > HBASE-20256.master.002.patch > > > Some commands do not work from 1.2 shell when running against 2.0 server. Add > a section to the reference guide mentioning the incompatibilities. > Some of these are collected in this document: > https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20256) Document commands that do not work with 1.2 shell
[ https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424230#comment-16424230 ] Sean Busbey commented on HBASE-20256: - +1 > Document commands that do not work with 1.2 shell > - > > Key: HBASE-20256 > URL: https://issues.apache.org/jira/browse/HBASE-20256 > Project: HBase > Issue Type: Sub-task > Components: documentation, shell >Affects Versions: 2.0.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Attachments: HBASE-20256.master.001.patch, > HBASE-20256.master.002.patch > > > Some commands do not work from 1.2 shell when running against 2.0 server. Add > a section to the reference guide mentioning the incompatibilities. > Some of these are collected in this document: > https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20193) Basic Replication Web UI - Regionserver
[ https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424228#comment-16424228 ] Sean Busbey commented on HBASE-20193: - thanks for the patch [~tianjingyun]! I'll take a look at the combination of this and HBASE-20194 later this week. Before I can do that I have to clear out some blockers on the upcoming 2.0 release. Anything you can do to review 2.0 blockers or to clean up the javac warning will make reviewing faster / easier. (I get that your patch may not have introduced the javac warning, but I'll still have to chase down who did and/or fix it before committing.) > Basic Replication Web UI - Regionserver > > > Key: HBASE-20193 > URL: https://issues.apache.org/jira/browse/HBASE-20193 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20193.master.001.patch, > HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, > HBASE-20193.master.004.patch > > > subtask of HBASE-15809. Implementation of replication UI on Regionserver web > page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19488) Remove Unused Code from CollectionUtils
[ https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424223#comment-16424223 ] Hadoop QA commented on HBASE-19488: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 24s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m 19s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 8s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} hbase-common: The patch generated 0 new + 89 unchanged - 1 fixed = 89 total (was 90) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} The patch hbase-client passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} The patch hbase-replication passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 17s{color} | {color:green} hbase-server: The patch generated 0 new + 294 unchanged - 1 fixed = 294 total (was 295) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 57s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 20m 57s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 36s{color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 26s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s{color} | {color:green} hbase-replication in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}117m 43s{color} | {color:green} hbase-server in the patch
[jira] [Updated] (HBASE-20329) Add note for operators to refguide on AsyncFSWAL
[ https://issues.apache.org/jira/browse/HBASE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20329: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: (was: 2.0.0) 3.0.0 Status: Resolved (was: Patch Available) Pushed to master branch. I'll pull it back to branch-2 and branch-2.0 when I copy the refguide from master to the release branch. Thanks for reviews. > Add note for operators to refguide on AsyncFSWAL > > > Key: HBASE-20329 > URL: https://issues.apache.org/jira/browse/HBASE-20329 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 3.0.0 > > Attachments: HBASE-20329.master.001.patch > > > Need a few notes in refguide on this new facility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19890) Canary usage should document hbase.canary.sink.class config
[ https://issues.apache.org/jira/browse/HBASE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424200#comment-16424200 ] Hadoop QA commented on HBASE-19890: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red}419m 52s{color} | {color:red} Docker failed to build yetus/hbase:d8b550f. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HBASE-19890 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12917326/HBASE-19890.master.001.patch | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/12273/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Canary usage should document hbase.canary.sink.class config > --- > > Key: HBASE-19890 > URL: https://issues.apache.org/jira/browse/HBASE-19890 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-19890.master.001.patch > > > Canary#main uses config hbase.canary.sink.class to instantiate Sink class. > The Sink instance affects creation of Monitor. > In the refguide for Canary, hbase.canary.sink.class was not mentioned. > We should document this config. > Additionally, we need to document that using the default sink is not > compatible with table parameters as input so the user must change it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20329) Add note for operators to refguide on AsyncFSWAL
[ https://issues.apache.org/jira/browse/HBASE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424195#comment-16424195 ] stack commented on HBASE-20329: --- Thanks boys. Let me add in your fixes on push. > Add note for operators to refguide on AsyncFSWAL > > > Key: HBASE-20329 > URL: https://issues.apache.org/jira/browse/HBASE-20329 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20329.master.001.patch > > > Need a few notes in refguide on this new facility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters
[ https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424167#comment-16424167 ] Hadoop QA commented on HBASE-16499: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 2s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 52s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 55s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 20m 20s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}117m 51s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}163m 54s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f | | JIRA Issue | HBASE-16499 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12917354/HBASE-16499.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8b3fa7e43440 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 219625233c | | maven | version: Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) | | Default Java | 1.8.0_162 | | findbugs | v3.1.0-RC3 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/12274/testReport/ | | Max. process+thread count | 4181 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/12274/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > slow replication for small HBase
[jira] [Created] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
Sean Busbey created HBASE-20334: --- Summary: add a test that expressly uses both our shaded client and the one from hadoop 3 Key: HBASE-20334 URL: https://issues.apache.org/jira/browse/HBASE-20334 Project: HBase Issue Type: Sub-task Components: hadoop3, shading Affects Versions: 2.0.0 Reporter: Sean Busbey Since we're making a shaded client that bleed out of our namespace and into Hadoop's, we should ensure that we can show our clients coexisting. Even if it's just an IT that successfully talks to both us and HDFS via our respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20333) break up shaded client into one with no Hadoop and one that's standalone
[ https://issues.apache.org/jira/browse/HBASE-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424161#comment-16424161 ] Sean Busbey commented on HBASE-20333: - artifact naming is difficult here. which is the more common use case: "just using HBase" or "using hbase around hadoop"? whichever should be the one that gets the less complicated name. I suspect "just using HBase" is the more common case, but names like "hbase-shaded-client-without-hadoop" are awkward. > break up shaded client into one with no Hadoop and one that's standalone > > > Key: HBASE-20333 > URL: https://issues.apache.org/jira/browse/HBASE-20333 > Project: HBase > Issue Type: Sub-task > Components: shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > there are contexts where we want to stay out of our downstream users way wrt > dependencies, but they need more Hadoop classes than we provide. i.e. any > downstream client that wants to use both HBase and HDFS in their application, > or any non-MR YARN application. > Now that Hadoop also has shaded client artifacts for Hadoop 3, we're also > providing less incremental benefit by including our own rewritten Hadoop > classes to avoid downstream needing to pull in all of Hadoop's transitive > dependencies. > right now those users need to ensure that any jars from the Hadoop project > are loaded in the classpath prior to our shaded client jar. This is brittle > and prone to weird debugging trouble. > instead, we should have two artifacts: one that just lists Hadoop as a > prerequisite and one that still includes the rewritten-but-not-relocated > Hadoop classes. > We can then use docs to emphasize when each of these is appropriate to use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20333) break up shaded client into one with no Hadoop and one that's standalone
Sean Busbey created HBASE-20333: --- Summary: break up shaded client into one with no Hadoop and one that's standalone Key: HBASE-20333 URL: https://issues.apache.org/jira/browse/HBASE-20333 Project: HBase Issue Type: Sub-task Components: shading Affects Versions: 2.0.0 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0 there are contexts where we want to stay out of our downstream users way wrt dependencies, but they need more Hadoop classes than we provide. i.e. any downstream client that wants to use both HBase and HDFS in their application, or any non-MR YARN application. Now that Hadoop also has shaded client artifacts for Hadoop 3, we're also providing less incremental benefit by including our own rewritten Hadoop classes to avoid downstream needing to pull in all of Hadoop's transitive dependencies. right now those users need to ensure that any jars from the Hadoop project are loaded in the classpath prior to our shaded client jar. This is brittle and prone to weird debugging trouble. instead, we should have two artifacts: one that just lists Hadoop as a prerequisite and one that still includes the rewritten-but-not-relocated Hadoop classes. We can then use docs to emphasize when each of these is appropriate to use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[ https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424156#comment-16424156 ] Sean Busbey commented on HBASE-20332: - I think the appropriate scope for the hadoop bits we need on this artifact is {{provided}}, given #1 and #2 above > shaded mapreduce module shouldn't include hadoop > > > Key: HBASE-20332 > URL: https://issues.apache.org/jira/browse/HBASE-20332 > Project: HBase > Issue Type: Sub-task > Components: mapreduce, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > AFAICT, we should just entirely skip including hadoop in our shaded mapreduce > module > 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}} > 2) those commands include all the needed Hadoop jars in your classpath by > default (both client side and in the containers) > 3) If you try to use "user classpath first" for your job as a workaround > (e.g. for some library your application needs that hadoop provides) then our > inclusion of *some but not all* hadoop classes then causes everything to fall > over because of mixing rewritten and non-rewritten hadoop classes > 4) if you don't use "user classpath first" then all of our > non-relocated-but-still-shaded hadoop classes are ignored anyways so we're > just wasting space -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[ https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-20332: --- Assignee: Sean Busbey > shaded mapreduce module shouldn't include hadoop > > > Key: HBASE-20332 > URL: https://issues.apache.org/jira/browse/HBASE-20332 > Project: HBase > Issue Type: Sub-task > Components: mapreduce, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > AFAICT, we should just entirely skip including hadoop in our shaded mapreduce > module > 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}} > 2) those commands include all the needed Hadoop jars in your classpath by > default (both client side and in the containers) > 3) If you try to use "user classpath first" for your job as a workaround > (e.g. for some library your application needs that hadoop provides) then our > inclusion of *some but not all* hadoop classes then causes everything to fall > over because of mixing rewritten and non-rewritten hadoop classes > 4) if you don't use "user classpath first" then all of our > non-relocated-but-still-shaded hadoop classes are ignored anyways so we're > just wasting space -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[ https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424152#comment-16424152 ] Sean Busbey commented on HBASE-20332: - re #1, you currently can't use our shaded mapreduce jar to submit MR applications directly without the use of the {{hadoop jar}} / {{yarn jar}} commands, because we need some Hadoop common bits for dealing with the local filesystem. so if we want a standalone jar for e.g. submitting MR jobs from a node that has no Hadoop installation, that's a different pile of work (and I'd argue a less common case that we should handle after making sure we have a simple "easy" path) > shaded mapreduce module shouldn't include hadoop > > > Key: HBASE-20332 > URL: https://issues.apache.org/jira/browse/HBASE-20332 > Project: HBase > Issue Type: Sub-task > Components: mapreduce, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > AFAICT, we should just entirely skip including hadoop in our shaded mapreduce > module > 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}} > 2) those commands include all the needed Hadoop jars in your classpath by > default (both client side and in the containers) > 3) If you try to use "user classpath first" for your job as a workaround > (e.g. for some library your application needs that hadoop provides) then our > inclusion of *some but not all* hadoop classes then causes everything to fall > over because of mixing rewritten and non-rewritten hadoop classes > 4) if you don't use "user classpath first" then all of our > non-relocated-but-still-shaded hadoop classes are ignored anyways so we're > just wasting space -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
Sean Busbey created HBASE-20332: --- Summary: shaded mapreduce module shouldn't include hadoop Key: HBASE-20332 URL: https://issues.apache.org/jira/browse/HBASE-20332 Project: HBase Issue Type: Sub-task Components: mapreduce, shading Affects Versions: 2.0.0 Reporter: Sean Busbey Fix For: 2.0.0 AFAICT, we should just entirely skip including hadoop in our shaded mapreduce module 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}} 2) those commands include all the needed Hadoop jars in your classpath by default (both client side and in the containers) 3) If you try to use "user classpath first" for your job as a workaround (e.g. for some library your application needs that hadoop provides) then our inclusion of *some but not all* hadoop classes then causes everything to fall over because of mixing rewritten and non-rewritten hadoop classes 4) if you don't use "user classpath first" then all of our non-relocated-but-still-shaded hadoop classes are ignored anyways so we're just wasting space -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient
[ https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-20244: --- Attachment: 20244.v1.txt > NoSuchMethodException when retrieving private method > decryptEncryptedDataEncryptionKey from DFSClient > - > > Key: HBASE-20244 > URL: https://issues.apache.org/jira/browse/HBASE-20244 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt > > > I was running unit test against hadoop 3.0.1 RC and saw the following in test > output: > {code} > ERROR [RS-EventLoopGroup-3-3] > asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly > initialize access to HDFS internals. Please update your WAL Provider to not > make use of the 'asyncfs' provider. See HBASE-16110 for more information. > java.lang.NoSuchMethodException: > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) > at java.lang.Class.getDeclaredMethod(Class.java:2130) > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232) > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262) > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661) > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118) > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720) > at > org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715) > at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) > at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) > at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) > at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) > at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) > at > org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) > at > org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306) > at > org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341) > at > org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) > at > org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > {code} > The private method was moved by HDFS-12574 to HdfsKMSUtil with different > signature. > To accommodate the above method movement, it seems we need to call the > following method of DFSClient : > {code} > public KeyProvider getKeyProvider() throws IOException { > {code} > Since the new decryptEncryptedDataEncryptionKey method has this signature: > {code} > static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo > feInfo, KeyProvider keyProvider) throws IOException { > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20095) Redesign single instance pool in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424133#comment-16424133 ] Ted Yu commented on HBASE-20095: Ping [~mdrob] Please take a look at the addendum at your convenience. > Redesign single instance pool in CleanerChore > - > > Key: HBASE-20095 > URL: https://issues.apache.org/jira/browse/HBASE-20095 > Project: HBase > Issue Type: Improvement >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Critical > Fix For: 3.0.0, 2.1.0 > > Attachments: 20095.addendum, 20095.addendum-2.txt, > HBASE-20095.master.001.patch, HBASE-20095.master.002.patch, > HBASE-20095.master.003.patch, HBASE-20095.master.004.patch, > HBASE-20095.master.005.patch, HBASE-20095.master.006.patch, > HBASE-20095.master.007.patch, HBASE-20095.master.008.patch, > HBASE-20095.master.009.patch, HBASE-20095.master.010.patch, > HBASE-20095.master.011.patch, HBASE-20095.master.012.patch, > HBASE-20095.master.013.patch, HBASE-20095.master.014.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20331) clean up shaded packaging for 2.0
Sean Busbey created HBASE-20331: --- Summary: clean up shaded packaging for 2.0 Key: HBASE-20331 URL: https://issues.apache.org/jira/browse/HBASE-20331 Project: HBase Issue Type: Umbrella Components: Client, mapreduce, shading Affects Versions: 2.0.0 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0 polishing pass on shaded modules for 2.0 based on trying to use them in more contexts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[ https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424123#comment-16424123 ] Hadoop QA commented on HBASE-19852: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 16s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 48s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 19m 22s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 59s{color} | {color:green} hbase-thrift in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 8s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 41m 8s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f | | JIRA Issue | HBASE-19852 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12917369/HBASE-19852.master.008.patch | | Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck xml compile findbugs hbaseanti checkstyle | | uname | Linux ecdcf200bb7c 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 219625233c | | maven | version: Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) | | Default Java | 1.8.0_162 | | findbugs | v3.1.0-RC3 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/12277/testReport/ | | Max. process+thread count | 1784 (vs. ulimit of 1) | | modules | C: hbase-thrift U: hbase-thrift | | Console output |
[jira] [Commented] (HBASE-19890) Canary usage should document hbase.canary.sink.class config
[ https://issues.apache.org/jira/browse/HBASE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424119#comment-16424119 ] Ted Yu commented on HBASE-19890: lgtm > Canary usage should document hbase.canary.sink.class config > --- > > Key: HBASE-19890 > URL: https://issues.apache.org/jira/browse/HBASE-19890 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-19890.master.001.patch > > > Canary#main uses config hbase.canary.sink.class to instantiate Sink class. > The Sink instance affects creation of Monitor. > In the refguide for Canary, hbase.canary.sink.class was not mentioned. > We should document this config. > Additionally, we need to document that using the default sink is not > compatible with table parameters as input so the user must change it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20253) Error message is missing for restore_snapshot
[ https://issues.apache.org/jira/browse/HBASE-20253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20253: Status: In Progress (was: Patch Available) I believe we still need to re-throw the exception even with the improved message. Otherwise, the shell will claim that the operation succeeded because the {{puts}} succeeds and is the last operation. example prior to this patch: {code} $ hbase shell --noninteractive < create 'test:some_table', 'family' > snapshot 'test:some_table', 'example_snapshot' > restore_snapshot 'example_snapshot' > EOF Created table test:some_table Took 1.1634 seconds Hbase::Table - test:some_table Took 0.5606 seconds Took 0.1246 seconds java exception ERROR Java::OrgApacheHadoopHbase::TableNotDisabledException: test:some_table [root@hbase19158-1 ~]# echo $? 1 {code} example with this patch: {code} $ hbase shell --noninteractive < create 'test:some_table', 'family' > snapshot 'test:some_table', 'example_snapshot' > restore_snapshot 'example_snapshot' > EOF Created table test:some_table Took 1.1208 seconds Hbase::Table - test:some_table Took 0.4879 seconds Table 'test:some_table' should be disabled to restore snapshot. Took 0.1080 seconds [root@hbase19158-1 ~]# echo $? 0 {code} It'd be good to add a test that checks this negative example while we're here. > Error message is missing for restore_snapshot > - > > Key: HBASE-20253 > URL: https://issues.apache.org/jira/browse/HBASE-20253 > Project: HBase > Issue Type: Sub-task > Components: shell >Affects Versions: 2.0.0 >Reporter: Peter Somogyi >Assignee: Gabor Bota >Priority: Minor > Attachments: HBASE-20253.master.001.patch, > HBASE-20253.master.002.patch > > > When the table is not disabled and restore_snapshot executed the error > message is useless, only displays the table name. > hbase(main):007:0> restore_snapshot 'tsnap' > ERROR: t > Restore a specified snapshot. > The restore will replace the content of the original table, > bringing back the content to the snapshot state. > The table must be disabled. > Examples: > hbase> restore_snapshot 'snapshotName' > Following command will restore all acl from snapshot table into the table. > hbase> restore_snapshot 'snapshotName', \{RESTORE_ACL=>true} > Took 0.1044 seconds -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[ https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424071#comment-16424071 ] Kevin Risden commented on HBASE-19852: -- Looks like the precommit test failures were due to something else. Re-uploaded patch with correct naming convention. Ping [~elserj] and [~carp84] - Would appreciate another review. > HBase Thrift 1 server SPNEGO Improvements > - > > Key: HBASE-19852 > URL: https://issues.apache.org/jira/browse/HBASE-19852 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Major > Attachments: HBASE-19852.master.001.patch, > HBASE-19852.master.002.patch, HBASE-19852.master.003.patch, > HBASE-19852.master.004.patch, HBASE-19852.master.006.patch, > HBASE-19852.master.007.patch.txt, HBASE-19852.master.008.patch > > > HBase Thrift1 server has some issues when trying to use SPNEGO. > From mailing list: > http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E > {quote}While setting up the HBase Thrift server with HTTP, there were a > significant amount of 401 errors where the HBase Thrift wasn't able to > handle the incoming Kerberos request. Documentation online is sparse when > it comes to setting up the principal/keytab for HTTP Kerberos. > I noticed that the HBase Thrift HTTP implementation was missing SPNEGO > principal/keytab like other Thrift based servers (HiveServer2). It looks > like HiveServer2 Thrift implementation and HBase Thrift v1 implementation > were very close to the same at one point. I made the following changes to > HBase Thrift v1 server implementation to make it work: > * add SPNEGO principal/keytab if in HTTP mode > * return 401 immediately if no authorization header instead of waiting for > try/catch down in program flow{quote} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[ https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Risden updated HBASE-19852: - Attachment: HBASE-19852.master.008.patch > HBase Thrift 1 server SPNEGO Improvements > - > > Key: HBASE-19852 > URL: https://issues.apache.org/jira/browse/HBASE-19852 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Major > Attachments: HBASE-19852.master.001.patch, > HBASE-19852.master.002.patch, HBASE-19852.master.003.patch, > HBASE-19852.master.004.patch, HBASE-19852.master.006.patch, > HBASE-19852.master.007.patch.txt, HBASE-19852.master.008.patch > > > HBase Thrift1 server has some issues when trying to use SPNEGO. > From mailing list: > http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E > {quote}While setting up the HBase Thrift server with HTTP, there were a > significant amount of 401 errors where the HBase Thrift wasn't able to > handle the incoming Kerberos request. Documentation online is sparse when > it comes to setting up the principal/keytab for HTTP Kerberos. > I noticed that the HBase Thrift HTTP implementation was missing SPNEGO > principal/keytab like other Thrift based servers (HiveServer2). It looks > like HiveServer2 Thrift implementation and HBase Thrift v1 implementation > were very close to the same at one point. I made the following changes to > HBase Thrift v1 server implementation to make it work: > * add SPNEGO principal/keytab if in HTTP mode > * return 401 immediately if no authorization header instead of waiting for > try/catch down in program flow{quote} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20276) [shell] confirm shell REPL change and document
[ https://issues.apache.org/jira/browse/HBASE-20276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424068#comment-16424068 ] Sean Busbey commented on HBASE-20276: - [I put up a DISCUSS thread pointing folks over here|https://lists.apache.org/thread.html/23cf8f132b0b87d34f14c5c7dee2a8458ce89af69e3bf0e9fda9e61a@%3Cdev.hbase.apache.org%3E]. I think we already have enough consensus to get a patch up for 1.5+ I'm happy to either get the patch together or review. I'll do the former after I wrap up some other stuff, or the latter if someone else has time to do it sooner. FWIW, I think we should keep the --return-values flag because it's been in a release. we should just have it print a message about how the change to not return values has been removed. > [shell] confirm shell REPL change and document > -- > > Key: HBASE-20276 > URL: https://issues.apache.org/jira/browse/HBASE-20276 > Project: HBase > Issue Type: Sub-task > Components: documentation, shell >Affects Versions: 1.4.0, 2.0.0 >Reporter: Sean Busbey >Priority: Blocker > Fix For: 1.4.4, 2.0.0 > > > Feedback from [~mdrob] on HBASE-19158: > {quote} > Shell: > HBASE-19770. There was another issue opened where this was identified as a > problem so maybe the shape will change further, but I can't find it now. > {quote} > New commentary from [~busbey]: > This was a follow on to HBASE-15965. That change effectively makes it so none > of our ruby wrappers can be used to build expressions in an interactive REPL. > This is a pretty severe change (most of my tips on HBASE-15611 will break, I > think). > I think we should > a) Have a DISCUSS thread, spanning dev@ and user@ > b) based on the outcome of that thread, either default to the new behavior or > the old behavior > c) if we keep the HBASE-15965 behavior as the default, flag it as > incompatible, call it out in the hbase 2.0 upgrade section, and update docs > (two examples: the output in the shell_exercises sections would be wrong, and > the _table_variables section won't work) > d) In either case document the new flag in the ref guide -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20258) Shell hangs when scanning a disabled table
[ https://issues.apache.org/jira/browse/HBASE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424066#comment-16424066 ] Hadoop QA commented on HBASE-20258: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 39s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 5s{color} | {color:red} The patch generated 4 new + 25 unchanged - 4 fixed = 29 total (was 29) {color} | | {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m 1s{color} | {color:red} The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 24s{color} | {color:green} hbase-shell in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 9s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 17m 38s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f | | JIRA Issue | HBASE-20258 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12917368/HBASE-20258.master.002.patch | | Optional Tests | asflicense javac javadoc unit rubocop ruby_lint | | uname | Linux 68b9ded4dae5 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 219625233c | | maven | version: Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) | | Default Java | 1.8.0_162 | | rubocop | v0.54.0 | | rubocop | https://builds.apache.org/job/PreCommit-HBASE-Build/12276/artifact/patchprocess/diff-patch-rubocop.txt | | ruby-lint | v2.3.1 | | ruby-lint | https://builds.apache.org/job/PreCommit-HBASE-Build/12276/artifact/patchprocess/diff-patch-ruby-lint.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/12276/testReport/ | | Max. process+thread count | 2137 (vs. ulimit of 1) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/12276/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Shell hangs when scanning a disabled table > -- > > Key: HBASE-20258 > URL: https://issues.apache.org/jira/browse/HBASE-20258 > Project: HBase > Issue Type: Sub-task >Reporter: Balazs Meszaros >Assignee: Balazs Meszaros >Priority: Major > Fix For: 3.0.0, 2.0.0 > > Attachments: HBASE-20258.branch-2.001.patch, > HBASE-20258.branch-2.002.patch, HBASE-20258.master.001.patch, > HBASE-20258.master.002.patch > > > I executed the following commands against a 2.0 server: > {code} > disable 't' > scan 't' > {code} > From client-1.2: it throws an error because the table is disabled -> OK. > From client-2.0: the shell hangs -> NOT OK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20305) Add option to SyncTable that skip deletes on target cluster
[ https://issues.apache.org/jira/browse/HBASE-20305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424058#comment-16424058 ] Wellington Chevreuil commented on HBASE-20305: -- Thanks for the review and suggestions [~davelatham]! Indeed, *doDeletes* sounds more intuitive. I'm gonna work on this and the additional *doPuts,* together with reviews for checkstyle violations. > Add option to SyncTable that skip deletes on target cluster > --- > > Key: HBASE-20305 > URL: https://issues.apache.org/jira/browse/HBASE-20305 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 2.0.0-alpha-4 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: 0001-HBASE-20305.master.001.patch > > > We had a situation where two clusters with active-active replication got out > of sync, but both had data that should be kept. The tables in question never > have data deleted, but ingestion had happened on the two different clusters, > some rows had been even updated. > In this scenario, a cell that is present in one of the table clusters should > not be deleted, but replayed on the other. Also, for cells with same > identifier but different values, the most recent value should be kept. > Current version of SyncTable would not be applicable here, because it would > simply copy the whole state from source to target, then losing any additional > rows that might be only in target, as well as cell values that got most > recent update. This could be solved by adding an option to skip deletes for > SyncTable. This way, the additional cells not present on source would still > be kept. For cells with same identifier but different values, it would just > perform a Put for the cell version from source, but client scans would still > fetch the most recent timestamp. > I'm attaching a patch with this additional option shortly. Please share your > thoughts. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20258) Shell hangs when scanning a disabled table
[ https://issues.apache.org/jira/browse/HBASE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-20258: Attachment: HBASE-20258.master.002.patch HBASE-20258.branch-2.002.patch > Shell hangs when scanning a disabled table > -- > > Key: HBASE-20258 > URL: https://issues.apache.org/jira/browse/HBASE-20258 > Project: HBase > Issue Type: Sub-task >Reporter: Balazs Meszaros >Assignee: Balazs Meszaros >Priority: Major > Fix For: 3.0.0, 2.0.0 > > Attachments: HBASE-20258.branch-2.001.patch, > HBASE-20258.branch-2.002.patch, HBASE-20258.master.001.patch, > HBASE-20258.master.002.patch > > > I executed the following commands against a 2.0 server: > {code} > disable 't' > scan 't' > {code} > From client-1.2: it throws an error because the table is disabled -> OK. > From client-2.0: the shell hangs -> NOT OK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20258) Shell hangs when scanning a disabled table
[ https://issues.apache.org/jira/browse/HBASE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-20258: Attachment: (was: HBASE-20258.master.002.patch) > Shell hangs when scanning a disabled table > -- > > Key: HBASE-20258 > URL: https://issues.apache.org/jira/browse/HBASE-20258 > Project: HBase > Issue Type: Sub-task >Reporter: Balazs Meszaros >Assignee: Balazs Meszaros >Priority: Major > Fix For: 3.0.0, 2.0.0 > > Attachments: HBASE-20258.branch-2.001.patch, > HBASE-20258.master.001.patch > > > I executed the following commands against a 2.0 server: > {code} > disable 't' > scan 't' > {code} > From client-1.2: it throws an error because the table is disabled -> OK. > From client-2.0: the shell hangs -> NOT OK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20258) Shell hangs when scanning a disabled table
[ https://issues.apache.org/jira/browse/HBASE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-20258: Attachment: (was: HBASE-20258.branch-2.002.patch) > Shell hangs when scanning a disabled table > -- > > Key: HBASE-20258 > URL: https://issues.apache.org/jira/browse/HBASE-20258 > Project: HBase > Issue Type: Sub-task >Reporter: Balazs Meszaros >Assignee: Balazs Meszaros >Priority: Major > Fix For: 3.0.0, 2.0.0 > > Attachments: HBASE-20258.branch-2.001.patch, > HBASE-20258.master.001.patch > > > I executed the following commands against a 2.0 server: > {code} > disable 't' > scan 't' > {code} > From client-1.2: it throws an error because the table is disabled -> OK. > From client-2.0: the shell hangs -> NOT OK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20258) Shell hangs when scanning a disabled table
[ https://issues.apache.org/jira/browse/HBASE-20258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424028#comment-16424028 ] Balazs Meszaros commented on HBASE-20258: - .002 correct the failures, but they weren't checked. > Shell hangs when scanning a disabled table > -- > > Key: HBASE-20258 > URL: https://issues.apache.org/jira/browse/HBASE-20258 > Project: HBase > Issue Type: Sub-task >Reporter: Balazs Meszaros >Assignee: Balazs Meszaros >Priority: Major > Fix For: 3.0.0, 2.0.0 > > Attachments: HBASE-20258.branch-2.001.patch, > HBASE-20258.master.001.patch > > > I executed the following commands against a 2.0 server: > {code} > disable 't' > scan 't' > {code} > From client-1.2: it throws an error because the table is disabled -> OK. > From client-2.0: the shell hangs -> NOT OK. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19488) Remove Unused Code from CollectionUtils
[ https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423980#comment-16423980 ] BELUGA BEHR commented on HBASE-19488: - [~appy] I just threw a new patch up. > Remove Unused Code from CollectionUtils > --- > > Key: HBASE-19488 > URL: https://issues.apache.org/jira/browse/HBASE-19488 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, > HBASE-19488.3.patch, HBASE-19488.4.patch > > > A bunch of unused code in CollectionUtils or code that can be found in Apache > Commons libraries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19488) Remove Unused Code from CollectionUtils
[ https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HBASE-19488: Status: Patch Available (was: Open) > Remove Unused Code from CollectionUtils > --- > > Key: HBASE-19488 > URL: https://issues.apache.org/jira/browse/HBASE-19488 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, > HBASE-19488.3.patch, HBASE-19488.4.patch > > > A bunch of unused code in CollectionUtils or code that can be found in Apache > Commons libraries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19488) Remove Unused Code from CollectionUtils
[ https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HBASE-19488: Status: Open (was: Patch Available) > Remove Unused Code from CollectionUtils > --- > > Key: HBASE-19488 > URL: https://issues.apache.org/jira/browse/HBASE-19488 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, > HBASE-19488.3.patch, HBASE-19488.4.patch > > > A bunch of unused code in CollectionUtils or code that can be found in Apache > Commons libraries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19488) Remove Unused Code from CollectionUtils
[ https://issues.apache.org/jira/browse/HBASE-19488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HBASE-19488: Attachment: HBASE-19488.4.patch > Remove Unused Code from CollectionUtils > --- > > Key: HBASE-19488 > URL: https://issues.apache.org/jira/browse/HBASE-19488 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HBASE-19488.1.patch, HBASE-19488.2.patch, > HBASE-19488.3.patch, HBASE-19488.4.patch > > > A bunch of unused code in CollectionUtils or code that can be found in Apache > Commons libraries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20259) Doc configs for in-memory-compaction and add detail to in-memory-compaction logging
[ https://issues.apache.org/jira/browse/HBASE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423950#comment-16423950 ] Peter Somogyi commented on HBASE-20259: --- Attached a simple addendum. > Doc configs for in-memory-compaction and add detail to in-memory-compaction > logging > --- > > Key: HBASE-20259 > URL: https://issues.apache.org/jira/browse/HBASE-20259 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20259.branch-2.0.ADDENDUM.patch, > HBASE-20259.master.001.patch, HBASE-20259.master.002.patch, > HBASE-20259.master.003.patch > > > I set {{hbase.systemtables.compacting.memstore.type}} to NONE but it seems > like in-memory is still on. My table looks like this: > {code} > Table ycsb is ENABLED > ycsb > COLUMN FAMILIES DESCRIPTION > {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', > NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', > CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => > 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER = > > 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', > > CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', > > COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} > {code} > Looks like table doesn't have it on either (IN_MEMORY_COMPACTION doesn't show > in the above). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20259) Doc configs for in-memory-compaction and add detail to in-memory-compaction logging
[ https://issues.apache.org/jira/browse/HBASE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-20259: -- Attachment: HBASE-20259.branch-2.0.ADDENDUM.patch > Doc configs for in-memory-compaction and add detail to in-memory-compaction > logging > --- > > Key: HBASE-20259 > URL: https://issues.apache.org/jira/browse/HBASE-20259 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20259.branch-2.0.ADDENDUM.patch, > HBASE-20259.master.001.patch, HBASE-20259.master.002.patch, > HBASE-20259.master.003.patch > > > I set {{hbase.systemtables.compacting.memstore.type}} to NONE but it seems > like in-memory is still on. My table looks like this: > {code} > Table ycsb is ENABLED > ycsb > COLUMN FAMILIES DESCRIPTION > {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', > NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', > CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => > 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER = > > 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', > > CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', > > COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} > {code} > Looks like table doesn't have it on either (IN_MEMORY_COMPACTION doesn't show > in the above). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-20259) Doc configs for in-memory-compaction and add detail to in-memory-compaction logging
[ https://issues.apache.org/jira/browse/HBASE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-20259: --- Reopening to fix merge conflict on branch-2.0. > Doc configs for in-memory-compaction and add detail to in-memory-compaction > logging > --- > > Key: HBASE-20259 > URL: https://issues.apache.org/jira/browse/HBASE-20259 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20259.master.001.patch, > HBASE-20259.master.002.patch, HBASE-20259.master.003.patch > > > I set {{hbase.systemtables.compacting.memstore.type}} to NONE but it seems > like in-memory is still on. My table looks like this: > {code} > Table ycsb is ENABLED > ycsb > COLUMN FAMILIES DESCRIPTION > {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', > NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', > CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => > 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER = > > 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', > > CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', > > COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} > {code} > Looks like table doesn't have it on either (IN_MEMORY_COMPACTION doesn't show > in the above). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16499) slow replication for small HBase clusters
[ https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-16499: -- Attachment: HBASE-16499.patch > slow replication for small HBase clusters > - > > Key: HBASE-16499 > URL: https://issues.apache.org/jira/browse/HBASE-16499 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Vikas Vishwakarma >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16499.patch, HBASE-16499.patch > > > For small clusters 10-20 nodes we recently observed that replication is > progressing very slowly when we do bulk writes and there is lot of lag > accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed > that the number of threads used for shipping wal edits in parallel comes from > the following equation in HBaseInterClusterReplicationEndpoint > int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1), > replicationSinkMgr.getSinks().size()); > ... > for (int i=0; ientryLists.add(new ArrayList(entries.size()/n+1)); <-- > batch size > } > ... > for (int i=0; i . > // RuntimeExceptions encountered here bubble up and are handled > in ReplicationSource > pool.submit(createReplicator(entryLists.get(i), i)); <-- > concurrency > futures++; > } > } > maxThreads is fixed & configurable and since we are taking min of the three > values n gets decided based replicationSinkMgr.getSinks().size() when we have > enough edits to replicate > replicationSinkMgr.getSinks().size() is decided based on > int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio); > where ratio is this.ratio = conf.getFloat("replication.source.ratio", > DEFAULT_REPLICATION_SOURCE_RATIO); > Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small > clusters of size 10-20 RegionServers the value we get for numSinks and hence > n is very small like 1 or 2. This substantially reduces the pool concurrency > used for shipping wal edits in parallel effectively slowing down replication > for small clusters and causing lot of lag accumulation in AgeOfLastShipped. > Sometimes it takes tens of hours to clear off the entire replication queue > even after the client has finished writing on the source side. > We are running tests by varying replication.source.ratio and have seen > multi-fold improvement in total replication time (will update the results > here). I wanted to propose here that we should increase the default value for > replication.source.ratio also so that we have sufficient concurrency even for > small clusters. We figured it out after lot of iterations and debugging so > probably slightly higher default will save the trouble. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters
[ https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423947#comment-16423947 ] Ashish Singhi commented on HBASE-16499: --- May be same RS in peer was getting selected for replication every time! > slow replication for small HBase clusters > - > > Key: HBASE-16499 > URL: https://issues.apache.org/jira/browse/HBASE-16499 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Vikas Vishwakarma >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16499.patch, HBASE-16499.patch > > > For small clusters 10-20 nodes we recently observed that replication is > progressing very slowly when we do bulk writes and there is lot of lag > accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed > that the number of threads used for shipping wal edits in parallel comes from > the following equation in HBaseInterClusterReplicationEndpoint > int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1), > replicationSinkMgr.getSinks().size()); > ... > for (int i=0; ientryLists.add(new ArrayList(entries.size()/n+1)); <-- > batch size > } > ... > for (int i=0; i . > // RuntimeExceptions encountered here bubble up and are handled > in ReplicationSource > pool.submit(createReplicator(entryLists.get(i), i)); <-- > concurrency > futures++; > } > } > maxThreads is fixed & configurable and since we are taking min of the three > values n gets decided based replicationSinkMgr.getSinks().size() when we have > enough edits to replicate > replicationSinkMgr.getSinks().size() is decided based on > int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio); > where ratio is this.ratio = conf.getFloat("replication.source.ratio", > DEFAULT_REPLICATION_SOURCE_RATIO); > Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small > clusters of size 10-20 RegionServers the value we get for numSinks and hence > n is very small like 1 or 2. This substantially reduces the pool concurrency > used for shipping wal edits in parallel effectively slowing down replication > for small clusters and causing lot of lag accumulation in AgeOfLastShipped. > Sometimes it takes tens of hours to clear off the entire replication queue > even after the client has finished writing on the source side. > We are running tests by varying replication.source.ratio and have seen > multi-fold improvement in total replication time (will update the results > here). I wanted to propose here that we should increase the default value for > replication.source.ratio also so that we have sufficient concurrency even for > small clusters. We figured it out after lot of iterations and debugging so > probably slightly higher default will save the trouble. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423946#comment-16423946 ] Ashish Singhi commented on HBASE-15291: --- Test failure is not related. > FileSystem not closed in secure bulkLoad > > > Key: HBASE-15291 > URL: https://issues.apache.org/jira/browse/HBASE-15291 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.2, 0.98.16.1 >Reporter: Yong Zhang >Assignee: Ashish Singhi >Priority: Major > Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, > HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, > HBASE-15291.addendum, HBASE-15291.patch, HBASE-15291.v1.patch, > HBASE-15291.v2.patch, HBASE-15291.v2.patch, patch > > > FileSystem not closed in secure bulkLoad after bulkLoad finish, it will > cause memory used more and more if too many bulkLoad . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20259) Doc configs for in-memory-compaction and add detail to in-memory-compaction logging
[ https://issues.apache.org/jira/browse/HBASE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423944#comment-16423944 ] Peter Somogyi commented on HBASE-20259: --- [~stack], your push to branch-2.0 caused a merge conflict. [https://github.com/apache/hbase/blob/branch-2.0/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L343] > Doc configs for in-memory-compaction and add detail to in-memory-compaction > logging > --- > > Key: HBASE-20259 > URL: https://issues.apache.org/jira/browse/HBASE-20259 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20259.master.001.patch, > HBASE-20259.master.002.patch, HBASE-20259.master.003.patch > > > I set {{hbase.systemtables.compacting.memstore.type}} to NONE but it seems > like in-memory is still on. My table looks like this: > {code} > Table ycsb is ENABLED > ycsb > COLUMN FAMILIES DESCRIPTION > {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', > NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', > CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => > 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER = > > 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', > > CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', > > COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} > {code} > Looks like table doesn't have it on either (IN_MEMORY_COMPACTION doesn't show > in the above). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20193) Basic Replication Web UI - Regionserver
[ https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423870#comment-16423870 ] Jingyun Tian edited comment on HBASE-20193 at 4/3/18 11:24 AM: --- [~stack] [~Apache9] can you help check this patch out? The compiling warning is not introduced by my patch. was (Author: tianjingyun): [~stack] can you help checko this patch out? The compiling warning is not introduced by my patch. > Basic Replication Web UI - Regionserver > > > Key: HBASE-20193 > URL: https://issues.apache.org/jira/browse/HBASE-20193 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20193.master.001.patch, > HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, > HBASE-20193.master.004.patch > > > subtask of HBASE-15809. Implementation of replication UI on Regionserver web > page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20194) Basic Replication WebUI - Master
[ https://issues.apache.org/jira/browse/HBASE-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423872#comment-16423872 ] Jingyun Tian commented on HBASE-20194: -- [~Apache9] Can you check this patch? Add a UT for it. > Basic Replication WebUI - Master > > > Key: HBASE-20194 > URL: https://issues.apache.org/jira/browse/HBASE-20194 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20194.master.001.patch, > HBASE-20194.master.002.patch, HBASE-20194.master.003.patch, > HBASE-20194.master.004.patch, HBASE-20194.master.005.patch > > > subtask of HBASE-15809. Implementation of Replication WebUI on Master webpage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20193) Basic Replication Web UI - Regionserver
[ https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423870#comment-16423870 ] Jingyun Tian commented on HBASE-20193: -- [~stack] can you help checko this patch out? The compiling warning is not introduced by my patch. > Basic Replication Web UI - Regionserver > > > Key: HBASE-20193 > URL: https://issues.apache.org/jira/browse/HBASE-20193 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20193.master.001.patch, > HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, > HBASE-20193.master.004.patch > > > subtask of HBASE-15809. Implementation of replication UI on Regionserver web > page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20194) Basic Replication WebUI - Master
[ https://issues.apache.org/jira/browse/HBASE-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423857#comment-16423857 ] Hadoop QA commented on HBASE-20194: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 10s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 55s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 20m 20s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}123m 26s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}170m 52s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f | | JIRA Issue | HBASE-20194 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12917324/HBASE-20194.master.005.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 60588cd137e3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh | | git revision | master / 219625233c | | maven | version: Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) | | Default Java | 1.8.0_162 | | findbugs | v3.1.0-RC3 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/12272/testReport/ | | Max. process+thread count | 4233 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/12272/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Basic Replication WebUI
[jira] [Commented] (HBASE-20193) Basic Replication Web UI - Regionserver
[ https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423847#comment-16423847 ] Hadoop QA commented on HBASE-20193: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 33s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 3s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 45s{color} | {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188 total (was 188) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 4s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 20m 23s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}122m 25s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}169m 8s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f | | JIRA Issue | HBASE-20193 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12917323/HBASE-20193.master.004.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 00d77ca7074f 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 219625233c | | maven | version: Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) | | Default Java | 1.8.0_162 | | findbugs | v3.1.0-RC3 | | javac | https://builds.apache.org/job/PreCommit-HBASE-Build/12271/artifact/patchprocess/diff-compile-javac-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/12271/testReport/ | | Max. process+thread count | 4176 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output |
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423788#comment-16423788 ] ramkrishna.s.vasudevan commented on HBASE-20188: The previous results were not apple to apple because offheap bucket cache does an onheap copy for hbase-1.2. So tried out the same experiment for a longer duration with L1 block cache and the avg was hbase -1.2 vs hbase -2 was 32393 ops/sec vs 28898 ops/sec. So we have a 4k difference still. In Stack's case it is much higher. Will try to figure out if there are any obvious bottlenecks that can be resolved to make up for this difference. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20328) Fix local backup master start command in documentation
[ https://issues.apache.org/jira/browse/HBASE-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423738#comment-16423738 ] Yuki Tawara commented on HBASE-20328: - Thanks for your review, [~uagashe]! > Fix local backup master start command in documentation > -- > > Key: HBASE-20328 > URL: https://issues.apache.org/jira/browse/HBASE-20328 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Yuki Tawara >Assignee: Yuki Tawara >Priority: Minor > Attachments: HBASE-20328.master.001.patch > > > In "2.3. Pseudo-Distributed Local Install" section of documentation, a > command for starting backup masters lacks "start" argument. > {noformat} > $ ./bin/local-master-backup.sh 2 3 5 > {noformat} > should > {noformat} > $ ./bin/local-master-backup.sh start 2 3 5 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19849) MetaEntry in HBaseFsck has inconsistent equals and hashCode
[ https://issues.apache.org/jira/browse/HBASE-19849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-19849: -- Affects Version/s: (was: 2.0.0-beta-1) 1.4.3 > MetaEntry in HBaseFsck has inconsistent equals and hashCode > --- > > Key: HBASE-19849 > URL: https://issues.apache.org/jira/browse/HBASE-19849 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.3 >Reporter: Peter Somogyi >Priority: Major > > [~balazs.meszaros] noticed that MetaEntry uses different fields in equals and > hashCode. The used fields should be unified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20298) Doc change in read/write/total accounting metrics
[ https://issues.apache.org/jira/browse/HBASE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423718#comment-16423718 ] Yu Li commented on HBASE-20298: --- +1 on the doc changes, thanks for taking care of this sir. [~stack] >From checkstyle output, we need to remove unused imports (caused by change in >this patch) from {{HRegionServer}} > Doc change in read/write/total accounting metrics > - > > Key: HBASE-20298 > URL: https://issues.apache.org/jira/browse/HBASE-20298 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20298.master.001.patch, > HBASE-20298.master.002.patch > > > Doc the change wrought by the parent issue. Get it up into the refguide as > part of the difference between old hbases and hbase2. > The change confused me and took me a while to untangle. > The read count is for reads that return a non-empty result now. In old > hbase1, we'd increment the read-count even if an empty result. This makes > reads look bad in YCSB runs when compared to hbase1 (see how > totalRequestCount in hbase2 can be way above the sum of reads+writes; it is > because it increments even if the row is not found). > Let me get this into refguide otherwise poor old operators will be baffled. > The release note on the parent is great; it just needs to be in our guide. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20236) [locking] Write-time worst offenders
[ https://issues.apache.org/jira/browse/HBASE-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423678#comment-16423678 ] Hadoop QA commented on HBASE-20236: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 1s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 9s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 7s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 35s{color} | {color:red} hbase-server generated 6 new + 182 unchanged - 6 fixed = 188 total (was 188) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 19s{color} | {color:red} hbase-server: The patch generated 2 new + 282 unchanged - 0 fixed = 284 total (was 282) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 34s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 17m 55s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}159m 36s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}200m 50s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.regionserver.TestPerColumnFamilyFlush | | | hadoop.hbase.regionserver.TestDefaultMemStore | | | hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController | | | hadoop.hbase.regionserver.TestCompactingToCellFlatMapMemStore | | | hadoop.hbase.regionserver.TestCompactingMemStore | | | hadoop.hbase.regionserver.TestHStore | | | hadoop.hbase.regionserver.TestWalAndCompactingMemStoreFlush | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f | | JIRA Issue | HBASE-20236 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12917304/HBASE-20236.master.001.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux cdfd0933ee08 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 219625233c |
[jira] [Updated] (HBASE-19890) Canary usage should document hbase.canary.sink.class config
[ https://issues.apache.org/jira/browse/HBASE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-19890: -- Attachment: HBASE-19890.master.001.patch > Canary usage should document hbase.canary.sink.class config > --- > > Key: HBASE-19890 > URL: https://issues.apache.org/jira/browse/HBASE-19890 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-19890.master.001.patch > > > Canary#main uses config hbase.canary.sink.class to instantiate Sink class. > The Sink instance affects creation of Monitor. > In the refguide for Canary, hbase.canary.sink.class was not mentioned. > We should document this config. > Additionally, we need to document that using the default sink is not > compatible with table parameters as input so the user must change it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19890) Canary usage should document hbase.canary.sink.class config
[ https://issues.apache.org/jira/browse/HBASE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-19890: -- Status: Patch Available (was: Open) > Canary usage should document hbase.canary.sink.class config > --- > > Key: HBASE-19890 > URL: https://issues.apache.org/jira/browse/HBASE-19890 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-19890.master.001.patch > > > Canary#main uses config hbase.canary.sink.class to instantiate Sink class. > The Sink instance affects creation of Monitor. > In the refguide for Canary, hbase.canary.sink.class was not mentioned. > We should document this config. > Additionally, we need to document that using the default sink is not > compatible with table parameters as input so the user must change it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20259) Doc configs for in-memory-compaction and add detail to in-memory-compaction logging
[ https://issues.apache.org/jira/browse/HBASE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423645#comment-16423645 ] Anastasia Braginsky commented on HBASE-20259: - {quote}Pushed to branch-2.0+ {quote} Is there some new branch "branch-2.0+"? I mean, if I want to take the code which is the most recent state of HBase-2.0, should I use branch-2 or anything else? > Doc configs for in-memory-compaction and add detail to in-memory-compaction > logging > --- > > Key: HBASE-20259 > URL: https://issues.apache.org/jira/browse/HBASE-20259 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20259.master.001.patch, > HBASE-20259.master.002.patch, HBASE-20259.master.003.patch > > > I set {{hbase.systemtables.compacting.memstore.type}} to NONE but it seems > like in-memory is still on. My table looks like this: > {code} > Table ycsb is ENABLED > ycsb > COLUMN FAMILIES DESCRIPTION > {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', > NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', > CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => > 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER = > > 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', > > CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', > > COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} > {code} > Looks like table doesn't have it on either (IN_MEMORY_COMPACTION doesn't show > in the above). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20194) Basic Replication WebUI - Master
[ https://issues.apache.org/jira/browse/HBASE-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-20194: - Attachment: HBASE-20194.master.005.patch > Basic Replication WebUI - Master > > > Key: HBASE-20194 > URL: https://issues.apache.org/jira/browse/HBASE-20194 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20194.master.001.patch, > HBASE-20194.master.002.patch, HBASE-20194.master.003.patch, > HBASE-20194.master.004.patch, HBASE-20194.master.005.patch > > > subtask of HBASE-15809. Implementation of Replication WebUI on Master webpage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20193) Basic Replication Web UI - Regionserver
[ https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-20193: - Attachment: HBASE-20193.master.004.patch > Basic Replication Web UI - Regionserver > > > Key: HBASE-20193 > URL: https://issues.apache.org/jira/browse/HBASE-20193 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20193.master.001.patch, > HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, > HBASE-20193.master.004.patch > > > subtask of HBASE-15809. Implementation of replication UI on Regionserver web > page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423591#comment-16423591 ] Chance Li commented on HBASE-20312: --- Thanks. [~carp84],[~stack] It's my fault. I will fix it on next patch. now, to use CompactingMemsotre, We must set "hbase.regionserver.memstore.class"=org.apache.hadoop.hbase.regionserver.DefaultMemStore. That's terrible. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch, > ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423585#comment-16423585 ] ramkrishna.s.vasudevan commented on HBASE-20188: I just reran the experiment with loading around 15G of data and loading everything to cache and doing pure read workloads. I used the hbase-1.2 client only with 100 threads (all single node) hbase-1.2.6 vs hbase-2 (31099 ops/sec vs 30838 ops/sec). (Almost same). I have 15 regions in my table and there are no splits that happen. All 15G of data is served from bucket cache (offheap mode). The results are not so different even if serve from L1 block cache also. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423581#comment-16423581 ] ramkrishna.s.vasudevan commented on HBASE-20188: bq.None really. Tried with data cached thinking it i/o that was responsible for the difference, but while it brings us closer, we are still down from hbase1.2.7 (52221.77 ops/second vs 33839.57). This is really surprising. Even after data cache you see hbase-2 to be so much slower? All are defaults in both cases? What about bloom filters? What is the cache size here? And you sure all data is served only from cache? > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0 > > Attachments: CAM-CONFIG-V01.patch, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423551#comment-16423551 ] Yu Li commented on HBASE-20312: --- I guess you didn't turn off CompactingMemsotre (set "hbase.hregion.compacting.memstore.type" to NONE) in your testing boss, could you check and retry if not? Thanks [~stack] [~chancelq] We need to make it work when {{hbase.hregion.compacting.memstore.type}} is set to BASIC or some other not NONE type. In current patch as long as {{hbase.regionserver.memstore.class}} is not set we will only initiate CCSMapChunkPool and MSLAB chunk creator won't get initialized, which will cause the NPE problem as stack saw in his testing. A naive fix might be adding check on {{hbase.hregion.compacting.memstore.type}} in {{CCSMapMemStore#isEnabled}} and only returns true if it's "NONE" > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch, > ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20256) Document commands that do not work with 1.2 shell
[ https://issues.apache.org/jira/browse/HBASE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423541#comment-16423541 ] Peter Somogyi commented on HBASE-20256: --- [~busbey], are you good with patch 2? > Document commands that do not work with 1.2 shell > - > > Key: HBASE-20256 > URL: https://issues.apache.org/jira/browse/HBASE-20256 > Project: HBase > Issue Type: Sub-task > Components: documentation, shell >Affects Versions: 2.0.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Attachments: HBASE-20256.master.001.patch, > HBASE-20256.master.002.patch > > > Some commands do not work from 1.2 shell when running against 2.0 server. Add > a section to the reference guide mentioning the incompatibilities. > Some of these are collected in this document: > https://docs.google.com/document/d/1l2ad5G_GUk0WQ02xKeKO6mTbpdyZjU1NGuPPI42Wbf4/edit -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423528#comment-16423528 ] Chance Li commented on HBASE-20312: --- stack {quote}regionserver.HStore: Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore ;{quote} This log shows the Meta is DefaultMemstore. Which patch did you apply? is it HBASE-20312-master.v3.patch? > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20312-1.3.2.patch, HBASE-20312-master.v1.patch, > HBASE-20312-master.v2.patch, HBASE-20312-master.v3.patch, > ccsmap-branch-1.1.patch, jira1.png, jira2.png, jira3.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20329) Add note for operators to refguide on AsyncFSWAL
[ https://issues.apache.org/jira/browse/HBASE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423522#comment-16423522 ] Peter Somogyi commented on HBASE-20329: --- Good summary [~stack]! nit: chained pipeline as the *default* *default* client does > Add note for operators to refguide on AsyncFSWAL > > > Key: HBASE-20329 > URL: https://issues.apache.org/jira/browse/HBASE-20329 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20329.master.001.patch > > > Need a few notes in refguide on this new facility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20329) Add note for operators to refguide on AsyncFSWAL
[ https://issues.apache.org/jira/browse/HBASE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423521#comment-16423521 ] Yu Li commented on HBASE-20329: --- +1, the doc looks great to me, thanks for writing this up boss. Minor: WALs edits are written concurrentl -> WAL's edits are written concurrently > Add note for operators to refguide on AsyncFSWAL > > > Key: HBASE-20329 > URL: https://issues.apache.org/jira/browse/HBASE-20329 > Project: HBase > Issue Type: Sub-task > Components: documentation, wal >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20329.master.001.patch > > > Need a few notes in refguide on this new facility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)