[jira] [Commented] (HBASE-20095) Redesign single instance pool in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413441#comment-16413441
]
Reid Chan commented on HBASE-20095:
---
{code}
javac -1, NettyRpcServer.java:[148,45] [FutureReturnValueIgnored] Return value
of methods returning Future must be checked. Ignoring returned Futures
suppresses exceptions thrown from the code that completes the Future.
{code}
{quote}
NettyRpcServer.java is not part of this patch.
{quote}
> Redesign single instance pool in CleanerChore
> -
>
> Key: HBASE-20095
> URL: https://issues.apache.org/jira/browse/HBASE-20095
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Attachments: HBASE-20095.master.001.patch,
> HBASE-20095.master.002.patch, HBASE-20095.master.003.patch,
> HBASE-20095.master.004.patch, HBASE-20095.master.005.patch,
> HBASE-20095.master.006.patch, HBASE-20095.master.007.patch,
> HBASE-20095.master.008.patch, HBASE-20095.master.009.patch,
> HBASE-20095.master.010.patch, HBASE-20095.master.011.patch,
> HBASE-20095.master.012.patch, HBASE-20095.master.013.patch,
> HBASE-20095.master.014.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20138) Find a way to deal with the conflicts when updating replication position
[ https://issues.apache.org/jira/browse/HBASE-20138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-20138: - Attachment: HBASE-20138.v5.patch > Find a way to deal with the conflicts when updating replication position > > > Key: HBASE-20138 > URL: https://issues.apache.org/jira/browse/HBASE-20138 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20138.v1.patch, HBASE-20138.v2.patch, > HBASE-20138.v3.patch, HBASE-20138.v4.patch, HBASE-20138.v5.patch > > > For now if a table is not created with SERIAL_REPLICATION_SCOPE and later > converted to SERIAL_REPLICATION_SCOPE , then we may have multiple replication > sources which replicate the different ranges for the same region and update > the queue storage concurrently. This will cause problem if the lower range > finish at last since the replication position will be wrong... > Need to find a way to deal with this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20270) Turn off command help that follows all errors in shell
[
https://issues.apache.org/jira/browse/HBASE-20270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413436#comment-16413436
]
Sakthi commented on HBASE-20270:
[~mdrob] , thanks for your comments. Working on them.
> Turn off command help that follows all errors in shell
> --
>
> Key: HBASE-20270
> URL: https://issues.apache.org/jira/browse/HBASE-20270
> Project: HBase
> Issue Type: Task
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sakthi
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: hbase-20270.master.001.patch
>
>
> Right now if a shell command gives an error, any error, it then echos the
> command help. It makes it harder to see the actual error text and is annoying.
> example:
> {code}
>
>
>
> hbase(main):007:0> create 'test:a_table', 'family', { NUMREGIONS => 20,
> SPLITALGO => 'HexStringSplit'}
> ERROR: Unknown namespace test!
> Creates a table. Pass a table name, and a set of column family
> specifications (at least one), and, optionally, table configuration.
> Column specification can be a simple string (name), or a dictionary
> (dictionaries are described below in main help output), necessarily
> including NAME attribute.
> Examples:
> Create a table with namespace=ns1 and table qualifier=t1
> hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5}
> Create a table with namespace=default and table qualifier=t1
> hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
> hbase> # The above in shorthand would be the following:
> hbase> create 't1', 'f1', 'f2', 'f3'
> hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000,
> BLOCKCACHE => true}
> hbase> create 't1', {NAME => 'f1', CONFIGURATION =>
> {'hbase.hstore.blockingStoreFiles' => '10'}}
> hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD =>
> 100, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
> Table configuration options can be put at the end.
> Examples:
> hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40']
> hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40']
> hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe'
> hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' =>
> 'myvalue' }
> hbase> # Optionally pre-split the table into NUMREGIONS, using
> hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname)
> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'}
> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit',
> REGION_REPLICATION => 2, CONFIGURATION =>
> {'hbase.hregion.scan.loadColumnFamiliesOnDemand' => 'true'}}
> hbase> create 't1', {NAME => 'f1', DFS_REPLICATION => 1}
> You can also keep around a reference to the created table:
> hbase> t1 = create 't1', 'f1'
> Which gives you a reference to the table named 't1', on which you can then
> call methods.
> Took 0.0221 seconds
>
>
> hbase(main):008:0> create_namespace 'test'
> Took 0.2554 seconds
>
>
> hbase(main):009:0> create 'test:a_table', 'family', { NUMREGIONS => 20,
> SPLITALGO => 'HexStringSplit'}
> Created table test:a_table
> Took 1.2264 seconds
> {code}
> I was trying to make a table in the test namespace before making the
> namespace. Much faster to recognize and move on when the error text isn't
> followed by 80x the text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20138) Find a way to deal with the conflicts when updating replication position
[ https://issues.apache.org/jira/browse/HBASE-20138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413435#comment-16413435 ] Zheng Hu commented on HBASE-20138: -- Patch.v5: Fix the checkstyle & failed UT. > Find a way to deal with the conflicts when updating replication position > > > Key: HBASE-20138 > URL: https://issues.apache.org/jira/browse/HBASE-20138 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20138.v1.patch, HBASE-20138.v2.patch, > HBASE-20138.v3.patch, HBASE-20138.v4.patch, HBASE-20138.v5.patch > > > For now if a table is not created with SERIAL_REPLICATION_SCOPE and later > converted to SERIAL_REPLICATION_SCOPE , then we may have multiple replication > sources which replicate the different ranges for the same region and update > the queue storage concurrently. This will cause problem if the lower range > finish at last since the replication position will be wrong... > Need to find a way to deal with this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20095) Redesign single instance pool in CleanerChore
[
https://issues.apache.org/jira/browse/HBASE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413426#comment-16413426
]
Hadoop QA commented on HBASE-20095:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
22s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 47s{color}
| {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
0s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 20s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}105m
45s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20095 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12916124/HBASE-20095.master.014.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux fff45a0f0bcc 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 3f7222df36 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/12130/artifact/patchprocess/diff-compile-javac-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12130/testReport/ |
| Max. process+thread count | 4439 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https
[jira] [Commented] (HBASE-20003) WALLess HBase on Persistent Memory
[ https://issues.apache.org/jira/browse/HBASE-20003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413417#comment-16413417 ] Duo Zhang commented on HBASE-20003: --- Seems like a HydraBase with 2PC instead of raft... The problem is very simple, why do we still need HDFS if we replicate data to each node by ourselves? And if no HDFS, then is this still 'HBase'? Maybe it will be much easier to start a new project, which is based on a 'WAL-Less' rocksdb... > WALLess HBase on Persistent Memory > -- > > Key: HBASE-20003 > URL: https://issues.apache.org/jira/browse/HBASE-20003 > Project: HBase > Issue Type: New Feature >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > > This JIRA aims to make use of persistent memory (pmem) technologies in HBase. > One such usage is to make the Memstore to reside on pmem. Making a persistent > memstore would remove the need for WAL and paves way for a WALLess HBase. > The existing region replica feature could be used here and ensure the data > written to memstores are synchronously replicated to the replicas and ensure > strong consistency of the data. (pipeline model) > Advantages : > - Data Availability : Since the data across replicas are consistent > (synchronously written) our data is always 100 % available. > - Lower MTTR : It becomes easier/faster to switch over to the replicas on a > primary region failure as there is no WAL replay involved. Building the > memstore map data also is much faster than reading the WAL and replaying the > WAL. > - Possibility of bigger memstores : These pmems are designed to have more > memory than DRAMs so it would also enable us to have bigger sized memstores > which leads to lesser flushes/compaction IO. > - Removes the dependency of HDFS on the write path > Initial PoC has been designed and developed. Testing is underway and we would > publish the PoC results along with the design doc sooner. The PoC doc will > talk about the design decisions, the libraries considered to work with these > pmem devices, pros and cons of those libraries and the performance results. > Note : Next gen memory technologies using 3DXPoint gives persistent memory > feature. Such memory DIMMs are soon to appear in the market. The PoC is done > around Intel's ApachePass (AEP) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20003) WALLess HBase on Persistent Memory
[ https://issues.apache.org/jira/browse/HBASE-20003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413378#comment-16413378 ] Chance Li commented on HBASE-20003: --- [~anoop.hbase] Sir, Excellent doc, I have a little question. To consider a much more special case where the Tertiary success but the Secondary failure(not down), and then the primary goes down without changing the META. This scenario looks like that the RS hosting the Secondary replica reaches the Global Memstore size upper limit(maybe because other table's regions' heavy workload). It's not similar to the case you mentioned. The questions are: 1) Which replica is eligible to become a new primary? (the last mutation(s)(batched) are only in Tertiary but not in Secondary). 2) Losing the state of replica, these two replica will keep the inconsistent state for a while until new replica goes up (the new primary will start flushing). That means the client sometimes can get data, sometimes can't. It's a very bad thing. How can we avoid it? Basically, the architecture looks like moving the pipline from the WAL to region replication, and it will improve MTTR , so it's acceptable. But the different is there will have more thing that will impact the pipeline, such as Global Memstore size upper limit, then the replica's pipeline may be more volatile than WAL's, and this may lead to uncontrolled replica failed, also lead to uncontrolled flush. Finally, Do Hbase consider the geographic locality? > WALLess HBase on Persistent Memory > -- > > Key: HBASE-20003 > URL: https://issues.apache.org/jira/browse/HBASE-20003 > Project: HBase > Issue Type: New Feature >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > > This JIRA aims to make use of persistent memory (pmem) technologies in HBase. > One such usage is to make the Memstore to reside on pmem. Making a persistent > memstore would remove the need for WAL and paves way for a WALLess HBase. > The existing region replica feature could be used here and ensure the data > written to memstores are synchronously replicated to the replicas and ensure > strong consistency of the data. (pipeline model) > Advantages : > - Data Availability : Since the data across replicas are consistent > (synchronously written) our data is always 100 % available. > - Lower MTTR : It becomes easier/faster to switch over to the replicas on a > primary region failure as there is no WAL replay involved. Building the > memstore map data also is much faster than reading the WAL and replaying the > WAL. > - Possibility of bigger memstores : These pmems are designed to have more > memory than DRAMs so it would also enable us to have bigger sized memstores > which leads to lesser flushes/compaction IO. > - Removes the dependency of HDFS on the write path > Initial PoC has been designed and developed. Testing is underway and we would > publish the PoC results along with the design doc sooner. The PoC doc will > talk about the design decisions, the libraries considered to work with these > pmem devices, pros and cons of those libraries and the performance results. > Note : Next gen memory technologies using 3DXPoint gives persistent memory > feature. Such memory DIMMs are soon to appear in the market. The PoC is done > around Intel's ApachePass (AEP) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20138) Find a way to deal with the conflicts when updating replication position
[
https://issues.apache.org/jira/browse/HBASE-20138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413374#comment-16413374
]
Hadoop QA commented on HBASE-20138:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 14s{color}
| {color:red} hbase-replication generated 1 new + 1 unchanged - 0 fixed = 2
total (was 1) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
8s{color} | {color:red} hbase-replication: The patch generated 3 new + 2
unchanged - 0 fixed = 5 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
9s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
9s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m
11s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m
23s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hbase-zookeeper in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 34s{color}
| {color:red} hbase-replication in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestZKReplicationQueueStorage |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12916128/HBASE-20138.v4.patch |
| Optional Tests | asflicense javac javadoc
[jira] [Commented] (HBASE-17919) HBase 2.x over hadoop 3.x umbrella
[ https://issues.apache.org/jira/browse/HBASE-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413373#comment-16413373 ] Sean Busbey commented on HBASE-17919: - Are we planning to mark HBase 2.0 on Hadoop 3.0.0-GA as S in the Hadoop support matrix? > HBase 2.x over hadoop 3.x umbrella > --- > > Key: HBASE-17919 > URL: https://issues.apache.org/jira/browse/HBASE-17919 > Project: HBase > Issue Type: Umbrella > Components: hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Priority: Critical > Fix For: 2.0.0 > > > We should try to get hbase 2.x branch working against the recently release > hadoop 3.0.0 alphas. These days 3.0.0-alpha2 is the latest. > HBASE-16733 and HBASE-17593 got the compile level checks in but we should > progress to getting unit tests to pass and a build against hadoop3 up. > This umbrella issue will capture issues around this project. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[
https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413365#comment-16413365
]
Hadoop QA commented on HBASE-20243:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
56s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 13 new + 439 unchanged - 5 fixed =
452 total (was 444) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m
5s{color} | {color:red} The patch generated 5 new + 737 unchanged - 0 fixed =
742 total (was 737) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
57s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 21s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 44s{color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
32s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestInterfaceAlign |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20243 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12916123/HBASE
[jira] [Updated] (HBASE-20138) Find a way to deal with the conflicts when updating replication position
[ https://issues.apache.org/jira/browse/HBASE-20138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-20138: - Attachment: HBASE-20138.v4.patch > Find a way to deal with the conflicts when updating replication position > > > Key: HBASE-20138 > URL: https://issues.apache.org/jira/browse/HBASE-20138 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20138.v1.patch, HBASE-20138.v2.patch, > HBASE-20138.v3.patch, HBASE-20138.v4.patch > > > For now if a table is not created with SERIAL_REPLICATION_SCOPE and later > converted to SERIAL_REPLICATION_SCOPE , then we may have multiple replication > sources which replicate the different ranges for the same region and update > the queue storage concurrently. This will cause problem if the lower range > finish at last since the replication position will be wrong... > Need to find a way to deal with this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20159) Support using separate ZK quorums for client
[
https://issues.apache.org/jira/browse/HBASE-20159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413362#comment-16413362
]
Yu Li commented on HBASE-20159:
---
Patch v2 addresses review comments and checkstyle/findbugs issues. Let's see
what HadoopQA says.
> Support using separate ZK quorums for client
>
>
> Key: HBASE-20159
> URL: https://issues.apache.org/jira/browse/HBASE-20159
> Project: HBase
> Issue Type: New Feature
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Attachments: HBASE-20159.patch, HBASE-20159.v2.patch
>
>
> Currently we are using the same zookeeper quorums for client and server,
> which makes us under risk that if some client connection boost exhausted
> zookeeper, RegionServer might abort due to zookeeper session loss. Actually
> we have suffered from this many times in production.
> Here we propose to allow client to use different ZK quorums, through below
> settings:
> {noformat}
> hbase.client.zookeeper.quorum
> hbase.client.zookeeper.property.clientPort
> hbase.client.zookeeper.observer.mode
> {noformat}
> The first two are for specifying client zookeeper properties, and the third
> one indicating whether the client ZK nodes are in observer mode. If the
> client ZK are not observer nodes, HMaster will take responsibility to
> synchronize necessary meta information (such as meta location and master
> address, etc.) from server to client ZK
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20159) Support using separate ZK quorums for client
[
https://issues.apache.org/jira/browse/HBASE-20159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-20159:
--
Attachment: HBASE-20159.v2.patch
> Support using separate ZK quorums for client
>
>
> Key: HBASE-20159
> URL: https://issues.apache.org/jira/browse/HBASE-20159
> Project: HBase
> Issue Type: New Feature
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Attachments: HBASE-20159.patch, HBASE-20159.v2.patch
>
>
> Currently we are using the same zookeeper quorums for client and server,
> which makes us under risk that if some client connection boost exhausted
> zookeeper, RegionServer might abort due to zookeeper session loss. Actually
> we have suffered from this many times in production.
> Here we propose to allow client to use different ZK quorums, through below
> settings:
> {noformat}
> hbase.client.zookeeper.quorum
> hbase.client.zookeeper.property.clientPort
> hbase.client.zookeeper.observer.mode
> {noformat}
> The first two are for specifying client zookeeper properties, and the third
> one indicating whether the client ZK nodes are in observer mode. If the
> client ZK are not observer nodes, HMaster will take responsibility to
> synchronize necessary meta information (such as meta location and master
> address, etc.) from server to client ZK
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20095) Redesign single instance pool in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-20095: -- Attachment: HBASE-20095.master.014.patch > Redesign single instance pool in CleanerChore > - > > Key: HBASE-20095 > URL: https://issues.apache.org/jira/browse/HBASE-20095 > Project: HBase > Issue Type: Improvement >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Critical > Attachments: HBASE-20095.master.001.patch, > HBASE-20095.master.002.patch, HBASE-20095.master.003.patch, > HBASE-20095.master.004.patch, HBASE-20095.master.005.patch, > HBASE-20095.master.006.patch, HBASE-20095.master.007.patch, > HBASE-20095.master.008.patch, HBASE-20095.master.009.patch, > HBASE-20095.master.010.patch, HBASE-20095.master.011.patch, > HBASE-20095.master.012.patch, HBASE-20095.master.013.patch, > HBASE-20095.master.014.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20095) Redesign single instance pool in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413342#comment-16413342 ] Reid Chan commented on HBASE-20095: --- Rebase master branch, and trigger QA again. > Redesign single instance pool in CleanerChore > - > > Key: HBASE-20095 > URL: https://issues.apache.org/jira/browse/HBASE-20095 > Project: HBase > Issue Type: Improvement >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Critical > Attachments: HBASE-20095.master.001.patch, > HBASE-20095.master.002.patch, HBASE-20095.master.003.patch, > HBASE-20095.master.004.patch, HBASE-20095.master.005.patch, > HBASE-20095.master.006.patch, HBASE-20095.master.007.patch, > HBASE-20095.master.008.patch, HBASE-20095.master.009.patch, > HBASE-20095.master.010.patch, HBASE-20095.master.011.patch, > HBASE-20095.master.012.patch, HBASE-20095.master.013.patch, > HBASE-20095.master.014.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20095) Redesign single instance pool in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-20095: -- Attachment: (was: HBASE-20095.master.014.patch) > Redesign single instance pool in CleanerChore > - > > Key: HBASE-20095 > URL: https://issues.apache.org/jira/browse/HBASE-20095 > Project: HBase > Issue Type: Improvement >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Critical > Attachments: HBASE-20095.master.001.patch, > HBASE-20095.master.002.patch, HBASE-20095.master.003.patch, > HBASE-20095.master.004.patch, HBASE-20095.master.005.patch, > HBASE-20095.master.006.patch, HBASE-20095.master.007.patch, > HBASE-20095.master.008.patch, HBASE-20095.master.009.patch, > HBASE-20095.master.010.patch, HBASE-20095.master.011.patch, > HBASE-20095.master.012.patch, HBASE-20095.master.013.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[ https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-20243: -- Status: Patch Available (was: Open) Attach 003 patch as [~yuzhih...@gmail.com] and [~appy] suggestions.Thanks > [Shell] Add shell command to create a new table by cloning the existent table > - > > Key: HBASE-20243 > URL: https://issues.apache.org/jira/browse/HBASE-20243 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20243.master.001.patch, > HBASE-20243.master.002.patch, HBASE-20243.master.003.patch > > > In the production environment, we need to create a new table every day. The > schema and the split keys of the table are the same as that of yesterday's > table, only the name of the table is different. For example, > x_20180321,x_20180322 etc.But now there is no convenient command to > do this. So we may need such a command(clone_table) to create a new table by > cloning the existent table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[ https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-20243: -- Attachment: HBASE-20243.master.003.patch > [Shell] Add shell command to create a new table by cloning the existent table > - > > Key: HBASE-20243 > URL: https://issues.apache.org/jira/browse/HBASE-20243 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20243.master.001.patch, > HBASE-20243.master.002.patch, HBASE-20243.master.003.patch > > > In the production environment, we need to create a new table every day. The > schema and the split keys of the table are the same as that of yesterday's > table, only the name of the table is different. For example, > x_20180321,x_20180322 etc.But now there is no convenient command to > do this. So we may need such a command(clone_table) to create a new table by > cloning the existent table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20243) [Shell] Add shell command to create a new table by cloning the existent table
[ https://issues.apache.org/jira/browse/HBASE-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-20243: -- Attachment: HBASE-20243.master.002.patch > [Shell] Add shell command to create a new table by cloning the existent table > - > > Key: HBASE-20243 > URL: https://issues.apache.org/jira/browse/HBASE-20243 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20243.master.001.patch, > HBASE-20243.master.002.patch > > > In the production environment, we need to create a new table every day. The > schema and the split keys of the table are the same as that of yesterday's > table, only the name of the table is different. For example, > x_20180321,x_20180322 etc.But now there is no convenient command to > do this. So we may need such a command(clone_table) to create a new table by > cloning the existent table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20285) Delete all last pushed sequence ids when removing a peer
Duo Zhang created HBASE-20285: - Summary: Delete all last pushed sequence ids when removing a peer Key: HBASE-20285 URL: https://issues.apache.org/jira/browse/HBASE-20285 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20191) Cleanup replication barrier for delete table in the chore
[ https://issues.apache.org/jira/browse/HBASE-20191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413282#comment-16413282 ] Duo Zhang commented on HBASE-20191: --- Actually I've already written some code for removing the replication barrier and last pushed sequence ids when deleting table. But the problem is that, we need to insert the related code to multiple places, for example, besides deleting table, truncating table, or remove the global scope for a table, or removing a table from a peer, etc. And it is not still not easy to make it clean since we may still hold the barriers and last pushed sequence id for split/merged regions... So, I think the correct way is to implement the logic in the cleaner chore so that we do not need to insert code everywhere. > Cleanup replication barrier for delete table in the chore > - > > Key: HBASE-20191 > URL: https://issues.apache.org/jira/browse/HBASE-20191 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20191) Cleanup replication barrier for delete table in the chore
[ https://issues.apache.org/jira/browse/HBASE-20191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20191: -- Summary: Cleanup replication barrier for delete table in the chore (was: Deal with the cleanup of replication barrier and last pushed sequence id for deleted table) > Cleanup replication barrier for delete table in the chore > - > > Key: HBASE-20191 > URL: https://issues.apache.org/jira/browse/HBASE-20191 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413186#comment-16413186 ] stack commented on HBASE-20188: --- [~anastas] Sorry. Should have been better w/ how to decode the graph. Look at the 'YCSB_IN_MEMORY_COMPACTION%3DNONE.ops.png' graph. The YCSB run is made up of 4 back-to-back runs. There is the load phase, then a 50/50 phase, then the 95% reads phase, and finally the 80% writes phase. Each phase lasts 20 minutes. If you look at the graph, there are four 'runs' each of 4 phases. The first run is an hbase 1.2.7 run. The second run is 2.0.0. The third run is me thinking I had disabled in-memory compaction by setting hbase-site.xml to NONE and fourth is me setting the property on the table so it was off. bq. ...if you say that turning in-memory-compaction off doesn't improve the reads we should look for the source for reads degradation anywhere else. Yes. On it. [~eshcar] bq. Can you give the same table with the none numbers? I will. I'll be back w/ the numbers. I'll post 50th and 99th too. bq. I ran them with CAM setting and no mslab. Pass me the settings you would have me try and I'll try them here to see if a difference. Sorry for stating the obvious but we run w/ MSLAB to avoid OOMEs because of heap fragmentation brought on by CSLM. If we were to turn it off, we'd have to do some legwork to ensure that we don't bring back the old OOMEs. Thanks. bq. This doesn't make much sense. If we flush "only" after aggregating 10% of the data in the active segment we can use a pipeline of length 1, this would be equivalent to a longer pipeline (of length 4) with smaller segments (consisting of 2% of the data). Ugh. sorry. I don't get it. I'm confused. Setting pipeline length to 1 but the pipeline would be longer Sorry for being thick. Tell me what to try. Would be good to get good defaults in now rather than later. Thanks. I looked at the blogs, old talks and at old issue Release Notes. Is there a paragraph write-up somewhere of what finally made it into hbase2 and the configurations an operator can twiddle? I'd like to surface something up into the refguide; e.g. default if MSLAB is... if w/o MSLAB its CAM, etc. I can find beautiful pictures and definitions but a summary by one of you two experts would help -- even if it was just pointers that'd be enough. Thanks. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20234) Expose in-memory compaction metrics
[
https://issues.apache.org/jira/browse/HBASE-20234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413180#comment-16413180
]
stack commented on HBASE-20234:
---
Thanks [~anastas] Simple incrementing counters work. Point at the location of
what needs exposing and I can do it. Per CompactingMemStore might be too many
counts...We do per region counts at the moment. The per-Store counts don't make
it up to Region-level?
> Expose in-memory compaction metrics
> ---
>
> Key: HBASE-20234
> URL: https://issues.apache.org/jira/browse/HBASE-20234
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
>
> Hard to glean insight from how well in-memory compaction is doing currently.
> It dumps stats into the logs but better if they were available to a
> dashboard. This issue is about exposing a couple of helpful counts. There are
> already by-region metrics. We can add a few for in-memory compaction (Help me
> out [~anastas]... what counts would be best to expose).
> Flush related metrics include
> {code}
> Namespace_default_table_tsdb-tree_region_cfbf23e7330a1a2bbde031f9583d3415_metric_flushesQueuedCount:
> {
> description: "Number flushes requested/queued for this region",
> value: 0
> {
> description: "The number of cells flushed to disk",
> value: 0
> },
> {
> description: "The total amount of data flushed to disk, in bytes",
> value: 0
> },
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17819) Reduce the heap overhead for BucketCache
[
https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413098#comment-16413098
]
Hudson commented on HBASE-17819:
Results for branch branch-2.0
[build #87 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/87/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/87//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/87//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/87//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Reduce the heap overhead for BucketCache
>
>
> Key: HBASE-17819
> URL: https://issues.apache.org/jira/browse/HBASE-17819
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch,
> HBASE-17819_V3.patch, HBASE-17819_new_V1.patch, HBASE-17819_new_V1.patch,
> HBASE-17819_new_V2.patch
>
>
> We keep Bucket entry map in BucketCache. Below is the math for heapSize for
> the key , value into this map.
> BlockCacheKey
> ---
> String hfileName - Ref - 4
> long offset - 8
> BlockType blockType - Ref - 4
> boolean isPrimaryReplicaBlock - 1
> Total = align(12 (Object) + 17 )= 32
> BucketEntry
>
> int offsetBase - 4
> int length - 4
> byte offset1 - 1
> byte deserialiserIndex - 1
> long accessCounter - 8
> BlockPriority priority - Ref - 4
> volatile boolean markedForEvict - 1
> AtomicInteger refCount - 16 + 4
> long cachedTime - 8
> Total = align(12 (Object) + 51) = 64
> ConcurrentHashMap Map.Entry - 40
> blocksByHFile ConcurrentSkipListSet Entry - 40
> Total = 32 + 64 + 80 = 176
> For 10 million blocks we will end up having 1.6GB of heap size.
> This jira aims to reduce this as much as possible
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17819) Reduce the heap overhead for BucketCache
[
https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413079#comment-16413079
]
Hudson commented on HBASE-17819:
Results for branch branch-2
[build #528 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/528/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/528//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/528//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/528//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Reduce the heap overhead for BucketCache
>
>
> Key: HBASE-17819
> URL: https://issues.apache.org/jira/browse/HBASE-17819
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch,
> HBASE-17819_V3.patch, HBASE-17819_new_V1.patch, HBASE-17819_new_V1.patch,
> HBASE-17819_new_V2.patch
>
>
> We keep Bucket entry map in BucketCache. Below is the math for heapSize for
> the key , value into this map.
> BlockCacheKey
> ---
> String hfileName - Ref - 4
> long offset - 8
> BlockType blockType - Ref - 4
> boolean isPrimaryReplicaBlock - 1
> Total = align(12 (Object) + 17 )= 32
> BucketEntry
>
> int offsetBase - 4
> int length - 4
> byte offset1 - 1
> byte deserialiserIndex - 1
> long accessCounter - 8
> BlockPriority priority - Ref - 4
> volatile boolean markedForEvict - 1
> AtomicInteger refCount - 16 + 4
> long cachedTime - 8
> Total = align(12 (Object) + 51) = 64
> ConcurrentHashMap Map.Entry - 40
> blocksByHFile ConcurrentSkipListSet Entry - 40
> Total = 32 + 64 + 80 = 176
> For 10 million blocks we will end up having 1.6GB of heap size.
> This jira aims to reduce this as much as possible
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20272) TestAsyncTable#testCheckAndMutateWithTimeRange fails due to TableExistsException
[
https://issues.apache.org/jira/browse/HBASE-20272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413036#comment-16413036
]
Hudson commented on HBASE-20272:
Results for branch master
[build #272 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/272/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> TestAsyncTable#testCheckAndMutateWithTimeRange fails due to
> TableExistsException
>
>
> Key: HBASE-20272
> URL: https://issues.apache.org/jira/browse/HBASE-20272
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20272.v1.txt, 20272.v2.txt
>
>
> The following test failure is reproducible:
> {code}
> org.apache.hadoop.hbase.TableExistsException: testCheckAndMutateWithTimeRange
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.prepareCreate(CreateTableProcedure.java:233)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:87)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1453)
> {code}
> The cause was that TestAsyncTable is parameterized while the
> testCheckAndMutateWithTimeRange uses the same table name without dropping the
> table after the first invocation finishes.
> This leads to second invocation failing with TableExistsException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20271) ReplicationSourceWALReader.switched should use the file name instead of the path object directly
[
https://issues.apache.org/jira/browse/HBASE-20271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413035#comment-16413035
]
Hudson commented on HBASE-20271:
Results for branch master
[build #272 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/272/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> ReplicationSourceWALReader.switched should use the file name instead of the
> path object directly
>
>
> Key: HBASE-20271
> URL: https://issues.apache.org/jira/browse/HBASE-20271
> Project: HBase
> Issue Type: Sub-task
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20271.patch
>
>
> {noformat}
> 2018-03-24 08:29:29,965 ERROR
> [RS_REFRESH_PEER-regionserver/ubuntu:0-0.replicationSource,2.replicationSource.shipperubuntu%2C35197%2C1521851267085,2]
> helpers.MarkerIgnoringBase(159): * ABORTING region server
> ubuntu,35197,1521851267085: Failed to operate on replication queue *
> org.apache.hadoop.hbase.replication.ReplicationException: Failed to set log
> position (serverName=ubuntu,35197,1521851267085, queueId=2,
> fileName=ubuntu%2C35197%2C1521851267085.1521851344947, position=2533)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.setWALPosition(ZKReplicationQueueStorage.java:237)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.lambda$9(ReplicationSourceManager.java:488)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.abortWhenFail(ReplicationSourceManager.java:455)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:488)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:232)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:134)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:104)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1006)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:663)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1670)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.setWALPosition(ZKReplicationQueueStorage.java:235)
> ... 6 more
> 2018-03-24 08:29:30,025 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37509]
> master.MasterRpcServices(508): Region server ubuntu,35197,1521851267085
> reported a fatal error:
> * ABORTING region server ubuntu,35197,1521851267085: Failed to operate on
> replication queue *
> Cause:
> org.apache.hadoop.hbase.replication.ReplicationException: Failed to set log
> position (serverName=ubuntu,35197,1521851267085, queueId=2,
> fileName=ubuntu%2C35197%2C1521851267085.1521851344947, position=2533)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.setWALPosition(ZKReplicationQueueStorage.java:237)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.lambda$9(ReplicationSourceManager.java:488)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.abortWhenFail(ReplicationSourceManager.java:455)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:488)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(Repli
[jira] [Commented] (HBASE-20227) Add UT for ReplicationUtils.contains method
[
https://issues.apache.org/jira/browse/HBASE-20227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413037#comment-16413037
]
Hudson commented on HBASE-20227:
Results for branch master
[build #272 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/272/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Add UT for ReplicationUtils.contains method
> ---
>
> Key: HBASE-20227
> URL: https://issues.apache.org/jira/browse/HBASE-20227
> Project: HBase
> Issue Type: Sub-task
> Components: Replication, test
>Reporter: Duo Zhang
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20227-v8.patch, HBASE-20227.master.001.patch,
> HBASE-20227.master.002.patch, HBASE-20227.master.003.patch,
> HBASE-20227.master.005.patch, HBASE-20227.master.006.patch,
> HBASE-20227.master.007.patch
>
>
> The method is extracted from NamespaceTableCfWALEntryFilter. But
> NamespaceTableCfWALEntryFilter is in the hbase-server module so the UTs which
> tests the function of this method are also in the hbase-server module. If
> later we modify ReplicationUtils.contains without touching any code in
> hbase-server, then we will not run the UTs for this method and may introduce
> bugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19158) Ref guide needs upgrade update
[
https://issues.apache.org/jira/browse/HBASE-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413038#comment-16413038
]
Hudson commented on HBASE-19158:
Results for branch master
[build #272 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/272/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Ref guide needs upgrade update
> --
>
> Key: HBASE-19158
> URL: https://issues.apache.org/jira/browse/HBASE-19158
> Project: HBase
> Issue Type: Task
> Components: documentation
>Affects Versions: 2.0.0-beta-1
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-19158.0.patch, HBASE-19158.1.patch,
> HBASE-19158.2.patch, HBASE-19158.3.patch
>
>
> Our ref guide has lots of references in the upgrade section to obsolete
> versions and no references to the 2.0.0-* releases. We should correct this
> for beta-1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20264) Update Java prerequisite section with LTS rec and status of current GA JDKs
[
https://issues.apache.org/jira/browse/HBASE-20264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413039#comment-16413039
]
Hudson commented on HBASE-20264:
Results for branch master
[build #272 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/272/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/272//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Update Java prerequisite section with LTS rec and status of current GA JDKs
> ---
>
> Key: HBASE-20264
> URL: https://issues.apache.org/jira/browse/HBASE-20264
> Project: HBase
> Issue Type: Task
> Components: documentation
>Affects Versions: 1.5.0, 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-20264.0.patch, HBASE-20264.1.patch
>
>
> Per the thread [\[DISCUSS\] strategy on Java
> versions|https://lists.apache.org/thread.html/b7417aa8359e1beaa7caf47e8fa524296c499974e7467a541c6df415@%3Cdev.hbase.apache.org%3E]
> * Add Java 9 and Java 10 to the support matrix as NT
> * Add a NOTE to Java prereqs about "use a LTS version"
> For now, leave out talk about planning for timelines on LTS additions or
> dropping older JDK support. Once we get over the initial hurdle of prepping
> for Java 11 we'll hopefully have enough info to know how realistic the things
> talked about in the thread are and we can include a writeup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20234) Expose in-memory compaction metrics
[
https://issues.apache.org/jira/browse/HBASE-20234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413001#comment-16413001
]
Anastasia Braginsky commented on HBASE-20234:
-
As I see the current metrics in the region are all related to on-disk flushes.I
am not sure how those metrics collection is working. Is it the counters that
are summarized all the time (just growing), or is it something reset in some
period of time? In general, I would suggest the following metrics
# Number of in-memory-flushes/rate of in-memory-flushes/average number of
in-memory-flushes per CompactingMemStore
# Number of bytes saved by CompactingMemStore (whether it is by flattening,
merging or compacting) per flush to disk. Meaning a per CompactingMemStore
counter that is zeroed upon flush to disk, and then collects again all the
saved bytes.
# Average pipeline length, averaged over the time and/or between stores
# Anything else?
> Expose in-memory compaction metrics
> ---
>
> Key: HBASE-20234
> URL: https://issues.apache.org/jira/browse/HBASE-20234
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
>
> Hard to glean insight from how well in-memory compaction is doing currently.
> It dumps stats into the logs but better if they were available to a
> dashboard. This issue is about exposing a couple of helpful counts. There are
> already by-region metrics. We can add a few for in-memory compaction (Help me
> out [~anastas]... what counts would be best to expose).
> Flush related metrics include
> {code}
> Namespace_default_table_tsdb-tree_region_cfbf23e7330a1a2bbde031f9583d3415_metric_flushesQueuedCount:
> {
> description: "Number flushes requested/queued for this region",
> value: 0
> {
> description: "The number of cells flushed to disk",
> value: 0
> },
> {
> description: "The total amount of data flushed to disk, in bytes",
> value: 0
> },
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20281) [DOC] upgrade section needs an explanation of changes to Bucket Cache
[
https://issues.apache.org/jira/browse/HBASE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412989#comment-16412989
]
Anoop Sam John commented on HBASE-20281:
As part of another issue under HBASE-11425 am changing this Sean. In middle
of the change now. Will give patch for it soon.
> [DOC] upgrade section needs an explanation of changes to Bucket Cache
> -
>
> Key: HBASE-20281
> URL: https://issues.apache.org/jira/browse/HBASE-20281
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache, documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0
>
>
> the first pass version of the upgrade docs has a brief note about the bucket
> cache changing:
> {quote}
> * hbase.bucketcache.combinedcache.enabled
> * hbase.bucketcache.ioengine no longer supports the 'heap' value.
> {quote}
> But the changes are substantial and warrant a section of their own under the
> "Changes of Note!" banner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20284) BucketCache reduce heap overhead : Investigate removal of NavigableSet 'blocksByHFile'
[ https://issues.apache.org/jira/browse/HBASE-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-20284: --- Summary: BucketCache reduce heap overhead : Investigate removal of NavigableSet 'blocksByHFile' (was: Reduce heap overhead : Investigate removal of NavigableSet 'blocksByHFile') > BucketCache reduce heap overhead : Investigate removal of NavigableSet > 'blocksByHFile' > -- > > Key: HBASE-20284 > URL: https://issues.apache.org/jira/browse/HBASE-20284 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > > This Set takes 40 bytes per entry (block). As of now the total heap > requirement per entry is 160. If we can avoid this Set it is 25% reduction. > This set is used for removal of blocks for a specific HFile after its > invalidation (Mostly because of its compaction or by Store close). Check > other ways to remove the blocks. May be in an async way after the compaction > is over by a dedicated cleaner thread (?) It might be ok not to remove the > invalidated file's entries immediately. When the cache is out of space, the > Eviction thread might select it and remove. Few things to consider/change > 1. When compaction process reads blocks , it might be delivered from cache. > We should not consider this access as a real block access for this block. > That will increase the chances of eviction thread selecting this block for > removal. We should be able to distinguish the Cache read by compaction > process/user read process clearly > 2. When the compaction process reads a block from cache, some way we can mark > this block (using one byte boolean) that it is just went with the compaction? > When later the Eviction thread to select a block and if there is tie because > of same access time/count, we can break this tie in favor of selecting the > already compacted block? Need to check its pros and cons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20284) Reduce heap overhead : Investigate removal of NavigableSet 'blocksByHFile'
Anoop Sam John created HBASE-20284: -- Summary: Reduce heap overhead : Investigate removal of NavigableSet 'blocksByHFile' Key: HBASE-20284 URL: https://issues.apache.org/jira/browse/HBASE-20284 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John This Set takes 40 bytes per entry (block). As of now the total heap requirement per entry is 160. If we can avoid this Set it is 25% reduction. This set is used for removal of blocks for a specific HFile after its invalidation (Mostly because of its compaction or by Store close). Check other ways to remove the blocks. May be in an async way after the compaction is over by a dedicated cleaner thread (?) It might be ok not to remove the invalidated file's entries immediately. When the cache is out of space, the Eviction thread might select it and remove. Few things to consider/change 1. When compaction process reads blocks , it might be delivered from cache. We should not consider this access as a real block access for this block. That will increase the chances of eviction thread selecting this block for removal. We should be able to distinguish the Cache read by compaction process/user read process clearly 2. When the compaction process reads a block from cache, some way we can mark this block (using one byte boolean) that it is just went with the compaction? When later the Eviction thread to select a block and if there is tie because of same access time/count, we can break this tie in favor of selecting the already compacted block? Need to check its pros and cons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17819) Reduce the heap overhead for BucketCache
[ https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17819: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the reviews. > Reduce the heap overhead for BucketCache > > > Key: HBASE-17819 > URL: https://issues.apache.org/jira/browse/HBASE-17819 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch, > HBASE-17819_V3.patch, HBASE-17819_new_V1.patch, HBASE-17819_new_V1.patch, > HBASE-17819_new_V2.patch > > > We keep Bucket entry map in BucketCache. Below is the math for heapSize for > the key , value into this map. > BlockCacheKey > --- > String hfileName - Ref - 4 > long offset - 8 > BlockType blockType - Ref - 4 > boolean isPrimaryReplicaBlock - 1 > Total = align(12 (Object) + 17 )= 32 > BucketEntry > > int offsetBase - 4 > int length - 4 > byte offset1 - 1 > byte deserialiserIndex - 1 > long accessCounter - 8 > BlockPriority priority - Ref - 4 > volatile boolean markedForEvict - 1 > AtomicInteger refCount - 16 + 4 > long cachedTime - 8 > Total = align(12 (Object) + 51) = 64 > ConcurrentHashMap Map.Entry - 40 > blocksByHFile ConcurrentSkipListSet Entry - 40 > Total = 32 + 64 + 80 = 176 > For 10 million blocks we will end up having 1.6GB of heap size. > This jira aims to reduce this as much as possible -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20271) ReplicationSourceWALReader.switched should use the file name instead of the path object directly
[
https://issues.apache.org/jira/browse/HBASE-20271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20271:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Seems worked. Resolve. Thanks [~tedyu] for reviewing.
> ReplicationSourceWALReader.switched should use the file name instead of the
> path object directly
>
>
> Key: HBASE-20271
> URL: https://issues.apache.org/jira/browse/HBASE-20271
> Project: HBase
> Issue Type: Sub-task
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20271.patch
>
>
> {noformat}
> 2018-03-24 08:29:29,965 ERROR
> [RS_REFRESH_PEER-regionserver/ubuntu:0-0.replicationSource,2.replicationSource.shipperubuntu%2C35197%2C1521851267085,2]
> helpers.MarkerIgnoringBase(159): * ABORTING region server
> ubuntu,35197,1521851267085: Failed to operate on replication queue *
> org.apache.hadoop.hbase.replication.ReplicationException: Failed to set log
> position (serverName=ubuntu,35197,1521851267085, queueId=2,
> fileName=ubuntu%2C35197%2C1521851267085.1521851344947, position=2533)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.setWALPosition(ZKReplicationQueueStorage.java:237)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.lambda$9(ReplicationSourceManager.java:488)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.abortWhenFail(ReplicationSourceManager.java:455)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:488)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:232)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:134)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:104)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1006)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:663)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1670)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.setWALPosition(ZKReplicationQueueStorage.java:235)
> ... 6 more
> 2018-03-24 08:29:30,025 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37509]
> master.MasterRpcServices(508): Region server ubuntu,35197,1521851267085
> reported a fatal error:
> * ABORTING region server ubuntu,35197,1521851267085: Failed to operate on
> replication queue *
> Cause:
> org.apache.hadoop.hbase.replication.ReplicationException: Failed to set log
> position (serverName=ubuntu,35197,1521851267085, queueId=2,
> fileName=ubuntu%2C35197%2C1521851267085.1521851344947, position=2533)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.setWALPosition(ZKReplicationQueueStorage.java:237)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.lambda$9(ReplicationSourceManager.java:488)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.abortWhenFail(ReplicationSourceManager.java:455)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:488)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:232)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:134)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:104)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1006)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:663)
> at
> org.apache.hadoop.hbase.
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[
https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412942#comment-16412942
]
Anastasia Braginsky commented on HBASE-20188:
-
{quote}bq. I opened HBASE-20259. Assigned you.
{quote}
Thanks [~stack]! I'll take care.
{quote}bq. Setting the table so in-memory compaction = NONE seems to make reads
worse w/ no real change to writes. See attached
YCSB_IN_MEMORY_COMPACTION=NONE.ops.png
{quote}
>From looking on YCSB_IN_MEMORY_COMPACTION=NONE.ops.png I cannot understand
>with which results should I compare. I mean where do we have reads with
>in-memory-compaction and where without...? Anyway, if you say that turning
>in-memory-compaction off doesn't improve the reads we should look for the
>source for reads degradation anywhere else.
In general I agree with Eshcar, if we want to improve reads (a little) we
should set hbase.hregion.compacting.pipeline.segments.limit to 1. However do
not expect a dramatic change there, according to my experiments it is about
1-4%.
> [TESTING] Performance
> -
>
> Key: HBASE-20188
> URL: https://issues.apache.org/jira/browse/HBASE-20188
> Project: HBase
> Issue Type: Umbrella
> Components: Performance
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: ITBLL2.5B_1.2.7vs2.0.0_cpu.png,
> ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png,
> ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png,
> ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png,
> ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png,
> YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png,
> YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png,
> flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt
>
>
> How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor
> that it is much slower, that the problem is the asyncwal writing. Does
> in-memory compaction slow us down or speed us up? What happens when you
> enable offheaping?
> Keep notes here in this umbrella issue. Need to be able to say something
> about perf when 2.0.0 ships.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
