[jira] [Updated] (HBASE-20119) Introduce a pojo class to carry coprocessor information in order to make TableDescriptorBuilder accept multiple cp at once
[
https://issues.apache.org/jira/browse/HBASE-20119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-20119:
---
Release Note:
1) Make all methods in TableDescriptorBuilder be setter pattern.
addCoprocessor -> setCoprocessor
addColumnFamily -> setColumnFamily
(addCoprocessor and addColumnFamily are still in branch-2 but they are marked
as deprecated)
2) add CoprocessorDescriptor to carry cp information
3) add CoprocessorDescriptorBuilder to build CoprocessorDescriptor
4) TD disallow user to set negative priority to coprocessor since parsing the
negative value will cause a exception
was:
1) Make all methods in TableDescriptorBuilder be setter pattern.
addCoprocessor -> setCoprocessor
addColumnFamily -> setColumnFamily
2) add CoprocessorDescriptor to carry cp information
3) add CoprocessorDescriptorBuilder to build CoprocessorDescriptor
4) TD disallow user to set negative priority to coprocessor since parsing the
negative value will cause a exception
> Introduce a pojo class to carry coprocessor information in order to make
> TableDescriptorBuilder accept multiple cp at once
> --
>
> Key: HBASE-20119
> URL: https://issues.apache.org/jira/browse/HBASE-20119
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 2.1.0
>
> Attachments: HBASE-20119.branch-2.0.v0.addendum.patch,
> HBASE-20119.branch-2.v0.patch, HBASE-20119.v0.patch.patch,
> HBASE-20119.v1.patch.patch, HBASE-20119.v2.patch, HBASE-20119.v3.patch
>
>
> The way to add cp to TableDescriptorBuilder is shown below.
> {code:java}
> public TableDescriptorBuilder addCoprocessor(String className) throws
> IOException {
> return addCoprocessor(className, null, Coprocessor.PRIORITY_USER, null);
> }
> public TableDescriptorBuilder addCoprocessor(String className, Path
> jarFilePath,
> int priority, final Map kvs) throws IOException {
> desc.addCoprocessor(className, jarFilePath, priority, kvs);
> return this;
> }
> public TableDescriptorBuilder addCoprocessorWithSpec(final String specStr)
> throws IOException {
> desc.addCoprocessorWithSpec(specStr);
> return this;
> }{code}
> When loading our config to create table with multiple cps, we have to write
> the ugly for-loop.
> {code:java}
> val builder = TableDescriptorBuilder.newBuilde(tableName)
> .setAAA()
> .setBBB()
> cps.map(toHBaseCp).foreach(builder.addCoprocessor)
> cfs.map(toHBaseCf).foreach(builder.addColumnFamily)
> admin.createTable(builder.build())
> {code}
> If we introduce a pojo to carry the cp data and add the method accepting
> multiple cps and cfs, it is easier to exercise the fluent interface of
> TableDescriptorBuilder.
> {code:java}
> admin.createTable(TableDescriptorBuilder.newBuilde(tableName)
> .addCoprocessor(cps.map(toHBaseCp).asJavaCollection)
> .addColumnFamily(cf.map(toHBaseCf).asJavaCollection)
> .setAAA()
> .setBBB()
> .build){code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20212) Make all Public classes have InterfaceAudience category
[
https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403480#comment-16403480
]
Chia-Ping Tsai edited comment on HBASE-20212 at 3/18/18 5:26 AM:
-
{quote}yeah, can do few modules at a time to chunk up the work.
{quote}
Try to simplify our rules. The regular expression is shown below.
{code:java}
(?!.*(.generated.|.tmpl.|\$)).*{code}
The following classes will be EXCLUDED
1) nested class
The nested class should default have same InterfaceAudience as with root class
unless it has another InterfaceAudience explicitly.
2) generated by protobuf
3) generated by jamon
4) generated by thrift
If we apply above regular expression, the count of invalid classes is about
200. It seems to me one patch can fix all.
Need suggestions (or +1)
was (Author: chia7712):
{quote}yeah, can do few modules at a time to chunk up the work.
{quote}
Try to simplify our rules. The regular expression is shown below.
{code:java}
(?!.*(.generated.|.tmpl.|\$)).*{code}
The following classes will be EXCLUDED
1) nested class
The nested class should default have same InterfaceAudience as with root class
unless it has another InterfaceAudience explicitly.
2) generated by protobuf
3) generated by jamon
If we apply above regular expression, the count of invalid classes is about
200. It seems to me one patch can fix all.
Need suggestions (or +1)
> Make all Public classes have InterfaceAudience category
> ---
>
> Key: HBASE-20212
> URL: https://issues.apache.org/jira/browse/HBASE-20212
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 2.1.0
>
>
> The tasks will be resolved are shown below.
> # add warbucks-maven-plugin to root pom
> # make sure all sub modules ref the warbucks-maven-plugin
> # remove old checker (TestInterfaceAudienceAnnotations)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20212) Make all Public classes have InterfaceAudience category
[
https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403872#comment-16403872
]
Chia-Ping Tsai commented on HBASE-20212:
{quote}Thrift classes are generated too.
{quote}
Thanks for the reminder [~stack]. The rule I posted also include the generated
classes in thrift module. Let me revise the previous post.
> Make all Public classes have InterfaceAudience category
> ---
>
> Key: HBASE-20212
> URL: https://issues.apache.org/jira/browse/HBASE-20212
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 2.1.0
>
>
> The tasks will be resolved are shown below.
> # add warbucks-maven-plugin to root pom
> # make sure all sub modules ref the warbucks-maven-plugin
> # remove old checker (TestInterfaceAudienceAnnotations)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20212) Make all Public classes have InterfaceAudience category
[
https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403480#comment-16403480
]
Chia-Ping Tsai edited comment on HBASE-20212 at 3/18/18 5:26 AM:
-
{quote}yeah, can do few modules at a time to chunk up the work.
{quote}
Try to simplify our rules. The regular expression is shown below.
{code:java}
(?!.*(.generated.|.tmpl.|\$)).*{code}
The following classes will be EXCLUDED
1) nested class
The nested class should default have same InterfaceAudience as with root class
unless it has another InterfaceAudience explicitly.
2) generated by protobuf
3) generated by jamon
4) generated by thrift (thank [~stack])
If we apply above regular expression, the count of invalid classes is about
200. It seems to me one patch can fix all.
Need suggestions (or +1)
was (Author: chia7712):
{quote}yeah, can do few modules at a time to chunk up the work.
{quote}
Try to simplify our rules. The regular expression is shown below.
{code:java}
(?!.*(.generated.|.tmpl.|\$)).*{code}
The following classes will be EXCLUDED
1) nested class
The nested class should default have same InterfaceAudience as with root class
unless it has another InterfaceAudience explicitly.
2) generated by protobuf
3) generated by jamon
4) generated by thrift
If we apply above regular expression, the count of invalid classes is about
200. It seems to me one patch can fix all.
Need suggestions (or +1)
> Make all Public classes have InterfaceAudience category
> ---
>
> Key: HBASE-20212
> URL: https://issues.apache.org/jira/browse/HBASE-20212
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 2.1.0
>
>
> The tasks will be resolved are shown below.
> # add warbucks-maven-plugin to root pom
> # make sure all sub modules ref the warbucks-maven-plugin
> # remove old checker (TestInterfaceAudienceAnnotations)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20119) Introduce a pojo class to carry coprocessor information in order to make TableDescriptorBuilder accept multiple cp at once
[
https://issues.apache.org/jira/browse/HBASE-20119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403871#comment-16403871
]
Chia-Ping Tsai commented on HBASE-20119:
{quote}Or is it because HTD implements TD?
{quote}
yep.
{quote}What you thinking, that this renaming is ok because it internal...
addColumnFamily to setColumnFamily?
{quote}
TableDescriptorBuilder is part of Public APIs. As [~elserj] suggested, I add
the old methods back and make them deprecated.
BTW, the addendum is pushed to only branch-2+. Given most ref to deprecated
methods are updated, it won't cause much code conflicts in the future.
> Introduce a pojo class to carry coprocessor information in order to make
> TableDescriptorBuilder accept multiple cp at once
> --
>
> Key: HBASE-20119
> URL: https://issues.apache.org/jira/browse/HBASE-20119
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 2.1.0
>
> Attachments: HBASE-20119.branch-2.0.v0.addendum.patch,
> HBASE-20119.branch-2.v0.patch, HBASE-20119.v0.patch.patch,
> HBASE-20119.v1.patch.patch, HBASE-20119.v2.patch, HBASE-20119.v3.patch
>
>
> The way to add cp to TableDescriptorBuilder is shown below.
> {code:java}
> public TableDescriptorBuilder addCoprocessor(String className) throws
> IOException {
> return addCoprocessor(className, null, Coprocessor.PRIORITY_USER, null);
> }
> public TableDescriptorBuilder addCoprocessor(String className, Path
> jarFilePath,
> int priority, final Map kvs) throws IOException {
> desc.addCoprocessor(className, jarFilePath, priority, kvs);
> return this;
> }
> public TableDescriptorBuilder addCoprocessorWithSpec(final String specStr)
> throws IOException {
> desc.addCoprocessorWithSpec(specStr);
> return this;
> }{code}
> When loading our config to create table with multiple cps, we have to write
> the ugly for-loop.
> {code:java}
> val builder = TableDescriptorBuilder.newBuilde(tableName)
> .setAAA()
> .setBBB()
> cps.map(toHBaseCp).foreach(builder.addCoprocessor)
> cfs.map(toHBaseCf).foreach(builder.addColumnFamily)
> admin.createTable(builder.build())
> {code}
> If we introduce a pojo to carry the cp data and add the method accepting
> multiple cps and cfs, it is easier to exercise the fluent interface of
> TableDescriptorBuilder.
> {code:java}
> admin.createTable(TableDescriptorBuilder.newBuilde(tableName)
> .addCoprocessor(cps.map(toHBaseCp).asJavaCollection)
> .addColumnFamily(cf.map(toHBaseCf).asJavaCollection)
> .setAAA()
> .setBBB()
> .build){code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403853#comment-16403853
]
Appy commented on HBASE-20151:
--
That sounds coherent to kinds of issues i saw with filter last time. Post a
patch [~reidchan]? I'll take a look at code soon.
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Reid Chan
>Priority: Major
> Fix For: 2.0.0
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20206) WALEntryStream should not switch WAL file silently
[
https://issues.apache.org/jira/browse/HBASE-20206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403843#comment-16403843
]
Hadoop QA commented on HBASE-20206:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
2s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hbase-replication passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} hbase-server: The patch generated 0 new + 14
unchanged - 1 fixed = 14 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
45s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 40s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
28s{color} | {color:red} hbase-server generated 5 new + 2 unchanged - 0 fixed =
7 total (was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}104m
33s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20206 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915040/HBASE-20206-v5.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1bb7e8885eb4 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-19024) provide a configurable option to hsync WAL edits to the disk for better durability
[ https://issues.apache.org/jira/browse/HBASE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403838#comment-16403838 ] Anoop Sam John commented on HBASE-19024: +1 for doing asyncfswal stuff in another jira. Just can make this jira subject clear this is for fs wal. Else +1 for the patch. > provide a configurable option to hsync WAL edits to the disk for better > durability > -- > > Key: HBASE-19024 > URL: https://issues.apache.org/jira/browse/HBASE-19024 > Project: HBase > Issue Type: Improvement > Components: wal > Environment: >Reporter: Vikas Vishwakarma >Assignee: Harshal Jain >Priority: Critical > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: HBASE-19024-master.v10.patch, > HBASE-19024.branch-1.2.001.patch, HBASE-19024.branch-1.2.002.patch, > HBASE-19024.branch-1.2.003.patch, HBASE-19024.branch-1.2.004.patch, > HBASE-19024.branch-1.2.005.patch, branch-1.branch-1.patch, > branch-1.v1.branch-1.patch, master.patch, master.v2.patch, master.v3.patch, > master.v5.patch, master.v5.patch, master.v6.patch, master.v9.patch > > > At present we do not have an option to hsync WAL edits to the disk for better > durability. In our local tests we see 10-15% latency impact of using hsync > instead of hflush which is not very high. > We should have a configurable option to hysnc WAL edits instead of just > sync/hflush which will call the corresponding API on the hadoop side. > Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling > FSDataOutputStream sync/hflush on the hadoop side. This can be modified to > let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can > keep the default value to sync as the current behavior and hsync can be > enabled based on explicit configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403836#comment-16403836 ] Anoop Sam John edited comment on HBASE-20188 at 3/18/18 2:19 AM: - In tests, we try flush to SDD or HDD boss? In writes we normally see this compares as hottest. It used to be this way. Now I can see the issue you face is the exception because memstore size is 4x larger than flush size. As you said yes the flush seems NOT speedy enough. I can think of following 1. Now with compacting memstore, the flush op as such more time taking. With default memstore, it just a matter of iterating over a map and write cells. But now we have to read from multiple segments in a heap way and so more compares there. Anyway at flush size the flush op was triggered. But till it reaches 4x flush was not complete. This can be one reason 2. The writes to CSLM as such became fast. With default memstore, when we are at flush size and started the flush op, still writes happening to CSLM. we allow it anyway. But then CSLM state is such that it already having so many cells and writes might be bit more delayed. So the pace of this might be low enough for flush to complete. But with compacting memstore, it will become fresh CSLM again and so very fast writes. How about your global memstore size limit. I guess this might be a very large number. Normally in tests what we see is this barrier breach and so forced flushes by blocking writes. Because there are enough regions in RS and write to all regions. So any region crossing this 4x mark is less likely compared to this global barrier breach. When I did tests normally will select this barrier as 2 * regions# * flush size. We very much need flush to be faster. If not on SSD, any chance for an SSD based tests? This is one reason why am a fan of that JMS issue of flush to SSD policy. was (Author: anoop.hbase): In tests, we try flush to SDD or HDD boss? In writes we normally see this compares as hottest. It used to be this way. Now I can see the issue you face is the exception because memstore size is 4x larger than flush size. As you said yes the flush is speedy enough. I can think of following 1. Now with compacting memstore, the flush op as such more time taking. With default memstore, it just a matter of iterating over a map and write cells. But now we have to read from multiple segments in a heap way and so more compares there. Anyway at flush size the flush op was triggered. But till it reaches 4x flush was not complete. This can be one reason 2. The writes to CSLM as such became fast. With default memstore, when we are at flush size and started the flush op, still writes happening to CSLM. we allow it anyway. But then CSLM state is such that it already having so many cells and writes might be bit more delayed. So the pace of this might be low enough for flush to complete. But with compacting memstore, it will become fresh CSLM again and so very fast writes. How about your global memstore size limit. I guess this might be a very large number. Normally in tests what we see is this barrier breach and so forced flushes by blocking writes. Because there are enough regions in RS and write to all regions. So any region crossing this 4x mark is less likely compared to this global barrier breach. When I did tests normally will select this barrier as 2 * regions# * flush size. We very much need flush to be faster. If not on SSD, any chance for an SSD based tests? This is one reason why am a fan of that JMS issue of flush to SSD policy. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403836#comment-16403836 ] Anoop Sam John commented on HBASE-20188: In tests, we try flush to SDD or HDD boss? In writes we normally see this compares as hottest. It used to be this way. Now I can see the issue you face is the exception because memstore size is 4x larger than flush size. As you said yes the flush is speedy enough. I can think of following 1. Now with compacting memstore, the flush op as such more time taking. With default memstore, it just a matter of iterating over a map and write cells. But now we have to read from multiple segments in a heap way and so more compares there. Anyway at flush size the flush op was triggered. But till it reaches 4x flush was not complete. This can be one reason 2. The writes to CSLM as such became fast. With default memstore, when we are at flush size and started the flush op, still writes happening to CSLM. we allow it anyway. But then CSLM state is such that it already having so many cells and writes might be bit more delayed. So the pace of this might be low enough for flush to complete. But with compacting memstore, it will become fresh CSLM again and so very fast writes. How about your global memstore size limit. I guess this might be a very large number. Normally in tests what we see is this barrier breach and so forced flushes by blocking writes. Because there are enough regions in RS and write to all regions. So any region crossing this 4x mark is less likely compared to this global barrier breach. When I did tests normally will select this barrier as 2 * regions# * flush size. We very much need flush to be faster. If not on SSD, any chance for an SSD based tests? This is one reason why am a fan of that JMS issue of flush to SSD policy. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20218) Proposed Perfromace Enhancements For TableSnapshotInputFomat
Saad Mufti created HBASE-20218: -- Summary: Proposed Perfromace Enhancements For TableSnapshotInputFomat Key: HBASE-20218 URL: https://issues.apache.org/jira/browse/HBASE-20218 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 1.4.0 Environment: HBase 1.4.0 running in AWS EMR 5.12.0 with the HBase rootdir set to a folder in S3 Reporter: Saad Mufti I have been testing a few Spark jobs we have at my company which work off of TableSnapshotInputFormat to read directly from the filesystem snapshots created on another EMR/Hbase cluster and stored in S3. During performance testing I found various small changes which would greatly enhance peformance. Right now we are running our jobs linked with a patched version of HBase 1.4.0 in which I made these changes, and I am hoping to submit my patch for review and eventual acceptance into the main codebase. The list of changes are : 1. a flag to control whether the snapshot restore uses a UUID based random temp dir in the specified restore directory. We use the flag to turn this off so that we can benefit from a AWS S3 specific bucket partitioning scheme we have provisioned. The way S3 partitioning works, you have to give a fixed path prefix and a pattern of files after that, and AWS can then partition on the paths after the fixed prefix into different resources to get more parallelization. We were advised by AWS that we could only get this good partitioning behavior if we didn't have that rancom directory in there. 2. a flag to turn off the code that tries to compute locality information for the splits. This is useless when dealing with S3 since the files are not on the cluster so there is no use in computing locality; and worse yet, it uses a single thread in the driver to iterate over all the files in the restored snapshot. For a very large table this was taking hours and hours iterating through S3 objects just to list them (about 360,000 of them for the our specific table). 3. a flag to override the column family schema setting to prefetch regions on open. This was causing the main executor thread on which a Spark task was running, which was trying to read through HFile's for its scan, compete for a lock on the underlying EMRFS stream object with prefetch threads trying to read the same file, so most tasks in the Spark stage would finish but the last few would linger half an hour or more competing with the prefetch threads alternately for a lock on an EMRFS stream object. This is the only change that had to be outside the mapreduce package as it directly affects the prefetch behavior in CacheConfig.java 4. a flag to turn off maintenance of Scan metrics. this was also causing a major slowdown, getting rid of this sped things up 4-5 times. What I observed in the thread dumps was that every call to update scan metrics was trying to get some HBase counter object and deep underneath was trying to access some Java resource bundle, and throwing an exception that it wasn't found. The exception was never visible at the application level and was swallowed underneath but whatever it was doing was causing a major slowdown. So we use this flag to avoid collecting those metrics because we never used them I am polishing my patch a bit more and hopefully will attach it tomorrow. One caveat, I tried but struggled with how to write any useful unit/component tests for these as these are invisible behaviors that do not affect the final result at all. And I am not that familiar with the HBase testing standards, so for now I am looking for guidance on what to tests. Would appreciate any feedback plus guidance on writing tests, provided of course there is interest in incorporating my patch into the main codebase. Cheers. Saad -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20218) Proposed Performance Enhancements For TableSnapshotInputFomat
[ https://issues.apache.org/jira/browse/HBASE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saad Mufti updated HBASE-20218: --- Summary: Proposed Performance Enhancements For TableSnapshotInputFomat (was: Proposed Perfromace Enhancements For TableSnapshotInputFomat) > Proposed Performance Enhancements For TableSnapshotInputFomat > - > > Key: HBASE-20218 > URL: https://issues.apache.org/jira/browse/HBASE-20218 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 1.4.0 > Environment: HBase 1.4.0 running in AWS EMR 5.12.0 with the HBase > rootdir set to a folder in S3 > >Reporter: Saad Mufti >Priority: Minor > > I have been testing a few Spark jobs we have at my company which work off of > TableSnapshotInputFormat to read directly from the filesystem snapshots > created on another EMR/Hbase cluster and stored in S3. During performance > testing I found various small changes which would greatly enhance peformance. > Right now we are running our jobs linked with a patched version of HBase > 1.4.0 in which I made these changes, and I am hoping to submit my patch for > review and eventual acceptance into the main codebase. > > The list of changes are : > > 1. a flag to control whether the snapshot restore uses a UUID based random > temp dir in the specified restore directory. We use the flag to turn this off > so that we can benefit from a AWS S3 specific bucket partitioning scheme we > have provisioned. The way S3 partitioning works, you have to give a fixed > path prefix and a pattern of files after that, and AWS can then partition on > the paths after the fixed prefix into different resources to get more > parallelization. We were advised by AWS that we could only get this good > partitioning behavior if we didn't have that rancom directory in there. > > 2. a flag to turn off the code that tries to compute locality information > for the splits. This is useless when dealing with S3 since the files are not > on the cluster so there is no use in computing locality; and worse yet, it > uses a single thread in the driver to iterate over all the files in the > restored snapshot. For a very large table this was taking hours and hours > iterating through S3 objects just to list them (about 360,000 of them for the > our specific table). > > 3. a flag to override the column family schema setting to prefetch regions on > open. This was causing the main executor thread on which a Spark task was > running, which was trying to read through HFile's for its scan, compete for a > lock on the underlying EMRFS stream object with prefetch threads trying to > read the same file, so most tasks in the Spark stage would finish but the > last few would linger half an hour or more competing with the prefetch > threads alternately for a lock on an EMRFS stream object. This is the only > change that had to be outside the mapreduce package as it directly affects > the prefetch behavior in CacheConfig.java > > 4. a flag to turn off maintenance of Scan metrics. this was also causing a > major slowdown, getting rid of this sped things up 4-5 times. What I observed > in the thread dumps was that every call to update scan metrics was trying to > get some HBase counter object and deep underneath was trying to access some > Java resource bundle, and throwing an exception that it wasn't found. The > exception was never visible at the application level and was swallowed > underneath but whatever it was doing was causing a major slowdown. So we use > this flag to avoid collecting those metrics because we never used them > > I am polishing my patch a bit more and hopefully will attach it tomorrow. One > caveat, I tried but struggled with how to write any useful unit/component > tests for these as these are invisible behaviors that do not affect the final > result at all. And I am not that familiar with the HBase testing standards, > so for now I am looking for guidance on what to tests. > > Would appreciate any feedback plus guidance on writing tests, provided of > course there is interest in incorporating my patch into the main codebase. > > Cheers. > > Saad > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20206) WALEntryStream should not switch WAL file silently
[ https://issues.apache.org/jira/browse/HBASE-20206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20206: -- Attachment: HBASE-20206-v5.patch > WALEntryStream should not switch WAL file silently > -- > > Key: HBASE-20206 > URL: https://issues.apache.org/jira/browse/HBASE-20206 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20206-v1.patch, HBASE-20206-v2.patch, > HBASE-20206-v3.patch, HBASE-20206-v4.patch, HBASE-20206-v5.patch, > HBASE-20206.patch > > > The returned WALEntryBatch contains a file name and position, if we allow > switching WAL file silently then the position maybe wrong since it may be for > another file... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403764#comment-16403764
]
Hadoop QA commented on HBASE-20202:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 58s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}101m
23s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-20202 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915022/HBASE-20202.branch-2.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0f1e21d9038e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 03e7b78260 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12007/testReport/ |
| Max. process+thread count | 5068 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Bu
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403721#comment-16403721
]
Hadoop QA commented on HBASE-20169:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
33s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 23s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}122m
40s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}189m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-20169 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915018/HBASE-20169.branch-2.003.patch
|
| Optional Tests | asflicense cc unit hbaseprotoc javac javadoc findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2808035087c3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build to
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403720#comment-16403720
]
Hadoop QA commented on HBASE-20169:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
20s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
59s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 53s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}121m
57s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-20169 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915016/HBASE-20169.branch-2.002.patch
|
| Optional Tests | asflicense cc unit hbaseprotoc javac javadoc findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b51b9ac80326 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build to
[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup
[
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403715#comment-16403715
]
Hudson commented on HBASE-20186:
Results for branch branch-1.4
[build #258 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.3
>
> Attachments: HBASE-20186.master.000.patch,
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map>
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions
> which have been already placed according to the rsgroup assignment, and it is
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite
> efficient, as it does not make good use of the fact that servers in
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
> for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
> groupClusterState.put(curr, correctedState.get(curr));
> }
> }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each
> rsgroup in average, the code above has time complexity as O(m * n), while
> using TreeSet's contains(), the time complexity could be reduced to O (m *
> logn).
> Another improvement is we do not need to scan every server for each rsgroup.
> If the processed server could be recorded, we could skip those.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20141) Fix TooManyFiles exception when RefreshingChannels in FileIOEngine
[
https://issues.apache.org/jira/browse/HBASE-20141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403714#comment-16403714
]
Hudson commented on HBASE-20141:
Results for branch branch-1.4
[build #258 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Fix TooManyFiles exception when RefreshingChannels in FileIOEngine
> --
>
> Key: HBASE-20141
> URL: https://issues.apache.org/jira/browse/HBASE-20141
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.0, 2.0.0-beta-1, 1.4.2
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Fix For: 2.0.0, 3.0.0, 1.5.0, 1.4.3
>
> Attachments: HBASE-20141.master.001.patch,
> HBASE-20141.master.002.patch, HBASE-20141.master.003.patch,
> HBASE-20141.master.004.patch
>
>
> HBASE-19435 implements a fix for reopening file channels when they are
> unnexpected closed
> to avoid disabling the BucketCache. However, it was missed that the the
> channels might not
> actually be completely closed (the write or read channel might still be open
> (see
> https://docs.oracle.com/javase/7/docs/api/java/nio/channels/ClosedChannelException.html)
> This commit closes any open channels before creating a new channel.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403703#comment-16403703
]
stack commented on HBASE-20202:
---
Retry. Test passes locally.
> [AMv2] Don't move region if its a split parent or offlined
> --
>
> Key: HBASE-20202
> URL: https://issues.apache.org/jira/browse/HBASE-20202
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20202.branch-2.001.patch,
> HBASE-20202.branch-2.002.patch, HBASE-20202.branch-2.003.patch,
> HBASE-20202.branch-2.003.patch
>
>
> Found this one running ITBLLs. We'd just finished splitting a region
> 91655de06786f786b0ee9c51280e1ee6 and then a move for it comes in. The move
> fails in an interesting way. The location has been removed from the
> regionnode kept by the Master. HBASE-20178 adds macro checks on context. Need
> to add a few checks to the likes of MoveRegionProcedure so we don't try to
> move an offlined/split parent.
> {code}
> 2018-03-14 10:21:45,678 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=3177, state=SUCCESS; SplitTableRegionProcedure
> table=IntegrationTestBigLinkedList, parent=91655de06786f786b0ee9c51280e1ee6,
> daughterA=b67bf6b79eaa83de788b0519f782ce8e,
> daughterB=99cf6ddb38cad08e3aa7635b6cac2e7b in 10.0210sec
> 2018-03-14 10:21:45,679 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.
> 2018-03-14 10:21:45,680 INFO [PEWorker-5]
> procedure.MasterProcedureScheduler: pid=3187,
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
> hri=IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.,
> source=ve0530.halxg.cloudera.com,16020,1521007509855,
> destination=ve0528.halxg.cloudera.com,16020,1521047890874,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,680 INFO [PEWorker-15] assignment.RegionStateStore:
> pid=3194 updating hbase:meta
> row=IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.,
> regionState=CLOSING
> 2018-03-14 10:21:45,680 INFO [PEWorker-5] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855}]
> 2018-03-14 10:21:45,683 INFO [PEWorker-15]
> assignment.RegionTransitionProcedure: Dispatch pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855; rit=CLOSING,
> location=ve0530.halxg.cloudera.com,16020,1521007509855
> 2018-03-14 10:21:45,752 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,753 ERROR [PEWorker-15] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855
> java.lang.NullPointerException
>
>
>
>at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:934)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer(RegionStates.java:962)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.
[jira] [Updated] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20202:
--
Attachment: HBASE-20202.branch-2.003.patch
> [AMv2] Don't move region if its a split parent or offlined
> --
>
> Key: HBASE-20202
> URL: https://issues.apache.org/jira/browse/HBASE-20202
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20202.branch-2.001.patch,
> HBASE-20202.branch-2.002.patch, HBASE-20202.branch-2.003.patch,
> HBASE-20202.branch-2.003.patch
>
>
> Found this one running ITBLLs. We'd just finished splitting a region
> 91655de06786f786b0ee9c51280e1ee6 and then a move for it comes in. The move
> fails in an interesting way. The location has been removed from the
> regionnode kept by the Master. HBASE-20178 adds macro checks on context. Need
> to add a few checks to the likes of MoveRegionProcedure so we don't try to
> move an offlined/split parent.
> {code}
> 2018-03-14 10:21:45,678 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=3177, state=SUCCESS; SplitTableRegionProcedure
> table=IntegrationTestBigLinkedList, parent=91655de06786f786b0ee9c51280e1ee6,
> daughterA=b67bf6b79eaa83de788b0519f782ce8e,
> daughterB=99cf6ddb38cad08e3aa7635b6cac2e7b in 10.0210sec
> 2018-03-14 10:21:45,679 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.
> 2018-03-14 10:21:45,680 INFO [PEWorker-5]
> procedure.MasterProcedureScheduler: pid=3187,
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
> hri=IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.,
> source=ve0530.halxg.cloudera.com,16020,1521007509855,
> destination=ve0528.halxg.cloudera.com,16020,1521047890874,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,680 INFO [PEWorker-15] assignment.RegionStateStore:
> pid=3194 updating hbase:meta
> row=IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.,
> regionState=CLOSING
> 2018-03-14 10:21:45,680 INFO [PEWorker-5] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855}]
> 2018-03-14 10:21:45,683 INFO [PEWorker-15]
> assignment.RegionTransitionProcedure: Dispatch pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855; rit=CLOSING,
> location=ve0530.halxg.cloudera.com,16020,1521007509855
> 2018-03-14 10:21:45,752 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,753 ERROR [PEWorker-15] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855
> java.lang.NullPointerException
>
>
>
>at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:934)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer(RegionStates.java:962)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.j
[jira] [Commented] (HBASE-20119) Introduce a pojo class to carry coprocessor information in order to make TableDescriptorBuilder accept multiple cp at once
[
https://issues.apache.org/jira/browse/HBASE-20119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403700#comment-16403700
]
stack commented on HBASE-20119:
---
The public List getCoprocessors() { is ok from HTD? I'm a bit lost.
Your original patch had getCoprocessors call getCoprocessorDescriptors but
this patch removes getCoprocessors. Or is it because HTD implements TD? That
would make sense. If so, patch good by me.
What you thinking, that this renaming is ok because it internal...
addColumnFamily to setColumnFamily?
> Introduce a pojo class to carry coprocessor information in order to make
> TableDescriptorBuilder accept multiple cp at once
> --
>
> Key: HBASE-20119
> URL: https://issues.apache.org/jira/browse/HBASE-20119
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 2.1.0
>
> Attachments: HBASE-20119.branch-2.0.v0.addendum.patch,
> HBASE-20119.branch-2.v0.patch, HBASE-20119.v0.patch.patch,
> HBASE-20119.v1.patch.patch, HBASE-20119.v2.patch, HBASE-20119.v3.patch
>
>
> The way to add cp to TableDescriptorBuilder is shown below.
> {code:java}
> public TableDescriptorBuilder addCoprocessor(String className) throws
> IOException {
> return addCoprocessor(className, null, Coprocessor.PRIORITY_USER, null);
> }
> public TableDescriptorBuilder addCoprocessor(String className, Path
> jarFilePath,
> int priority, final Map kvs) throws IOException {
> desc.addCoprocessor(className, jarFilePath, priority, kvs);
> return this;
> }
> public TableDescriptorBuilder addCoprocessorWithSpec(final String specStr)
> throws IOException {
> desc.addCoprocessorWithSpec(specStr);
> return this;
> }{code}
> When loading our config to create table with multiple cps, we have to write
> the ugly for-loop.
> {code:java}
> val builder = TableDescriptorBuilder.newBuilde(tableName)
> .setAAA()
> .setBBB()
> cps.map(toHBaseCp).foreach(builder.addCoprocessor)
> cfs.map(toHBaseCf).foreach(builder.addColumnFamily)
> admin.createTable(builder.build())
> {code}
> If we introduce a pojo to carry the cp data and add the method accepting
> multiple cps and cfs, it is easier to exercise the fluent interface of
> TableDescriptorBuilder.
> {code:java}
> admin.createTable(TableDescriptorBuilder.newBuilde(tableName)
> .addCoprocessor(cps.map(toHBaseCp).asJavaCollection)
> .addColumnFamily(cf.map(toHBaseCf).asJavaCollection)
> .setAAA()
> .setBBB()
> .build){code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403698#comment-16403698
]
Hadoop QA commented on HBASE-20202:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 7s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 33s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestReplicationDroppedTables |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-20202 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915017/HBASE-20202.branch-2.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 352f14884291 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 03e7b78260 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/12006/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org
[jira] [Commented] (HBASE-20212) Make all Public classes have InterfaceAudience category
[ https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403697#comment-16403697 ] stack commented on HBASE-20212: --- Thrift classes are generated too. Otherwise sounds good sir. > Make all Public classes have InterfaceAudience category > --- > > Key: HBASE-20212 > URL: https://issues.apache.org/jira/browse/HBASE-20212 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.1.0 > > > The tasks will be resolved are shown below. > # add warbucks-maven-plugin to root pom > # make sure all sub modules ref the warbucks-maven-plugin > # remove old checker (TestInterfaceAudienceAnnotations) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403692#comment-16403692 ] stack commented on HBASE-20188: --- [~carp84] ... FYI sir. > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403691#comment-16403691 ] stack commented on HBASE-20188: --- [~eshcar] and [~anastas] You two see the svgs and the tree.txt file attached here? The CSLM is the umbrella under which much of the CPU usage on a mostly write basis occurs (ITBLL). The leafs are Cell compares (rowCompare in particular given rows are mostly unique in this dataset). You can see some CPU usage by in-memory compaction... as an umbrella again over compares. I was wondering what your thoughts were regards our doing MORE aggressive in-memory compaction moving Cells from CSLM to your flat structures, could it save on the number of overall compares (and hence CPU) or, if not on compares, overhead from the CSLM itself shows up as a pretty big CPU user too? What you reckon? > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19024) provide a configurable option to hsync WAL edits to the disk for better durability
[
https://issues.apache.org/jira/browse/HBASE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403681#comment-16403681
]
Andrew Purtell commented on HBASE-19024:
{quote}So we can enable hsync over hflush at server level by hbase.wal.hsync
new config. And individually a Mutation can mark any lower Durability also.
Same way even if this config is NOT ON at RS level, still Mutation can mark so.
Correct?
{quote}
Yes
{quote}isForceSync - Not used. Do we need this? I can not see even in tests
been used.
{quote}
It seems to be a bean thing. I just rebased the patch. Can remove it. Not a big
deal either way.
{quote}In case of AsyncFSWAL, we do not use this new config and seems we use
only hflush way there.(Offline confirmed with [~Apache9]). Another issue to fix
that or in this issue only?
{quote}
We don't have asyncfswal in branch-1 so had a blind spot here. Could do it.
Could wait for another JIRA. Might be less confusing to add support there too.
> provide a configurable option to hsync WAL edits to the disk for better
> durability
> --
>
> Key: HBASE-19024
> URL: https://issues.apache.org/jira/browse/HBASE-19024
> Project: HBase
> Issue Type: Improvement
> Components: wal
> Environment:
>Reporter: Vikas Vishwakarma
>Assignee: Harshal Jain
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 1.5.0
>
> Attachments: HBASE-19024-master.v10.patch,
> HBASE-19024.branch-1.2.001.patch, HBASE-19024.branch-1.2.002.patch,
> HBASE-19024.branch-1.2.003.patch, HBASE-19024.branch-1.2.004.patch,
> HBASE-19024.branch-1.2.005.patch, branch-1.branch-1.patch,
> branch-1.v1.branch-1.patch, master.patch, master.v2.patch, master.v3.patch,
> master.v5.patch, master.v5.patch, master.v6.patch, master.v9.patch
>
>
> At present we do not have an option to hsync WAL edits to the disk for better
> durability. In our local tests we see 10-15% latency impact of using hsync
> instead of hflush which is not very high.
> We should have a configurable option to hysnc WAL edits instead of just
> sync/hflush which will call the corresponding API on the hadoop side.
> Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling
> FSDataOutputStream sync/hflush on the hadoop side. This can be modified to
> let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can
> keep the default value to sync as the current behavior and hsync can be
> enabled based on explicit configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403548#comment-16403548
]
stack commented on HBASE-20169:
---
.003 purifies the patch. I had mixed in fix for HBASE-20202. Removed it.
> NPE when calling HBTU.shutdownMiniCluster
> -
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch, HBASE-20169.branch-2.003.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20169:
--
Attachment: HBASE-20169.branch-2.003.patch
> NPE when calling HBTU.shutdownMiniCluster
> -
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch, HBASE-20169.branch-2.003.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403546#comment-16403546
]
stack commented on HBASE-20202:
---
.003 is an addendum. The prepare step is not being called at all. Fix. TODO:
Add test failing the prepare and play with setting lock for life of procedure.
> [AMv2] Don't move region if its a split parent or offlined
> --
>
> Key: HBASE-20202
> URL: https://issues.apache.org/jira/browse/HBASE-20202
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20202.branch-2.001.patch,
> HBASE-20202.branch-2.002.patch, HBASE-20202.branch-2.003.patch
>
>
> Found this one running ITBLLs. We'd just finished splitting a region
> 91655de06786f786b0ee9c51280e1ee6 and then a move for it comes in. The move
> fails in an interesting way. The location has been removed from the
> regionnode kept by the Master. HBASE-20178 adds macro checks on context. Need
> to add a few checks to the likes of MoveRegionProcedure so we don't try to
> move an offlined/split parent.
> {code}
> 2018-03-14 10:21:45,678 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=3177, state=SUCCESS; SplitTableRegionProcedure
> table=IntegrationTestBigLinkedList, parent=91655de06786f786b0ee9c51280e1ee6,
> daughterA=b67bf6b79eaa83de788b0519f782ce8e,
> daughterB=99cf6ddb38cad08e3aa7635b6cac2e7b in 10.0210sec
> 2018-03-14 10:21:45,679 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.
> 2018-03-14 10:21:45,680 INFO [PEWorker-5]
> procedure.MasterProcedureScheduler: pid=3187,
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
> hri=IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.,
> source=ve0530.halxg.cloudera.com,16020,1521007509855,
> destination=ve0528.halxg.cloudera.com,16020,1521047890874,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,680 INFO [PEWorker-15] assignment.RegionStateStore:
> pid=3194 updating hbase:meta
> row=IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.,
> regionState=CLOSING
> 2018-03-14 10:21:45,680 INFO [PEWorker-5] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855}]
> 2018-03-14 10:21:45,683 INFO [PEWorker-15]
> assignment.RegionTransitionProcedure: Dispatch pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855; rit=CLOSING,
> location=ve0530.halxg.cloudera.com,16020,1521007509855
> 2018-03-14 10:21:45,752 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,753 ERROR [PEWorker-15] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855
> java.lang.NullPointerException
>
>
>
>at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:934)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer
[jira] [Updated] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20202:
--
Status: Patch Available (was: Reopened)
> [AMv2] Don't move region if its a split parent or offlined
> --
>
> Key: HBASE-20202
> URL: https://issues.apache.org/jira/browse/HBASE-20202
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20202.branch-2.001.patch,
> HBASE-20202.branch-2.002.patch, HBASE-20202.branch-2.003.patch
>
>
> Found this one running ITBLLs. We'd just finished splitting a region
> 91655de06786f786b0ee9c51280e1ee6 and then a move for it comes in. The move
> fails in an interesting way. The location has been removed from the
> regionnode kept by the Master. HBASE-20178 adds macro checks on context. Need
> to add a few checks to the likes of MoveRegionProcedure so we don't try to
> move an offlined/split parent.
> {code}
> 2018-03-14 10:21:45,678 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=3177, state=SUCCESS; SplitTableRegionProcedure
> table=IntegrationTestBigLinkedList, parent=91655de06786f786b0ee9c51280e1ee6,
> daughterA=b67bf6b79eaa83de788b0519f782ce8e,
> daughterB=99cf6ddb38cad08e3aa7635b6cac2e7b in 10.0210sec
> 2018-03-14 10:21:45,679 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.
> 2018-03-14 10:21:45,680 INFO [PEWorker-5]
> procedure.MasterProcedureScheduler: pid=3187,
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
> hri=IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.,
> source=ve0530.halxg.cloudera.com,16020,1521007509855,
> destination=ve0528.halxg.cloudera.com,16020,1521047890874,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,680 INFO [PEWorker-15] assignment.RegionStateStore:
> pid=3194 updating hbase:meta
> row=IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.,
> regionState=CLOSING
> 2018-03-14 10:21:45,680 INFO [PEWorker-5] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855}]
> 2018-03-14 10:21:45,683 INFO [PEWorker-15]
> assignment.RegionTransitionProcedure: Dispatch pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855; rit=CLOSING,
> location=ve0530.halxg.cloudera.com,16020,1521007509855
> 2018-03-14 10:21:45,752 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,753 ERROR [PEWorker-15] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855
> java.lang.NullPointerException
>
>
>
>at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:934)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer(RegionStates.java:962)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1548)
>
[jira] [Updated] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20202:
--
Attachment: HBASE-20202.branch-2.003.patch
> [AMv2] Don't move region if its a split parent or offlined
> --
>
> Key: HBASE-20202
> URL: https://issues.apache.org/jira/browse/HBASE-20202
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20202.branch-2.001.patch,
> HBASE-20202.branch-2.002.patch, HBASE-20202.branch-2.003.patch
>
>
> Found this one running ITBLLs. We'd just finished splitting a region
> 91655de06786f786b0ee9c51280e1ee6 and then a move for it comes in. The move
> fails in an interesting way. The location has been removed from the
> regionnode kept by the Master. HBASE-20178 adds macro checks on context. Need
> to add a few checks to the likes of MoveRegionProcedure so we don't try to
> move an offlined/split parent.
> {code}
> 2018-03-14 10:21:45,678 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=3177, state=SUCCESS; SplitTableRegionProcedure
> table=IntegrationTestBigLinkedList, parent=91655de06786f786b0ee9c51280e1ee6,
> daughterA=b67bf6b79eaa83de788b0519f782ce8e,
> daughterB=99cf6ddb38cad08e3aa7635b6cac2e7b in 10.0210sec
> 2018-03-14 10:21:45,679 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.
> 2018-03-14 10:21:45,680 INFO [PEWorker-5]
> procedure.MasterProcedureScheduler: pid=3187,
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
> hri=IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.,
> source=ve0530.halxg.cloudera.com,16020,1521007509855,
> destination=ve0528.halxg.cloudera.com,16020,1521047890874,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,680 INFO [PEWorker-15] assignment.RegionStateStore:
> pid=3194 updating hbase:meta
> row=IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.,
> regionState=CLOSING
> 2018-03-14 10:21:45,680 INFO [PEWorker-5] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855}]
> 2018-03-14 10:21:45,683 INFO [PEWorker-15]
> assignment.RegionTransitionProcedure: Dispatch pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855; rit=CLOSING,
> location=ve0530.halxg.cloudera.com,16020,1521007509855
> 2018-03-14 10:21:45,752 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,753 ERROR [PEWorker-15] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855
> java.lang.NullPointerException
>
>
>
>at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:934)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer(RegionStates.java:962)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1548)
>
[jira] [Updated] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20169:
--
Attachment: HBASE-20169.branch-2.002.patch
> NPE when calling HBTU.shutdownMiniCluster
> -
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403544#comment-16403544
]
stack commented on HBASE-20169:
---
.002 was missing the set of next step in procedure so we were stuck spinning.
> NPE when calling HBTU.shutdownMiniCluster
> -
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reopened HBASE-20202:
---
bq. This is wrong. Let me fix.
I filed HBASE-20217 to make Move hold lock across unassign and assign but now
am having second thoughts -- as I reconsider your feedback more -- so closed it
for now. Let me try it first to make sure it a good idea (I've been running
these patches on my little cluster trying to test for scale which has been good
at shaking out corner cases...). Meantime reopening this issue till I address
your suggestions.
Note to self, look at RecoverMetaProcedure too. Will have same issue.
Thanks again for taking a look.
> [AMv2] Don't move region if its a split parent or offlined
> --
>
> Key: HBASE-20202
> URL: https://issues.apache.org/jira/browse/HBASE-20202
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20202.branch-2.001.patch,
> HBASE-20202.branch-2.002.patch
>
>
> Found this one running ITBLLs. We'd just finished splitting a region
> 91655de06786f786b0ee9c51280e1ee6 and then a move for it comes in. The move
> fails in an interesting way. The location has been removed from the
> regionnode kept by the Master. HBASE-20178 adds macro checks on context. Need
> to add a few checks to the likes of MoveRegionProcedure so we don't try to
> move an offlined/split parent.
> {code}
> 2018-03-14 10:21:45,678 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=3177, state=SUCCESS; SplitTableRegionProcedure
> table=IntegrationTestBigLinkedList, parent=91655de06786f786b0ee9c51280e1ee6,
> daughterA=b67bf6b79eaa83de788b0519f782ce8e,
> daughterB=99cf6ddb38cad08e3aa7635b6cac2e7b in 10.0210sec
> 2018-03-14 10:21:45,679 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.
> 2018-03-14 10:21:45,680 INFO [PEWorker-5]
> procedure.MasterProcedureScheduler: pid=3187,
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
> hri=IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.,
> source=ve0530.halxg.cloudera.com,16020,1521007509855,
> destination=ve0528.halxg.cloudera.com,16020,1521047890874,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,680 INFO [PEWorker-15] assignment.RegionStateStore:
> pid=3194 updating hbase:meta
> row=IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.,
> regionState=CLOSING
> 2018-03-14 10:21:45,680 INFO [PEWorker-5] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855}]
> 2018-03-14 10:21:45,683 INFO [PEWorker-15]
> assignment.RegionTransitionProcedure: Dispatch pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855; rit=CLOSING,
> location=ve0530.halxg.cloudera.com,16020,1521007509855
> 2018-03-14 10:21:45,752 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,753 ERROR [PEWorker-15] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855
> java.lang.NullPointerException
>
>
>
[jira] [Resolved] (HBASE-20217) Make MoveRegionProcedure hold region lock for life of the procedure
[ https://issues.apache.org/jira/browse/HBASE-20217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-20217. --- Resolution: Later Hang on... let me think a bit more about this... > Make MoveRegionProcedure hold region lock for life of the procedure > --- > > Key: HBASE-20217 > URL: https://issues.apache.org/jira/browse/HBASE-20217 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > > From HBASE-20202, make procedure hold lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20206) WALEntryStream should not switch WAL file silently
[
https://issues.apache.org/jira/browse/HBASE-20206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403501#comment-16403501
]
Hadoop QA commented on HBASE-20206:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
6s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hbase-replication passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} hbase-server: The patch generated 0 new + 3
unchanged - 1 fixed = 3 total (was 4) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
46s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 39s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 35s{color}
| {color:red} hbase-replication in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 45s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestReplicationStateZKImpl |
| | hadoop.hbase.replication.TestZKReplicationQueueStorage |
| | hadoop.hbase.replication.regionserver.TestReplicationSourceManagerZkImpl |
| | hadoop.hbase.replication.TestReplicationDroppedTables |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20206 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915004/HBASE-20206-v4.patch |
| Optional Tests | asflicens
[jira] [Commented] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403483#comment-16403483
]
stack commented on HBASE-20202:
---
Thanks for taking a look [~Apache9]
bq. And since we do not set holdLock to true for MoveRegionProcedure
This is wrong. Let me fix.
I added prepare because thats easy to reason about come rollback time. In
general our steps are too macro (e.g. HBASE-20103). Filed new issue.
> [AMv2] Don't move region if its a split parent or offlined
> --
>
> Key: HBASE-20202
> URL: https://issues.apache.org/jira/browse/HBASE-20202
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20202.branch-2.001.patch,
> HBASE-20202.branch-2.002.patch
>
>
> Found this one running ITBLLs. We'd just finished splitting a region
> 91655de06786f786b0ee9c51280e1ee6 and then a move for it comes in. The move
> fails in an interesting way. The location has been removed from the
> regionnode kept by the Master. HBASE-20178 adds macro checks on context. Need
> to add a few checks to the likes of MoveRegionProcedure so we don't try to
> move an offlined/split parent.
> {code}
> 2018-03-14 10:21:45,678 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=3177, state=SUCCESS; SplitTableRegionProcedure
> table=IntegrationTestBigLinkedList, parent=91655de06786f786b0ee9c51280e1ee6,
> daughterA=b67bf6b79eaa83de788b0519f782ce8e,
> daughterB=99cf6ddb38cad08e3aa7635b6cac2e7b in 10.0210sec
> 2018-03-14 10:21:45,679 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.
> 2018-03-14 10:21:45,680 INFO [PEWorker-5]
> procedure.MasterProcedureScheduler: pid=3187,
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
> hri=IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.,
> source=ve0530.halxg.cloudera.com,16020,1521007509855,
> destination=ve0528.halxg.cloudera.com,16020,1521047890874,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,680 INFO [PEWorker-15] assignment.RegionStateStore:
> pid=3194 updating hbase:meta
> row=IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.,
> regionState=CLOSING
> 2018-03-14 10:21:45,680 INFO [PEWorker-5] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855}]
> 2018-03-14 10:21:45,683 INFO [PEWorker-15]
> assignment.RegionTransitionProcedure: Dispatch pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855; rit=CLOSING,
> location=ve0530.halxg.cloudera.com,16020,1521007509855
> 2018-03-14 10:21:45,752 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,753 ERROR [PEWorker-15] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855
> java.lang.NullPointerException
>
>
>
>at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStat
[jira] [Created] (HBASE-20217) Make MoveRegionProcedure hold region lock for life of the procedure
stack created HBASE-20217: - Summary: Make MoveRegionProcedure hold region lock for life of the procedure Key: HBASE-20217 URL: https://issues.apache.org/jira/browse/HBASE-20217 Project: HBase Issue Type: Sub-task Reporter: stack Assignee: stack Fix For: 2.0.0 >From HBASE-20202, make procedure hold lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20212) Make all Public classes have InterfaceAudience category
[
https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403480#comment-16403480
]
Chia-Ping Tsai commented on HBASE-20212:
{quote}yeah, can do few modules at a time to chunk up the work.
{quote}
Try to simplify our rules. The regular expression is shown below.
{code:java}
(?!.*(.generated.|.tmpl.|\$)).*{code}
The following classes will be EXCLUDED
1) nested class
The nested class should default have same InterfaceAudience as with root class
unless it has another InterfaceAudience explicitly.
2) generated by protobuf
3) generated by jamon
If we apply above regular expression, the count of invalid classes is about
200. It seems to me one patch can fix all.
Need suggestions (or +1)
> Make all Public classes have InterfaceAudience category
> ---
>
> Key: HBASE-20212
> URL: https://issues.apache.org/jira/browse/HBASE-20212
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 2.1.0
>
>
> The tasks will be resolved are shown below.
> # add warbucks-maven-plugin to root pom
> # make sure all sub modules ref the warbucks-maven-plugin
> # remove old checker (TestInterfaceAudienceAnnotations)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403467#comment-16403467
]
Reid Chan commented on HBASE-20151:
---
Because SCVF returns {{INCLUDE}} and FF returns {{NEXT_ROW}}, then
{{FilterListAnd}} returns {{NEXT_ROW}}, so after {{'a','1'}} it skips to
{{'b','2'}}. This leads to what i said.
If FF returns {{NEXT_COL}}(tested), results are as expected.
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Reid Chan
>Priority: Major
> Fix For: 2.0.0
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20206) WALEntryStream should not switch WAL file silently
[ https://issues.apache.org/jira/browse/HBASE-20206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20206: -- Attachment: HBASE-20206-v4.patch > WALEntryStream should not switch WAL file silently > -- > > Key: HBASE-20206 > URL: https://issues.apache.org/jira/browse/HBASE-20206 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20206-v1.patch, HBASE-20206-v2.patch, > HBASE-20206-v3.patch, HBASE-20206-v4.patch, HBASE-20206.patch > > > The returned WALEntryBatch contains a file name and position, if we allow > switching WAL file silently then the position maybe wrong since it may be for > another file... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20213) [LOGGING] Aligning formatting and logging less (compactions, in-memory compactions)
[
https://issues.apache.org/jira/browse/HBASE-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403431#comment-16403431
]
Hudson commented on HBASE-20213:
Results for branch master
[build #264 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/264/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [LOGGING] Aligning formatting and logging less (compactions, in-memory
> compactions)
> ---
>
> Key: HBASE-20213
> URL: https://issues.apache.org/jira/browse/HBASE-20213
> Project: HBase
> Issue Type: Bug
> Components: logging
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20213.branch-2.001.patch,
> HBASE-20213.branch-2.002.patch, HBASE-20213.branch-2.003.patch,
> HBASE-20213.branch-2.004.patch
>
>
> Here is some logging cleanup that came of a study session this afternoon.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20214) Review of RegionLocationFinder Class
[
https://issues.apache.org/jira/browse/HBASE-20214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403432#comment-16403432
]
Hudson commented on HBASE-20214:
Results for branch master
[build #264 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/264/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Review of RegionLocationFinder Class
>
>
> Key: HBASE-20214
> URL: https://issues.apache.org/jira/browse/HBASE-20214
> Project: HBase
> Issue Type: Improvement
> Components: Balancer, master
>Affects Versions: 2.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20214.1.patch
>
>
> # Use SLF4J parameter logging
> # Remove superfluous code
> # Replace code with re-usable libraries where possible
> # Use different data structure
> # Small perf improvements
> # Fix some checkstyle
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403429#comment-16403429
]
Hudson commented on HBASE-20202:
Results for branch master
[build #264 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/264/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [AMv2] Don't move region if its a split parent or offlined
> --
>
> Key: HBASE-20202
> URL: https://issues.apache.org/jira/browse/HBASE-20202
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20202.branch-2.001.patch,
> HBASE-20202.branch-2.002.patch
>
>
> Found this one running ITBLLs. We'd just finished splitting a region
> 91655de06786f786b0ee9c51280e1ee6 and then a move for it comes in. The move
> fails in an interesting way. The location has been removed from the
> regionnode kept by the Master. HBASE-20178 adds macro checks on context. Need
> to add a few checks to the likes of MoveRegionProcedure so we don't try to
> move an offlined/split parent.
> {code}
> 2018-03-14 10:21:45,678 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=3177, state=SUCCESS; SplitTableRegionProcedure
> table=IntegrationTestBigLinkedList, parent=91655de06786f786b0ee9c51280e1ee6,
> daughterA=b67bf6b79eaa83de788b0519f782ce8e,
> daughterB=99cf6ddb38cad08e3aa7635b6cac2e7b in 10.0210sec
> 2018-03-14 10:21:45,679 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.
> 2018-03-14 10:21:45,680 INFO [PEWorker-5]
> procedure.MasterProcedureScheduler: pid=3187,
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
> hri=IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.,
> source=ve0530.halxg.cloudera.com,16020,1521007509855,
> destination=ve0528.halxg.cloudera.com,16020,1521047890874,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,680 INFO [PEWorker-15] assignment.RegionStateStore:
> pid=3194 updating hbase:meta
> row=IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.,
> regionState=CLOSING
> 2018-03-14 10:21:45,680 INFO [PEWorker-5] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855}]
> 2018-03-14 10:21:45,683 INFO [PEWorker-15]
> assignment.RegionTransitionProcedure: Dispatch pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855; rit=CLOSING,
> location=ve0530.halxg.cloudera.com,16020,1521007509855
> 2018-03-14 10:21:45,752 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,753 ERROR [PEWorker-15] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1
[jira] [Commented] (HBASE-20141) Fix TooManyFiles exception when RefreshingChannels in FileIOEngine
[
https://issues.apache.org/jira/browse/HBASE-20141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403430#comment-16403430
]
Hudson commented on HBASE-20141:
Results for branch master
[build #264 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/264/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/264//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Fix TooManyFiles exception when RefreshingChannels in FileIOEngine
> --
>
> Key: HBASE-20141
> URL: https://issues.apache.org/jira/browse/HBASE-20141
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.0, 2.0.0-beta-1, 1.4.2
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Fix For: 2.0.0, 3.0.0, 1.5.0, 1.4.3
>
> Attachments: HBASE-20141.master.001.patch,
> HBASE-20141.master.002.patch, HBASE-20141.master.003.patch,
> HBASE-20141.master.004.patch
>
>
> HBASE-19435 implements a fix for reopening file channels when they are
> unnexpected closed
> to avoid disabling the BucketCache. However, it was missed that the the
> channels might not
> actually be completely closed (the write or read channel might still be open
> (see
> https://docs.oracle.com/javase/7/docs/api/java/nio/channels/ClosedChannelException.html)
> This commit closes any open channels before creating a new channel.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20202) [AMv2] Don't move region if its a split parent or offlined
[
https://issues.apache.org/jira/browse/HBASE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403421#comment-16403421
]
Duo Zhang commented on HBASE-20202:
---
Sorry a bit late, but is it necessary to have an extra prepare state? Just
check it in the MOVE_REGION_UNASSIGN state is enough?
And since we do not set holdLock to true for MoveRegionProcedure, there could
be holes between the execution of different states where other procedures may
grab the region lock since we schedule sub procedure and the parent procedure
will be suspended and release the region lock?
Thanks.
> [AMv2] Don't move region if its a split parent or offlined
> --
>
> Key: HBASE-20202
> URL: https://issues.apache.org/jira/browse/HBASE-20202
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20202.branch-2.001.patch,
> HBASE-20202.branch-2.002.patch
>
>
> Found this one running ITBLLs. We'd just finished splitting a region
> 91655de06786f786b0ee9c51280e1ee6 and then a move for it comes in. The move
> fails in an interesting way. The location has been removed from the
> regionnode kept by the Master. HBASE-20178 adds macro checks on context. Need
> to add a few checks to the likes of MoveRegionProcedure so we don't try to
> move an offlined/split parent.
> {code}
> 2018-03-14 10:21:45,678 INFO [PEWorker-2] procedure2.ProcedureExecutor:
> Finished pid=3177, state=SUCCESS; SplitTableRegionProcedure
> table=IntegrationTestBigLinkedList, parent=91655de06786f786b0ee9c51280e1ee6,
> daughterA=b67bf6b79eaa83de788b0519f782ce8e,
> daughterB=99cf6ddb38cad08e3aa7635b6cac2e7b in 10.0210sec
> 2018-03-14 10:21:45,679 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.
> 2018-03-14 10:21:45,680 INFO [PEWorker-5]
> procedure.MasterProcedureScheduler: pid=3187,
> state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure
> hri=IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.,
> source=ve0530.halxg.cloudera.com,16020,1521007509855,
> destination=ve0528.halxg.cloudera.com,16020,1521047890874,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,680 INFO [PEWorker-15] assignment.RegionStateStore:
> pid=3194 updating hbase:meta
> row=IntegrationTestBigLinkedList,\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA0,1521047891276.af198ca64b196fb3d2f5b3e815b2dad0.,
> regionState=CLOSING
> 2018-03-14 10:21:45,680 INFO [PEWorker-5] procedure2.ProcedureExecutor:
> Initialized subprocedures=[{pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855}]
> 2018-03-14 10:21:45,683 INFO [PEWorker-15]
> assignment.RegionTransitionProcedure: Dispatch pid=3194, ppid=3193,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=af198ca64b196fb3d2f5b3e815b2dad0,
> server=ve0530.halxg.cloudera.com,16020,1521007509855; rit=CLOSING,
> location=ve0530.halxg.cloudera.com,16020,1521007509855
> 2018-03-14 10:21:45,752 INFO [PEWorker-15]
> procedure.MasterProcedureScheduler: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855,
> IntegrationTestBigLinkedList,\x0C0\xC3\x0C0\xC3\x0C0,1521045713137.91655de06786f786b0ee9c51280e1ee6.
> 2018-03-14 10:21:45,753 ERROR [PEWorker-15] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=3195, ppid=3187,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=91655de06786f786b0ee9c51280e1ee6,
> server=ve0530.halxg.cloudera.com,16020,1521007509855
> java.lang.NullPointerException
>
>
>
>at
> java.util.concu
[jira] [Updated] (HBASE-19665) Add table based replication peers/queues storage back
[ https://issues.apache.org/jira/browse/HBASE-19665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19665: -- Fix Version/s: HBASE-15867 > Add table based replication peers/queues storage back > - > > Key: HBASE-19665 > URL: https://issues.apache.org/jira/browse/HBASE-19665 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Guanghao Zhang >Assignee: Zheng Hu >Priority: Major > Fix For: HBASE-15867 > > Attachments: HBASE-19665.v1.patch, HBASE-19665.v2.patch, > HBASE-19665.v3.patch, HBASE-19665.v3.patch, HBASE-19665.v4.patch, > HBASE-19665.v4.patch > > > We removed them after HBASE-19397. So open a issue to track this thing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15867) Move HBase replication tracking from ZooKeeper to HBase
[ https://issues.apache.org/jira/browse/HBASE-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403414#comment-16403414 ] Duo Zhang commented on HBASE-15867: --- As per this discussion https://lists.apache.org/thread.html/483ee387d80e9d2d914733db44e8236ca74065905efbacdb21cec615@%3Cdev.hbase.apache.org%3E I've reverted HBASE-19665 on master and created a feature branch HBASE-15867 for this feature. FYI [~openinx]. > Move HBase replication tracking from ZooKeeper to HBase > --- > > Key: HBASE-15867 > URL: https://issues.apache.org/jira/browse/HBASE-15867 > Project: HBase > Issue Type: New Feature > Components: Replication >Affects Versions: 2.1.0 >Reporter: Joseph >Assignee: Zheng Hu >Priority: Major > Fix For: 2.1.0 > > > Move the WAL file and offset tracking out of ZooKeeper and into an HBase > table called hbase:replication. > The largest three new changes will be two classes ReplicationTableBase, > TableBasedReplicationQueues, and TableBasedReplicationQueuesClient. As of now > ReplicationPeers and HFileRef's tracking will not be implemented. Subtasks > have been filed for these two jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20119) Introduce a pojo class to carry coprocessor information in order to make TableDescriptorBuilder accept multiple cp at once
[
https://issues.apache.org/jira/browse/HBASE-20119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403373#comment-16403373
]
Chia-Ping Tsai commented on HBASE-20119:
[~elserj] [~stack] need +1, bosses
> Introduce a pojo class to carry coprocessor information in order to make
> TableDescriptorBuilder accept multiple cp at once
> --
>
> Key: HBASE-20119
> URL: https://issues.apache.org/jira/browse/HBASE-20119
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.0.0, 3.0.0, 2.1.0
>
> Attachments: HBASE-20119.branch-2.0.v0.addendum.patch,
> HBASE-20119.branch-2.v0.patch, HBASE-20119.v0.patch.patch,
> HBASE-20119.v1.patch.patch, HBASE-20119.v2.patch, HBASE-20119.v3.patch
>
>
> The way to add cp to TableDescriptorBuilder is shown below.
> {code:java}
> public TableDescriptorBuilder addCoprocessor(String className) throws
> IOException {
> return addCoprocessor(className, null, Coprocessor.PRIORITY_USER, null);
> }
> public TableDescriptorBuilder addCoprocessor(String className, Path
> jarFilePath,
> int priority, final Map kvs) throws IOException {
> desc.addCoprocessor(className, jarFilePath, priority, kvs);
> return this;
> }
> public TableDescriptorBuilder addCoprocessorWithSpec(final String specStr)
> throws IOException {
> desc.addCoprocessorWithSpec(specStr);
> return this;
> }{code}
> When loading our config to create table with multiple cps, we have to write
> the ugly for-loop.
> {code:java}
> val builder = TableDescriptorBuilder.newBuilde(tableName)
> .setAAA()
> .setBBB()
> cps.map(toHBaseCp).foreach(builder.addCoprocessor)
> cfs.map(toHBaseCf).foreach(builder.addColumnFamily)
> admin.createTable(builder.build())
> {code}
> If we introduce a pojo to carry the cp data and add the method accepting
> multiple cps and cfs, it is easier to exercise the fluent interface of
> TableDescriptorBuilder.
> {code:java}
> admin.createTable(TableDescriptorBuilder.newBuilde(tableName)
> .addCoprocessor(cps.map(toHBaseCp).asJavaCollection)
> .addColumnFamily(cf.map(toHBaseCf).asJavaCollection)
> .setAAA()
> .setBBB()
> .build){code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup
[
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403369#comment-16403369
]
Hudson commented on HBASE-20186:
Results for branch branch-2
[build #495 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/495/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/495//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/495//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/495//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.3
>
> Attachments: HBASE-20186.master.000.patch,
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map>
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions
> which have been already placed according to the rsgroup assignment, and it is
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite
> efficient, as it does not make good use of the fact that servers in
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
> for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
> groupClusterState.put(curr, correctedState.get(curr));
> }
> }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each
> rsgroup in average, the code above has time complexity as O(m * n), while
> using TreeSet's contains(), the time complexity could be reduced to O (m *
> logn).
> Another improvement is we do not need to scan every server for each rsgroup.
> If the processed server could be recorded, we could skip those.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup
[
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403366#comment-16403366
]
Hudson commented on HBASE-20186:
Results for branch branch-1
[build #252 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.3
>
> Attachments: HBASE-20186.master.000.patch,
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map>
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions
> which have been already placed according to the rsgroup assignment, and it is
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite
> efficient, as it does not make good use of the fact that servers in
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
> for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
> groupClusterState.put(curr, correctedState.get(curr));
> }
> }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each
> rsgroup in average, the code above has time complexity as O(m * n), while
> using TreeSet's contains(), the time complexity could be reduced to O (m *
> logn).
> Another improvement is we do not need to scan every server for each rsgroup.
> If the processed server could be recorded, we could skip those.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20141) Fix TooManyFiles exception when RefreshingChannels in FileIOEngine
[
https://issues.apache.org/jira/browse/HBASE-20141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403365#comment-16403365
]
Hudson commented on HBASE-20141:
Results for branch branch-1
[build #252 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Fix TooManyFiles exception when RefreshingChannels in FileIOEngine
> --
>
> Key: HBASE-20141
> URL: https://issues.apache.org/jira/browse/HBASE-20141
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.0, 2.0.0-beta-1, 1.4.2
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Fix For: 2.0.0, 3.0.0, 1.5.0, 1.4.3
>
> Attachments: HBASE-20141.master.001.patch,
> HBASE-20141.master.002.patch, HBASE-20141.master.003.patch,
> HBASE-20141.master.004.patch
>
>
> HBASE-19435 implements a fix for reopening file channels when they are
> unnexpected closed
> to avoid disabling the BucketCache. However, it was missed that the the
> channels might not
> actually be completely closed (the write or read channel might still be open
> (see
> https://docs.oracle.com/javase/7/docs/api/java/nio/channels/ClosedChannelException.html)
> This commit closes any open channels before creating a new channel.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19024) provide a configurable option to hsync WAL edits to the disk for better durability
[ https://issues.apache.org/jira/browse/HBASE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403324#comment-16403324 ] Anoop Sam John commented on HBASE-19024: setForceSync - to be forceSync ? isForceSync - Not used. Do we need this? I can not see even in tests been used. So we can enable hsync over hflush at server level by hbase.wal.hsync new config. And individually a Mutation can mark any lower Durability also. Same way even if this config is NOT ON at RS level, still Mutation can mark so. Correct? In case of AsyncFSWAL, we do not use this new config and seems we use only hflush way there.(Offline confirmed with [~Apache9]). Another issue to fix that or in this issue only? > provide a configurable option to hsync WAL edits to the disk for better > durability > -- > > Key: HBASE-19024 > URL: https://issues.apache.org/jira/browse/HBASE-19024 > Project: HBase > Issue Type: Improvement > Components: wal > Environment: >Reporter: Vikas Vishwakarma >Assignee: Harshal Jain >Priority: Critical > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: HBASE-19024-master.v10.patch, > HBASE-19024.branch-1.2.001.patch, HBASE-19024.branch-1.2.002.patch, > HBASE-19024.branch-1.2.003.patch, HBASE-19024.branch-1.2.004.patch, > HBASE-19024.branch-1.2.005.patch, branch-1.branch-1.patch, > branch-1.v1.branch-1.patch, master.patch, master.v2.patch, master.v3.patch, > master.v5.patch, master.v5.patch, master.v6.patch, master.v9.patch > > > At present we do not have an option to hsync WAL edits to the disk for better > durability. In our local tests we see 10-15% latency impact of using hsync > instead of hflush which is not very high. > We should have a configurable option to hysnc WAL edits instead of just > sync/hflush which will call the corresponding API on the hadoop side. > Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling > FSDataOutputStream sync/hflush on the hadoop side. This can be modified to > let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can > keep the default value to sync as the current behavior and hsync can be > enabled based on explicit configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403313#comment-16403313
]
Hadoop QA commented on HBASE-20169:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
16s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 9s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}227m 40s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}292m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestWarmupRegion |
| | hadoop.hbase.regionserver.TestRemoveRegionMetrics |
| | hadoop.hbase.regionserver.wal.TestSecureAsyncWALReplay |
| | hadoop.hbase.regionserver.TestHRegionFileSystem |
| | hadoop.hbase.security.access.TestAccessController |
| | hadoop.hbase.coprocessor.TestMasterObserver |
| | hadoop.hbase.client.TestFromClientSide |
| | hadoop.hbase.client.TestAsyncMetaRegionLocator |
| | hadoop.hbase.TestAcidGuaranteesWithEagerPo
[jira] [Commented] (HBASE-20212) Make all Public classes have InterfaceAudience category
[ https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403310#comment-16403310 ] Appy commented on HBASE-20212: -- yeah, can do few modules at a time to chunk up the work. > Make all Public classes have InterfaceAudience category > --- > > Key: HBASE-20212 > URL: https://issues.apache.org/jira/browse/HBASE-20212 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.1.0 > > > The tasks will be resolved are shown below. > # add warbucks-maven-plugin to root pom > # make sure all sub modules ref the warbucks-maven-plugin > # remove old checker (TestInterfaceAudienceAnnotations) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
