[jira] [Updated] (HBASE-15635) Mean age of Blocks in cache (seconds) on webUI should be greater than zero

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-15635:
--
Attachment: 7BFFAF68-0807-400C-853F-706B498449E1.png

> Mean age of Blocks in cache (seconds) on webUI should be greater than zero
> --
>
> Key: HBASE-15635
> URL: https://issues.apache.org/jira/browse/HBASE-15635
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.17
>Reporter: Heng Chen
> Attachments: 7BFFAF68-0807-400C-853F-706B498449E1.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15579) Remove synchronized around nonce in Procedure submit

[Stephen Yuan Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236644#comment-15236644
 ] 

Stephen Yuan Jiang commented on HBASE-15579:


i know this.  What I am trying to say is that we unnecessary generate some ids 
and might not use it.  During debugging and looking at log, we might wonder why 
some id is missing ( a bug or from a RPC retry with same nonce).

> Remove synchronized around nonce in Procedure submit
> 
>
> Key: HBASE-15579
> URL: https://issues.apache.org/jira/browse/HBASE-15579
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.3.0, 1.1.5, 1.2.2
>
> Attachments: HBASE-15579-v0.patch
>
>
> There is a synchronized in ProcedureExecutor.submitProcedure() that protect 
> the nonce insert. We can get rid of that and just use the nonce concurrent 
> map pufIfMissing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15635) Mean age of Blocks in cache (seconds) on webUI should be greater than zero

[Heng Chen (JIRA)]

Heng Chen created HBASE-15635:
-

 Summary: Mean age of Blocks in cache (seconds) on webUI should be 
greater than zero
 Key: HBASE-15635
 URL: https://issues.apache.org/jira/browse/HBASE-15635
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.17
Reporter: Heng Chen






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15633) Backport HBASE-15507 to branch-1

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236643#comment-15236643
 ] 

Hadoop QA commented on HBASE-15633:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s {color} 
| {color:red} HBASE-15633 does not apply to master. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.2.1/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12798182/HBASE-15633-branch1-v1.patch
 |
| JIRA Issue | HBASE-15633 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/1368/console |
| Powered by | Apache Yetus 0.2.1   http://yetus.apache.org |


This message was automatically generated.



> Backport HBASE-15507 to branch-1
> 
>
> Key: HBASE-15633
> URL: https://issues.apache.org/jira/browse/HBASE-15633
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication, shell
>Affects Versions: 1.3.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
> Attachments: HBASE-15633-branch-1.patch, HBASE-15633-branch1-v1.patch
>
>
> HBASE-15507 introduced an update_peer_config method to allow online changes 
> to replication config. It depended on HBASE-11393, which so far exists only 
> in master and can only be in 2.0 and above because of some incompatible 
> changes to replication peer config serialization. 
> To get this in branch-1 will require two things:
> 1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
> incompatible tableCF serialization changes.
> 2. Port HBASE-15507's logic on top of this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15454) Archive store files older than max age

[Clara Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236634#comment-15236634
 ] 

Clara Xiong commented on HBASE-15454:
-

hbase.hstore.compaction.date.tiered.max.storefile.age.millis is the config you 
need. In my implementation, windows will not be tiered/combined once they cross 
this point of time, whether for minor or major compaction. Archive process can 
use this too. Could you explain more why you need to concat the windows across 
the max age point? My understanding is that once the maxTimestamp of file is 
older than max age, it can be archived. It doesn't mean a file with a time span 
across the point MUST be archived. This way the files will stay in the same 
size of the highest tier.

I am out on a trip. I will read more closely  tomorrow evening on the new 
window implementation to see if I miss anything.

> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454.patch
>
>
> Sometimes the old data is rarely touched but we can not remove it. So archive 
> it to several big files(by year or something) and use EC to reduce the 
> redundancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15518) Add Per-Table metrics back

[Alicia Ying Shu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alicia Ying Shu updated HBASE-15518:

Attachment: HBASE-15518-v3.patch

> Add Per-Table metrics back
> --
>
> Key: HBASE-15518
> URL: https://issues.apache.org/jira/browse/HBASE-15518
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Alicia Ying Shu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15518-v1.patch, HBASE-15518-v2.patch, 
> HBASE-15518-v3.patch, HBASE-15518.patch
>
>
> We used to have per-table metrics, but it was removed in some restructuring. 
> We have per-region metrics, and per-regionserver metrics, but nothing in 
> between. 
> For majority of users, per-region is too granular, they are mostly interested 
> in table level aggregates. This is especially useful in multi-tenant cases 
> where a table's disk usage, number of requests, etc can be made much more 
> visible. 
> In this jira, we'll add the basic infrastructure to add a single (or a few) 
> per-table metrics. Than we can improve on that by adding remaining metrics 
> from the region server level. 
> The plan is to NOT aggregate per-table metrics at master for now. Just 
> aggregation of per-region metrics at the per-table level for every 
> regionserver. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15454) Archive store files older than max age

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236618#comment-15236618
 ] 

Duo Zhang commented on HBASE-15454:
---

Or that, let's do not combine calendar based window and archive together?
Add a new config called archive, it will archive data older than 'max age', no 
matter how the window is implemented.

And for the window, I think we have different window implementation before and 
after 'max age' is reasonable as in your tiered window implementation, you will 
not expand window when crossing the 'max age', right? I do not know how to 
concat the windows before and after 'max age' smoothly if I do not know the 
implementation of each window right now, but I could have a try.

What do you think? [~davelatham] [~clarax98007]

Thanks.

> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454.patch
>
>
> Sometimes the old data is rarely touched but we can not remove it. So archive 
> it to several big files(by year or something) and use EC to reduce the 
> redundancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15454) Archive store files older than max age

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236605#comment-15236605
 ] 

Duo Zhang commented on HBASE-15454:
---

{quote}
it seems very inefficient that we need a routine to slice the data along the 
exponential windows for minor/major compaction and another concurrent routine 
to slice the data along the calendar windows to archive them.
{quote}
For major compaction, windows before 'max age' will be ArchiveWindow. The 'max 
age' is a split point, newer windows are tiered and older windows are archived.

{quote}
A user should only need either layout, not both.
{quote}
That's not true. For example, you want to archive data by year, and today is 
Jan 1, I think most of people do not want to archive the data written 
yesterday, right? So there will be a 'max age' config which means we only 
archive data older than it.

{quote}
And please add the EC manager code and make it work with both types of windows.
{quote}
The EC manager is part of HDFS...As I said, it is transparent to HBase 
currently...And you can use it for any file you want, but it can not perform 
well on small files.

Thanks.

> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454.patch
>
>
> Sometimes the old data is rarely touched but we can not remove it. So archive 
> it to several big files(by year or something) and use EC to reduce the 
> redundancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15454) Archive store files older than max age

[Clara Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236596#comment-15236596
 ] 

Clara Xiong commented on HBASE-15454:
-

To be specific, it seems very inefficient that we need a routine to slice the 
data  along the exponential windows for minor/major compaction and another 
concurrent routine to slice the data along the calendar windows to archive 
them.  A user should only need either layout, not both. Either layout satisfies 
time-range scan efficiency and archive/TTL efficiency. This is the same idea as 
Dave's pluggable window algorithm.

And please add the EC manager code and make it work with both types of windows. 
To answer your question that the order of archiving differs from compaction, it 
should be in EC's logic that scan the store file's time range to pick the files 
to archive. It can share the TTL logic.

> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454.patch
>
>
> Sometimes the old data is rarely touched but we can not remove it. So archive 
> it to several big files(by year or something) and use EC to reduce the 
> redundancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15633) Backport HBASE-15507 to branch-1

[Geoffrey Jacoby (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby updated HBASE-15633:

Status: Patch Available  (was: Open)

Fixed checkstyle warnings

> Backport HBASE-15507 to branch-1
> 
>
> Key: HBASE-15633
> URL: https://issues.apache.org/jira/browse/HBASE-15633
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication, shell
>Affects Versions: 1.3.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
> Attachments: HBASE-15633-branch-1.patch, HBASE-15633-branch1-v1.patch
>
>
> HBASE-15507 introduced an update_peer_config method to allow online changes 
> to replication config. It depended on HBASE-11393, which so far exists only 
> in master and can only be in 2.0 and above because of some incompatible 
> changes to replication peer config serialization. 
> To get this in branch-1 will require two things:
> 1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
> incompatible tableCF serialization changes.
> 2. Port HBASE-15507's logic on top of this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15633) Backport HBASE-15507 to branch-1

[Geoffrey Jacoby (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby updated HBASE-15633:

Attachment: HBASE-15633-branch1-v1.patch

> Backport HBASE-15507 to branch-1
> 
>
> Key: HBASE-15633
> URL: https://issues.apache.org/jira/browse/HBASE-15633
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication, shell
>Affects Versions: 1.3.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
> Attachments: HBASE-15633-branch-1.patch, HBASE-15633-branch1-v1.patch
>
>
> HBASE-15507 introduced an update_peer_config method to allow online changes 
> to replication config. It depended on HBASE-11393, which so far exists only 
> in master and can only be in 2.0 and above because of some incompatible 
> changes to replication peer config serialization. 
> To get this in branch-1 will require two things:
> 1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
> incompatible tableCF serialization changes.
> 2. Port HBASE-15507's logic on top of this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15633) Backport HBASE-15507 to branch-1

[Geoffrey Jacoby (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby updated HBASE-15633:

Status: Open  (was: Patch Available)

> Backport HBASE-15507 to branch-1
> 
>
> Key: HBASE-15633
> URL: https://issues.apache.org/jira/browse/HBASE-15633
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication, shell
>Affects Versions: 1.3.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
> Attachments: HBASE-15633-branch-1.patch
>
>
> HBASE-15507 introduced an update_peer_config method to allow online changes 
> to replication config. It depended on HBASE-11393, which so far exists only 
> in master and can only be in 2.0 and above because of some incompatible 
> changes to replication peer config serialization. 
> To get this in branch-1 will require two things:
> 1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
> incompatible tableCF serialization changes.
> 2. Port HBASE-15507's logic on top of this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15454) Archive store files older than max age

[Clara Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236572#comment-15236572
 ] 

Clara Xiong commented on HBASE-15454:
-

Thanks to [~davelatham] I left some comments on RB. But I have the same bigger 
questions you asked.

In general, I like the idea of extending window class but I am concerned about 
mingling needsCompaction/needsArchive and selectCompaction/selectArchive. DTCP 
has already provided a way to archive data, except that it is not by calendar 
windows. I understand for some use cases, people may want calendar windows than 
exponential ones. Extending window class is a good idea. If that is not enough, 
we can have a separate policy that takes calendar windows for minor, major and 
archive. It can share some logic from DTCP. 

> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454.patch
>
>
> Sometimes the old data is rarely touched but we can not remove it. So archive 
> it to several big files(by year or something) and use EC to reduce the 
> redundancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14985) TimeRange constructors should set allTime when appropriate

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236571#comment-15236571
 ] 

Hudson commented on HBASE-14985:


FAILURE: Integrated in HBase-Trunk_matrix #840 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/840/])
HBASE-14985 TimeRange constructors should set allTime when appropriate (tedyu: 
rev ff9c92e16831fe350904ac99f92619fb97ba2bef)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTimeRangeTracker.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java


> TimeRange constructors should set allTime when appropriate
> --
>
> Key: HBASE-14985
> URL: https://issues.apache.org/jira/browse/HBASE-14985
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.1.3, 0.98.17
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-14985-v1.patch, HBASE-14985-v1.patch, 
> HBASE-14985.patch
>
>
> The default TimeRange constructor creates a range from 0 to Long.MAX_VALUE 
> and sets an allTime flag to true. This flag allows some performance 
> optimizations when comparing or using TimeRanges.
> This flag is not set, however, if you call "new TimeRange(0L)" or "new 
> TimeRange(0L, Long.MAX_VALUE)", even though both of these create a logically 
> equivalent TimeRange to "new TimeRange()". Since TimeRanges are immutable and 
> detecting this condition is trivial, we should set the flag automatically in 
> the explicit constructors when appropriate. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15634) TestDateTieredCompactionPolicy#negativeForMajor is flaky

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15634:
---
Description: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1365/artifact/patchprocess/patch-unit-hbase-server.txt
 :
{code}
negativeForMajor(org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy)
  Time elapsed: 0.48 sec  <<< FAILURE!
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy.compactEquals(TestDateTieredCompactionPolicy.java:94)
at 
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy.negativeForMajor(TestDateTieredCompactionPolicy.java:301)
{code}
Similar test failure occurred in master JDK 8 build as well 
(https://builds.apache.org/job/HBase-TRUNK_matrix/839/jdk=latest1.8,label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.regionserver/TestDateTieredCompactionPolicy/negativeForMajor/).

Since TestDateTieredCompactionPolicy is a small test, its failure prevented 
large tests from running.

  was:
https://builds.apache.org/job/PreCommit-HBASE-Build/1365/artifact/patchprocess/patch-unit-hbase-server.txt
 :
{code}
negativeForMajor(org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy)
  Time elapsed: 0.48 sec  <<< FAILURE!
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy.compactEquals(TestDateTieredCompactionPolicy.java:94)
at 
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy.negativeForMajor(TestDateTieredCompactionPolicy.java:301)
{code}
Similar test failure occurred in master JDK 8 build as well.

Since TestDateTieredCompactionPolicy is a small test, its failure prevented 
large tests from running.


> TestDateTieredCompactionPolicy#negativeForMajor is flaky
> 
>
> Key: HBASE-15634
> URL: https://issues.apache.org/jira/browse/HBASE-15634
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Affects Versions: 1.4.0
>Reporter: Ted Yu
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/1365/artifact/patchprocess/patch-unit-hbase-server.txt
>  :
> {code}
> negativeForMajor(org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy)
>   Time elapsed: 0.48 sec  <<< FAILURE!
> java.lang.AssertionError: expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy.compactEquals(TestDateTieredCompactionPolicy.java:94)
>   at 
> org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy.negativeForMajor(TestDateTieredCompactionPolicy.java:301)
> {code}
> Similar test failure occurred in master JDK 8 build as well 
> (https://builds.apache.org/job/HBase-TRUNK_matrix/839/jdk=latest1.8,label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.regionserver/TestDateTieredCompactionPolicy/negativeForMajor/).
> Since TestDateTieredCompactionPolicy is a small test, its failure prevented 
> large tests from running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15634) TestDateTieredCompactionPolicy#negativeForMajor is flaky

[Ted Yu (JIRA)]

Ted Yu created HBASE-15634:
--

 Summary: TestDateTieredCompactionPolicy#negativeForMajor is flaky
 Key: HBASE-15634
 URL: https://issues.apache.org/jira/browse/HBASE-15634
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 1.4.0
Reporter: Ted Yu


https://builds.apache.org/job/PreCommit-HBASE-Build/1365/artifact/patchprocess/patch-unit-hbase-server.txt
 :
{code}
negativeForMajor(org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy)
  Time elapsed: 0.48 sec  <<< FAILURE!
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy.compactEquals(TestDateTieredCompactionPolicy.java:94)
at 
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy.negativeForMajor(TestDateTieredCompactionPolicy.java:301)
{code}
Similar test failure occurred in master JDK 8 build as well.

Since TestDateTieredCompactionPolicy is a small test, its failure prevented 
large tests from running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15629) Backport HBASE-14703 to 0.98+

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236559#comment-15236559
 ] 

Heng Chen commented on HBASE-15629:
---

[~jesse_yates] Mind take a look?

> Backport HBASE-14703 to 0.98+
> -
>
> Key: HBASE-15629
> URL: https://issues.apache.org/jira/browse/HBASE-15629
> Project: HBase
>  Issue Type: Task
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 0.98.19
>
> Attachments: HBASE-15629-0.98.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15315) Remove always set super user call as high priority

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236554#comment-15236554
 ] 

Anoop Sam John commented on HBASE-15315:


So this jira undo the patch in HBASE-13375 completely right?

> Remove always set super user call as high priority
> --
>
> Key: HBASE-15315
> URL: https://issues.apache.org/jira/browse/HBASE-15315
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yong Zhang
>Assignee: Yong Zhang
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15315.001.patch
>
>
> Current implementation set superuser call as ADMIN_QOS, but we have many 
> customers use superuser to do normal table operation such as put/get data and 
> so on. If client put much data during region assignment, RPC from HMaster may 
> timeout because of no handle. so it is better to remove always set super user 
> call as high priority. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15633) Backport HBASE-15507 to branch-1

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236551#comment-15236551
 ] 

Hadoop QA commented on HBASE-15633:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 1s 
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 1s 
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 50s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
38s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 2s 
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 21s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
30s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
4s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 12m 
56s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 46s 
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 42s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 43s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 14s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 2s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 2s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 56s 
{color} | {color:red} hbase-client: patch generated 5 new + 11 unchanged - 0 
fixed = 16 total (was 11) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 53s 
{color} | {color:red} hbase-server: patch generated 5 new + 11 unchanged - 0 
fixed = 16 total (was 11) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 56s 
{color} | {color:red} hbase-shell: patch generated 5 new + 11 unchanged - 0 
fixed = 16 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
60m 23s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 17m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 44s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 44s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 57s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 11s {color} 
| {color:red} 

[jira] [Commented] (HBASE-15619) Performance regression observed: Empty random read(get) performance of branch-1 worse than 0.98

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236485#comment-15236485
 ] 

Yu Li commented on HBASE-15619:
---

Sharp eyes! But it's indeed the case. We checked the jstack and found some 
locks on the HDFS layer (which I guess only emerges with PCIe-SSD), and our 
people already find a way to improve that. Will link the issue here as soon as 
we create any HDFS JIRA (probably days later though).

> Performance regression observed: Empty random read(get) performance of 
> branch-1 worse than 0.98
> ---
>
> Key: HBASE-15619
> URL: https://issues.apache.org/jira/browse/HBASE-15619
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: compare.png, flamegraph-108588.098.svg, 
> flamegraph-1221.branch-1.svg, flamegraph-135684.1.1.svg
>
>
> As titled, I observed the perf regression in the final stress testing before 
> upgrading our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is 
> the most important perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>  1) 1.1.2 with important perf fixes/improvements including HBASE-15031 
> and HBASE-14465
>  2) 1.1.4 release
>  3) 1.2.1RC1
> 2. Test environment
> * YCSB: 0.7.0 with 
> [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
> * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
> 100 threads
> * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
> * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
> hardware for client and server
> 3. Test cases
> * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
> * case #1: read against empty table
> * -case #2: lrucache 100% hit-
> * -case #3: BLOCKCACHE=>false-
> 4. Test result
> * 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so 
> will only paste comparison data of 0.98.12+ and 1.1.2+
> * per-RS Throughput(ops/s)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|383562|-257493-|-47594-|
> |1.1.2+|363050|-232757-|-35872-|
> * AverageLatency(us)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|2774|-4134-|-22371-|
> |1.1.2+|2930|-4572-|-29690-|
> It seems there's perf regression on RPCServer (we tried 0.98 client against 
> 1.x server and observed a similar perf to 1.x client)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15619) Performance regression observed: Empty random read(get) performance of branch-1 worse than 0.98

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236482#comment-15236482
 ] 

Yu Li commented on HBASE-15619:
---

Thanks for the double check [~stack]!

bq. I'm thinking that us being bad at reading non-existent values is a problem 
but not a critical issue. What you think Yu Li?
Agreed, and make sense to knock it down from critical.

bq. Seems like 1.1 is about the same as 0.98 otherwise (I thought it was much 
better).
Same feeling, it's a pity, but also means we still have the space to improve. 
:-)

bq. flight recording might give better detail. I didn't do this. Let me know if 
you want me to (you'd probably be better-off doing it yourself Yu Li if bad 
performance reading an empty table is important for you).
Let me further dig into the empty reading case and get back here if any 
findings, what you've done is already very helpful, thank you sir!

> Performance regression observed: Empty random read(get) performance of 
> branch-1 worse than 0.98
> ---
>
> Key: HBASE-15619
> URL: https://issues.apache.org/jira/browse/HBASE-15619
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: compare.png, flamegraph-108588.098.svg, 
> flamegraph-1221.branch-1.svg, flamegraph-135684.1.1.svg
>
>
> As titled, I observed the perf regression in the final stress testing before 
> upgrading our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is 
> the most important perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>  1) 1.1.2 with important perf fixes/improvements including HBASE-15031 
> and HBASE-14465
>  2) 1.1.4 release
>  3) 1.2.1RC1
> 2. Test environment
> * YCSB: 0.7.0 with 
> [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
> * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
> 100 threads
> * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
> * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
> hardware for client and server
> 3. Test cases
> * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
> * case #1: read against empty table
> * -case #2: lrucache 100% hit-
> * -case #3: BLOCKCACHE=>false-
> 4. Test result
> * 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so 
> will only paste comparison data of 0.98.12+ and 1.1.2+
> * per-RS Throughput(ops/s)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|383562|-257493-|-47594-|
> |1.1.2+|363050|-232757-|-35872-|
> * AverageLatency(us)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|2774|-4134-|-22371-|
> |1.1.2+|2930|-4572-|-29690-|
> It seems there's perf regression on RPCServer (we tried 0.98 client against 
> 1.x server and observed a similar perf to 1.x client)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15454) Archive store files older than max age

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236457#comment-15236457
 ] 

Duo Zhang commented on HBASE-15454:
---

[~stack] Your comment is another big story :)

In our current design, 'Get' request on archived old files could be eliminated 
by bloom filter. And for 'Scan' request, it requires the user who construct the 
'Scan' object to set timerange on the 'Scan'.

For 'Get', I think it is possible to do some optimizations which make us touch 
the archived file as less as possible, such as read hot files first, if not 
found then open all files to read. But for 'Scan', I haven't find an easy way 
to do this without user specified timerange, and unfortunately, our customer 
uses 'Scan' much more than 'Get' :(

And it is a good idea to not exclude archived files when considering split. I 
think it could be generalized as 'do not include fold files when computing size 
for splitting'? File another issue on this?

Thanks.

> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454.patch
>
>
> Sometimes the old data is rarely touched but we can not remove it. So archive 
> it to several big files(by year or something) and use EC to reduce the 
> redundancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15454) Archive store files older than max age

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236439#comment-15236439
 ] 

Duo Zhang commented on HBASE-15454:
---

{quote}
if it's orthogonal and should be mostly independent of compaction policies.
{quote}
Mostly independent, especially on the code. In the current design, we will do 
EC in background with a RaidManager, it is transparent to HBase. But EC has 
some requirements on the file. First, large files, as large as possible. 
Second, keep the file there as long as possible. So there does have something 
to do when implementing compaction policies if you want to support EC.
In our plan, we have 3 stages.
1. Just do EC without any changes on the HDFS architecture(we are still working 
on this now).
2. Deploy an HDFS with different storage types. For example, 12 disks, 3 of 
them are SSD and 9 of them are hard disks. Data is written to SSD first and 
will be moved to hard disks when it is archived. This is still transparent  to 
HBase.
3. Deploy another HDFS whose machines are designed for storing large files 
only(24 or more disks typically, low power cpu, small memory) and move archived 
file to that HDFS. I think this will require some modifications on HBase to 
support operating on multiple HDFSes?

{quote}
I don’t see how this achieves no overlapping store files. Can you explain that 
part?
{quote}
I find the first and last files that overlapping with current archive window, 
and then compact all files between them. These makes sure that all data belongs 
to this window are contained in the output file.

{quote}
Can we instead allow the windowing algorithm to be pluggable? 
{quote}
I think it is a bit hard to this. First the configuration will become more 
complicated. And for the implementation, different window implementation is a 
bit different. For example, TieredWindow must be iterated from new to old. But 
for ArchiveWindow, you can construct them at any place of the timeline. Do you 
have a more specific design of the pluggable window?

Thanks. [~davelatham]

> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454.patch
>
>
> Sometimes the old data is rarely touched but we can not remove it. So archive 
> it to several big files(by year or something) and use EC to reduce the 
> redundancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15405) Fix PE logging and wrong defaults in help message

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236406#comment-15236406
 ] 

Hadoop QA commented on HBASE-15405:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 1s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 12s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 18s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12798137/HBASE-15405-master-v5.patch
 |
| JIRA Issue | HBASE-15405 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux pietas.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT 
Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 8964573 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions |  /home/jenkins/tools/java/jdk1.8.0:1.8.0 
/usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/1363/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  

[jira] [Commented] (HBASE-15632) Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236407#comment-15236407
 ] 

Duo Zhang commented on HBASE-15632:
---

I'm OK with it.

> Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145
> --
>
> Key: HBASE-15632
> URL: https://issues.apache.org/jira/browse/HBASE-15632
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-15632-v001.patch
>
>
> HBASE-13145 introduce the following check
> {code}
> diff --git 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
>  
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> index 215069c..8f73af5 100644
> --- 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++ 
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -1574,7 +1574,8 @@ public class HRegion implements HeapSize, 
> PropagatingConfigurationObserver { //
> */
>@VisibleForTesting
>public long getEarliestFlushTimeForAllStores() {
> -return Collections.min(lastStoreFlushTimeMap.values());
> +return lastStoreFlushTimeMap.isEmpty() ? Long.MAX_VALUE : 
> Collections.min(lastStoreFlushTimeMap
> +.values());
>}
>  {code}
> I think the reason for the check is that table creation without family is 
> allowed before HBASE-15456. With HBASE-15456, table creation without family 
> is not allowed. We have one user claimed that they run into the same 
> HRegionServer$PeriodicMemstoreFlusher exception, and the table was created 
> with family. The log was not kept so could not find more info there.  By 
> checking the code, it seems impossible. Can we undo this check so the real 
> issue is not hidden in case there is one, [~Apache9]?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14985) TimeRange constructors should set allTime when appropriate

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14985:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks for the patch, Geoffrey

> TimeRange constructors should set allTime when appropriate
> --
>
> Key: HBASE-14985
> URL: https://issues.apache.org/jira/browse/HBASE-14985
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.1.3, 0.98.17
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-14985-v1.patch, HBASE-14985-v1.patch, 
> HBASE-14985.patch
>
>
> The default TimeRange constructor creates a range from 0 to Long.MAX_VALUE 
> and sets an allTime flag to true. This flag allows some performance 
> optimizations when comparing or using TimeRanges.
> This flag is not set, however, if you call "new TimeRange(0L)" or "new 
> TimeRange(0L, Long.MAX_VALUE)", even though both of these create a logically 
> equivalent TimeRange to "new TimeRange()". Since TimeRanges are immutable and 
> detecting this condition is trivial, we should set the flag automatically in 
> the explicit constructors when appropriate. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15632) Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236346#comment-15236346
 ] 

Hadoop QA commented on HBASE-15632:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
51m 7s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 123m 33s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
36s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 216m 3s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 
|
| Timed out junit tests | 
org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
|   | 
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery |
|   | org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2 |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12798101/HBASE-15632-v001.patch
 |
| JIRA Issue | HBASE-15632 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 

[jira] [Commented] (HBASE-15627) Miss space and closing quote in AccessController#checkSystemOrSuperUser

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236344#comment-15236344
 ] 

Hudson commented on HBASE-15627:


SUCCESS: Integrated in HBase-1.1-JDK7 #1698 (See 
[https://builds.apache.org/job/HBase-1.1-JDK7/1698/])
HBASE-15627 Miss space and closing quote in (matteo.bertozzi: rev 
da021be17158e49ff41f23e914519efd80f9a0ae)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java


>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> 
>
> Key: HBASE-15627
> URL: https://issues.apache.org/jira/browse/HBASE-15627
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15627-v001.patch
>
>
>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> {code}
>   private void checkSystemOrSuperUser() throws IOException {
> // No need to check if we're not going to throw
> if (!authorizationEnabled) {
>   return;
> }
> User activeUser = getActiveUser();
> if (!Superusers.isSuperUser(activeUser)) {
> //***, miss closing ' and space in the AccessDeniedException string
>   throw new AccessDeniedException("User '" + (activeUser != null ?
> activeUser.getShortName() : "null") + "is not system or super user.");
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15619) Performance regression observed: Empty random read(get) performance of branch-1 worse than 0.98

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15619:
--
Attachment: flamegraph-108588.098.svg
flamegraph-1221.branch-1.svg
flamegraph-135684.1.1.svg

Flame graphs from when were were doing the C` workload where we were mostly 
random reading values with about 40% of the time reading nothing from hbase.

I don't think these help  much. The shape of the server has changed too much. 
Would be more useful if they'd been for the empty table. Then we might see 
where they are spending the extra time... though flight recording might give 
better detail. I didn't do this. Let me know if you want me to (you'd probably 
be better-off doing it yourself [~carp84] if bad performance reading an empty 
table is important for you). 

> Performance regression observed: Empty random read(get) performance of 
> branch-1 worse than 0.98
> ---
>
> Key: HBASE-15619
> URL: https://issues.apache.org/jira/browse/HBASE-15619
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: compare.png, flamegraph-108588.098.svg, 
> flamegraph-1221.branch-1.svg, flamegraph-135684.1.1.svg
>
>
> As titled, I observed the perf regression in the final stress testing before 
> upgrading our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is 
> the most important perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>  1) 1.1.2 with important perf fixes/improvements including HBASE-15031 
> and HBASE-14465
>  2) 1.1.4 release
>  3) 1.2.1RC1
> 2. Test environment
> * YCSB: 0.7.0 with 
> [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
> * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
> 100 threads
> * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
> * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
> hardware for client and server
> 3. Test cases
> * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
> * case #1: read against empty table
> * -case #2: lrucache 100% hit-
> * -case #3: BLOCKCACHE=>false-
> 4. Test result
> * 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so 
> will only paste comparison data of 0.98.12+ and 1.1.2+
> * per-RS Throughput(ops/s)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|383562|-257493-|-47594-|
> |1.1.2+|363050|-232757-|-35872-|
> * AverageLatency(us)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|2774|-4134-|-22371-|
> |1.1.2+|2930|-4572-|-29690-|
> It seems there's perf regression on RPCServer (we tried 0.98 client against 
> 1.x server and observed a similar perf to 1.x client)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15619) Performance regression observed: Empty random read(get) performance of branch-1 worse than 0.98

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15619:
--
Priority: Major  (was: Critical)

Knocking it down from critical. Let me know if you think different [~carp84]

> Performance regression observed: Empty random read(get) performance of 
> branch-1 worse than 0.98
> ---
>
> Key: HBASE-15619
> URL: https://issues.apache.org/jira/browse/HBASE-15619
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: compare.png
>
>
> As titled, I observed the perf regression in the final stress testing before 
> upgrading our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is 
> the most important perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>  1) 1.1.2 with important perf fixes/improvements including HBASE-15031 
> and HBASE-14465
>  2) 1.1.4 release
>  3) 1.2.1RC1
> 2. Test environment
> * YCSB: 0.7.0 with 
> [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
> * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
> 100 threads
> * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
> * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
> hardware for client and server
> 3. Test cases
> * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
> * case #1: read against empty table
> * -case #2: lrucache 100% hit-
> * -case #3: BLOCKCACHE=>false-
> 4. Test result
> * 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so 
> will only paste comparison data of 0.98.12+ and 1.1.2+
> * per-RS Throughput(ops/s)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|383562|-257493-|-47594-|
> |1.1.2+|363050|-232757-|-35872-|
> * AverageLatency(us)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|2774|-4134-|-22371-|
> |1.1.2+|2930|-4572-|-29690-|
> It seems there's perf regression on RPCServer (we tried 0.98 client against 
> 1.x server and observed a similar perf to 1.x client)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15619) Performance regression observed: Empty random read(get) performance of branch-1 worse than 0.98

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15619:
--
Attachment: compare.png

I ran loadings comparing 0.98.13 (works w/ jdk8) to tip of 1.1 and branch-1.

The diagram shows three versions of the software all undergoing 5 loadings.

 # C-Empty is running ycsb workload c against an empty table
 # load is running 20minutes of loading of table
 # A is workload 'a'; i.e. 50/50 read/write for 20minutes
 # C is workload 'c' against loaded table
 # C` is running workload c alone where we are get keys most of the time but 
fail to find values almost as much.

>From the diagram we can see:

# For empty table, indeed there is regression.
# For load phase, 1.1 and branch-1 tip are a little slower
# For workload 'A', 50/50, all are about same.
# For workload 'C' when our random read is actually fetching keys, 1.1 and 
branch-1 are about 25% better.
# For the case where we are reading values about 60% of the time and doing 
reads of non-existent values about 40% of the time, we are about 15% slower in 
1.1 and branch-1.

I'm thinking that us being bad at reading non-existent values is a problem but 
not a critical issue. What you think [~carp84]? Seems like 1.1 is about the 
same as 0.98 otherwise (I thought it was much better).


> Performance regression observed: Empty random read(get) performance of 
> branch-1 worse than 0.98
> ---
>
> Key: HBASE-15619
> URL: https://issues.apache.org/jira/browse/HBASE-15619
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: compare.png
>
>
> As titled, I observed the perf regression in the final stress testing before 
> upgrading our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is 
> the most important perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>  1) 1.1.2 with important perf fixes/improvements including HBASE-15031 
> and HBASE-14465
>  2) 1.1.4 release
>  3) 1.2.1RC1
> 2. Test environment
> * YCSB: 0.7.0 with 
> [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
> * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
> 100 threads
> * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
> * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
> hardware for client and server
> 3. Test cases
> * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
> * case #1: read against empty table
> * -case #2: lrucache 100% hit-
> * -case #3: BLOCKCACHE=>false-
> 4. Test result
> * 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so 
> will only paste comparison data of 0.98.12+ and 1.1.2+
> * per-RS Throughput(ops/s)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|383562|-257493-|-47594-|
> |1.1.2+|363050|-232757-|-35872-|
> * AverageLatency(us)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|2774|-4134-|-22371-|
> |1.1.2+|2930|-4572-|-29690-|
> It seems there's perf regression on RPCServer (we tried 0.98 client against 
> 1.x server and observed a similar perf to 1.x client)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14985) TimeRange constructors should set allTime when appropriate

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14985:
---
 Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
   2.0.0

> TimeRange constructors should set allTime when appropriate
> --
>
> Key: HBASE-14985
> URL: https://issues.apache.org/jira/browse/HBASE-14985
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.1.3, 0.98.17
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-14985-v1.patch, HBASE-14985-v1.patch, 
> HBASE-14985.patch
>
>
> The default TimeRange constructor creates a range from 0 to Long.MAX_VALUE 
> and sets an allTime flag to true. This flag allows some performance 
> optimizations when comparing or using TimeRanges.
> This flag is not set, however, if you call "new TimeRange(0L)" or "new 
> TimeRange(0L, Long.MAX_VALUE)", even though both of these create a logically 
> equivalent TimeRange to "new TimeRange()". Since TimeRanges are immutable and 
> detecting this condition is trivial, we should set the flag automatically in 
> the explicit constructors when appropriate. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15615) Wrong sleep time when RegionServerCallable need retry

[Matt Warhaftig (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236282#comment-15236282
 ] 

Matt Warhaftig commented on HBASE-15615:


Hi [~zghaobac], Thanks for finding this bug and submitting a patch.  I had 
actually glanced at this ticket before you grabbed it and noticed that some of 
the other classes implementing RetryingCallable (ex MasterCallable) do not do 
tries +1 and so was thinking about if all implementation should have tries +1.

Anyway, this is just a stream of consciousness question from my head, so please 
disregard if you know better. :-). 

Cheers,
Matt

> Wrong sleep time when RegionServerCallable need retry
> -
>
> Key: HBASE-15615
> URL: https://issues.apache.org/jira/browse/HBASE-15615
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-15615.patch
>
>
> In RpcRetryingCallerImpl, it get pause time by expectedSleep = 
> callable.sleep(pause, tries + 1); And in RegionServerCallable, it get pasue 
> time by sleep = ConnectionUtils.getPauseTime(pause, tries + 1). So tries will 
> be bumped up twice. And the pasue time is 3 * hbase.client.pause when tries 
> is 0.
> RETRY_BACKOFF = {1, 2, 3, 5, 10, 20, 40, 100, 100, 100, 100, 200, 200}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14985) TimeRange constructors should set allTime when appropriate

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236267#comment-15236267
 ] 

Hadoop QA commented on HBASE-14985:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 42s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 39s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
31m 51s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 2s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 133m 12s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 198m 18s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12798095/HBASE-14985-v1.patch |
| JIRA Issue | HBASE-14985 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency 

[jira] [Commented] (HBASE-15296) Break out writer and reader from StoreFile

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236215#comment-15236215
 ] 

Hadoop QA commented on HBASE-15296:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color} 
| {color:red} HBASE-15296 does not apply to master. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.2.1/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12798142/HBASE-15296-master-v2.patch
 |
| JIRA Issue | HBASE-15296 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/1364/console |
| Powered by | Apache Yetus 0.2.1   http://yetus.apache.org |


This message was automatically generated.



> Break out writer and reader from StoreFile
> --
>
> Key: HBASE-15296
> URL: https://issues.apache.org/jira/browse/HBASE-15296
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15296-branch-1.1.patch, 
> HBASE-15296-branch-1.2.patch, HBASE-15296-branch-1.patch, 
> HBASE-15296-master-v2.patch, HBASE-15296-master.patch
>
>
> StoreFile.java is trending to become a monolithic class, it's ~1800 lines. 
> Would it make sense to break out reader and writer (~500 lines each) into 
> separate files.
> We are doing so many different things in a single class: comparators, reader, 
> writer, other stuff; and it hurts readability a lot, to the point that just 
> reading through a piece of code require scrolling up and down to see which 
> level (reader/writer/base class level) it belongs to. These small-small 
> things really don't help while trying to understanding the code. There are 
> good reasons we don't do these often (affects existing patches, needs to be 
> done for all branches, etc). But this and a few other classes can really use 
> a single iteration of refactoring to make things a lot better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15633) Backport HBASE-15507 to branch-1

[Geoffrey Jacoby (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby updated HBASE-15633:

Release Note: Added update_peer_config to the HBase shell and 
ReplicationAdmin, and provided a callback for custom replication endpoints to 
be notified of changes to their configuration and peer data
  Status: Patch Available  (was: In Progress)

Backported PeerConfigTracker from HBASE-11393 and all of HBASE-15507 to 
branch-1. 

> Backport HBASE-15507 to branch-1
> 
>
> Key: HBASE-15633
> URL: https://issues.apache.org/jira/browse/HBASE-15633
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication, shell
>Affects Versions: 1.3.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
> Attachments: HBASE-15633-branch-1.patch
>
>
> HBASE-15507 introduced an update_peer_config method to allow online changes 
> to replication config. It depended on HBASE-11393, which so far exists only 
> in master and can only be in 2.0 and above because of some incompatible 
> changes to replication peer config serialization. 
> To get this in branch-1 will require two things:
> 1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
> incompatible tableCF serialization changes.
> 2. Port HBASE-15507's logic on top of this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15633) Backport HBASE-15507 to branch-1

[Geoffrey Jacoby (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby updated HBASE-15633:

Attachment: HBASE-15633-branch-1.patch

> Backport HBASE-15507 to branch-1
> 
>
> Key: HBASE-15633
> URL: https://issues.apache.org/jira/browse/HBASE-15633
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication, shell
>Affects Versions: 1.3.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
> Attachments: HBASE-15633-branch-1.patch
>
>
> HBASE-15507 introduced an update_peer_config method to allow online changes 
> to replication config. It depended on HBASE-11393, which so far exists only 
> in master and can only be in 2.0 and above because of some incompatible 
> changes to replication peer config serialization. 
> To get this in branch-1 will require two things:
> 1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
> incompatible tableCF serialization changes.
> 2. Port HBASE-15507's logic on top of this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15627) Miss space and closing quote in AccessController#checkSystemOrSuperUser

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236190#comment-15236190
 ] 

Hudson commented on HBASE-15627:


SUCCESS: Integrated in HBase-1.1-JDK8 #1784 (See 
[https://builds.apache.org/job/HBase-1.1-JDK8/1784/])
HBASE-15627 Miss space and closing quote in (matteo.bertozzi: rev 
da021be17158e49ff41f23e914519efd80f9a0ae)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java


>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> 
>
> Key: HBASE-15627
> URL: https://issues.apache.org/jira/browse/HBASE-15627
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15627-v001.patch
>
>
>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> {code}
>   private void checkSystemOrSuperUser() throws IOException {
> // No need to check if we're not going to throw
> if (!authorizationEnabled) {
>   return;
> }
> User activeUser = getActiveUser();
> if (!Superusers.isSuperUser(activeUser)) {
> //***, miss closing ' and space in the AccessDeniedException string
>   throw new AccessDeniedException("User '" + (activeUser != null ?
> activeUser.getShortName() : "null") + "is not system or super user.");
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15296) Break out writer and reader from StoreFile

[Appy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Appy updated HBASE-15296:
-
Attachment: HBASE-15296-master-v2.patch

> Break out writer and reader from StoreFile
> --
>
> Key: HBASE-15296
> URL: https://issues.apache.org/jira/browse/HBASE-15296
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15296-branch-1.1.patch, 
> HBASE-15296-branch-1.2.patch, HBASE-15296-branch-1.patch, 
> HBASE-15296-master-v2.patch, HBASE-15296-master.patch
>
>
> StoreFile.java is trending to become a monolithic class, it's ~1800 lines. 
> Would it make sense to break out reader and writer (~500 lines each) into 
> separate files.
> We are doing so many different things in a single class: comparators, reader, 
> writer, other stuff; and it hurts readability a lot, to the point that just 
> reading through a piece of code require scrolling up and down to see which 
> level (reader/writer/base class level) it belongs to. These small-small 
> things really don't help while trying to understanding the code. There are 
> good reasons we don't do these often (affects existing patches, needs to be 
> done for all branches, etc). But this and a few other classes can really use 
> a single iteration of refactoring to make things a lot better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15405) Fix PE logging and wrong defaults in help message

[Appy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Appy updated HBASE-15405:
-
Attachment: HBASE-15405-master-v5.patch

> Fix PE logging and wrong defaults in help message
> -
>
> Key: HBASE-15405
> URL: https://issues.apache.org/jira/browse/HBASE-15405
> Project: HBase
>  Issue Type: Bug
>  Components: Performance
>Reporter: Appy
>Assignee: Appy
>Priority: Minor
> Attachments: HBASE-15405-master-v2.patch, 
> HBASE-15405-master-v3.patch, HBASE-15405-master-v4 (1).patch, 
> HBASE-15405-master-v4 (1).patch, HBASE-15405-master-v4.patch, 
> HBASE-15405-master-v4.patch, HBASE-15405-master-v5.patch, 
> HBASE-15405-master.patch, HBASE-15405-master.patch
>
>
> Corrects wrong default values for few options in the help message.
> Final stats from multiple clients are intermingled making it hard to 
> understand. Also the logged stats aren't very machine readable. It can be 
> helpful in a daily perf testing rig which scraps logs for results.
> Example of logs before the change.
> {noformat}
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log 
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log 
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log 
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log 
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log 
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log 
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log 
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576, 
> latency mean=953.98, min=359.00, max=324050.00, stdDev=851.82, 95th=1368.00, 
> 99th=1625.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576, 
> latency mean=953.92, min=356.00, max=323394.00, stdDev=817.55, 95th=1370.00, 
> 99th=1618.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576, 
> latency mean=953.98, min=367.00, max=322745.00, stdDev=840.43, 95th=1369.00, 
> 99th=1622.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log 
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log 
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log 
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min  = 
> 375.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min  = 
> 363.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg  = 
> 953.6624126434326
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg  = 
> 953.4124526977539
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev   = 
> 781.3929776087633
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev   = 
> 742.8027916717297
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th = 
> 894.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th = 
> 894.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th = 
> 1070.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th = 
> 1071.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th = 
> 1369.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th = 
> 1369.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99th = 
> 1623.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99th = 
> 1624.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min  = 
> 372.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.9th   = 
> 3013.998000214
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg  = 
> 953.2451229095459
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.9th   = 
> 3043.998000214
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev   = 
> 725.4744472152282
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.99th  = 
> 25282.38016755
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th = 
> 895.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.99th  = 
> 

[jira] [Updated] (HBASE-15631) Backport Regionserver Groups (HBASE-6721) to branch-1

[Francis Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francis Liu updated HBASE-15631:

Attachment: HBASE-15631_1_branch-1.patch

fixed a small typo in get_rsgroup.rb. will forward port to trunk once this 
patch is ok. 

> Backport Regionserver Groups (HBASE-6721) to branch-1 
> --
>
> Key: HBASE-15631
> URL: https://issues.apache.org/jira/browse/HBASE-15631
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 1.4.0
>Reporter: Francis Liu
>Assignee: Francis Liu
> Attachments: HBASE-15631.patch, HBASE-15631_1_branch-1.patch
>
>
> Based on dev list discussion backporting region server group should not be an 
> issue as it does not: 1. destabilize the code. 2. cause backward 
> incompatibility. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15631) Backport Regionserver Groups (HBASE-6721) to branch-1

[Francis Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236152#comment-15236152
 ] 

Francis Liu commented on HBASE-15631:
-

RSGroup integration test passes on my local machine. The tests pass tho I had 
to manually restart a region server to restore the cluster between tests as the 
restart command in the test harness does not use local-regionserver.sh.

> Backport Regionserver Groups (HBASE-6721) to branch-1 
> --
>
> Key: HBASE-15631
> URL: https://issues.apache.org/jira/browse/HBASE-15631
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 1.4.0
>Reporter: Francis Liu
>Assignee: Francis Liu
> Attachments: HBASE-15631.patch, HBASE-15631_1_branch-1.patch
>
>
> Based on dev list discussion backporting region server group should not be an 
> issue as it does not: 1. destabilize the code. 2. cause backward 
> incompatibility. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15579) Remove synchronized around nonce in Procedure submit

[Stephen Yuan Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236148#comment-15236148
 ] 

Stephen Yuan Jiang commented on HBASE-15579:


fine with me either only ProcedureExecutor.java or all tests.

> Remove synchronized around nonce in Procedure submit
> 
>
> Key: HBASE-15579
> URL: https://issues.apache.org/jira/browse/HBASE-15579
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.3.0, 1.1.5, 1.2.2
>
> Attachments: HBASE-15579-v0.patch
>
>
> There is a synchronized in ProcedureExecutor.submitProcedure() that protect 
> the nonce insert. We can get rid of that and just use the nonce concurrent 
> map pufIfMissing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236139#comment-15236139
 ] 

stack commented on HBASE-15236:
---

What a dirty bug. That is some pretty amazing digging on your part [~appy] 
figuring out the root cause. You the man.

On the patch, from your finding, seems like the KeyValueScanner#getSequenceId 
was a bit wrong-headed in its original implementation... or at least, it can 
get messy as you've found trying to stamp a KVScanner with a 'sequenceId'. (A 
KVScanner 'readPoint' seems more natural but no reason two layers of KVScanners 
couldn't have same readPoint so that won't help you here).

You use the notion of priority... of one KVScanner having priority over another 
KVScanner. While 'correct', when fellas see priority in hbase codebase, they 
are thinking scheduling. Suggest you use another term... I was thinking 
'ordinal' as in the position the KVScanner holds in the tiering of 
KVScanners... but is it true that two KVScanners might return same ordinal or 
that ordinal is not always available when scanning? (memstore would be at 
position 0, so its ordinal would be zero, and so on back through the HFiles... 
though in the patch memstore is MAX_INT so I have the wrong order here).

Why is the MemStore#getPriorityID synchornized? Does the value change post 
construction of the MemStore object?

For Segments, can we not set the 'priority' on construction rather than open up 
a setPriorityID? As you do for StoreFileScanner.

Suggest you also say why we have this priority id at all in the KVScanner#get* 
method its to break ties.

If every Cell had a sequenceId, we'd not need this?





> Inconsistent cell reads over multiple bulk-loaded HFiles
> 
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
>  Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch, 
> HBASE-15236-master-v3.patch, HBASE-15236-master-v4.patch, 
> HBASE-15236-master-v5.patch, HBASE-15236-master-v6.patch, 
> HBASE-15236-master-v7.patch, HBASE-15236.patch, TestWithSingleHRegion.java, 
> tables_data.zip
>
>
>  If there are two bulkloaded hfiles in a region with same seqID, same 
> timestamps and duplicate keys*, get and scan may return different values for 
> a key. Not sure how this would happen, but one of our customer uploaded a 
> dataset with 2 files in a single region and both having same bulk load 
> timestamp. These files are small ~50M (I couldn't find any setting for max 
> file size that could lead to 2 files). The range of keys in two hfiles are 
> overlapping to some extent, but not fully (so the two files are because of 
> region merge).
> In such a case, depending on file sizes (used in 
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell 
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r" 
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same 
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a 
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv  
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6 
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1 
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached 
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables 
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2) 
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5  where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66 
> bigger in size than second hfile, then:
> Get returns 55  where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of 
> store files which is kept sorted in DefaultStoreFileManger using 
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id, 
> then by decreasing filesize. So our larger files sizes are always in the 
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek() 
> scanners (type is StoreFileScanner in this case) point to KVs which compare 
> as equal, we put back the current scanner into heap, and 

[jira] [Commented] (HBASE-15579) Remove synchronized around nonce in Procedure submit

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236122#comment-15236122
 ] 

Matteo Bertozzi commented on HBASE-15579:
-

the test wasn't supposed to be part of the patch, since doesn't really change 
things. 
do you want me to fix every assertTrue() instead of assertEquals() here or 
should I just keep the fix only ProcedureExecutor.java?

> Remove synchronized around nonce in Procedure submit
> 
>
> Key: HBASE-15579
> URL: https://issues.apache.org/jira/browse/HBASE-15579
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.3.0, 1.1.5, 1.2.2
>
> Attachments: HBASE-15579-v0.patch
>
>
> There is a synchronized in ProcedureExecutor.submitProcedure() that protect 
> the nonce insert. We can get rid of that and just use the nonce concurrent 
> map pufIfMissing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15579) Remove synchronized around nonce in Procedure submit

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236118#comment-15236118
 ] 

Matteo Bertozzi commented on HBASE-15579:
-

yeah, but the IDs are not really supposed to be user sequential. 
you'll see that especially with sub-procedures. 

e.g. create may have procId=1, and you run disable after it may have procId=10 
where the 9 missing are for assignment sub-procedures.

> Remove synchronized around nonce in Procedure submit
> 
>
> Key: HBASE-15579
> URL: https://issues.apache.org/jira/browse/HBASE-15579
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.3.0, 1.1.5, 1.2.2
>
> Attachments: HBASE-15579-v0.patch
>
>
> There is a synchronized in ProcedureExecutor.submitProcedure() that protect 
> the nonce insert. We can get rid of that and just use the nonce concurrent 
> map pufIfMissing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15579) Remove synchronized around nonce in Procedure submit

[Stephen Yuan Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236101#comment-15236101
 ] 

Stephen Yuan Jiang commented on HBASE-15579:


The draw back of this change is that we would have non-consecutive proc id - if 
nonce exists, the generated proc id would be unused.


> Remove synchronized around nonce in Procedure submit
> 
>
> Key: HBASE-15579
> URL: https://issues.apache.org/jira/browse/HBASE-15579
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.3.0, 1.1.5, 1.2.2
>
> Attachments: HBASE-15579-v0.patch
>
>
> There is a synchronized in ProcedureExecutor.submitProcedure() that protect 
> the nonce insert. We can get rid of that and just use the nonce concurrent 
> map pufIfMissing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15579) Remove synchronized around nonce in Procedure submit

[Stephen Yuan Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236075#comment-15236075
 ] 

Stephen Yuan Jiang commented on HBASE-15579:


There are more places in test code that needs to be changed.  For example, 
TestCreateTableProcedure#testCreateTwiceWithSameNonce.

> Remove synchronized around nonce in Procedure submit
> 
>
> Key: HBASE-15579
> URL: https://issues.apache.org/jira/browse/HBASE-15579
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.3.0, 1.1.5, 1.2.2
>
> Attachments: HBASE-15579-v0.patch
>
>
> There is a synchronized in ProcedureExecutor.submitProcedure() that protect 
> the nonce insert. We can get rid of that and just use the nonce concurrent 
> map pufIfMissing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15633) Backport HBASE-15507 to branch-1

[Geoffrey Jacoby (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby updated HBASE-15633:

Description: 
HBASE-15507 introduced an update_peer_config method to allow online changes to 
replication config. It depended on HBASE-11393, which so far exists only in 
master and can only be in 2.0 and above because of some incompatible changes to 
replication peer config serialization. 

To get this in branch-1 will require two things:
1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
incompatible tableCF serialization changes.
2. Port HBASE-15507's logic on top of this. 

  was:
HBase-15507 introduced an update_peer_config method to allow online changes to 
replication config. It depended on HBASE-11393, which so far exists only in 
master and can only be in 2.0 and above because of some incompatible changes to 
replication peer config serialization. 

To get this in branch-1 will require two things:
1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
incompatible tableCF serialization changes.
2. Port HBASE-15507's logic on top of this. 


> Backport HBASE-15507 to branch-1
> 
>
> Key: HBASE-15633
> URL: https://issues.apache.org/jira/browse/HBASE-15633
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication, shell
>Affects Versions: 1.3.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>
> HBASE-15507 introduced an update_peer_config method to allow online changes 
> to replication config. It depended on HBASE-11393, which so far exists only 
> in master and can only be in 2.0 and above because of some incompatible 
> changes to replication peer config serialization. 
> To get this in branch-1 will require two things:
> 1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
> incompatible tableCF serialization changes.
> 2. Port HBASE-15507's logic on top of this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HBASE-15633) Backport HBASE-15507 to branch-1

[Geoffrey Jacoby (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-15633 started by Geoffrey Jacoby.
---
> Backport HBASE-15507 to branch-1
> 
>
> Key: HBASE-15633
> URL: https://issues.apache.org/jira/browse/HBASE-15633
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication, shell
>Affects Versions: 1.3.0
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>
> HBASE-15507 introduced an update_peer_config method to allow online changes 
> to replication config. It depended on HBASE-11393, which so far exists only 
> in master and can only be in 2.0 and above because of some incompatible 
> changes to replication peer config serialization. 
> To get this in branch-1 will require two things:
> 1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
> incompatible tableCF serialization changes.
> 2. Port HBASE-15507's logic on top of this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15633) Backport HBASE-15507 to branch-1

[Geoffrey Jacoby (JIRA)]

Geoffrey Jacoby created HBASE-15633:
---

 Summary: Backport HBASE-15507 to branch-1
 Key: HBASE-15633
 URL: https://issues.apache.org/jira/browse/HBASE-15633
 Project: HBase
  Issue Type: New Feature
  Components: Replication, shell
Affects Versions: 1.3.0
Reporter: Geoffrey Jacoby
Assignee: Geoffrey Jacoby


HBase-15507 introduced an update_peer_config method to allow online changes to 
replication config. It depended on HBASE-11393, which so far exists only in 
master and can only be in 2.0 and above because of some incompatible changes to 
replication peer config serialization. 

To get this in branch-1 will require two things:
1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the 
incompatible tableCF serialization changes.
2. Port HBASE-15507's logic on top of this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15454) Archive store files older than max age

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236001#comment-15236001
 ] 

stack commented on HBASE-15454:
---

I just read the design doc too and had [~davelatham] questions. 

I was just talking to a user and here's what they want (I was reading this 
issue thinking I could get some of it here?)

 # Heavy write across the whole key-space. UUID for key.
 # Only data written in the last 30 days is likely to be viewed again.
 # Only in extremely rare cases, older data will be looked at. In this latter 
case, access times can be 100x those of the 30day working set.
 # All queries are random reads.. no Scanning.

We chatted about a few options. I see here there is talk of archiving old data 
only in my scenario, the data still needs to be accessible. One thought was a 
'live' table and an 'archive' table where 'live' table had configuration 
amenable to low-latency serving and then the archive table would be configured 
to carry lots of data at the expense of being slow to read. What I wanted was 
an atomic way of moving files > 30 days from one table to the other (close 
region, move, open region is all we have).

Thinking some here, would be cool if there was a category of file that was 
'cold'. These files would not participate in compactions nor in region size 
accounting (to provent split) and would be 'closed' normally so no resources 
consumed. If a seek came in for one of these files, we'd open it, satisfy the 
seek, keep it open in case a new seek came in again but otherwise, we'd let it 
close again. This or just an atomic means of moving files form online to cold 
(table) storage.

> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454.patch
>
>
> Sometimes the old data is rarely touched but we can not remove it. So archive 
> it to several big files(by year or something) and use EC to reduce the 
> redundancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15632) Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-15632:
-
Status: Patch Available  (was: Open)

> Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145
> --
>
> Key: HBASE-15632
> URL: https://issues.apache.org/jira/browse/HBASE-15632
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-15632-v001.patch
>
>
> HBASE-13145 introduce the following check
> {code}
> diff --git 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
>  
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> index 215069c..8f73af5 100644
> --- 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++ 
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -1574,7 +1574,8 @@ public class HRegion implements HeapSize, 
> PropagatingConfigurationObserver { //
> */
>@VisibleForTesting
>public long getEarliestFlushTimeForAllStores() {
> -return Collections.min(lastStoreFlushTimeMap.values());
> +return lastStoreFlushTimeMap.isEmpty() ? Long.MAX_VALUE : 
> Collections.min(lastStoreFlushTimeMap
> +.values());
>}
>  {code}
> I think the reason for the check is that table creation without family is 
> allowed before HBASE-15456. With HBASE-15456, table creation without family 
> is not allowed. We have one user claimed that they run into the same 
> HRegionServer$PeriodicMemstoreFlusher exception, and the table was created 
> with family. The log was not kept so could not find more info there.  By 
> checking the code, it seems impossible. Can we undo this check so the real 
> issue is not hidden in case there is one, [~Apache9]?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15467) Remove 1.x/2.0 TableDescriptor incompatibility

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235975#comment-15235975
 ] 

Hadoop QA commented on HBASE-15467:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m 
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 4s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 5s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s 
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 59s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | 

[jira] [Updated] (HBASE-15632) Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-15632:
-
Attachment: HBASE-15632-v001.patch

> Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145
> --
>
> Key: HBASE-15632
> URL: https://issues.apache.org/jira/browse/HBASE-15632
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-15632-v001.patch
>
>
> HBASE-13145 introduce the following check
> {code}
> diff --git 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
>  
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> index 215069c..8f73af5 100644
> --- 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++ 
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -1574,7 +1574,8 @@ public class HRegion implements HeapSize, 
> PropagatingConfigurationObserver { //
> */
>@VisibleForTesting
>public long getEarliestFlushTimeForAllStores() {
> -return Collections.min(lastStoreFlushTimeMap.values());
> +return lastStoreFlushTimeMap.isEmpty() ? Long.MAX_VALUE : 
> Collections.min(lastStoreFlushTimeMap
> +.values());
>}
>  {code}
> I think the reason for the check is that table creation without family is 
> allowed before HBASE-15456. With HBASE-15456, table creation without family 
> is not allowed. We have one user claimed that they run into the same 
> HRegionServer$PeriodicMemstoreFlusher exception, and the table was created 
> with family. The log was not kept so could not find more info there.  By 
> checking the code, it seems impossible. Can we undo this check so the real 
> issue is not hidden in case there is one, [~Apache9]?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15454) Archive store files older than max age

[Dave Latham (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235958#comment-15235958
 ] 

Dave Latham commented on HBASE-15454:
-

Clara's out for a few days.

I read over the design doc and the patch and am still trying to understand how 
this all fits together, and if this is the best way to add a new set of logic.
The design doc mentions 3 goals:
1. Use EC (to reduce the redundancy of old data)
2. No overlapping store files (to facilitate easier comparisons of data)
3. Calendar boundaries between windows (presumably for analysis, comparison, or 
operations on files to match expectations of people or other systems)

Some questions and thoughts:
* Does EC mean erasure coding?  I don't see anything related in the patch, and 
am not familiar with any existing support within HBase to take advantage of it 
with HDFS.  Does it actually relate to this work or are you just mentioning 
your intention to use this feature and EC at the same time.  I'd love to hear 
your thoughts on how you'd do it if so.  I also wonder if it's orthogonal and 
should be mostly independent of compaction policies.
* I don’t see how this achieves no overlapping store files.  Can you explain 
that part?
* Treating the archive windows as separate concept feels unnecessary to me.   
Can we instead allow the windowing algorithm to be pluggable?  The current 
implementation is exponential growing windows across tiers to some limit, but 
others could be entirely calendar based or fixed size windows.  Then to achieve 
what’s here you could implement a custom one window schedule that starts with 
exponential and then transitions to calendar based.

> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454.patch
>
>
> Sometimes the old data is rarely touched but we can not remove it. So archive 
> it to several big files(by year or something) and use EC to reduce the 
> redundancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235941#comment-15235941
 ] 

Hudson commented on HBASE-15093:


FAILURE: Integrated in HBase-Trunk_matrix #839 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/839/])
HBASE-15093 Replication can report incorrect size of log queue for the (tedyu: 
rev 8541fe4ad10737efbec3734d3ba4d835c51afa7d)
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java


> Replication can report incorrect size of log queue for the global source when 
> multiwal is enabled
> -
>
> Key: HBASE-15093
> URL: https://issues.apache.org/jira/browse/HBASE-15093
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15093-V0.patch, HBASE-15093-V1.patch, 
> HBASE-15093-V1.patch, HBASE-15093-V2.patch
>
>
> Replication can  report incorrect size for the size of log queue for the 
> global source when multiwal is enabled. This happens because the method 
> MetricsSource#setSizeofLogQueue performs non-trivial operations in a 
> multithreaded world, even though it is not synchronized. 
> We can simply divide MetricsSource#setSizeofLogQueue into 
> MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not 
> sure why we are currently directly setting the size instead of 
> incrementing/decrementing it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15627) Miss space and closing quote in AccessController#checkSystemOrSuperUser

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235940#comment-15235940
 ] 

Hudson commented on HBASE-15627:


FAILURE: Integrated in HBase-Trunk_matrix #839 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/839/])
HBASE-15627 Miss space and closing quote in (matteo.bertozzi: rev 
8964573394bcf2026bb5514c08918bd272784ed0)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java


>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> 
>
> Key: HBASE-15627
> URL: https://issues.apache.org/jira/browse/HBASE-15627
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15627-v001.patch
>
>
>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> {code}
>   private void checkSystemOrSuperUser() throws IOException {
> // No need to check if we're not going to throw
> if (!authorizationEnabled) {
>   return;
> }
> User activeUser = getActiveUser();
> if (!Superusers.isSuperUser(activeUser)) {
> //***, miss closing ' and space in the AccessDeniedException string
>   throw new AccessDeniedException("User '" + (activeUser != null ?
> activeUser.getShortName() : "null") + "is not system or super user.");
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14985) TimeRange constructors should set allTime when appropriate

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14985:
---
Attachment: HBASE-14985-v1.patch

Re-attach for QA.

> TimeRange constructors should set allTime when appropriate
> --
>
> Key: HBASE-14985
> URL: https://issues.apache.org/jira/browse/HBASE-14985
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.1.3, 0.98.17
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Minor
> Attachments: HBASE-14985-v1.patch, HBASE-14985-v1.patch, 
> HBASE-14985.patch
>
>
> The default TimeRange constructor creates a range from 0 to Long.MAX_VALUE 
> and sets an allTime flag to true. This flag allows some performance 
> optimizations when comparing or using TimeRanges.
> This flag is not set, however, if you call "new TimeRange(0L)" or "new 
> TimeRange(0L, Long.MAX_VALUE)", even though both of these create a logically 
> equivalent TimeRange to "new TimeRange()". Since TimeRanges are immutable and 
> detecting this condition is trivial, we should set the flag automatically in 
> the explicit constructors when appropriate. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15630) Improve checksum files for releases for easier verification

[Christopher Tubbs (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235887#comment-15235887
 ] 

Christopher Tubbs commented on HBASE-15630:
---

Ah, so that's how it's done. That's an interesting way to use gpg... especially 
since if a user has GPG in order to do this trick, it'd be better if they 
ignore this file and check the signature instead. The hashes seem more useful 
for people who don't have, or don't want to use, gpg. In that case, it seems 
preferable to use the standardized coreutils format.

A few other comments about the gpg output: if one wishes gpg output to be 
consistent and its output scriptable, one should specify {{ gpg --with-colons 
}}. In this case, order of output also matters, as does the supported 
algorithms. Unfortunately, the {{ gpg --with-colons }} option also substitutes 
the algorithm names with a numeric ID, which is less useful for human readers. 
That's unfortunate.

At least now I know how it's done. Thanks for the tip!

> Improve checksum files for releases for easier verification
> ---
>
> Key: HBASE-15630
> URL: https://issues.apache.org/jira/browse/HBASE-15630
> Project: HBase
>  Issue Type: Wish
>Affects Versions: 1.2.1
>Reporter: Christopher Tubbs
>Priority: Trivial
>
> Trying to verify latest release (1.2.1), and I found it a bit inconvenient to 
> parse the *.mds checksum file. The line wrapping, white space, and the 
> general format of the file does not lend itself for easy verification.
> I suggest using the standard "coreutils" format for md5sum, sha*sum, etc., 
> instead: 
> {code}
> # md5
> 3d66c0dd4f38fa881046fe64dd680a7a *hbase-1.2.1-src.tar.gz
> # sha1
> 3666a4829d9a8d9285173bfa8e8d0ff5423a22d6 *hbase-1.2.1-src.tar.gz
> # rmd160
> #fb318e84b6256492cfb990aec2238a64c2da21ad *hbase-1.2.1-src.tar.gz
> # sha224
> 89d341a55069e4875f9e6859737062fd7a4c11596811731c4ba95ca0 
> *hbase-1.2.1-src.tar.gz
> # sha256
> e8000a65e98d4c5db7bab54da99a57209fe4ea777ab41e91ae8ccf7bfa2d50dd 
> *hbase-1.2.1-src.tar.gz
> # sha384
> 49aa0620bf0fbe20bbde66cecabb76b22defb9ee609936edc3952889e6484e55c88f1c93d6258a2eaab4a9d5188b6170
>  *hbase-1.2.1-src.tar.gz
> # sha512
> 28956a35a01ae87e9f733664c52c6fd25f9a60a1ff7047bbf306cd433c2a5b863c9bf05aba1d58792b86eec9943ae00e772c4b76fb81c5d210cf256cd074189b
>  *hbase-1.2.1-src.tar.gz
> {code}
> (comment lines added for humans, but ignored by tools; commented out rmd160, 
> because not a coreutils supported algorithm; binary flag optional, could use 
> another space instead... probably only matters for some dos tools)
> This makes it very easy to verify multiple files and hashes using: {{shasum 
> -c file.mds}} or {{sha1sum -c file.mds}} or {{md5sum -c file.mds}}.
> In addition to the file format change, I suggest these two additional changes:
> 1. Drop rmd160. It's not nearly as popular as the others, and it doesn't lend 
> itself to easy verification (no coreutils equivalent command like md5sum, 
> sha1sum, etc.)
> 2. Concatenate hashes from all files into a single file. This makes it easier 
> to verify all downloads at once.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15627) Miss space and closing quote in AccessController#checkSystemOrSuperUser

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235837#comment-15235837
 ] 

Hudson commented on HBASE-15627:


SUCCESS: Integrated in HBase-1.2-IT #482 (See 
[https://builds.apache.org/job/HBase-1.2-IT/482/])
HBASE-15627 Miss space and closing quote in (matteo.bertozzi: rev 
53b94c140ab8700d2930663c2b060030a602a364)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java


>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> 
>
> Key: HBASE-15627
> URL: https://issues.apache.org/jira/browse/HBASE-15627
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15627-v001.patch
>
>
>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> {code}
>   private void checkSystemOrSuperUser() throws IOException {
> // No need to check if we're not going to throw
> if (!authorizationEnabled) {
>   return;
> }
> User activeUser = getActiveUser();
> if (!Superusers.isSuperUser(activeUser)) {
> //***, miss closing ' and space in the AccessDeniedException string
>   throw new AccessDeniedException("User '" + (activeUser != null ?
> activeUser.getShortName() : "null") + "is not system or super user.");
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15627) Miss space and closing quote in AccessController#checkSystemOrSuperUser

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235762#comment-15235762
 ] 

Hudson commented on HBASE-15627:


SUCCESS: Integrated in HBase-1.3-IT #606 (See 
[https://builds.apache.org/job/HBase-1.3-IT/606/])
HBASE-15627 Miss space and closing quote in (matteo.bertozzi: rev 
a4cd35338502b83e28318e3a93009159db3133d5)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java


>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> 
>
> Key: HBASE-15627
> URL: https://issues.apache.org/jira/browse/HBASE-15627
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15627-v001.patch
>
>
>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> {code}
>   private void checkSystemOrSuperUser() throws IOException {
> // No need to check if we're not going to throw
> if (!authorizationEnabled) {
>   return;
> }
> User activeUser = getActiveUser();
> if (!Superusers.isSuperUser(activeUser)) {
> //***, miss closing ' and space in the AccessDeniedException string
>   throw new AccessDeniedException("User '" + (activeUser != null ?
> activeUser.getShortName() : "null") + "is not system or super user.");
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235763#comment-15235763
 ] 

Hudson commented on HBASE-15093:


SUCCESS: Integrated in HBase-1.3-IT #606 (See 
[https://builds.apache.org/job/HBase-1.3-IT/606/])
HBASE-15093 Replication can report incorrect size of log queue for the (tedyu: 
rev 9473ab4bace61e9ca6f75fa4d0edb32d1b8d417e)
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java


> Replication can report incorrect size of log queue for the global source when 
> multiwal is enabled
> -
>
> Key: HBASE-15093
> URL: https://issues.apache.org/jira/browse/HBASE-15093
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15093-V0.patch, HBASE-15093-V1.patch, 
> HBASE-15093-V1.patch, HBASE-15093-V2.patch
>
>
> Replication can  report incorrect size for the size of log queue for the 
> global source when multiwal is enabled. This happens because the method 
> MetricsSource#setSizeofLogQueue performs non-trivial operations in a 
> multithreaded world, even though it is not synchronized. 
> We can simply divide MetricsSource#setSizeofLogQueue into 
> MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not 
> sure why we are currently directly setting the size instead of 
> incrementing/decrementing it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot

[Vladimir Rodionov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235744#comment-15235744
 ] 

Vladimir Rodionov commented on HBASE-7912:
--

{quote}
Isn't the justification above enough? We should have consistency in the file 
layout, no matter it is root directory, exported snapshots or backups. Even 
between backup types, full and increment have different layout making it a 
maintenance burden for no reason.
{quote}
[~enis], what maintenance burden are referring to?

{quote}
Plus there are even more things to consider:
For incremental backups, lacking the extra directory level for ServerName means 
that every WAL in the backup will end up in the same directory. For 1000 
servers, where every servers rolls the WAL every half an hour, that is 48K WAL 
files per day. If you have incremental backup every week, it is 336K files 
under one directory. We already know that the NN will fail with this.
{quote}

This looks like artificial number manipulation. Nobody runs incremental backups 
once a week and increasing WAL size from default 128MB can help as well. 
Although I agree: having additional subdirectory is better.

{quote}
In the full backup layout, we are repeating the ns + table for no good reason 
at all.
{quote}

We are not repeating, ns is upper directory level, similar to hbase layout. I 
am not sure I am following you here, [~enis].

> HBase Backup/Restore Based on HBase Snapshot
> 
>
> Key: HBASE-7912
> URL: https://issues.apache.org/jira/browse/HBASE-7912
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Richard Ding
>Assignee: Vladimir Rodionov
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf, 
> HBaseBackupRestore-Jira-7912-DesignDoc-v2.pdf, 
> HBaseBackupRestore-Jira-7912-v4.pdf, HBaseBackupRestore-Jira-7912-v5 .pdf, 
> HBaseBackupRestore-Jira-7912-v6.pdf, HBaseBackupandRestore.pdf, 
> HBase_BackupRestore-Jira-7912-CLI-v1.pdf
>
>
> Finally, we completed the implementation of our backup/restore solution, and 
> would like to share with community through this jira. 
> We are leveraging existing hbase snapshot feature, and provide a general 
> solution to common users. Our full backup is using snapshot to capture 
> metadata locally and using exportsnapshot to move data to another cluster; 
> the incremental backup is using offline-WALplayer to backup HLogs; we also 
> leverage global distribution rolllog and flush to improve performance; other 
> added-on values such as convert, merge, progress report, and CLI commands. So 
> that a common user can backup hbase data without in-depth knowledge of hbase. 
>  Our solution also contains some usability features for enterprise users. 
> The detail design document and CLI command will be attached in this jira. We 
> plan to use 10~12 subtasks to share each of the following features, and 
> document the detail implement in the subtasks: 
> * *Full Backup* : provide local and remote back/restore for a list of tables
> * *offline-WALPlayer* to convert HLog to HFiles offline (for incremental 
> backup)
> * *distributed* Logroll and distributed flush 
> * Backup *Manifest* and history
> * *Incremental* backup: to build on top of full backup as daily/weekly backup 
> * *Convert*  incremental backup WAL files into hfiles
> * *Merge* several backup images into one(like merge weekly into monthly)
> * *add and remove* table to and from Backup image
> * *Cancel* a backup process
> * backup progress *status*
> * full backup based on *existing snapshot*
> *-*
> *Below is the original description, to keep here as the history for the 
> design and discussion back in 2013*
> There have been attempts in the past to come up with a viable HBase 
> backup/restore solution (e.g., HBASE-4618).  Recently, there are many 
> advancements and new features in HBase, for example, FileLink, Snapshot, and 
> Distributed Barrier Procedure. This is a proposal for a backup/restore 
> solution that utilizes these new features to achieve better performance and 
> consistency. 
>  
> A common practice of backup and restore in database is to first take full 
> baseline backup, and then periodically take incremental backup that capture 
> the changes since the full baseline backup. HBase cluster can store massive 
> amount data.  Combination of full backups with incremental backups has 
> tremendous benefit for HBase as well.  The following is a typical scenario 
> for full and incremental backup.
> # The user takes a full backup of a table or a set of tables in HBase. 
> # The user schedules periodical incremental backups to capture the changes 
> from the full backup, or from last 

[jira] [Commented] (HBASE-15454) Archive store files older than max age

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235741#comment-15235741
 ] 

Ted Yu commented on HBASE-15454:


[~claraxiong]:
Can you take a look at the patch ?

> Archive store files older than max age
> --
>
> Key: HBASE-15454
> URL: https://issues.apache.org/jira/browse/HBASE-15454
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15454-v1.patch, HBASE-15454.patch
>
>
> Sometimes the old data is rarely touched but we can not remove it. So archive 
> it to several big files(by year or something) and use EC to reduce the 
> redundancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15627) Miss space and closing quote in AccessController#checkSystemOrSuperUser

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235715#comment-15235715
 ] 

Hudson commented on HBASE-15627:


SUCCESS: Integrated in HBase-1.2 #601 (See 
[https://builds.apache.org/job/HBase-1.2/601/])
HBASE-15627 Miss space and closing quote in (matteo.bertozzi: rev 
53b94c140ab8700d2930663c2b060030a602a364)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java


>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> 
>
> Key: HBASE-15627
> URL: https://issues.apache.org/jira/browse/HBASE-15627
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15627-v001.patch
>
>
>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> {code}
>   private void checkSystemOrSuperUser() throws IOException {
> // No need to check if we're not going to throw
> if (!authorizationEnabled) {
>   return;
> }
> User activeUser = getActiveUser();
> if (!Superusers.isSuperUser(activeUser)) {
> //***, miss closing ' and space in the AccessDeniedException string
>   throw new AccessDeniedException("User '" + (activeUser != null ?
> activeUser.getShortName() : "null") + "is not system or super user.");
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15619) Performance regression observed: Empty random read(get) performance of branch-1 worse than 0.98

[Vladimir Rodionov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235702#comment-15235702
 ] 

Vladimir Rodionov commented on HBASE-15619:
---

With hardware:
{quote}
Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
hardware for client and server
{quote}

only 47K reads from file system? for 3 RS? 15.7K per server? for PCIe-SSD? That 
is ridiculously low number.  

> Performance regression observed: Empty random read(get) performance of 
> branch-1 worse than 0.98
> ---
>
> Key: HBASE-15619
> URL: https://issues.apache.org/jira/browse/HBASE-15619
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
>
> As titled, I observed the perf regression in the final stress testing before 
> upgrading our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is 
> the most important perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>  1) 1.1.2 with important perf fixes/improvements including HBASE-15031 
> and HBASE-14465
>  2) 1.1.4 release
>  3) 1.2.1RC1
> 2. Test environment
> * YCSB: 0.7.0 with 
> [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
> * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
> 100 threads
> * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
> * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
> hardware for client and server
> 3. Test cases
> * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
> * case #1: read against empty table
> * -case #2: lrucache 100% hit-
> * -case #3: BLOCKCACHE=>false-
> 4. Test result
> * 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so 
> will only paste comparison data of 0.98.12+ and 1.1.2+
> * per-RS Throughput(ops/s)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|383562|-257493-|-47594-|
> |1.1.2+|363050|-232757-|-35872-|
> * AverageLatency(us)
> ||HBaseVersion||case#1||-case#2-||-case#3-||
> |0.98.12+|2774|-4134-|-22371-|
> |1.1.2+|2930|-4572-|-29690-|
> It seems there's perf regression on RPCServer (we tried 0.98 client against 
> 1.x server and observed a similar perf to 1.x client)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15631) Backport Regionserver Groups (HBASE-6721) to branch-1

[Francis Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235692#comment-15235692
 ] 

Francis Liu commented on HBASE-15631:
-

Forgot to mention I had to add a small change to LocalHBaseCluster. Since it's 
preventing a test from running two masters due to a regionserver info port 
conflict. 

> Backport Regionserver Groups (HBASE-6721) to branch-1 
> --
>
> Key: HBASE-15631
> URL: https://issues.apache.org/jira/browse/HBASE-15631
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 1.4.0
>Reporter: Francis Liu
>Assignee: Francis Liu
> Attachments: HBASE-15631.patch
>
>
> Based on dev list discussion backporting region server group should not be an 
> issue as it does not: 1. destabilize the code. 2. cause backward 
> incompatibility. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15631) Backport Regionserver Groups (HBASE-6721) to branch-1

[Francis Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francis Liu updated HBASE-15631:

Attachment: HBASE-15631.patch

Working patch. Backport mainly involved using different apis that got moved or 
deprecated. I also removed a try-catch block in assignmentmanager as it seemed 
redundant something we can alsoremove in trunk actually. No significant code 
changes. Need to run IT tests. 

> Backport Regionserver Groups (HBASE-6721) to branch-1 
> --
>
> Key: HBASE-15631
> URL: https://issues.apache.org/jira/browse/HBASE-15631
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 1.4.0
>Reporter: Francis Liu
>Assignee: Francis Liu
> Attachments: HBASE-15631.patch
>
>
> Based on dev list discussion backporting region server group should not be an 
> issue as it does not: 1. destabilize the code. 2. cause backward 
> incompatibility. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15632) Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235673#comment-15235673
 ] 

Matteo Bertozzi commented on HBASE-15632:
-

In HBASE-13145 the check was added because the test was creating the table 
without families, which was later fixed by HBASE-15456.

lastStoreFlushTimeMap should contains at least 1 value (N families) after 
HRegion.initialize()
the PeriodicMemstoreFlusher should only run on the region, if the region is 
marked online. 
region online should happen after HRegion.initialize() is called.

so, I'm +1 on removing it. we should remove that check and avoid to hide the 
problem if there is any case where we add the region to the online servers 
before calling initialize()

> Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145
> --
>
> Key: HBASE-15632
> URL: https://issues.apache.org/jira/browse/HBASE-15632
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
>
> HBASE-13145 introduce the following check
> {code}
> diff --git 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
>  
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> index 215069c..8f73af5 100644
> --- 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++ 
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -1574,7 +1574,8 @@ public class HRegion implements HeapSize, 
> PropagatingConfigurationObserver { //
> */
>@VisibleForTesting
>public long getEarliestFlushTimeForAllStores() {
> -return Collections.min(lastStoreFlushTimeMap.values());
> +return lastStoreFlushTimeMap.isEmpty() ? Long.MAX_VALUE : 
> Collections.min(lastStoreFlushTimeMap
> +.values());
>}
>  {code}
> I think the reason for the check is that table creation without family is 
> allowed before HBASE-15456. With HBASE-15456, table creation without family 
> is not allowed. We have one user claimed that they run into the same 
> HRegionServer$PeriodicMemstoreFlusher exception, and the table was created 
> with family. The log was not kept so could not find more info there.  By 
> checking the code, it seems impossible. Can we undo this check so the real 
> issue is not hidden in case there is one, [~Apache9]?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15627) Miss space and closing quote in AccessController#checkSystemOrSuperUser

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235671#comment-15235671
 ] 

Hudson commented on HBASE-15627:


SUCCESS: Integrated in HBase-1.4 #80 (See 
[https://builds.apache.org/job/HBase-1.4/80/])
HBASE-15627 Miss space and closing quote in (matteo.bertozzi: rev 
2dbbe8960a1e07d109dae8affe326d3aa3acca76)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java


>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> 
>
> Key: HBASE-15627
> URL: https://issues.apache.org/jira/browse/HBASE-15627
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15627-v001.patch
>
>
>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> {code}
>   private void checkSystemOrSuperUser() throws IOException {
> // No need to check if we're not going to throw
> if (!authorizationEnabled) {
>   return;
> }
> User activeUser = getActiveUser();
> if (!Superusers.isSuperUser(activeUser)) {
> //***, miss closing ' and space in the AccessDeniedException string
>   throw new AccessDeniedException("User '" + (activeUser != null ?
> activeUser.getShortName() : "null") + "is not system or super user.");
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235672#comment-15235672
 ] 

Hudson commented on HBASE-15093:


SUCCESS: Integrated in HBase-1.4 #80 (See 
[https://builds.apache.org/job/HBase-1.4/80/])
HBASE-15093 Replication can report incorrect size of log queue for the (tedyu: 
rev b7502feff3259dcfb27ce28838122fb1b240eede)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java


> Replication can report incorrect size of log queue for the global source when 
> multiwal is enabled
> -
>
> Key: HBASE-15093
> URL: https://issues.apache.org/jira/browse/HBASE-15093
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15093-V0.patch, HBASE-15093-V1.patch, 
> HBASE-15093-V1.patch, HBASE-15093-V2.patch
>
>
> Replication can  report incorrect size for the size of log queue for the 
> global source when multiwal is enabled. This happens because the method 
> MetricsSource#setSizeofLogQueue performs non-trivial operations in a 
> multithreaded world, even though it is not synchronized. 
> We can simply divide MetricsSource#setSizeofLogQueue into 
> MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not 
> sure why we are currently directly setting the size instead of 
> incrementing/decrementing it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15632) Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145

[huaxiang sun (JIRA)]

huaxiang sun created HBASE-15632:


 Summary: Undo the checking of lastStoreFlushTimeMap.isEmpty() 
introduced in HBASE-13145
 Key: HBASE-15632
 URL: https://issues.apache.org/jira/browse/HBASE-15632
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Affects Versions: 2.0.0
Reporter: huaxiang sun
Assignee: huaxiang sun
Priority: Minor


HBASE-13145 introduce the following check
{code}
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index 215069c..8f73af5 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1574,7 +1574,8 @@ public class HRegion implements HeapSize, 
PropagatingConfigurationObserver { //
*/
   @VisibleForTesting
   public long getEarliestFlushTimeForAllStores() {
-return Collections.min(lastStoreFlushTimeMap.values());
+return lastStoreFlushTimeMap.isEmpty() ? Long.MAX_VALUE : 
Collections.min(lastStoreFlushTimeMap
+.values());
   }
 {code}

I think the reason for the check is that table creation without family is 
allowed before HBASE-15456. With HBASE-15456, table creation without family is 
not allowed. We have one user claimed that they run into the same 
HRegionServer$PeriodicMemstoreFlusher exception, and the table was created with 
family. The log was not kept so could not find more info there.  By checking 
the code, it seems impossible. Can we undo this check so the real issue is not 
hidden in case there is one, [~Apache9]?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15631) Backport Regionserver Groups (HBASE-6721) to branch-1

[Francis Liu (JIRA)]

Francis Liu created HBASE-15631:
---

 Summary: Backport Regionserver Groups (HBASE-6721) to branch-1 
 Key: HBASE-15631
 URL: https://issues.apache.org/jira/browse/HBASE-15631
 Project: HBase
  Issue Type: New Feature
Affects Versions: 1.4.0
Reporter: Francis Liu
Assignee: Francis Liu


Based on dev list discussion backporting region server group should not be an 
issue as it does not: 1. destabilize the code. 2. cause backward 
incompatibility. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15624) Move master branch/hbase-2.0.0 to jdk-8 only

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235648#comment-15235648
 ] 

Sean Busbey commented on HBASE-15624:
-

As a part of raising the visibility of this decision, could we start adding 
HBase 2.0.0 to our prereq lists in the ref guide? With a disclaimer of 
"tentative" or some such?

Maybe also a note to user@ once that change is live on the website?

> Move master branch/hbase-2.0.0 to jdk-8 only
> 
>
> Key: HBASE-15624
> URL: https://issues.apache.org/jira/browse/HBASE-15624
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Set build and pom target jvm version as jdk8. We chatted about it here: 
> http://osdir.com/ml/general/2016-04/msg09691.html Set it as blocker on 2.0.0.
> We need to work on YETUS-369 before we can finish up this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15627) Miss space and closing quote in AccessController#checkSystemOrSuperUser

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235645#comment-15235645
 ] 

Hudson commented on HBASE-15627:


FAILURE: Integrated in HBase-1.3 #643 (See 
[https://builds.apache.org/job/HBase-1.3/643/])
HBASE-15627 Miss space and closing quote in (matteo.bertozzi: rev 
a4cd35338502b83e28318e3a93009159db3133d5)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java


>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> 
>
> Key: HBASE-15627
> URL: https://issues.apache.org/jira/browse/HBASE-15627
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15627-v001.patch
>
>
>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> {code}
>   private void checkSystemOrSuperUser() throws IOException {
> // No need to check if we're not going to throw
> if (!authorizationEnabled) {
>   return;
> }
> User activeUser = getActiveUser();
> if (!Superusers.isSuperUser(activeUser)) {
> //***, miss closing ' and space in the AccessDeniedException string
>   throw new AccessDeniedException("User '" + (activeUser != null ?
> activeUser.getShortName() : "null") + "is not system or super user.");
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235646#comment-15235646
 ] 

Hudson commented on HBASE-15093:


FAILURE: Integrated in HBase-1.3 #643 (See 
[https://builds.apache.org/job/HBase-1.3/643/])
HBASE-15093 Replication can report incorrect size of log queue for the (tedyu: 
rev 9473ab4bace61e9ca6f75fa4d0edb32d1b8d417e)
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java


> Replication can report incorrect size of log queue for the global source when 
> multiwal is enabled
> -
>
> Key: HBASE-15093
> URL: https://issues.apache.org/jira/browse/HBASE-15093
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15093-V0.patch, HBASE-15093-V1.patch, 
> HBASE-15093-V1.patch, HBASE-15093-V2.patch
>
>
> Replication can  report incorrect size for the size of log queue for the 
> global source when multiwal is enabled. This happens because the method 
> MetricsSource#setSizeofLogQueue performs non-trivial operations in a 
> multithreaded world, even though it is not synchronized. 
> We can simply divide MetricsSource#setSizeofLogQueue into 
> MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not 
> sure why we are currently directly setting the size instead of 
> incrementing/decrementing it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15518) Add Per-Table metrics back

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235613#comment-15235613
 ] 

Hadoop QA commented on HBASE-15518:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
19s {color} | {color:green} hbase-hadoop-compat: patch generated 0 new + 0 
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
12s {color} | {color:green} hbase-hadoop2-compat: patch generated 0 new + 0 
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
12s {color} | {color:green} hbase-server: patch generated 0 new + 0 unchanged - 
1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
37s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
30m 31s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 5s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s 
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 32s 
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 34s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | 

[jira] [Updated] (HBASE-15467) Remove 1.x/2.0 TableDescriptor incompatibility

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-15467:

Attachment: HBASE-15467-v1.patch

v1 is just a rebase on HBASE-15605

> Remove 1.x/2.0 TableDescriptor incompatibility
> --
>
> Key: HBASE-15467
> URL: https://issues.apache.org/jira/browse/HBASE-15467
> Project: HBase
>  Issue Type: Bug
>  Components: master, regionserver
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-15467-v0.patch, HBASE-15467-v1.patch
>
>
> I'm experimenting with Master on "2.0" and RSs on 1.x and the first problem 
> that I get is on createTable where the Master is trying to write the HTD as 
> TableDescriptor instead of TableSchema and the RS is not able to read it.
> Since TableDescriptor does nothing for now. I'd say we can remove it. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15621) Suppress Hbase SnapshotHFile cleaner error messages when a snaphot is going on

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235588#comment-15235588
 ] 

Matteo Bertozzi commented on HBASE-15621:
-

+1

> Suppress Hbase SnapshotHFile cleaner error  messages when a snaphot is going 
> on
> ---
>
> Key: HBASE-15621
> URL: https://issues.apache.org/jira/browse/HBASE-15621
> Project: HBase
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15621-v001.patch
>
>
> Run into the following exception when a snapshot is going on.
> partial file of region-manifest and data.manifest could be read and parsed by 
> the cleaner which results in InvalidProtocolBufferException, which needs to 
> be ignored for in-progress snapshot.
> {code}
> 2016-04-01 00:31:50,200 ERROR 
> org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner: Exception while 
> checking if: *** was valid, keeping it just in case.
> com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol 
> message, the input ended unexpectedly in the middle of a field.  This could 
> mean either than the input has been truncated or that an embedded message 
> misreported its own length.
> at 
> com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:70)
> at 
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:746)
> at 
> com.google.protobuf.CodedInputStream.readRawByte(CodedInputStream.java:769)
> at 
> com.google.protobuf.CodedInputStream.readRawVarint64(CodedInputStream.java:462)
> at 
> com.google.protobuf.CodedInputStream.readUInt64(CodedInputStream.java:188)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile.(SnapshotProtos.java:1331)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile.(SnapshotProtos.java:1263)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile$1.parsePartialFrom(SnapshotProtos.java:1364)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile$1.parsePartialFrom(SnapshotProtos.java:1359)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles.(SnapshotProtos.java:2161)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles.(SnapshotProtos.java:2103)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(SnapshotProtos.java:2197)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(SnapshotProtos.java:2192)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1165)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at 
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at 
> 

[jira] [Reopened] (HBASE-15621) Suppress Hbase SnapshotHFile cleaner error messages when a snaphot is going on

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi reopened HBASE-15621:
-

> Suppress Hbase SnapshotHFile cleaner error  messages when a snaphot is going 
> on
> ---
>
> Key: HBASE-15621
> URL: https://issues.apache.org/jira/browse/HBASE-15621
> Project: HBase
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15621-v001.patch
>
>
> Run into the following exception when a snapshot is going on.
> partial file of region-manifest and data.manifest could be read and parsed by 
> the cleaner which results in InvalidProtocolBufferException, which needs to 
> be ignored for in-progress snapshot.
> {code}
> 2016-04-01 00:31:50,200 ERROR 
> org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner: Exception while 
> checking if: *** was valid, keeping it just in case.
> com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol 
> message, the input ended unexpectedly in the middle of a field.  This could 
> mean either than the input has been truncated or that an embedded message 
> misreported its own length.
> at 
> com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:70)
> at 
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:746)
> at 
> com.google.protobuf.CodedInputStream.readRawByte(CodedInputStream.java:769)
> at 
> com.google.protobuf.CodedInputStream.readRawVarint64(CodedInputStream.java:462)
> at 
> com.google.protobuf.CodedInputStream.readUInt64(CodedInputStream.java:188)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile.(SnapshotProtos.java:1331)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile.(SnapshotProtos.java:1263)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile$1.parsePartialFrom(SnapshotProtos.java:1364)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile$1.parsePartialFrom(SnapshotProtos.java:1359)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles.(SnapshotProtos.java:2161)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles.(SnapshotProtos.java:2103)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(SnapshotProtos.java:2197)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(SnapshotProtos.java:2192)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1165)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at 
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
> at 
> 

[jira] [Updated] (HBASE-15627) Miss space and closing quote in AccessController#checkSystemOrSuperUser

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-15627:

   Resolution: Fixed
Fix Version/s: 1.2.2
   1.1.5
   0.98.19
   1.3.0
   2.0.0
   Status: Resolved  (was: Patch Available)

>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> 
>
> Key: HBASE-15627
> URL: https://issues.apache.org/jira/browse/HBASE-15627
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15627-v001.patch
>
>
>  Miss space and closing quote in AccessController#checkSystemOrSuperUser
> {code}
>   private void checkSystemOrSuperUser() throws IOException {
> // No need to check if we're not going to throw
> if (!authorizationEnabled) {
>   return;
> }
> User activeUser = getActiveUser();
> if (!Superusers.isSuperUser(activeUser)) {
> //***, miss closing ' and space in the AccessDeniedException string
>   throw new AccessDeniedException("User '" + (activeUser != null ?
> activeUser.getShortName() : "null") + "is not system or super user.");
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15621) Suppress Hbase SnapshotHFile cleaner error messages when a snaphot is going on

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-15621:

Status: Patch Available  (was: Reopened)

> Suppress Hbase SnapshotHFile cleaner error  messages when a snaphot is going 
> on
> ---
>
> Key: HBASE-15621
> URL: https://issues.apache.org/jira/browse/HBASE-15621
> Project: HBase
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15621-v001.patch
>
>
> Run into the following exception when a snapshot is going on.
> partial file of region-manifest and data.manifest could be read and parsed by 
> the cleaner which results in InvalidProtocolBufferException, which needs to 
> be ignored for in-progress snapshot.
> {code}
> 2016-04-01 00:31:50,200 ERROR 
> org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner: Exception while 
> checking if: *** was valid, keeping it just in case.
> com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol 
> message, the input ended unexpectedly in the middle of a field.  This could 
> mean either than the input has been truncated or that an embedded message 
> misreported its own length.
> at 
> com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:70)
> at 
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:746)
> at 
> com.google.protobuf.CodedInputStream.readRawByte(CodedInputStream.java:769)
> at 
> com.google.protobuf.CodedInputStream.readRawVarint64(CodedInputStream.java:462)
> at 
> com.google.protobuf.CodedInputStream.readUInt64(CodedInputStream.java:188)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile.(SnapshotProtos.java:1331)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile.(SnapshotProtos.java:1263)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile$1.parsePartialFrom(SnapshotProtos.java:1364)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile$1.parsePartialFrom(SnapshotProtos.java:1359)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles.(SnapshotProtos.java:2161)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles.(SnapshotProtos.java:2103)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(SnapshotProtos.java:2197)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(SnapshotProtos.java:2192)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1165)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at 
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at 
> 

[jira] [Commented] (HBASE-15586) Unify human readable numbers in the web UI

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235572#comment-15235572
 ] 

Sean Busbey commented on HBASE-15586:
-

Thanks for the ping. Unless [~ndimiduk] strongly wants it in 1.1.z, I'd just as 
soon leave it in 1.3+.

> Unify human readable numbers in the web UI 
> ---
>
> Key: HBASE-15586
> URL: https://issues.apache.org/jira/browse/HBASE-15586
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: Screen Shot 2016-04-01 at 3.52.00 PM.png, Screen Shot 
> 2016-04-01 at 3.52.12 PM.png, Screen Shot 2016-04-01 at 3.52.32 PM.png, 
> Screen Shot 2016-04-01 at 3.52.38 PM.png, Screen Shot 2016-04-01 at 3.52.49 
> PM.png, hbase-15586-v1.patch, hbase-15586-v1.patch, hbase-15586-v2.patch
>
>
> I was looking at something else in the regionserver web ui trying to 
> understand the WAL size, and we are reporting that as raw bytes, not in MB / 
> GB range. Did a quick sweep to unify all the human-readable representations 
> in the UI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15586) Unify human readable numbers in the web UI

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-15586:

Issue Type: Improvement  (was: Bug)

> Unify human readable numbers in the web UI 
> ---
>
> Key: HBASE-15586
> URL: https://issues.apache.org/jira/browse/HBASE-15586
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: Screen Shot 2016-04-01 at 3.52.00 PM.png, Screen Shot 
> 2016-04-01 at 3.52.12 PM.png, Screen Shot 2016-04-01 at 3.52.32 PM.png, 
> Screen Shot 2016-04-01 at 3.52.38 PM.png, Screen Shot 2016-04-01 at 3.52.49 
> PM.png, hbase-15586-v1.patch, hbase-15586-v1.patch, hbase-15586-v2.patch
>
>
> I was looking at something else in the regionserver web ui trying to 
> understand the WAL size, and we are reporting that as raw bytes, not in MB / 
> GB range. Did a quick sweep to unify all the human-readable representations 
> in the UI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15630) Improve checksum files for releases for easier verification

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235525#comment-15235525
 ] 

Elliott Clark commented on HBASE-15630:
---

Try this:
{code}
(gpg --print-mds hbase-1.2.1-src.tar.gz | diff - hbase-1.2.1-src.tar.gz.mds ) 
&& echo "Passed"
{code}

> Improve checksum files for releases for easier verification
> ---
>
> Key: HBASE-15630
> URL: https://issues.apache.org/jira/browse/HBASE-15630
> Project: HBase
>  Issue Type: Wish
>Affects Versions: 1.2.1
>Reporter: Christopher Tubbs
>Priority: Trivial
>
> Trying to verify latest release (1.2.1), and I found it a bit inconvenient to 
> parse the *.mds checksum file. The line wrapping, white space, and the 
> general format of the file does not lend itself for easy verification.
> I suggest using the standard "coreutils" format for md5sum, sha*sum, etc., 
> instead: 
> {code}
> # md5
> 3d66c0dd4f38fa881046fe64dd680a7a *hbase-1.2.1-src.tar.gz
> # sha1
> 3666a4829d9a8d9285173bfa8e8d0ff5423a22d6 *hbase-1.2.1-src.tar.gz
> # rmd160
> #fb318e84b6256492cfb990aec2238a64c2da21ad *hbase-1.2.1-src.tar.gz
> # sha224
> 89d341a55069e4875f9e6859737062fd7a4c11596811731c4ba95ca0 
> *hbase-1.2.1-src.tar.gz
> # sha256
> e8000a65e98d4c5db7bab54da99a57209fe4ea777ab41e91ae8ccf7bfa2d50dd 
> *hbase-1.2.1-src.tar.gz
> # sha384
> 49aa0620bf0fbe20bbde66cecabb76b22defb9ee609936edc3952889e6484e55c88f1c93d6258a2eaab4a9d5188b6170
>  *hbase-1.2.1-src.tar.gz
> # sha512
> 28956a35a01ae87e9f733664c52c6fd25f9a60a1ff7047bbf306cd433c2a5b863c9bf05aba1d58792b86eec9943ae00e772c4b76fb81c5d210cf256cd074189b
>  *hbase-1.2.1-src.tar.gz
> {code}
> (comment lines added for humans, but ignored by tools; commented out rmd160, 
> because not a coreutils supported algorithm; binary flag optional, could use 
> another space instead... probably only matters for some dos tools)
> This makes it very easy to verify multiple files and hashes using: {{shasum 
> -c file.mds}} or {{sha1sum -c file.mds}} or {{md5sum -c file.mds}}.
> In addition to the file format change, I suggest these two additional changes:
> 1. Drop rmd160. It's not nearly as popular as the others, and it doesn't lend 
> itself to easy verification (no coreutils equivalent command like md5sum, 
> sha1sum, etc.)
> 2. Concatenate hashes from all files into a single file. This makes it easier 
> to verify all downloads at once.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15615) Wrong sleep time when RegionServerCallable need retry

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235428#comment-15235428
 ] 

Hadoop QA commented on HBASE-15615:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
29m 39s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 4s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
8s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 57s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12797983/HBASE-15615.patch |
| JIRA Issue | HBASE-15615 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 8964573 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions |  /home/jenkins/tools/java/jdk1.8.0:1.8.0 
/usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79 |
| findbugs | v3.0.0 

[jira] [Updated] (HBASE-15518) Add Per-Table metrics back

[Alicia Ying Shu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alicia Ying Shu updated HBASE-15518:

Attachment: HBASE-15518-v2.patch

> Add Per-Table metrics back
> --
>
> Key: HBASE-15518
> URL: https://issues.apache.org/jira/browse/HBASE-15518
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Alicia Ying Shu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15518-v1.patch, HBASE-15518-v2.patch, 
> HBASE-15518.patch
>
>
> We used to have per-table metrics, but it was removed in some restructuring. 
> We have per-region metrics, and per-regionserver metrics, but nothing in 
> between. 
> For majority of users, per-region is too granular, they are mostly interested 
> in table level aggregates. This is especially useful in multi-tenant cases 
> where a table's disk usage, number of requests, etc can be made much more 
> visible. 
> In this jira, we'll add the basic infrastructure to add a single (or a few) 
> per-table metrics. Than we can improve on that by adding remaining metrics 
> from the region server level. 
> The plan is to NOT aggregate per-table metrics at master for now. Just 
> aggregation of per-region metrics at the per-table level for every 
> regionserver. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15630) Improve checksum files for releases for easier verification

[Christopher Tubbs (JIRA)]

Christopher Tubbs created HBASE-15630:
-

 Summary: Improve checksum files for releases for easier 
verification
 Key: HBASE-15630
 URL: https://issues.apache.org/jira/browse/HBASE-15630
 Project: HBase
  Issue Type: Wish
Affects Versions: 1.2.1
Reporter: Christopher Tubbs
Priority: Trivial


Trying to verify latest release (1.2.1), and I found it a bit inconvenient to 
parse the *.mds checksum file. The line wrapping, white space, and the general 
format of the file does not lend itself for easy verification.

I suggest using the standard "coreutils" format for md5sum, sha*sum, etc., 
instead: 
{code}
# md5
3d66c0dd4f38fa881046fe64dd680a7a *hbase-1.2.1-src.tar.gz
# sha1
3666a4829d9a8d9285173bfa8e8d0ff5423a22d6 *hbase-1.2.1-src.tar.gz
# rmd160
#fb318e84b6256492cfb990aec2238a64c2da21ad *hbase-1.2.1-src.tar.gz
# sha224
89d341a55069e4875f9e6859737062fd7a4c11596811731c4ba95ca0 *hbase-1.2.1-src.tar.gz
# sha256
e8000a65e98d4c5db7bab54da99a57209fe4ea777ab41e91ae8ccf7bfa2d50dd 
*hbase-1.2.1-src.tar.gz
# sha384
49aa0620bf0fbe20bbde66cecabb76b22defb9ee609936edc3952889e6484e55c88f1c93d6258a2eaab4a9d5188b6170
 *hbase-1.2.1-src.tar.gz
# sha512
28956a35a01ae87e9f733664c52c6fd25f9a60a1ff7047bbf306cd433c2a5b863c9bf05aba1d58792b86eec9943ae00e772c4b76fb81c5d210cf256cd074189b
 *hbase-1.2.1-src.tar.gz
{code}
(comment lines added for humans, but ignored by tools; commented out rmd160, 
because not a coreutils supported algorithm; binary flag optional, could use 
another space instead... probably only matters for some dos tools)

This makes it very easy to verify multiple files and hashes using: {{shasum -c 
file.mds}} or {{sha1sum -c file.mds}} or {{md5sum -c file.mds}}.

In addition to the file format change, I suggest these two additional changes:

1. Drop rmd160. It's not nearly as popular as the others, and it doesn't lend 
itself to easy verification (no coreutils equivalent command like md5sum, 
sha1sum, etc.)
2. Concatenate hashes from all files into a single file. This makes it easier 
to verify all downloads at once.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15630) Improve checksum files for releases for easier verification

[Christopher Tubbs (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235381#comment-15235381
 ] 

Christopher Tubbs commented on HBASE-15630:
---

Out of curiosity:  what tool(s) are generating this format? I've noticed Hadoop 
checksums also suffer from these problems.

> Improve checksum files for releases for easier verification
> ---
>
> Key: HBASE-15630
> URL: https://issues.apache.org/jira/browse/HBASE-15630
> Project: HBase
>  Issue Type: Wish
>Affects Versions: 1.2.1
>Reporter: Christopher Tubbs
>Priority: Trivial
>
> Trying to verify latest release (1.2.1), and I found it a bit inconvenient to 
> parse the *.mds checksum file. The line wrapping, white space, and the 
> general format of the file does not lend itself for easy verification.
> I suggest using the standard "coreutils" format for md5sum, sha*sum, etc., 
> instead: 
> {code}
> # md5
> 3d66c0dd4f38fa881046fe64dd680a7a *hbase-1.2.1-src.tar.gz
> # sha1
> 3666a4829d9a8d9285173bfa8e8d0ff5423a22d6 *hbase-1.2.1-src.tar.gz
> # rmd160
> #fb318e84b6256492cfb990aec2238a64c2da21ad *hbase-1.2.1-src.tar.gz
> # sha224
> 89d341a55069e4875f9e6859737062fd7a4c11596811731c4ba95ca0 
> *hbase-1.2.1-src.tar.gz
> # sha256
> e8000a65e98d4c5db7bab54da99a57209fe4ea777ab41e91ae8ccf7bfa2d50dd 
> *hbase-1.2.1-src.tar.gz
> # sha384
> 49aa0620bf0fbe20bbde66cecabb76b22defb9ee609936edc3952889e6484e55c88f1c93d6258a2eaab4a9d5188b6170
>  *hbase-1.2.1-src.tar.gz
> # sha512
> 28956a35a01ae87e9f733664c52c6fd25f9a60a1ff7047bbf306cd433c2a5b863c9bf05aba1d58792b86eec9943ae00e772c4b76fb81c5d210cf256cd074189b
>  *hbase-1.2.1-src.tar.gz
> {code}
> (comment lines added for humans, but ignored by tools; commented out rmd160, 
> because not a coreutils supported algorithm; binary flag optional, could use 
> another space instead... probably only matters for some dos tools)
> This makes it very easy to verify multiple files and hashes using: {{shasum 
> -c file.mds}} or {{sha1sum -c file.mds}} or {{md5sum -c file.mds}}.
> In addition to the file format change, I suggest these two additional changes:
> 1. Drop rmd160. It's not nearly as popular as the others, and it doesn't lend 
> itself to easy verification (no coreutils equivalent command like md5sum, 
> sha1sum, etc.)
> 2. Concatenate hashes from all files into a single file. This makes it easier 
> to verify all downloads at once.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-14061) Support CF-level Storage Policy

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li reassigned HBASE-14061:
-

Assignee: Yu Li  (was: Victor Xu)

> Support CF-level Storage Policy
> ---
>
> Key: HBASE-14061
> URL: https://issues.apache.org/jira/browse/HBASE-14061
> Project: HBase
>  Issue Type: Sub-task
>  Components: HFile, regionserver
> Environment: hadoop-2.6.0
>Reporter: Victor Xu
>Assignee: Yu Li
> Attachments: HBASE-14061-master-v1.patch
>
>
> After reading [HBASE-12848|https://issues.apache.org/jira/browse/HBASE-12848] 
> and [HBASE-12934|https://issues.apache.org/jira/browse/HBASE-12934], I wrote 
> a patch to implement cf-level storage policy. 
> My main purpose is to improve random-read performance for some really hot 
> data, which usually locates in certain column family of a big table.
> Usage:
> $ hbase shell
> > alter 'TABLE_NAME', METADATA => {'hbase.hstore.block.storage.policy' => 
> > 'POLICY_NAME'}
> > alter 'TABLE_NAME', {NAME=>'CF_NAME', METADATA => 
> > {'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}}
> HDFS's setStoragePolicy can only take effect when new hfile is created in a 
> configured directory, so I had to make sub directories(for each cf) in 
> region's .tmp directory and set storage policy for them.
> Besides, I had to upgrade hadoop version to 2.6.0 because 
> dfs.getStoragePolicy cannot be easily written in reflection, and I needed 
> this api to finish my unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15619) Performance regression observed: Empty random read(get) performance of branch-1 worse than 0.98

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-15619:
--
Description: 
As titled, I observed the perf regression in the final stress testing before 
upgrading our online cluster to 1.x. More details as follows:

1. HBase version in the comparison test:
  * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is the 
most important perf-related one (especially under high stress)
  * 1.x: checked 3 releases in total
 1) 1.1.2 with important perf fixes/improvements including HBASE-15031 and 
HBASE-14465
 2) 1.1.4 release
 3) 1.2.1RC1

2. Test environment
* YCSB: 0.7.0 with 
[YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
* Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
100 threads
* Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
* Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
hardware for client and server

3. Test cases
* -p fieldcount=1 -p fieldlength=128 -p readproportion=1
* case #1: read against empty table
* -case #2: lrucache 100% hit-
* -case #3: BLOCKCACHE=>false-

4. Test result
* 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so 
will only paste comparison data of 0.98.12+ and 1.1.2+
* per-RS Throughput(ops/s)
||HBaseVersion||case#1||-case#2-||-case#3-||
|0.98.12+|383562|-257493-|-47594-|
|1.1.2+|363050|-232757-|-35872-|
* AverageLatency(us)
||HBaseVersion||case#1||-case#2-||-case#3-||
|0.98.12+|2774|-4134-|-22371-|
|1.1.2+|2930|-4572-|-29690-|

It seems there's perf regression on RPCServer (we tried 0.98 client against 1.x 
server and observed a similar perf to 1.x client)

  was:
As titled, I observed the perf regression in the final stress testing before 
upgrading our online cluster to 1.x. More details as follows:

1. HBase version in the comparison test:
  * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is the 
most important perf-related one (especially under high stress)
  * 1.x: checked 3 releases in total
 1) 1.1.2 with important perf fixes/improvements including HBASE-15031 and 
HBASE-14465 (we planed to upgrade to 1.x since Oct. last year, by when 1.1.2 
was the latest stable release in branch-1)
 2) 1.1.4 release
 3) 1.2.1RC1

2. Test environment
* YCSB: 0.7.0 with 
[YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
* Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
100 threads
* Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
* Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
hardware for client and server

3. Test cases
* -p fieldcount=1 -p fieldlength=128 -p readproportion=1
* case #1: read against empty table
* case #2: lrucache 100% hit
* case #3: BLOCKCACHE=>false

4. Test result
* 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so 
will only paste comparison data of 0.98.12+ and 1.1.2+
* per-RS Throughput(ops/s)
||HBaseVersion||case#1||case#2||case#3||
|0.98.12+|383562|257493|47594|
|1.1.2+|363050|232757|35872|
* AverageLatency(us)
||HBaseVersion||case#1||case#2||case#3||
|0.98.12+|2774|4134|22371|
|1.1.2+|2930|4572|29690|

It seems to me each part on the read path has perf regression: RPCServer for 
case#1, lrucache read for case#2, and hfile read for case#3...


> Performance regression observed: Empty random read(get) performance of 
> branch-1 worse than 0.98
> ---
>
> Key: HBASE-15619
> URL: https://issues.apache.org/jira/browse/HBASE-15619
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
>
> As titled, I observed the perf regression in the final stress testing before 
> upgrading our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is 
> the most important perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>  1) 1.1.2 with important perf fixes/improvements including HBASE-15031 
> and HBASE-14465
>  2) 1.1.4 release
>  3) 1.2.1RC1
> 2. Test environment
> * YCSB: 0.7.0 with 
> [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
> * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
> 100 threads
> * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
> * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
> hardware for client and server
> 3. Test cases
> * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
> * case #1: 

[jira] [Updated] (HBASE-15621) Suppress Hbase SnapshotHFile cleaner error messages when a snaphot is going on

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-15621:

   Resolution: Fixed
Fix Version/s: 1.2.2
   1.1.5
   0.98.19
   1.3.0
   2.0.0
   Status: Resolved  (was: Patch Available)

> Suppress Hbase SnapshotHFile cleaner error  messages when a snaphot is going 
> on
> ---
>
> Key: HBASE-15621
> URL: https://issues.apache.org/jira/browse/HBASE-15621
> Project: HBase
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.1.5, 1.2.2
>
> Attachments: HBASE-15621-v001.patch
>
>
> Run into the following exception when a snapshot is going on.
> partial file of region-manifest and data.manifest could be read and parsed by 
> the cleaner which results in InvalidProtocolBufferException, which needs to 
> be ignored for in-progress snapshot.
> {code}
> 2016-04-01 00:31:50,200 ERROR 
> org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner: Exception while 
> checking if: *** was valid, keeping it just in case.
> com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol 
> message, the input ended unexpectedly in the middle of a field.  This could 
> mean either than the input has been truncated or that an embedded message 
> misreported its own length.
> at 
> com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:70)
> at 
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:746)
> at 
> com.google.protobuf.CodedInputStream.readRawByte(CodedInputStream.java:769)
> at 
> com.google.protobuf.CodedInputStream.readRawVarint64(CodedInputStream.java:462)
> at 
> com.google.protobuf.CodedInputStream.readUInt64(CodedInputStream.java:188)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile.(SnapshotProtos.java:1331)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile.(SnapshotProtos.java:1263)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile$1.parsePartialFrom(SnapshotProtos.java:1364)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$StoreFile$1.parsePartialFrom(SnapshotProtos.java:1359)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles.(SnapshotProtos.java:2161)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles.(SnapshotProtos.java:2103)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(SnapshotProtos.java:2197)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(SnapshotProtos.java:2192)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1165)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at 
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at 
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at 
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at 
> 

[jira] [Updated] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15093:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
   1.3.0
   2.0.0
   Status: Resolved  (was: Patch Available)

Thanks for the patch, Ashu.

Thanks for the review, Enis.

> Replication can report incorrect size of log queue for the global source when 
> multiwal is enabled
> -
>
> Key: HBASE-15093
> URL: https://issues.apache.org/jira/browse/HBASE-15093
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15093-V0.patch, HBASE-15093-V1.patch, 
> HBASE-15093-V1.patch, HBASE-15093-V2.patch
>
>
> Replication can  report incorrect size for the size of log queue for the 
> global source when multiwal is enabled. This happens because the method 
> MetricsSource#setSizeofLogQueue performs non-trivial operations in a 
> multithreaded world, even though it is not synchronized. 
> We can simply divide MetricsSource#setSizeofLogQueue into 
> MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not 
> sure why we are currently directly setting the size instead of 
> incrementing/decrementing it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15619) Performance regression observed: Empty random read(get) performance of branch-1 worse than 0.98

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-15619:
--
Summary: Performance regression observed: Empty random read(get) 
performance of branch-1 worse than 0.98  (was: Performance regression observed: 
Random read(get) performance of branch-1 worse than 0.98)

> Performance regression observed: Empty random read(get) performance of 
> branch-1 worse than 0.98
> ---
>
> Key: HBASE-15619
> URL: https://issues.apache.org/jira/browse/HBASE-15619
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
>
> As titled, I observed the perf regression in the final stress testing before 
> upgrading our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is 
> the most important perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>  1) 1.1.2 with important perf fixes/improvements including HBASE-15031 
> and HBASE-14465 (we planed to upgrade to 1.x since Oct. last year, by when 
> 1.1.2 was the latest stable release in branch-1)
>  2) 1.1.4 release
>  3) 1.2.1RC1
> 2. Test environment
> * YCSB: 0.7.0 with 
> [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
> * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
> 100 threads
> * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
> * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
> hardware for client and server
> 3. Test cases
> * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
> * case #1: read against empty table
> * case #2: lrucache 100% hit
> * case #3: BLOCKCACHE=>false
> 4. Test result
> * 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so 
> will only paste comparison data of 0.98.12+ and 1.1.2+
> * per-RS Throughput(ops/s)
> ||HBaseVersion||case#1||case#2||case#3||
> |0.98.12+|383562|257493|47594|
> |1.1.2+|363050|232757|35872|
> * AverageLatency(us)
> ||HBaseVersion||case#1||case#2||case#3||
> |0.98.12+|2774|4134|22371|
> |1.1.2+|2930|4572|29690|
> It seems to me each part on the read path has perf regression: RPCServer for 
> case#1, lrucache read for case#2, and hfile read for case#3...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15619) Performance regression observed: Random read(get) performance of branch-1 worse than 0.98

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235283#comment-15235283
 ] 

Yu Li commented on HBASE-15619:
---

[~stack] and all:
Please ignore case#2 and case#3, today I just found that HBASE-14061 is not 
committed yet thus not included in release 1.1.4/1.2.1, so I was actually 
comparing ONE_SSD read for 0.98.12+ and HOT(all HDD) for 1.1.4/1.2.1. After 
loading data with our 1.1.2+ to generate ONE_SSD hfiles, no more regression 
observed (the issue does exist in our 1.1.2+, but that's my own problem and 
will resolve it in private). Sorry for the bothering.

However, for case#1 the regression does exist, so the RPCServer part may still 
have some problem. Will update the JIRA title and description.

> Performance regression observed: Random read(get) performance of branch-1 
> worse than 0.98
> -
>
> Key: HBASE-15619
> URL: https://issues.apache.org/jira/browse/HBASE-15619
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
>
> As titled, I observed the perf regression in the final stress testing before 
> upgrading our online cluster to 1.x. More details as follows:
> 1. HBase version in the comparison test:
>   * 0.98: based on 0.98.12 with some backports, among which HBASE-11297 is 
> the most important perf-related one (especially under high stress)
>   * 1.x: checked 3 releases in total
>  1) 1.1.2 with important perf fixes/improvements including HBASE-15031 
> and HBASE-14465 (we planed to upgrade to 1.x since Oct. last year, by when 
> 1.1.2 was the latest stable release in branch-1)
>  2) 1.1.4 release
>  3) 1.2.1RC1
> 2. Test environment
> * YCSB: 0.7.0 with 
> [YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651] applied
> * Client: 4 physical nodes, each with 8 YCSB instance, each instance with 
> 100 threads
> * Server: 1 Master with 3 RS, each RS with 256 handlers and 64G heap
> * Hardware: 64-core CPU, 256GB Mem, 10Gb Net, 1 PCIe-SSD and 11 HDD, same 
> hardware for client and server
> 3. Test cases
> * -p fieldcount=1 -p fieldlength=128 -p readproportion=1
> * case #1: read against empty table
> * case #2: lrucache 100% hit
> * case #3: BLOCKCACHE=>false
> 4. Test result
> * 1.1.4 and 1.2.1 have a similar perf (less than 2% deviation) as 1.1.2+, so 
> will only paste comparison data of 0.98.12+ and 1.1.2+
> * per-RS Throughput(ops/s)
> ||HBaseVersion||case#1||case#2||case#3||
> |0.98.12+|383562|257493|47594|
> |1.1.2+|363050|232757|35872|
> * AverageLatency(us)
> ||HBaseVersion||case#1||case#2||case#3||
> |0.98.12+|2774|4134|22371|
> |1.1.2+|2930|4572|29690|
> It seems to me each part on the read path has perf regression: RPCServer for 
> case#1, lrucache read for case#2, and hfile read for case#3...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235269#comment-15235269
 ] 

Ted Yu commented on HBASE-15093:


Failure from TestRegionMergeTransactionOnCluster was not related.

> Replication can report incorrect size of log queue for the global source when 
> multiwal is enabled
> -
>
> Key: HBASE-15093
> URL: https://issues.apache.org/jira/browse/HBASE-15093
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>Priority: Minor
> Attachments: HBASE-15093-V0.patch, HBASE-15093-V1.patch, 
> HBASE-15093-V1.patch, HBASE-15093-V2.patch
>
>
> Replication can  report incorrect size for the size of log queue for the 
> global source when multiwal is enabled. This happens because the method 
> MetricsSource#setSizeofLogQueue performs non-trivial operations in a 
> multithreaded world, even though it is not synchronized. 
> We can simply divide MetricsSource#setSizeofLogQueue into 
> MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not 
> sure why we are currently directly setting the size instead of 
> incrementing/decrementing it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15605) Remove PB references from HCD and HTD for 2.0

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235260#comment-15235260
 ] 

Hudson commented on HBASE-15605:


SUCCESS: Integrated in HBase-1.3 #642 (See 
[https://builds.apache.org/job/HBase-1.3/642/])
HBASE-15605 Remove PB references from HCD and HTD for 2.0 (Ram) 
(ramkrishna.s.vasudevan: rev 672c427ce4da7f01168e1e0be221602d34cc7324)
* hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java


> Remove PB references from HCD and HTD for 2.0
> -
>
> Key: HBASE-15605
> URL: https://issues.apache.org/jira/browse/HBASE-15605
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-15605.patch, HBASE-15605_1.patch, 
> HBASE-15605_2.patch, HBASE-15605_branch-1.patch
>
>
> This task is sub-task for HBASE-15174. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235243#comment-15235243
 ] 

Ted Yu commented on HBASE-15406:


[~mantonov]:
Can you take a look at the patch ?

> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch, 
> HBASE-15406_v1.patch, HBASE-15406_v2.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default 
> value after hbck exits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-15500) Cut HBase 1.2.1 release

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-15500.
-
Resolution: Fixed

release tagged as rel/1.2.1, announcement email sent.

> Cut HBase 1.2.1 release
> ---
>
> Key: HBASE-15500
> URL: https://issues.apache.org/jira/browse/HBASE-15500
> Project: HBase
>  Issue Type: Task
>  Components: community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Fix For: 1.2.1
>
>
> we're due for 1.2.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)