[jira] [Commented] (HBASE-11393) Replication TableCfs should be a PB object rather than a string
[
https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213938#comment-15213938
]
Hadoop QA commented on HBASE-11393:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 55s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 44s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 27m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 49s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 13s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 0s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 28m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 11 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 54s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color
[jira] [Commented] (HBASE-11393) Replication TableCfs should be a PB object rather than a string
[ https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213981#comment-15213981 ] Heng Chen commented on HBASE-11393: --- Failing test : org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster has no relates with current patch. If no other suggestions, i will commit it later. > Replication TableCfs should be a PB object rather than a string > --- > > Key: HBASE-11393 > URL: https://issues.apache.org/jira/browse/HBASE-11393 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-11393.patch, HBASE-11393_v1.patch, > HBASE-11393_v10.patch, HBASE-11393_v11.patch, HBASE-11393_v12.patch, > HBASE-11393_v14.patch, HBASE-11393_v15.patch, HBASE-11393_v16.patch, > HBASE-11393_v2.patch, HBASE-11393_v3.patch, HBASE-11393_v4.patch, > HBASE-11393_v5.patch, HBASE-11393_v6.patch, HBASE-11393_v7.patch, > HBASE-11393_v8.patch, HBASE-11393_v9.patch > > > We concatenate the list of tables and column families in format > "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer > mapping. > This results in ugly parsing code. We should do this a PB object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15499) Add multiple data type support for increment
[ https://issues.apache.org/jira/browse/HBASE-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Liangliang updated HBASE-15499: -- Attachment: HBASE-15499-V2.diff indent switch-case > Add multiple data type support for increment > > > Key: HBASE-15499 > URL: https://issues.apache.org/jira/browse/HBASE-15499 > Project: HBase > Issue Type: New Feature > Components: API >Reporter: He Liangliang >Assignee: He Liangliang > Attachments: HBASE-15499-V2.diff, HBASE-15499.diff > > > Currently the increment assumes long with byte-wise serialization. It's > useful to support flexible data type/serializations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15499) Add multiple data type support for increment
[ https://issues.apache.org/jira/browse/HBASE-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Liangliang updated HBASE-15499: -- Attachment: (was: HBASE-15499-V2.diff) > Add multiple data type support for increment > > > Key: HBASE-15499 > URL: https://issues.apache.org/jira/browse/HBASE-15499 > Project: HBase > Issue Type: New Feature > Components: API >Reporter: He Liangliang >Assignee: He Liangliang > Attachments: HBASE-15499.diff > > > Currently the increment assumes long with byte-wise serialization. It's > useful to support flexible data type/serializations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15499) Add multiple data type support for increment
[ https://issues.apache.org/jira/browse/HBASE-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Liangliang updated HBASE-15499: -- Attachment: HBASE-15499-V2.diff > Add multiple data type support for increment > > > Key: HBASE-15499 > URL: https://issues.apache.org/jira/browse/HBASE-15499 > Project: HBase > Issue Type: New Feature > Components: API >Reporter: He Liangliang >Assignee: He Liangliang > Attachments: HBASE-15499-V2.diff, HBASE-15499.diff > > > Currently the increment assumes long with byte-wise serialization. It's > useful to support flexible data type/serializations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15499) Add multiple data type support for increment
[ https://issues.apache.org/jira/browse/HBASE-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214044#comment-15214044 ] He Liangliang commented on HBASE-15499: --- added missing annotation > Add multiple data type support for increment > > > Key: HBASE-15499 > URL: https://issues.apache.org/jira/browse/HBASE-15499 > Project: HBase > Issue Type: New Feature > Components: API >Reporter: He Liangliang >Assignee: He Liangliang > Attachments: HBASE-15499-V2.diff, HBASE-15499.diff > > > Currently the increment assumes long with byte-wise serialization. It's > useful to support flexible data type/serializations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214150#comment-15214150 ] Matteo Bertozzi commented on HBASE-15508: - didn't notice the "snapshot" command to take a snapshot. just snapshotinfo. sounds good to me to revert. > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.2.2 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15187) Integrate CSRF prevention filter to REST gateway
[ https://issues.apache.org/jira/browse/HBASE-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214224#comment-15214224 ] Ted Yu commented on HBASE-15187: [~apurtell]: Gentle ping. > Integrate CSRF prevention filter to REST gateway > > > Key: HBASE-15187 > URL: https://issues.apache.org/jira/browse/HBASE-15187 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: HBASE-15187-branch-1.v13.patch, HBASE-15187.v1.patch, > HBASE-15187.v10.patch, HBASE-15187.v11.patch, HBASE-15187.v12.patch, > HBASE-15187.v13.patch, HBASE-15187.v2.patch, HBASE-15187.v3.patch, > HBASE-15187.v4.patch, HBASE-15187.v5.patch, HBASE-15187.v6.patch, > HBASE-15187.v7.patch, HBASE-15187.v8.patch, HBASE-15187.v9.patch > > > HADOOP-12691 introduced a filter in Hadoop Common to help REST APIs guard > against cross-site request forgery attacks. > This issue tracks the integration of that filter into HBase REST gateway. > From REST section of refguide: > To delete a table, use a DELETE request with the /schema endpoint: > http://example.com:8000/schema > Suppose an attacker hosts a malicious web form on a domain under his control. > The form uses the DELETE action targeting a REST URL. Through social > engineering, the attacker tricks an authenticated user into accessing the > form and submitting it. > The browser sends the HTTP DELETE request to the REST gateway. > At REST gateway, the call is executed and user table is dropped -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214227#comment-15214227
]
Ted Yu commented on HBASE-15406:
[~mantonov]:
Mind spending 10 minutes on Heng's patch ?
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15546) improvements to consistency
Walter Koetke created HBASE-15546: - Summary: improvements to consistency Key: HBASE-15546 URL: https://issues.apache.org/jira/browse/HBASE-15546 Project: HBase Issue Type: Bug Affects Versions: 1.0.3 Reporter: Walter Koetke Assignee: Walter Koetke Fix For: 2.0.0, 1.0.4, 1.4.0 Improvements needed to consistency logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15534) A client with basic Get and Put functionality
[ https://issues.apache.org/jira/browse/HBASE-15534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-15534: --- Attachment: hbase-cpp-client.v1-master.patch > A client with basic Get and Put functionality > - > > Key: HBASE-15534 > URL: https://issues.apache.org/jira/browse/HBASE-15534 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Vamsi Mohan V S Thattikota >Assignee: Vamsi Mohan V S Thattikota > Attachments: hbase-cpp-client.v1-master.patch > > > This is a C++ client with basic Get and Put API functionality implemented. > The API is implemented to match close to Java API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Memory optimizations
[
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214273#comment-15214273
]
Anastasia Braginsky commented on HBASE-14921:
-
Thanks [~ram_krish], you raise the really good questions about ChunkPool!
bq. So the CellBlockSerialized (I think we can have a better name) - will work
only if the MSLAB Chunk pool is ON right? If there is no MSLAB chunk pool still
it can work? I may be missing something.
Indeed this requires ChunkPool to be ON. Please pay attention that we can
design ChunkPool not to pre-allocate Chunks if this is what configuration
wants. It can be just a global bookkeeper for all allocated Chunks. We just
require ChunkPool not to be NULL.
bq. Now in your shallowCellsToBuffer even the shallowBuffer is created with the
same 'chunkSize' as we create the deepbuffer also? Is that really needed?
The “chunkSize” used in shallowCellsToBuffer holds the size of any Chunk
created by ChunkPool. I assume (as it is now in MemStoreChunkPool) that all
Chunks created from the same instance of Pool are of same size. This is why one
can reuse the Chunks. If Chunks are "tailor-made" with different sizes, they
almost cannot be reused.
bq. How does things work when the chunkpool gets reset and initialized?
This would be a problem. What is the case when ChunkPool need to be reset and
initialized?
And BTW regarding the names, do you like the names that I have suggested above?
{quote}
Regarding the names, as “CellBlock” is already in some other use, we suggest
the following names (almost as you suggested).
CellBlock -> CellFlatMap
CellBlockObjectArray -> CellArrayMap
CellBlockSerialized -> CellChunkMap
Why “flat”? Because this is how we call the process of changing the
ConcurrentSkipListMap to CellMap. When we have in-memory-flash and there is no
need to compact (unnecessary copies) we flatten the segment. Meaning we create
ImmutableSegment with the same MSLAB, but with different Map, replacing
ConcurrentSkipListMap to CellMap. What do you think?
{quote}
> Memory optimizations
>
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf,
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch,
> HBASE-14921-V02.patch
>
>
> Memory optimizations including compressed format representation and offheap
> allocations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15547) Balancer should take into account number of PENDING_OPEN regions
[ https://issues.apache.org/jira/browse/HBASE-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15547: Fix Version/s: 1.4.0 2.0.0 > Balancer should take into account number of PENDING_OPEN regions > > > Key: HBASE-15547 > URL: https://issues.apache.org/jira/browse/HBASE-15547 > Project: HBase > Issue Type: Improvement > Components: Balancer, Operability, Region Assignment >Affects Versions: 0.98.0, 1.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.4.0 > > > We recently had a cluster get into a bad state where a subset of region > servers consistently could not open new regions (but could continue serving > the regions they already hosted). > Recovering the cluster was just a matter of restarting region servers in > sequence. However, this led to things getting substantially worse before they > got better since the bulk assigner continued to place an uniform number of > recovered regions across all servers, including onto those that could not > open regions. > It would be useful if the balancer could penalize regionservers with a > backlog of pending_open regions and place more work on those regionservers > that are properly serving. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15547) Balancer should take into account number of PENDING_OPEN regions
Sean Busbey created HBASE-15547: --- Summary: Balancer should take into account number of PENDING_OPEN regions Key: HBASE-15547 URL: https://issues.apache.org/jira/browse/HBASE-15547 Project: HBase Issue Type: Improvement Components: Balancer, Operability, Region Assignment Reporter: Sean Busbey Priority: Critical We recently had a cluster get into a bad state where a subset of region servers consistently could not open new regions (but could continue serving the regions they already hosted). Recovering the cluster was just a matter of restarting region servers in sequence. However, this led to things getting substantially worse before they got better since the bulk assigner continued to place an uniform number of recovered regions across all servers, including onto those that could not open regions. It would be useful if the balancer could penalize regionservers with a backlog of pending_open regions and place more work on those regionservers that are properly serving. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15547) Balancer should take into account number of PENDING_OPEN regions
[ https://issues.apache.org/jira/browse/HBASE-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15547: Affects Version/s: 0.98.0 1.0.0 > Balancer should take into account number of PENDING_OPEN regions > > > Key: HBASE-15547 > URL: https://issues.apache.org/jira/browse/HBASE-15547 > Project: HBase > Issue Type: Improvement > Components: Balancer, Operability, Region Assignment >Affects Versions: 0.98.0, 1.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.4.0 > > > We recently had a cluster get into a bad state where a subset of region > servers consistently could not open new regions (but could continue serving > the regions they already hosted). > Recovering the cluster was just a matter of restarting region servers in > sequence. However, this led to things getting substantially worse before they > got better since the bulk assigner continued to place an uniform number of > recovered regions across all servers, including onto those that could not > open regions. > It would be useful if the balancer could penalize regionservers with a > backlog of pending_open regions and place more work on those regionservers > that are properly serving. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15546) improvements to consistency
[ https://issues.apache.org/jira/browse/HBASE-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214340#comment-15214340 ] Dave Latham commented on HBASE-15546: - What sort of improvements? > improvements to consistency > --- > > Key: HBASE-15546 > URL: https://issues.apache.org/jira/browse/HBASE-15546 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.3 >Reporter: Walter Koetke >Assignee: Walter Koetke > Fix For: 2.0.0, 1.0.4, 1.4.0 > > > Improvements needed to consistency logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-15508. - Resolution: Fixed Fix Version/s: (was: 1.2.2) (was: 1.1.4) (was: 1.2.1) 1.4.0 reverted from branch-1.1 and branch-1.2 > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
[ https://issues.apache.org/jira/browse/HBASE-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214385#comment-15214385 ] Vladimir Rodionov commented on HBASE-15506: --- [~lhofhansl] Yes, this JIRA is for enabling HDFS-7276, by default. As for why we need to decrease garbage allocation ... this affects latency and server stability in a long term (with high memory allocation rate chances you get eventually full GC is much higher). > FSDataOutputStream.write() allocates new byte buffer on each operation > -- > > Key: HBASE-15506 > URL: https://issues.apache.org/jira/browse/HBASE-15506 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Deep inside stack trace in DFSOutputStream.createPacket. > This should be opened in HDFS. This JIRA is to track HDFS work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
[
https://issues.apache.org/jira/browse/HBASE-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214393#comment-15214393
]
Vladimir Rodionov commented on HBASE-15506:
---
{quote}
t's possible I am missing something. We're talking about 64KB buffers
allocating for the HDFS stream packets, right? 100MB of garbage in 64KB chunks
is a mere 1600 objects. Surely we're not going to be worried about that.
{quote}
100MB of garbage here, 200MB of garbage there ... adds up to GBs everywhere.
> FSDataOutputStream.write() allocates new byte buffer on each operation
> --
>
> Key: HBASE-15506
> URL: https://issues.apache.org/jira/browse/HBASE-15506
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>
> Deep inside stack trace in DFSOutputStream.createPacket.
> This should be opened in HDFS. This JIRA is to track HDFS work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
[ https://issues.apache.org/jira/browse/HBASE-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214414#comment-15214414 ] stack commented on HBASE-15506: --- HDFS-7276 adds 4 new configs (5 including the enable) where you adjust the manager to suit loading. A bit of work to figure good defaults that make it so this feature a benefit rather than a drawback would help this issue along; would be great if it ergonomically adjusted to the loading rather than require manual adjustment. We'd probably be first users too so some long running tests with this enabled would help this issue too (were some interesting issues during its development/testing). > FSDataOutputStream.write() allocates new byte buffer on each operation > -- > > Key: HBASE-15506 > URL: https://issues.apache.org/jira/browse/HBASE-15506 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Deep inside stack trace in DFSOutputStream.createPacket. > This should be opened in HDFS. This JIRA is to track HDFS work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214437#comment-15214437 ] stack commented on HBASE-14921: --- bq. As you have mentioned, CellBlock is implementing ConcurrentNavigbleMap because delegatee of the CellSet is of type ConcurrentNavigbleMap and we want just to give CellSet another delegatee instead of ConcurrentSkipListMap. Please say more on this. CellSet is a NavigableMap (not a ConcurrentNavigableMap) so I'm missing where we need the 'Concurrent' (is it in this patch?) Your new names are better. I considered 'flat' Map but shied away given its meaning over in spark/scala; I think it will be ok as long as you stick why its a 'flat' map in the javadoc on CellFlatMap. bq. ... it is Cells per chunk and it should be constant for all Chunks in a CellBlockSerialized, because MSLAB always allocates Chunks with same size How do you see this working? We do not control the size of inbound Cells. They could have some regularity and they could also be erratic to the extreme (What to do when a 1G cell arrives into a column family that up to this has been taking on metrics?) I still do not see how the 3 * int is BYTES_IN_CELL. Not important. bq. But we are not sure what are the configurations where ChunkPool is off on purpose and what we can do with such configurations. Do you know? It was introduced and off by default as is usual when new features. But as also happens this is our practice, the facility was 'forgotten'. It came up then when our Lars noticed it and wanted to remove it since it was not being used. It came up again recently in HBASE-15513 It would seem to make sense enabling it by default if we come up w/ a proper sizing. Having it on seems to mess w/ G1GC too. Would need to figure that. We need to do up a memory management doc. Between your work on Segments, Segment pipelines, MSLAB chunks, chunk pools and bytebufferpools to host requests read from sockets, bucket cache and reference counting bucketcache bucket blocks at read time, it would be good if we had a map so we could trace a Cell on its travels. > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15546) improvements to consistency
[ https://issues.apache.org/jira/browse/HBASE-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214445#comment-15214445 ] Mikhail Antonov commented on HBASE-15546: - Yeah, what are they? If some important bugfixes, shall they also go in other 1.* branches? > improvements to consistency > --- > > Key: HBASE-15546 > URL: https://issues.apache.org/jira/browse/HBASE-15546 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.3 >Reporter: Walter Koetke >Assignee: Walter Koetke > Fix For: 2.0.0, 1.0.4, 1.4.0 > > > Improvements needed to consistency logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15499) Add multiple data type support for increment
[
https://issues.apache.org/jira/browse/HBASE-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214487#comment-15214487
]
Hadoop QA commented on HBASE-15499:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.2.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 37s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 32s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 16s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 39s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 36s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
23s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 15m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 17s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 4s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 5s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 14s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 11s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 140m 28s
{color} | {color:red} hbase-
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214526#comment-15214526 ] stack commented on HBASE-15525: --- So in your patch, ByteBufferPool is a FixedSizeBBP? So we allocate in a step function adding a new BB of fixed size to the OS to fill? If FixedSizeBBP is full, we allocate from Heap. Sounds good. Tie the new pool to the new Stream? Give them same prefix. They work together. Looks good [~anoop.hbase] > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Attachments: WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15384) Avoid using '/tmp' directory in TestBulkLoad
[ https://issues.apache.org/jira/browse/HBASE-15384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214534#comment-15214534 ] Enis Soztutar commented on HBASE-15384: --- Does this apply only to master? Otherwise we should commit UT fixes to every applicable release. > Avoid using '/tmp' directory in TestBulkLoad > > > Key: HBASE-15384 > URL: https://issues.apache.org/jira/browse/HBASE-15384 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 2.0.0 >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-15384.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214538#comment-15214538 ] stack commented on HBASE-15525: --- Oh, so, it addresses this issue because sizing of the offheap pool is fixed and if a write burst, new allocations will be from heap, not from offheap? How we come up what the fixedsize of a ByteBuffer will be? In this case, the write s usually 12k in size but there were some sprinkling of 512k writes so the BBs took the 512k form. Would the FixedSize buffers be 512k or 12k? Or something else? And how many of these BBs will we keep around? > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Attachments: WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15337) Document FIFO and date tiered compaction in the book
[ https://issues.apache.org/jira/browse/HBASE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15337: -- Issue Type: Sub-task (was: Improvement) Parent: HBASE-15339 > Document FIFO and date tiered compaction in the book > > > Key: HBASE-15337 > URL: https://issues.apache.org/jira/browse/HBASE-15337 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > > We have two new compaction algorithms FIFO and Date tiered that are for time > series data. We should document how to use them in the book. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15327) Canary will always invoke admin.balancer() in each sniffing period when writeSniffing is enabled
[
https://issues.apache.org/jira/browse/HBASE-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15327:
---
Attachment: HBASE-15327-trunk.patch
Previous test output is no longer accessible.
Re-attach.
> Canary will always invoke admin.balancer() in each sniffing period when
> writeSniffing is enabled
>
>
> Key: HBASE-15327
> URL: https://issues.apache.org/jira/browse/HBASE-15327
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 2.0.0
>Reporter: Jianwei Cui
>Priority: Minor
> Attachments: HBASE-15327-trunk.patch, HBASE-15327-trunk.patch
>
>
> When Canary#writeSniffing is enabled, Canary#checkWriteTableDistribution will
> make sure the regions of write table distributed on all region servers as:
> {code}
> int numberOfServers = admin.getClusterStatus().getServers().size();
> ..
> int numberOfCoveredServers = serverSet.size();
> if (numberOfCoveredServers < numberOfServers) {
> admin.balancer();
> }
> {code}
> The master will also work as a regionserver, so that ClusterStatus#getServers
> will contain the master. On the other hand, write table of Canary will not be
> assigned to master, making numberOfCoveredServers always smaller than
> numberOfServers and admin.balancer always be invoked in each sniffing period.
> This may cause frequent region moves. A simple fix is excluding master from
> numberOfServers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15548) SyncTable: sourceHashDir is supposed to be optional but won't work without
My Ho created HBASE-15548: - Summary: SyncTable: sourceHashDir is supposed to be optional but won't work without Key: HBASE-15548 URL: https://issues.apache.org/jira/browse/HBASE-15548 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 1.2.0 Environment: Ubuntu 12.04 Reporter: My Ho 1) SyncTable code is contradictory. Usage said sourcehashdir is optional (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L687). However, the command won't run if sourcehashdir is missing (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L83-L85) 2) There is no documentation on how to create the desired sourcehash -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15548) SyncTable: sourceHashDir is supposed to be optional but won't work without
[ https://issues.apache.org/jira/browse/HBASE-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] My Ho updated HBASE-15548: -- Priority: Blocker (was: Major) > SyncTable: sourceHashDir is supposed to be optional but won't work without > --- > > Key: HBASE-15548 > URL: https://issues.apache.org/jira/browse/HBASE-15548 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 > Environment: Ubuntu 12.04 >Reporter: My Ho >Priority: Blocker > > 1) SyncTable code is contradictory. Usage said sourcehashdir is optional > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L687). > However, the command won't run if sourcehashdir is missing > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L83-L85) > > 2) There is no documentation on how to create the desired sourcehash -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15546) improvements to consistency
[ https://issues.apache.org/jira/browse/HBASE-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-15546. Resolution: Invalid Assignee: (was: Walter Koetke) Fix Version/s: (was: 1.4.0) (was: 1.0.4) (was: 2.0.0) We can't accept proposals without actionable scope. Especially for this topic, please start a discussion on d...@hbase.apache.org first. > improvements to consistency > --- > > Key: HBASE-15546 > URL: https://issues.apache.org/jira/browse/HBASE-15546 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.3 >Reporter: Walter Koetke > > Improvements needed to consistency logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15548) SyncTable: sourceHashDir is supposed to be optional but won't work without
[ https://issues.apache.org/jira/browse/HBASE-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214653#comment-15214653 ] Dave Latham commented on HBASE-15548: - Sorry for the discrepency between the usage information and the actual behavior. As you discovered, the sourcehashdir is required at the moment. You may have found it already, but to generate the hash data, you can run `org.apache.hadoop.hbase.mapreduce.HashTable` > SyncTable: sourceHashDir is supposed to be optional but won't work without > --- > > Key: HBASE-15548 > URL: https://issues.apache.org/jira/browse/HBASE-15548 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 > Environment: Ubuntu 12.04 >Reporter: My Ho >Priority: Blocker > > 1) SyncTable code is contradictory. Usage said sourcehashdir is optional > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L687). > However, the command won't run if sourcehashdir is missing > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L83-L85) > > 2) There is no documentation on how to create the desired sourcehash -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-15548) SyncTable: sourceHashDir is supposed to be optional but won't work without
[ https://issues.apache.org/jira/browse/HBASE-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham reassigned HBASE-15548: --- Assignee: Dave Latham > SyncTable: sourceHashDir is supposed to be optional but won't work without > --- > > Key: HBASE-15548 > URL: https://issues.apache.org/jira/browse/HBASE-15548 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 > Environment: Ubuntu 12.04 >Reporter: My Ho >Assignee: Dave Latham >Priority: Minor > Attachments: HBASE-15548.patch > > > 1) SyncTable code is contradictory. Usage said sourcehashdir is optional > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L687). > However, the command won't run if sourcehashdir is missing > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L83-L85) > > 2) There is no documentation on how to create the desired sourcehash -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15548) SyncTable: sourceHashDir is supposed to be optional but won't work without
[ https://issues.apache.org/jira/browse/HBASE-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-15548: Priority: Minor (was: Blocker) > SyncTable: sourceHashDir is supposed to be optional but won't work without > --- > > Key: HBASE-15548 > URL: https://issues.apache.org/jira/browse/HBASE-15548 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 > Environment: Ubuntu 12.04 >Reporter: My Ho >Priority: Minor > Attachments: HBASE-15548.patch > > > 1) SyncTable code is contradictory. Usage said sourcehashdir is optional > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L687). > However, the command won't run if sourcehashdir is missing > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L83-L85) > > 2) There is no documentation on how to create the desired sourcehash -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15548) SyncTable: sourceHashDir is supposed to be optional but won't work without
[ https://issues.apache.org/jira/browse/HBASE-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-15548: Attachment: HBASE-15548.patch > SyncTable: sourceHashDir is supposed to be optional but won't work without > --- > > Key: HBASE-15548 > URL: https://issues.apache.org/jira/browse/HBASE-15548 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 > Environment: Ubuntu 12.04 >Reporter: My Ho >Assignee: Dave Latham >Priority: Minor > Attachments: HBASE-15548.patch > > > 1) SyncTable code is contradictory. Usage said sourcehashdir is optional > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L687). > However, the command won't run if sourcehashdir is missing > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L83-L85) > > 2) There is no documentation on how to create the desired sourcehash -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15548) SyncTable: sourceHashDir is supposed to be optional but won't work without
[ https://issues.apache.org/jira/browse/HBASE-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-15548: Fix Version/s: 2.0.0 Status: Patch Available (was: Open) > SyncTable: sourceHashDir is supposed to be optional but won't work without > --- > > Key: HBASE-15548 > URL: https://issues.apache.org/jira/browse/HBASE-15548 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 > Environment: Ubuntu 12.04 >Reporter: My Ho >Assignee: Dave Latham >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15548.patch > > > 1) SyncTable code is contradictory. Usage said sourcehashdir is optional > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L687). > However, the command won't run if sourcehashdir is missing > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L83-L85) > > 2) There is no documentation on how to create the desired sourcehash -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15548) SyncTable: sourceHashDir is supposed to be optional but won't work without
[ https://issues.apache.org/jira/browse/HBASE-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214673#comment-15214673 ] Ted Yu commented on HBASE-15548: +1 > SyncTable: sourceHashDir is supposed to be optional but won't work without > --- > > Key: HBASE-15548 > URL: https://issues.apache.org/jira/browse/HBASE-15548 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 > Environment: Ubuntu 12.04 >Reporter: My Ho >Assignee: Dave Latham >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15548.patch > > > 1) SyncTable code is contradictory. Usage said sourcehashdir is optional > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L687). > However, the command won't run if sourcehashdir is missing > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L83-L85) > > 2) There is no documentation on how to create the desired sourcehash -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214710#comment-15214710 ] Ted Yu commented on HBASE-15400: Clara: When you have the next patch, mind uploading it onto review board ? Thanks > Use DateTieredCompactor for Date Tiered Compaction > -- > > Key: HBASE-15400 > URL: https://issues.apache.org/jira/browse/HBASE-15400 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15400-15389-v12.patch, HBASE-15400-v1.pa, > HBASE-15400.patch > > > When we compact, we can output multiple files along the current window > boundaries. There are two use cases: > 1. Major compaction: We want to output date tiered store files with data > older than max age archived in trunks of the window size on the higher > tier.1. Once a file is old enough to be out of the range that we do tiered > minor compaction, we don't compact them any further. So they retain the same > timespan as they were compacted last time, which is the window size of the > highest tier. Major compaction will touch these files and we want to maintain > the same layout. > 2. Bulk load files and the old file generated by major compaction before > upgrading to DTCP. > Pros: > 1. Restore locality, process versioning, updates and deletes while > maintaining the tiered layout. > 2. The best way to fix a skewed layout. > > This work is based on a prototype of DateTieredCompactor from HBASE-15389 and > focused on the part to meet needs for these two use cases while supporting > others. I have to call out a few design decisions: > 1. We only want to output the files along all windows for major compaction. > And we want to output multiple files older than max age in the trunks of the > maximum tier window size determined by base window size, windows per tier and > max age. > 2. For minor compaction, we don't want to output too many files, which will > remain around because of current restriction of contiguous compaction by seq > id. I will only output two files if all the files in the windows are being > combined, one for the data within window and the other for the out-of-window > tail. If there is any file in the window excluded from compaction, only one > file will be output from compaction. When the windows are promoted, the > situation of out of order data will gradually improve. For the incoming > window, we need to accommodate the case with user-specified future data. > 3. We have to pass the boundaries with the list of store file as a complete > time snapshot instead of two separate calls because window layout is > determined by the time the computation is called. So we will need new type of > compaction request. > 4. Since we will assign the same seq id for all output files, we need to sort > by maxTimestamp subsequently. Right now all compaction policy gets the files > sorted for StoreFileManager which sorts by seq id and other criteria. I will > use this order for DTCP only, to avoid impacting other compaction policies. > 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214719#comment-15214719 ] Clara Xiong commented on HBASE-15400: - [~tedyu]The one on RB and here are the same as I am testing right now. I will upload again once testing is done. > Use DateTieredCompactor for Date Tiered Compaction > -- > > Key: HBASE-15400 > URL: https://issues.apache.org/jira/browse/HBASE-15400 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15400-15389-v12.patch, HBASE-15400-v1.pa, > HBASE-15400.patch > > > When we compact, we can output multiple files along the current window > boundaries. There are two use cases: > 1. Major compaction: We want to output date tiered store files with data > older than max age archived in trunks of the window size on the higher > tier.1. Once a file is old enough to be out of the range that we do tiered > minor compaction, we don't compact them any further. So they retain the same > timespan as they were compacted last time, which is the window size of the > highest tier. Major compaction will touch these files and we want to maintain > the same layout. > 2. Bulk load files and the old file generated by major compaction before > upgrading to DTCP. > Pros: > 1. Restore locality, process versioning, updates and deletes while > maintaining the tiered layout. > 2. The best way to fix a skewed layout. > > This work is based on a prototype of DateTieredCompactor from HBASE-15389 and > focused on the part to meet needs for these two use cases while supporting > others. I have to call out a few design decisions: > 1. We only want to output the files along all windows for major compaction. > And we want to output multiple files older than max age in the trunks of the > maximum tier window size determined by base window size, windows per tier and > max age. > 2. For minor compaction, we don't want to output too many files, which will > remain around because of current restriction of contiguous compaction by seq > id. I will only output two files if all the files in the windows are being > combined, one for the data within window and the other for the out-of-window > tail. If there is any file in the window excluded from compaction, only one > file will be output from compaction. When the windows are promoted, the > situation of out of order data will gradually improve. For the incoming > window, we need to accommodate the case with user-specified future data. > 3. We have to pass the boundaries with the list of store file as a complete > time snapshot instead of two separate calls because window layout is > determined by the time the computation is called. So we will need new type of > compaction request. > 4. Since we will assign the same seq id for all output files, we need to sort > by maxTimestamp subsequently. Right now all compaction policy gets the files > sorted for StoreFileManager which sorts by seq id and other criteria. I will > use this order for DTCP only, to avoid impacting other compaction policies. > 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[
https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214720#comment-15214720
]
Ted Yu commented on HBASE-15400:
Do you get the following compilation error on v12 ?
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on
project hbase-server: Compilation failure: Compilation failure:
[ERROR]
/Users/tyu/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java:[273,20]
method cloneForReader() is already defined in class
org.apache.hadoop.hbase.regionserver.StoreFile
[ERROR]
/Users/tyu/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java:[52,8]
org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy is not
abstract and does not override abstract method
shouldPerformMajorCompaction(java.util.Collection)
in org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy
[ERROR]
/Users/tyu/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java:[168,3]
method does not override or implement a method from a supertype
[ERROR]
/Users/tyu/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DateTieredStoreEngine.java:[74,5]
method does not override or implement a method from a supertype
{code}
> Use DateTieredCompactor for Date Tiered Compaction
> --
>
> Key: HBASE-15400
> URL: https://issues.apache.org/jira/browse/HBASE-15400
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Reporter: Clara Xiong
>Assignee: Clara Xiong
> Fix For: 2.0.0
>
> Attachments: HBASE-15400-15389-v12.patch, HBASE-15400-v1.pa,
> HBASE-15400.patch
>
>
> When we compact, we can output multiple files along the current window
> boundaries. There are two use cases:
> 1. Major compaction: We want to output date tiered store files with data
> older than max age archived in trunks of the window size on the higher
> tier.1. Once a file is old enough to be out of the range that we do tiered
> minor compaction, we don't compact them any further. So they retain the same
> timespan as they were compacted last time, which is the window size of the
> highest tier. Major compaction will touch these files and we want to maintain
> the same layout.
> 2. Bulk load files and the old file generated by major compaction before
> upgrading to DTCP.
> Pros:
> 1. Restore locality, process versioning, updates and deletes while
> maintaining the tiered layout.
> 2. The best way to fix a skewed layout.
>
> This work is based on a prototype of DateTieredCompactor from HBASE-15389 and
> focused on the part to meet needs for these two use cases while supporting
> others. I have to call out a few design decisions:
> 1. We only want to output the files along all windows for major compaction.
> And we want to output multiple files older than max age in the trunks of the
> maximum tier window size determined by base window size, windows per tier and
> max age.
> 2. For minor compaction, we don't want to output too many files, which will
> remain around because of current restriction of contiguous compaction by seq
> id. I will only output two files if all the files in the windows are being
> combined, one for the data within window and the other for the out-of-window
> tail. If there is any file in the window excluded from compaction, only one
> file will be output from compaction. When the windows are promoted, the
> situation of out of order data will gradually improve. For the incoming
> window, we need to accommodate the case with user-specified future data.
> 3. We have to pass the boundaries with the list of store file as a complete
> time snapshot instead of two separate calls because window layout is
> determined by the time the computation is called. So we will need new type of
> compaction request.
> 4. Since we will assign the same seq id for all output files, we need to sort
> by maxTimestamp subsequently. Right now all compaction policy gets the files
> sorted for StoreFileManager which sorts by seq id and other criteria. I will
> use this order for DTCP only, to avoid impacting other compaction policies.
> 5. We need some cleanup of current design of StoreEngine and CompactionPolicy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214779#comment-15214779 ] Hudson commented on HBASE-15508: SUCCESS: Integrated in HBase-1.2-IT #472 (See [https://builds.apache.org/job/HBase-1.2-IT/472/]) Revert "HBASE-15508 Add command for exporting snapshot in hbase command (busbey: rev 142003e237776f57079ae7de1b8d15537a025bba) * bin/hbase * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15293) Handle TableNotFound and IllegalArgument exceptions in table.jsp.
[ https://issues.apache.org/jira/browse/HBASE-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214785#comment-15214785 ] Ted Yu commented on HBASE-15293: Have you checked whether TestRegionServerMetrics failure was related to the patch ? > Handle TableNotFound and IllegalArgument exceptions in table.jsp. > - > > Key: HBASE-15293 > URL: https://issues.apache.org/jira/browse/HBASE-15293 > Project: HBase > Issue Type: Sub-task > Components: UI >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-15293_v0.patch, HBASE-15293_v1.patch > > > table.jsp accepts "name" parameter for table name and in case when table does > not exist or empty parameter is passed it will throw 500 error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15191) CopyTable and VerifyReplication - Option to specify batch size, versions
[ https://issues.apache.org/jira/browse/HBASE-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15191: --- Attachment: HBASE_15191.patch Re-attach for QA run. > CopyTable and VerifyReplication - Option to specify batch size, versions > > > Key: HBASE-15191 > URL: https://issues.apache.org/jira/browse/HBASE-15191 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 0.98.16.1 >Reporter: Parth Shah >Priority: Minor > Attachments: HBASE_15191.patch, HBASE_15191.patch > > > Need option to specify batch size for CopyTable and VerifyReplication. We > are working on patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214791#comment-15214791 ] Clara Xiong commented on HBASE-15400: - Ah, let me sync with master since HBASE-15389 is already committed and will upload again. > Use DateTieredCompactor for Date Tiered Compaction > -- > > Key: HBASE-15400 > URL: https://issues.apache.org/jira/browse/HBASE-15400 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15400-15389-v12.patch, HBASE-15400-v1.pa, > HBASE-15400.patch > > > When we compact, we can output multiple files along the current window > boundaries. There are two use cases: > 1. Major compaction: We want to output date tiered store files with data > older than max age archived in trunks of the window size on the higher > tier.1. Once a file is old enough to be out of the range that we do tiered > minor compaction, we don't compact them any further. So they retain the same > timespan as they were compacted last time, which is the window size of the > highest tier. Major compaction will touch these files and we want to maintain > the same layout. > 2. Bulk load files and the old file generated by major compaction before > upgrading to DTCP. > Pros: > 1. Restore locality, process versioning, updates and deletes while > maintaining the tiered layout. > 2. The best way to fix a skewed layout. > > This work is based on a prototype of DateTieredCompactor from HBASE-15389 and > focused on the part to meet needs for these two use cases while supporting > others. I have to call out a few design decisions: > 1. We only want to output the files along all windows for major compaction. > And we want to output multiple files older than max age in the trunks of the > maximum tier window size determined by base window size, windows per tier and > max age. > 2. For minor compaction, we don't want to output too many files, which will > remain around because of current restriction of contiguous compaction by seq > id. I will only output two files if all the files in the windows are being > combined, one for the data within window and the other for the out-of-window > tail. If there is any file in the window excluded from compaction, only one > file will be output from compaction. When the windows are promoted, the > situation of out of order data will gradually improve. For the incoming > window, we need to accommodate the case with user-specified future data. > 3. We have to pass the boundaries with the list of store file as a complete > time snapshot instead of two separate calls because window layout is > determined by the time the computation is called. So we will need new type of > compaction request. > 4. Since we will assign the same seq id for all output files, we need to sort > by maxTimestamp subsequently. Right now all compaction policy gets the files > sorted for StoreFileManager which sorts by seq id and other criteria. I will > use this order for DTCP only, to avoid impacting other compaction policies. > 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15191) CopyTable and VerifyReplication - Option to specify batch size, versions
[ https://issues.apache.org/jira/browse/HBASE-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15191: --- Assignee: Parth Shah Fix Version/s: 1.4.0 2.0.0 > CopyTable and VerifyReplication - Option to specify batch size, versions > > > Key: HBASE-15191 > URL: https://issues.apache.org/jira/browse/HBASE-15191 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 0.98.16.1 >Reporter: Parth Shah >Assignee: Parth Shah >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE_15191.patch, HBASE_15191.patch > > > Need option to specify batch size for CopyTable and VerifyReplication. We > are working on patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15546) improvements to consistency
[ https://issues.apache.org/jira/browse/HBASE-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214807#comment-15214807 ] Walter Koetke commented on HBASE-15546: --- Fair enough guys. Sorry. Needed a quick placeholder jira for an extension we have internally at Splice, but I'll create a new Jira that describes it right off the bat. > improvements to consistency > --- > > Key: HBASE-15546 > URL: https://issues.apache.org/jira/browse/HBASE-15546 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.3 >Reporter: Walter Koetke > > Improvements needed to consistency logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15549) Change TABLES_ON_MASTER to false in master branch
stack created HBASE-15549: - Summary: Change TABLES_ON_MASTER to false in master branch Key: HBASE-15549 URL: https://issues.apache.org/jira/browse/HBASE-15549 Project: HBase Issue Type: Bug Components: Balancer, master Reporter: stack I thought we had an issue to do this but I can't find it so here is a new one. Currently, in master branch, HMaster is a RegionServer and carries meta regions such as hbase:meta. Disable this facility by default until better thought through (I think we should undo master ever carrying regions; FBers are thinking we should go forward with notion that master carries meta/system regions. TODO). I want to disable it because this facility is not finished and meantime I want to test new stuff coming in on master branch w/o this new feature getting in the way. Parking here a patch that has how to disable master carrying regions and a probably redundant test that ensures all works when master is not carrying meta regions (added because it WASN'T working for me -- but was pilot error). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15549) Change TABLES_ON_MASTER to 'none' in master branch
[ https://issues.apache.org/jira/browse/HBASE-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15549: -- Summary: Change TABLES_ON_MASTER to 'none' in master branch (was: Change TABLES_ON_MASTER to false in master branch) > Change TABLES_ON_MASTER to 'none' in master branch > -- > > Key: HBASE-15549 > URL: https://issues.apache.org/jira/browse/HBASE-15549 > Project: HBase > Issue Type: Bug > Components: Balancer, master >Reporter: stack > > I thought we had an issue to do this but I can't find it so here is a new one. > Currently, in master branch, HMaster is a RegionServer and carries meta > regions such as hbase:meta. Disable this facility by default until better > thought through (I think we should undo master ever carrying regions; FBers > are thinking we should go forward with notion that master carries meta/system > regions. TODO). I want to disable it because this facility is not finished > and meantime I want to test new stuff coming in on master branch w/o this new > feature getting in the way. > Parking here a patch that has how to disable master carrying regions and a > probably redundant test that ensures all works when master is not carrying > meta regions (added because it WASN'T working for me -- but was pilot error). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214822#comment-15214822 ] Hudson commented on HBASE-15508: ABORTED: Integrated in HBase-1.2 #587 (See [https://builds.apache.org/job/HBase-1.2/587/]) Revert "HBASE-15508 Add command for exporting snapshot in hbase command (busbey: rev 142003e237776f57079ae7de1b8d15537a025bba) * bin/hbase * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15333) Enhance the filter to handle short, integer, long, float and double
[ https://issues.apache.org/jira/browse/HBASE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhan Zhang updated HBASE-15333: --- Attachment: HBASE-15333-3.patch > Enhance the filter to handle short, integer, long, float and double > --- > > Key: HBASE-15333 > URL: https://issues.apache.org/jira/browse/HBASE-15333 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: HBASE-15333-1.patch, HBASE-15333-2.patch, > HBASE-15333-3.patch > > > Currently, the range filter is based on the order of bytes. But for java > primitive type, such as short, int, long, double, float, etc, their order is > not consistent with their byte order, extra manipulation has to be in place > to take care of them correctly. > For example, for the integer range (-100, 100), the filter <= 1, the current > filter will return 0 and 1, and the right return value should be (-100, 1] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15549) Undo HMaster carrying regions in master branch as default
[ https://issues.apache.org/jira/browse/HBASE-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15549: -- Summary: Undo HMaster carrying regions in master branch as default (was: Change TABLES_ON_MASTER to 'none' in master branch) > Undo HMaster carrying regions in master branch as default > - > > Key: HBASE-15549 > URL: https://issues.apache.org/jira/browse/HBASE-15549 > Project: HBase > Issue Type: Bug > Components: Balancer, master >Reporter: stack > > I thought we had an issue to do this but I can't find it so here is a new one. > Currently, in master branch, HMaster is a RegionServer and carries meta > regions such as hbase:meta. Disable this facility by default until better > thought through (I think we should undo master ever carrying regions; FBers > are thinking we should go forward with notion that master carries meta/system > regions. TODO). I want to disable it because this facility is not finished > and meantime I want to test new stuff coming in on master branch w/o this new > feature getting in the way. > Parking here a patch that has how to disable master carrying regions and a > probably redundant test that ensures all works when master is not carrying > meta regions (added because it WASN'T working for me -- but was pilot error). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15549) Undo HMaster carrying regions in master branch as default
[ https://issues.apache.org/jira/browse/HBASE-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214858#comment-15214858 ] Elliott Clark commented on HBASE-15549: --- bq.I want to disable it because this facility is not finished and meantime What's not finished? What issues are you seeing with it? For us meta on master is the only thing that gets IT tests to pass on a more consistent basis. I'd prefer to fix any issues if possible. > Undo HMaster carrying regions in master branch as default > - > > Key: HBASE-15549 > URL: https://issues.apache.org/jira/browse/HBASE-15549 > Project: HBase > Issue Type: Bug > Components: Balancer, master >Reporter: stack > > I thought we had an issue to do this but I can't find it so here is a new one. > Currently, in master branch, HMaster is a RegionServer and carries meta > regions such as hbase:meta. Disable this facility by default until better > thought through (I think we should undo master ever carrying regions; FBers > are thinking we should go forward with notion that master carries meta/system > regions. TODO). I want to disable it because this facility is not finished > and meantime I want to test new stuff coming in on master branch w/o this new > feature getting in the way. > Parking here a patch that has how to disable master carrying regions and a > probably redundant test that ensures all works when master is not carrying > meta regions (added because it WASN'T working for me -- but was pilot error). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15333) Enhance the filter to handle short, integer, long, float and double
[
https://issues.apache.org/jira/browse/HBASE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214946#comment-15214946
]
Hadoop QA commented on HBASE-15333:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 57s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
43m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 0s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 27s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795696/HBASE-15333-3.patch |
| JIRA Issue | HBASE-15333 |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / c96b642 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/1192/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1192/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Enhance the filter to handle short, integer, long, float and double
> ---
>
> Key: HBASE-15333
> URL: https://issues.apache.org/jira/browse/HBASE-15333
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zhan Zhang
>Assignee: Zhan Zhang
> Attachments: HBASE-15333-1.patch, HBASE-15333-2.patch,
> HBASE-15333-3.patch
>
>
> Currently, the range filter is based on the order of bytes. But for java
> primitive type, such as short, int, long, double, float, etc, their order is
> not consistent with their byte order, extra manipulation has to be in place
> to take care of them correctly.
> For example, for the integer range (-100, 100), the filter <= 1, the current
> filter will return 0 and 1, and the right return value should be (-100, 1]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15549) Undo HMaster carrying regions in master branch as default
[ https://issues.apache.org/jira/browse/HBASE-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15549: -- Attachment: 15549.patch Add a means of disabling master-as-regionserver. Default in patch is that master CAN carry regions as it currently does but would change this on commit. Adds test of this new config. > Undo HMaster carrying regions in master branch as default > - > > Key: HBASE-15549 > URL: https://issues.apache.org/jira/browse/HBASE-15549 > Project: HBase > Issue Type: Bug > Components: Balancer, master >Reporter: stack > Attachments: 15549.patch > > > I thought we had an issue to do this but I can't find it so here is a new one. > Currently, in master branch, HMaster is a RegionServer and carries meta > regions such as hbase:meta. Disable this facility by default until better > thought through (I think we should undo master ever carrying regions; FBers > are thinking we should go forward with notion that master carries meta/system > regions. TODO). I want to disable it because this facility is not finished > and meantime I want to test new stuff coming in on master branch w/o this new > feature getting in the way. > Parking here a patch that has how to disable master carrying regions and a > probably redundant test that ensures all works when master is not carrying > meta regions (added because it WASN'T working for me -- but was pilot error). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14123: -- Attachment: HBASE-14123-v13.patch > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-14123-v1.patch, HBASE-14123-v10.patch, > HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v13.patch, > HBASE-14123-v2.patch, HBASE-14123-v3.patch, HBASE-14123-v4.patch, > HBASE-14123-v5.patch, HBASE-14123-v6.patch, HBASE-14123-v7.patch, > HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15549) Undo HMaster carrying regions in master branch as default
[ https://issues.apache.org/jira/browse/HBASE-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214953#comment-15214953 ] Elliott Clark commented on HBASE-15549: --- How is this different from hbase.balancer.tablesOnMaster ? Seems weird to have to configs that do the same thing. > Undo HMaster carrying regions in master branch as default > - > > Key: HBASE-15549 > URL: https://issues.apache.org/jira/browse/HBASE-15549 > Project: HBase > Issue Type: Bug > Components: Balancer, master >Reporter: stack > Attachments: 15549.patch > > > I thought we had an issue to do this but I can't find it so here is a new one. > Currently, in master branch, HMaster is a RegionServer and carries meta > regions such as hbase:meta. Disable this facility by default until better > thought through (I think we should undo master ever carrying regions; FBers > are thinking we should go forward with notion that master carries meta/system > regions. TODO). I want to disable it because this facility is not finished > and meantime I want to test new stuff coming in on master branch w/o this new > feature getting in the way. > Parking here a patch that has how to disable master carrying regions and a > probably redundant test that ensures all works when master is not carrying > meta regions (added because it WASN'T working for me -- but was pilot error). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15549) Undo HMaster carrying regions in master branch as default
[ https://issues.apache.org/jira/browse/HBASE-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214956#comment-15214956 ] stack commented on HBASE-15549: --- bq. What's not finished? What Master's personality is going forward. What we have now is just wrong with master carrying regions but only some regions and then the ones it does carry are the critical ones. Master is now inline with writes. This sort of deploy won't scale (one server only carrying metadata), etc. I'm game for entertaining Master being a RegionServer or Master not being a RegionServer but this half-baked notion where Master is sort of a RegionServer. Testing, ITBLL doesn't work for me. Trying to figure why. Want to be able to undo this behavior. > Undo HMaster carrying regions in master branch as default > - > > Key: HBASE-15549 > URL: https://issues.apache.org/jira/browse/HBASE-15549 > Project: HBase > Issue Type: Bug > Components: Balancer, master >Reporter: stack > Attachments: 15549.patch > > > I thought we had an issue to do this but I can't find it so here is a new one. > Currently, in master branch, HMaster is a RegionServer and carries meta > regions such as hbase:meta. Disable this facility by default until better > thought through (I think we should undo master ever carrying regions; FBers > are thinking we should go forward with notion that master carries meta/system > regions. TODO). I want to disable it because this facility is not finished > and meantime I want to test new stuff coming in on master branch w/o this new > feature getting in the way. > Parking here a patch that has how to disable master carrying regions and a > probably redundant test that ensures all works when master is not carrying > meta regions (added because it WASN'T working for me -- but was pilot error). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15411) Rewrite backup with Procedure V2
[
https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214960#comment-15214960
]
Ted Yu commented on HBASE-15411:
TestFullBackup based on patch v27:
{code}
Running org.apache.hadoop.hbase.backup.TestFullBackup
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.667 sec - in
org.apache.hadoop.hbase.backup.TestFullBackup
{code}
> Rewrite backup with Procedure V2
>
>
> Key: HBASE-15411
> URL: https://issues.apache.org/jira/browse/HBASE-15411
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt,
> 15411-v13.txt, 15411-v14.txt, 15411-v15.txt, 15411-v16.txt, 15411-v18.txt,
> 15411-v22.txt, 15411-v3.txt, 15411-v5.txt, 15411-v6.txt, 15411-v7.txt,
> 15411-v9.txt, FullTableBackupProcedure.java
>
>
> Currently full / incremental backup is driven by BackupHandler (see call()
> method for flow).
> This issue is to rewrite the flow using Procedure V2.
> States (enum) for full / incremental backup would be introduced in
> Backup.proto which correspond to the steps performed in BackupHandler#call().
> executeFromState() would pace the backup based on the current state.
> serializeStateData() / deserializeStateData() would be used to persist state
> into procedure WAL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15549) Undo HMaster carrying regions in master branch as default
[ https://issues.apache.org/jira/browse/HBASE-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214963#comment-15214963 ] stack commented on HBASE-15549: --- bq. How is this different from hbase.balancer.tablesOnMaster ? Seems weird to have to configs that do the same thing. Thanks. [~mbertozzi] pointed out same thing. tablesOnMaster is kinda wonky. You pass 'none' to have no tables on master (As Matteo notes, you better not name your table 'none'). If you pass nothing, you get 'tables on master' (I passed 'false' because I misread that its supposed to be a list of Strings and got extra confused because it 'worked'). Matteo also notes for this to work all needs to subclass the base load balancer... not all balancers do. Will do another version of patch. Wanted to get an issue in to start catching the master-is-a-regionserver flak too. > Undo HMaster carrying regions in master branch as default > - > > Key: HBASE-15549 > URL: https://issues.apache.org/jira/browse/HBASE-15549 > Project: HBase > Issue Type: Bug > Components: Balancer, master >Reporter: stack > Attachments: 15549.patch > > > I thought we had an issue to do this but I can't find it so here is a new one. > Currently, in master branch, HMaster is a RegionServer and carries meta > regions such as hbase:meta. Disable this facility by default until better > thought through (I think we should undo master ever carrying regions; FBers > are thinking we should go forward with notion that master carries meta/system > regions. TODO). I want to disable it because this facility is not finished > and meantime I want to test new stuff coming in on master branch w/o this new > feature getting in the way. > Parking here a patch that has how to disable master carrying regions and a > probably redundant test that ensures all works when master is not carrying > meta regions (added because it WASN'T working for me -- but was pilot error). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15549) Undo HMaster carrying regions in master branch as default
[ https://issues.apache.org/jira/browse/HBASE-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214965#comment-15214965 ] Elliott Clark commented on HBASE-15549: --- bq.What we have now is just wrong with master carrying regions but only some regions and then the ones it does carry are the critical ones. I disagree with this so strongly. With a large enough cluster I get double assignments, failed splits, and every nasty horrible thing if master can't talk to meta. Without meta on master I can fail ITBLL in less than 2 days. Every single time. So without meta on master there's something seriously wrong. But even if we fixed that meta and master are wedded. Master at it's heart is something that's there to alter system tables. It doesn't really do much except read and write meta and security related tables and tell others that it's done that. Why would you put that somewhere else? For data locality it makes sense. However it is different from a regionserver in that if it goes down things in the cluster are bad. So not putting user tables on that totally makes sense. > Undo HMaster carrying regions in master branch as default > - > > Key: HBASE-15549 > URL: https://issues.apache.org/jira/browse/HBASE-15549 > Project: HBase > Issue Type: Bug > Components: Balancer, master >Reporter: stack > Attachments: 15549.patch > > > I thought we had an issue to do this but I can't find it so here is a new one. > Currently, in master branch, HMaster is a RegionServer and carries meta > regions such as hbase:meta. Disable this facility by default until better > thought through (I think we should undo master ever carrying regions; FBers > are thinking we should go forward with notion that master carries meta/system > regions. TODO). I want to disable it because this facility is not finished > and meantime I want to test new stuff coming in on master branch w/o this new > feature getting in the way. > Parking here a patch that has how to disable master carrying regions and a > probably redundant test that ensures all works when master is not carrying > meta regions (added because it WASN'T working for me -- but was pilot error). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15544) HFileReaderImpl.readBlock allocates memory for HFileBlock on every call
[ https://issues.apache.org/jira/browse/HBASE-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214983#comment-15214983 ] stack commented on HBASE-15544: --- Yeah, we allocate buffers to do compression. What you suggest [~vrodionov]? > HFileReaderImpl.readBlock allocates memory for HFileBlock on every call > --- > > Key: HBASE-15544 > URL: https://issues.apache.org/jira/browse/HBASE-15544 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > This accounts for 25% of memory allocation during compaction when compression > is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214984#comment-15214984 ] Elliott Clark commented on HBASE-14613: --- bq.No, no, not set to lower, instead, calculate the size of pinned in Old and set this number higher. Yeah sorry that's what I meant. If we upped it then we were not initiating fast enough. Too low and things were getting promoted before mixed gens. We run our regionservers pretty much at the edge of what the > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214984#comment-15214984 ] Elliott Clark edited comment on HBASE-14613 at 3/28/16 10:11 PM: - bq.No, no, not set to lower, instead, calculate the size of pinned in Old and set this number higher. Yeah sorry that's what I meant. If we upped it then we were not initiating fast enough. Too low and things were getting promoted before mixed gens. We run our regionservers pretty much at the edge of what the gc seems capable. was (Author: eclark): bq.No, no, not set to lower, instead, calculate the size of pinned in Old and set this number higher. Yeah sorry that's what I meant. If we upped it then we were not initiating fast enough. Too low and things were getting promoted before mixed gens. We run our regionservers pretty much at the edge of what the > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215009#comment-15215009 ] stack commented on HBASE-15525: --- So, just to note, on read side, we are already reading in 'chunks'. If you look at RpcServer, when we read in the request, we are currently reading into an onheap BB -- until HBASE-14490 [RpcServer] reuse request read buffer goes in at least -- that is allocated just to hold the response. We fill this BB by reading off the socket channel in chunks of NIO_BUFFER_LIMIT (64kb). Seems like we could plug in an MBB here that referenced fixed-size BBs gotten from a pool as you are doing here -- so both read and write-side. > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Attachments: WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215010#comment-15215010 ] Ted Yu commented on HBASE-15362: [~dvdreddy]: How much reduction in garbage did you observe with the above change ? Thanks > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15544) HFileReaderImpl.readBlock allocates memory for HFileBlock on every call
[ https://issues.apache.org/jira/browse/HBASE-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215019#comment-15215019 ] Vladimir Rodionov commented on HBASE-15544: --- Reuse them if cache-on-read is disabled (including compaction case). > HFileReaderImpl.readBlock allocates memory for HFileBlock on every call > --- > > Key: HBASE-15544 > URL: https://issues.apache.org/jira/browse/HBASE-15544 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > This accounts for 25% of memory allocation during compaction when compression > is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15339) Improve DateTieredCompactionPolicy
[
https://issues.apache.org/jira/browse/HBASE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215020#comment-15215020
]
Enis Soztutar commented on HBASE-15339:
---
[~clarax98007], [~Apache9] do you guys think that we can address all subtasks
of this by 1.3 timeframe. HBASE-15454 needs a design doc, so maybe that is out,
but it would be good if we can get everything in before 1.3.
BTW, looking at
{{hbase.hstore.compaction.date.tiered.max.storefile.age.millis}} is the
description in the document correct? After this age, we do not compact files,
so if you have a TTL larger than
{{hbase.hstore.compaction.date.tiered.max.storefile.age.millis}}, our TTL-based
file deletion does not work anymore? Something to check / fix.
> Improve DateTieredCompactionPolicy
> --
>
> Key: HBASE-15339
> URL: https://issues.apache.org/jira/browse/HBASE-15339
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Reporter: Duo Zhang
>
> Add multi-output support.
> Add archive old data support.
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Clara Xiong updated HBASE-15400: Attachment: HBASE-15400-v3.patch > Use DateTieredCompactor for Date Tiered Compaction > -- > > Key: HBASE-15400 > URL: https://issues.apache.org/jira/browse/HBASE-15400 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15400-15389-v12.patch, HBASE-15400-v1.pa, > HBASE-15400-v3.patch, HBASE-15400.patch > > > When we compact, we can output multiple files along the current window > boundaries. There are two use cases: > 1. Major compaction: We want to output date tiered store files with data > older than max age archived in trunks of the window size on the higher > tier.1. Once a file is old enough to be out of the range that we do tiered > minor compaction, we don't compact them any further. So they retain the same > timespan as they were compacted last time, which is the window size of the > highest tier. Major compaction will touch these files and we want to maintain > the same layout. > 2. Bulk load files and the old file generated by major compaction before > upgrading to DTCP. > Pros: > 1. Restore locality, process versioning, updates and deletes while > maintaining the tiered layout. > 2. The best way to fix a skewed layout. > > This work is based on a prototype of DateTieredCompactor from HBASE-15389 and > focused on the part to meet needs for these two use cases while supporting > others. I have to call out a few design decisions: > 1. We only want to output the files along all windows for major compaction. > And we want to output multiple files older than max age in the trunks of the > maximum tier window size determined by base window size, windows per tier and > max age. > 2. For minor compaction, we don't want to output too many files, which will > remain around because of current restriction of contiguous compaction by seq > id. I will only output two files if all the files in the windows are being > combined, one for the data within window and the other for the out-of-window > tail. If there is any file in the window excluded from compaction, only one > file will be output from compaction. When the windows are promoted, the > situation of out of order data will gradually improve. For the incoming > window, we need to accommodate the case with user-specified future data. > 3. We have to pass the boundaries with the list of store file as a complete > time snapshot instead of two separate calls because window layout is > determined by the time the computation is called. So we will need new type of > compaction request. > 4. Since we will assign the same seq id for all output files, we need to sort > by maxTimestamp subsequently. Right now all compaction policy gets the files > sorted for StoreFileManager which sorts by seq id and other criteria. I will > use this order for DTCP only, to avoid impacting other compaction policies. > 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215028#comment-15215028 ] Clara Xiong commented on HBASE-15400: - Resolved a couple of merge conflict after HBASE-15389 is committed. Also uploaded to RB for review. > Use DateTieredCompactor for Date Tiered Compaction > -- > > Key: HBASE-15400 > URL: https://issues.apache.org/jira/browse/HBASE-15400 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15400-15389-v12.patch, HBASE-15400-v1.pa, > HBASE-15400-v3.patch, HBASE-15400.patch > > > When we compact, we can output multiple files along the current window > boundaries. There are two use cases: > 1. Major compaction: We want to output date tiered store files with data > older than max age archived in trunks of the window size on the higher > tier.1. Once a file is old enough to be out of the range that we do tiered > minor compaction, we don't compact them any further. So they retain the same > timespan as they were compacted last time, which is the window size of the > highest tier. Major compaction will touch these files and we want to maintain > the same layout. > 2. Bulk load files and the old file generated by major compaction before > upgrading to DTCP. > Pros: > 1. Restore locality, process versioning, updates and deletes while > maintaining the tiered layout. > 2. The best way to fix a skewed layout. > > This work is based on a prototype of DateTieredCompactor from HBASE-15389 and > focused on the part to meet needs for these two use cases while supporting > others. I have to call out a few design decisions: > 1. We only want to output the files along all windows for major compaction. > And we want to output multiple files older than max age in the trunks of the > maximum tier window size determined by base window size, windows per tier and > max age. > 2. For minor compaction, we don't want to output too many files, which will > remain around because of current restriction of contiguous compaction by seq > id. I will only output two files if all the files in the windows are being > combined, one for the data within window and the other for the out-of-window > tail. If there is any file in the window excluded from compaction, only one > file will be output from compaction. When the windows are promoted, the > situation of out of order data will gradually improve. For the incoming > window, we need to accommodate the case with user-specified future data. > 3. We have to pass the boundaries with the list of store file as a complete > time snapshot instead of two separate calls because window layout is > determined by the time the computation is called. So we will need new type of > compaction request. > 4. Since we will assign the same seq id for all output files, we need to sort > by maxTimestamp subsequently. Right now all compaction policy gets the files > sorted for StoreFileManager which sorts by seq id and other criteria. I will > use this order for DTCP only, to avoid impacting other compaction policies. > 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215047#comment-15215047 ] deepankar commented on HBASE-15362: --- Depends on the default block size we have a default block size of 16 kb and this allocates a buffer of 64 kb , so in per read we are saving 64 - 16 kb of buffer I think. > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215052#comment-15215052 ] Ted Yu commented on HBASE-15362: Do you want to include the above change in a separate JIRA ? > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215056#comment-15215056 ] deepankar commented on HBASE-15362: --- But this change is very specific to LZO and specialized (hacky) , is it ok include it ? We hacked it as a temporary solution. > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Bug >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15550) Backport HBASE-12220 hedgedRead metrics to 1.2+
stack created HBASE-15550: - Summary: Backport HBASE-12220 hedgedRead metrics to 1.2+ Key: HBASE-15550 URL: https://issues.apache.org/jira/browse/HBASE-15550 Project: HBase Issue Type: Sub-task Reporter: stack hedgedRead metrics are in master branch only. They were put in branch-1 but then reverted because old versions of hadoop didn't supported hedgedReads. 1.2+ has minimal hadoop version. Retry backport. FYI [~phobos182] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15550) Backport HBASE-12220 hedgedRead metrics to 1.2+
[ https://issues.apache.org/jira/browse/HBASE-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15550: -- Attachment: 15550.branch-1.2.txt Lets see how this patch does. > Backport HBASE-12220 hedgedRead metrics to 1.2+ > --- > > Key: HBASE-15550 > URL: https://issues.apache.org/jira/browse/HBASE-15550 > Project: HBase > Issue Type: Sub-task >Reporter: stack > Fix For: 2.0.0 > > Attachments: 15550.branch-1.2.txt > > > hedgedRead metrics are in master branch only. They were put in branch-1 but > then reverted because old versions of hadoop didn't supported hedgedReads. > 1.2+ has minimal hadoop version. Retry backport. > FYI [~phobos182] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15550) Backport HBASE-12220 hedgedRead metrics to 1.2+
[ https://issues.apache.org/jira/browse/HBASE-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15550: -- Assignee: stack Affects Version/s: 1.2.2 1.3.0 Status: Patch Available (was: Open) > Backport HBASE-12220 hedgedRead metrics to 1.2+ > --- > > Key: HBASE-15550 > URL: https://issues.apache.org/jira/browse/HBASE-15550 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.0, 1.2.2 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 15550.branch-1.2.txt > > > hedgedRead metrics are in master branch only. They were put in branch-1 but > then reverted because old versions of hadoop didn't supported hedgedReads. > 1.2+ has minimal hadoop version. Retry backport. > FYI [~phobos182] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15550) Backport HBASE-12220 hedgedRead metrics to 1.2+
[ https://issues.apache.org/jira/browse/HBASE-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215081#comment-15215081 ] Sean Busbey commented on HBASE-15550: - Should we move Hadoop 2.2 and 2.3 from NT to X in the [basic prerequisites list for hadoop|http://hbase.apache.org/book.html#hadoop] since we know it'll fail with this patch? > Backport HBASE-12220 hedgedRead metrics to 1.2+ > --- > > Key: HBASE-15550 > URL: https://issues.apache.org/jira/browse/HBASE-15550 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.0, 1.2.2 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 15550.branch-1.2.txt > > > hedgedRead metrics are in master branch only. They were put in branch-1 but > then reverted because old versions of hadoop didn't supported hedgedReads. > 1.2+ has minimal hadoop version. Retry backport. > FYI [~phobos182] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15548) SyncTable: sourceHashDir is supposed to be optional but won't work without
[
https://issues.apache.org/jira/browse/HBASE-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215100#comment-15215100
]
Hadoop QA commented on HBASE-15548:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 126m 24s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 181m 17s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795674/HBASE-15548.patch |
| JIRA Issue | HBASE-15548 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c96b642 |
| Default Java
[jira] [Updated] (HBASE-15548) SyncTable: sourceHashDir is supposed to be optional but won't work without
[ https://issues.apache.org/jira/browse/HBASE-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15548: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 Status: Resolved (was: Patch Available) Thanks for the patch, Dave. > SyncTable: sourceHashDir is supposed to be optional but won't work without > --- > > Key: HBASE-15548 > URL: https://issues.apache.org/jira/browse/HBASE-15548 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 > Environment: Ubuntu 12.04 >Reporter: My Ho >Assignee: Dave Latham >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15548.patch > > > 1) SyncTable code is contradictory. Usage said sourcehashdir is optional > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L687). > However, the command won't run if sourcehashdir is missing > (https://github.com/apache/hbase/blob/ad3feaa44800f10d102255a240c38ccf23a82d49/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/SyncTable.java#L83-L85) > > 2) There is no documentation on how to create the desired sourcehash -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215113#comment-15215113
]
Enis Soztutar commented on HBASE-15295:
---
Thanks Stack for the review. I was trying to commit this last week, but had to
adjust for branch-1, 1.2, run unit tests, etc which takes some time.
bq. RPC while holding a lock is always going to be fun. Should we open an issue
to undo this? It is always going to be problematic. This fixes one means by
which the RPC can fail but there'll be others to follow?
Phoenix defines its own RPCScheduler and controller which adds a handler pool
for the index updates versus regular updates. The rpc scheduler + rpc
controller used properly should in-theory make it so that this kind of
deadlocks do not happen. In this particular case, it is an HBase bug that the
coprocessor call to update meta is not routed with the high priority. But I
agree on the general statement that we may have to re-think how secondary
indexes work in Phoenix. Did not check how the transactional stuff works with
secondary indexes. Without looking at the code, I am assuming that we do not
block the handlers / MVCC for the secondary index updates.
There is one more issue that I did not open a jira for, which is that due to
the fact that we have to replay the secondary index updates when recovering
regions for the data table, there is a soft deadlock that can happen in the
region opening. We have seen this recently in a cluster where region opener
threads are blocked waiting on index regions to come online, but index regions
are just sitting in the same region opening queue waiting for the data regions
to be opened. Will create the jira.
bq. Are you referring to the fact that hbase:meta still has the old-style
encoding? Yeah, would need a migration step to move from old to new. Could
clean up some code that does the check for old vs new encoding if this were
done.
Yep. I don't have the context for why we did not update the meta to have an
encoded name, but now we have to handle the non-encoded regions and encoded
regions differently. A pity we did not realize this in singularity.
bq. That seems like 'proper' use of an RPC 'controller' type object So,
ignore this comment (smile). We'll need an RPC-type controller thing whatever
our RPC. And it seems like it is all over our client code already. I suppose
you are just formalizing the 'damage' done already.
Agreed. my understanding is that the RPCController is now passed around via the
factory, and we are controlling the priority, deadline, etc with the
controller. The patch just follows the same thing that we do elsewhere (other
admin methods, htable methods, etc).
Will commit shortly.
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_v1.patch, hbase-15295_v1.patch,
> hbase-15295_v2.patch, hbase-15295_v3.patch, hbase-15295_v4.patch,
> hbase-15295_v5.patch, hbase-15295_v5.patch, hbase-15295_v6.patch,
> hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndex
[jira] [Commented] (HBASE-15191) CopyTable and VerifyReplication - Option to specify batch size, versions
[
https://issues.apache.org/jira/browse/HBASE-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215118#comment-15215118
]
Hadoop QA commented on HBASE-15191:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 6m 52s
{color} | {color:red} hbase-server: patch generated 3 new + 19 unchanged - 0
fixed = 22 total (was 19) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
43m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 116m 41s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 200m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2
|
| Timed out junit tests |
org.apache.hadoop.hbase.security.access.TestAccessController |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795689/HBASE_15191.patch |
| JIRA Issue | HBASE-15191 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf900.gq1.ygridco
[jira] [Commented] (HBASE-15545) org.apache.hadoop.io.compress.DecompressorStream allocates too much memory
[ https://issues.apache.org/jira/browse/HBASE-15545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215126#comment-15215126 ] deepankar commented on HBASE-15545: --- We also had a similar issue, it also allocates a lot of memory in the RPC path also, we fixed it an hacky way, will be good if we can come up with a better fix Relevant Comment : https://issues.apache.org/jira/browse/HBASE-15362?focusedCommentId=15174292&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15174292 > org.apache.hadoop.io.compress.DecompressorStream allocates too much memory > -- > > Key: HBASE-15545 > URL: https://issues.apache.org/jira/browse/HBASE-15545 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > It accounts for ~ 11% of overall memory allocation during compaction when > compression (GZ) is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215144#comment-15215144 ] Hudson commented on HBASE-15508: SUCCESS: Integrated in HBase-1.1-JDK7 #1690 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1690/]) Revert "HBASE-15508 Add command for exporting snapshot in hbase command (busbey: rev e959eb36ffb7feb249294491ba8037fd9a63404b) * bin/hbase * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215149#comment-15215149
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-14123 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.2.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795715/HBASE-14123-v13.patch
|
| JIRA Issue | HBASE-14123 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1197/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> HBase Backup/Restore Phase 2
>
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
> Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14123-v1.patch, HBASE-14123-v10.patch,
> HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v13.patch,
> HBASE-14123-v2.patch, HBASE-14123-v3.patch, HBASE-14123-v4.patch,
> HBASE-14123-v5.patch, HBASE-14123-v6.patch, HBASE-14123-v7.patch,
> HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15551) Make call queue too big exception use servername
Elliott Clark created HBASE-15551: - Summary: Make call queue too big exception use servername Key: HBASE-15551 URL: https://issues.apache.org/jira/browse/HBASE-15551 Project: HBase Issue Type: Bug Reporter: Elliott Clark If the rpc server is listening to something other than the hostname ( 0.0.0.0 for example ) then the exception thrown isn't very helpful. We should make that more helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15551) Make call queue too big exception use servername
[ https://issues.apache.org/jira/browse/HBASE-15551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15551: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) > Make call queue too big exception use servername > > > Key: HBASE-15551 > URL: https://issues.apache.org/jira/browse/HBASE-15551 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Priority: Minor > > If the rpc server is listening to something other than the hostname ( 0.0.0.0 > for example ) then the exception thrown isn't very helpful. We should make > that more helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15551) Make call queue too big exception use servername
[ https://issues.apache.org/jira/browse/HBASE-15551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15551: -- Affects Version/s: 1.2.0 > Make call queue too big exception use servername > > > Key: HBASE-15551 > URL: https://issues.apache.org/jira/browse/HBASE-15551 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark > > If the rpc server is listening to something other than the hostname ( 0.0.0.0 > for example ) then the exception thrown isn't very helpful. We should make > that more helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-15295:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed this. Thanks everyone.
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_v1.patch, hbase-15295_v1.patch,
> hbase-15295_v2.patch, hbase-15295_v3.patch, hbase-15295_v4.patch,
> hbase-15295_v5.patch, hbase-15295_v5.patch, hbase-15295_v6.patch,
> hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> And the index region is trying to split, and is trying to do a meta update:
> {code}
> "regionserver//10.132.70.191:16020-splits-1454693389669" #1779
> prio=5 os_prio=0 t
[jira] [Commented] (HBASE-15550) Backport HBASE-12220 hedgedRead metrics to 1.2+
[
https://issues.apache.org/jira/browse/HBASE-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215208#comment-15215208
]
Hadoop QA commented on HBASE-15550:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.2.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
21s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 16s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
42s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
21s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 27s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 39s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 11s
{color} | {color:red} hbase-hadoop2-compat in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 14s
{color} | {color:red} hbase-hadoop2-compat in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 50s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 41s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 26s
{color} | {color:red} hbase-hadoop2-compat in the patch failed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 25s
{color} | {color:red} hbase-hadoop2-compat in the patch failed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 59s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 0s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 27s {color}
| {color:red} hbase-hadoop2-compat in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s {color}
| {color:red} hbase-hadoop2-compat in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 59s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 0s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 12s
{color} | {color:red} hbase-hadoop2-compat in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 15s
{color} | {color:red} hbase-hadoop2-compat in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 48s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 48s
{color} |
[jira] [Commented] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215217#comment-15215217 ] Hudson commented on HBASE-15508: FAILURE: Integrated in HBase-1.1-JDK8 #1776 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1776/]) Revert "HBASE-15508 Add command for exporting snapshot in hbase command (busbey: rev e959eb36ffb7feb249294491ba8037fd9a63404b) * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * bin/hbase * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15339) Improve DateTieredCompactionPolicy
[ https://issues.apache.org/jira/browse/HBASE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215260#comment-15215260 ] Duo Zhang commented on HBASE-15339: --- HBASE-15454 maybe out but I think it is OK to backport it to 1.3 later, it does not change any behavior of existing code. And for the max age, IMO, it does not make sense to have some files that out of compaction control. The max age will work with HBASE-15454 together since the files older than max age will be archived. And if you use TTL, then you'd better not use max age? Thanks. > Improve DateTieredCompactionPolicy > -- > > Key: HBASE-15339 > URL: https://issues.apache.org/jira/browse/HBASE-15339 > Project: HBase > Issue Type: Improvement > Components: Compaction >Reporter: Duo Zhang > > Add multi-output support. > Add archive old data support. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15339) Improve DateTieredCompactionPolicy
[
https://issues.apache.org/jira/browse/HBASE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215275#comment-15215275
]
Enis Soztutar commented on HBASE-15339:
---
bq. And for the max age, IMO, it does not make sense to have some files that
out of compaction control
You think that we should not have
{{hbase.hstore.compaction.date.tiered.max.storefile.age.millis}} as a conf
option at all? [~clarax98007] wdyt?
bq. And if you use TTL, then you'd better not use max age?
See FifoCompactionPolicy.selectCompaction(). We select only expired files in
the compaction selection so that TTL'ed data can still be deleted even though
fifo compaction does not perform any compactions. We can do the same trick
here.
> Improve DateTieredCompactionPolicy
> --
>
> Key: HBASE-15339
> URL: https://issues.apache.org/jira/browse/HBASE-15339
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Reporter: Duo Zhang
>
> Add multi-output support.
> Add archive old data support.
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15339) Improve DateTieredCompactionPolicy
[ https://issues.apache.org/jira/browse/HBASE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215289#comment-15215289 ] Duo Zhang commented on HBASE-15339: --- I mean we can have the conf option, but we should only set it when archive is enabled. > Improve DateTieredCompactionPolicy > -- > > Key: HBASE-15339 > URL: https://issues.apache.org/jira/browse/HBASE-15339 > Project: HBase > Issue Type: Improvement > Components: Compaction >Reporter: Duo Zhang > > Add multi-output support. > Add archive old data support. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15339) Improve DateTieredCompactionPolicy
[ https://issues.apache.org/jira/browse/HBASE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215300#comment-15215300 ] Clara Xiong commented on HBASE-15339: - maxAge, in short words, is the configuration to control how far you go when promoting tiers. Once a window is out of this range, say more than a year old from current time, we don't combine the windows to promote to the next tiers any more. It doesn't mean they are out of compaction control. Within DTC, minor compaction won't touch these files any more but major compaction will still touch them. It actually makes TTL or archiving much simpler and more efficient since any files older than max age but younger than TTL (or no TTL) will not be combined(I used the term "compacted" but this might confusing) into larger sizes.This way, TTL and archiving process can just scan from oldest files to remove/archive files based on their time range instead of scanning a very large file from major compaction by Ratio-based compaction. The design here is similar to FIFOCompactionPolicy you described. > Improve DateTieredCompactionPolicy > -- > > Key: HBASE-15339 > URL: https://issues.apache.org/jira/browse/HBASE-15339 > Project: HBase > Issue Type: Improvement > Components: Compaction >Reporter: Duo Zhang > > Add multi-output support. > Add archive old data support. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Clara Xiong updated HBASE-15400: Description: When we compact, we can output multiple files along the current window boundaries. There are two use cases: 1. Major compaction: We want to output date tiered store files with data older than max age archived in trunks of the window size on the higher tier. Once a window is old enough, we don't combine the windows to promote to the next tier any further. So files in these windows retain the same timespan as they were minor-compacted last time, which is the window size of the highest tier. Major compaction will touch these files and we want to maintain the same layout. 2. Bulk load files and the old file generated by major compaction before upgrading to DTCP. Pros: 1. Restore locality, process versioning, updates and deletes while maintaining the tiered layout. 2. The best way to fix a skewed layout. This work is based on a prototype of DateTieredCompactor from HBASE-15389 and focused on the part to meet needs for these two use cases while supporting others. I have to call out a few design decisions: 1. We only want to output the files along all windows for major compaction. And we want to output multiple files older than max age in the trunks of the maximum tier window size determined by base window size, windows per tier and max age. 2. For minor compaction, we don't want to output too many files, which will remain around because of current restriction of contiguous compaction by seq id. I will only output two files if all the files in the windows are being combined, one for the data within window and the other for the out-of-window tail. If there is any file in the window excluded from compaction, only one file will be output from compaction. When the windows are promoted, the situation of out of order data will gradually improve. For the incoming window, we need to accommodate the case with user-specified future data. 3. We have to pass the boundaries with the list of store file as a complete time snapshot instead of two separate calls because window layout is determined by the time the computation is called. So we will need new type of compaction request. 4. Since we will assign the same seq id for all output files, we need to sort by maxTimestamp subsequently. Right now all compaction policy gets the files sorted for StoreFileManager which sorts by seq id and other criteria. I will use this order for DTCP only, to avoid impacting other compaction policies. 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. was: When we compact, we can output multiple files along the current window boundaries. There are two use cases: 1. Major compaction: We want to output date tiered store files with data older than max age archived in trunks of the window size on the higher tier.1. Once a file is old enough to be out of the range that we do tiered minor compaction, we don't compact them any further. So they retain the same timespan as they were compacted last time, which is the window size of the highest tier. Major compaction will touch these files and we want to maintain the same layout. 2. Bulk load files and the old file generated by major compaction before upgrading to DTCP. Pros: 1. Restore locality, process versioning, updates and deletes while maintaining the tiered layout. 2. The best way to fix a skewed layout. This work is based on a prototype of DateTieredCompactor from HBASE-15389 and focused on the part to meet needs for these two use cases while supporting others. I have to call out a few design decisions: 1. We only want to output the files along all windows for major compaction. And we want to output multiple files older than max age in the trunks of the maximum tier window size determined by base window size, windows per tier and max age. 2. For minor compaction, we don't want to output too many files, which will remain around because of current restriction of contiguous compaction by seq id. I will only output two files if all the files in the windows are being combined, one for the data within window and the other for the out-of-window tail. If there is any file in the window excluded from compaction, only one file will be output from compaction. When the windows are promoted, the situation of out of order data will gradually improve. For the incoming window, we need to accommodate the case with user-specified future data. 3. We have to pass the boundaries with the list of store file as a complete time snapshot instead of two separate calls because window layout is determined by the time the computation is called. So we will need new type of compaction request. 4. Since we will assign the same seq id for all output files, we need to sort by maxTimestamp subsequently. Right now all com
[jira] [Updated] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Clara Xiong updated HBASE-15400: Description: When we compact, we can output multiple files along the current window boundaries. There are two use cases: 1. Major compaction: We want to output date tiered store files with data older than max age archived in trunks of the window size on the higher tier. Once a window is old enough, we don't combine the windows to promote to the next tier any further. So files in these windows retain the same timespan as they were minor-compacted last time, which is the window size of the highest tier. Major compaction will touch these files and we want to maintain the same layout. This way, TTL and archiving will be simpler and more efficient. 2. Bulk load files and the old file generated by major compaction before upgrading to DTCP. Pros: 1. Restore locality, process versioning, updates and deletes while maintaining the tiered layout. 2. The best way to fix a skewed layout. This work is based on a prototype of DateTieredCompactor from HBASE-15389 and focused on the part to meet needs for these two use cases while supporting others. I have to call out a few design decisions: 1. We only want to output the files along all windows for major compaction. And we want to output multiple files older than max age in the trunks of the maximum tier window size determined by base window size, windows per tier and max age. 2. For minor compaction, we don't want to output too many files, which will remain around because of current restriction of contiguous compaction by seq id. I will only output two files if all the files in the windows are being combined, one for the data within window and the other for the out-of-window tail. If there is any file in the window excluded from compaction, only one file will be output from compaction. When the windows are promoted, the situation of out of order data will gradually improve. For the incoming window, we need to accommodate the case with user-specified future data. 3. We have to pass the boundaries with the list of store file as a complete time snapshot instead of two separate calls because window layout is determined by the time the computation is called. So we will need new type of compaction request. 4. Since we will assign the same seq id for all output files, we need to sort by maxTimestamp subsequently. Right now all compaction policy gets the files sorted for StoreFileManager which sorts by seq id and other criteria. I will use this order for DTCP only, to avoid impacting other compaction policies. 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. was: When we compact, we can output multiple files along the current window boundaries. There are two use cases: 1. Major compaction: We want to output date tiered store files with data older than max age archived in trunks of the window size on the higher tier. Once a window is old enough, we don't combine the windows to promote to the next tier any further. So files in these windows retain the same timespan as they were minor-compacted last time, which is the window size of the highest tier. Major compaction will touch these files and we want to maintain the same layout. 2. Bulk load files and the old file generated by major compaction before upgrading to DTCP. Pros: 1. Restore locality, process versioning, updates and deletes while maintaining the tiered layout. 2. The best way to fix a skewed layout. This work is based on a prototype of DateTieredCompactor from HBASE-15389 and focused on the part to meet needs for these two use cases while supporting others. I have to call out a few design decisions: 1. We only want to output the files along all windows for major compaction. And we want to output multiple files older than max age in the trunks of the maximum tier window size determined by base window size, windows per tier and max age. 2. For minor compaction, we don't want to output too many files, which will remain around because of current restriction of contiguous compaction by seq id. I will only output two files if all the files in the windows are being combined, one for the data within window and the other for the out-of-window tail. If there is any file in the window excluded from compaction, only one file will be output from compaction. When the windows are promoted, the situation of out of order data will gradually improve. For the incoming window, we need to accommodate the case with user-specified future data. 3. We have to pass the boundaries with the list of store file as a complete time snapshot instead of two separate calls because window layout is determined by the time the computation is called. So we will need new type of compaction request. 4. Since we will assign the same seq id for all output files
[jira] [Updated] (HBASE-13590) TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey
[ https://issues.apache.org/jira/browse/HBASE-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13590: Fix Version/s: (was: 1.2.1) 1.2.0 > TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey > --- > > Key: HBASE-13590 > URL: https://issues.apache.org/jira/browse/HBASE-13590 > Project: HBase > Issue Type: Test > Components: master >Reporter: Nick Dimiduk >Assignee: Yu Li > Fix For: 1.2.0, 1.3.0, 1.1.4 > > Attachments: HBASE-13590.branch-1.1.patch, > HBASE-13590.branch-1.1.patch, HBASE-13590.branch-1.1.patch, > HBASE-13590.branch-1.patch, HBASE-13590.branch-1.v2.patch, > testEnableTableHandler_branch-1.1.log.zip, > testEnableTableHandler_branch-1.log.zip > > > Looking at our [build > history|https://builds.apache.org/job/HBase-1.1/buildTimeTrend], it seems > this test is flakey. See builds 429, 431, 439. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[
https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215320#comment-15215320
]
Hadoop QA commented on HBASE-15400:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 17s
{color} | {color:red} hbase-server: patch generated 2 new + 110 unchanged - 28
fixed = 112 total (was 138) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 13s
{color} | {color:red} hbase-server generated 5 new + 0 unchanged - 0 fixed = 5
total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 40s
{color} | {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 1 new + 1
unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 13s
{color} | {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 1
new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 112m 32s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 30s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Boxed value is unboxed and then immediately reboxed in
org.apache.hadoop.hbase.regionserver.StoreFile$
[jira] [Commented] (HBASE-11393) Replication TableCfs should be a PB object rather than a string
[ https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215324#comment-15215324 ] Heng Chen commented on HBASE-11393: --- push to master. Need backport it to branch-1, [~enis] ? > Replication TableCfs should be a PB object rather than a string > --- > > Key: HBASE-11393 > URL: https://issues.apache.org/jira/browse/HBASE-11393 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-11393.patch, HBASE-11393_v1.patch, > HBASE-11393_v10.patch, HBASE-11393_v11.patch, HBASE-11393_v12.patch, > HBASE-11393_v14.patch, HBASE-11393_v15.patch, HBASE-11393_v16.patch, > HBASE-11393_v2.patch, HBASE-11393_v3.patch, HBASE-11393_v4.patch, > HBASE-11393_v5.patch, HBASE-11393_v6.patch, HBASE-11393_v7.patch, > HBASE-11393_v8.patch, HBASE-11393_v9.patch > > > We concatenate the list of tables and column families in format > "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer > mapping. > This results in ugly parsing code. We should do this a PB object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11393) Replication TableCfs should be a PB object rather than a string
[ https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215336#comment-15215336 ] Enis Soztutar commented on HBASE-11393: --- I think we can leave this in master only unless we need it otherwise. > Replication TableCfs should be a PB object rather than a string > --- > > Key: HBASE-11393 > URL: https://issues.apache.org/jira/browse/HBASE-11393 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-11393.patch, HBASE-11393_v1.patch, > HBASE-11393_v10.patch, HBASE-11393_v11.patch, HBASE-11393_v12.patch, > HBASE-11393_v14.patch, HBASE-11393_v15.patch, HBASE-11393_v16.patch, > HBASE-11393_v2.patch, HBASE-11393_v3.patch, HBASE-11393_v4.patch, > HBASE-11393_v5.patch, HBASE-11393_v6.patch, HBASE-11393_v7.patch, > HBASE-11393_v8.patch, HBASE-11393_v9.patch > > > We concatenate the list of tables and column families in format > "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer > mapping. > This results in ugly parsing code. We should do this a PB object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11393) Replication TableCfs should be a PB object rather than a string
[ https://issues.apache.org/jira/browse/HBASE-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-11393: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Replication TableCfs should be a PB object rather than a string > --- > > Key: HBASE-11393 > URL: https://issues.apache.org/jira/browse/HBASE-11393 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-11393.patch, HBASE-11393_v1.patch, > HBASE-11393_v10.patch, HBASE-11393_v11.patch, HBASE-11393_v12.patch, > HBASE-11393_v14.patch, HBASE-11393_v15.patch, HBASE-11393_v16.patch, > HBASE-11393_v2.patch, HBASE-11393_v3.patch, HBASE-11393_v4.patch, > HBASE-11393_v5.patch, HBASE-11393_v6.patch, HBASE-11393_v7.patch, > HBASE-11393_v8.patch, HBASE-11393_v9.patch > > > We concatenate the list of tables and column families in format > "table1:cf1,cf2;table2:cfA,cfB" in zookeeper for table-cf to replication peer > mapping. > This results in ugly parsing code. We should do this a PB object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
