[jira] [Commented] (HBASE-13150) TestMasterObserver failing disable table at end of test
[ https://issues.apache.org/jira/browse/HBASE-13150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348352#comment-14348352 ] Andrey Stepachev commented on HBASE-13150: -- [~zhangduo] at first I tried to make thread BulkUnassign more intelligent, it peeks regions and later rechecks that all regions not OPENED, if so it rescheduled. But later I found piece of code removed in HBASE-13076 in this test never failed in my setup (2/3 times it failed before) flackery > TestMasterObserver failing disable table at end of test > --- > > Key: HBASE-13150 > URL: https://issues.apache.org/jira/browse/HBASE-13150 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > > I see in > https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6202/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestMasterObserver/testRegionTransitionOperations/ > , now we have added in timeouts, that we are failing to disable a table. It > looks like table is disabled but regions are being opened on the disabled > table still, like HBASE-6537 > Let me see if can figure why this happening. Will be back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13012) Add shell commands to trigger the mob file compactor
[ https://issues.apache.org/jira/browse/HBASE-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348354#comment-14348354 ] Jonathan Hsieh commented on HBASE-13012: v9 lgtm. +1. thanks for the patch jiajia and jingcheng. committed to hbase-11339 branch. > Add shell commands to trigger the mob file compactor > - > > Key: HBASE-13012 > URL: https://issues.apache.org/jira/browse/HBASE-13012 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13012-V5.diff, HBASE-13012-V6.diff, > HBASE-13012-V7.diff, HBASE-13012-V9.diff, HBASE-13012.v8.patch, > hbase-13012-V1.patch, hbase-13012-V2.diff, hbase-13012-V3.diff, > hbase-13012-V4.diff > > > Currently the MobFileCompactor is run by HMaster periodically, we need to add > a shell to trigger the compactor by commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-13012) Add shell commands to trigger the mob file compactor
[ https://issues.apache.org/jira/browse/HBASE-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh resolved HBASE-13012. Resolution: Fixed Hadoop Flags: Reviewed > Add shell commands to trigger the mob file compactor > - > > Key: HBASE-13012 > URL: https://issues.apache.org/jira/browse/HBASE-13012 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13012-V5.diff, HBASE-13012-V6.diff, > HBASE-13012-V7.diff, HBASE-13012-V9.diff, HBASE-13012.v8.patch, > hbase-13012-V1.patch, hbase-13012-V2.diff, hbase-13012-V3.diff, > hbase-13012-V4.diff > > > Currently the MobFileCompactor is run by HMaster periodically, we need to add > a shell to trigger the compactor by commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13012) Add shell commands to trigger the mob file compactor
[ https://issues.apache.org/jira/browse/HBASE-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13012: --- Release Note: This adds two new shell commands -- compact_mob and major_compact_mob to the hbase shell. Run compaction on a mob enabled column family or all mob enabled column families within a table Examples: Compact a column family within a table: hbase> compact_mob 't1', 'c1' Compact all mob enabled column families hbase> compact_mob 't1' Run major compaction on a mob enabled column family or all mob enabled column families within a table Examples: Compact a column family within a table: hbase> major_compact_mob 't1', 'c1' Compact all mob enabled column families within a table hbase> major_compact_mob 't1' was: This adds two new shell commands -- compact_mob and major_compact_mob to the hbase shell. Run compaction on a mob enabled column family or all mob enabled column families within a table Examples: Compact a column family within a table: hbase> compact_mob 't1', 'c1' Compact all mob enabled column families hbase> compact_mob 't1' Run major compaction on a mob enabled column family or all mob enabled column families within a table Examples: Compact a column family within a table: hbase> major_compact_mob 't1', 'c1' Compact all mob enabled column families within a table hbase> major_compact_mob 't1' > Add shell commands to trigger the mob file compactor > - > > Key: HBASE-13012 > URL: https://issues.apache.org/jira/browse/HBASE-13012 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13012-V5.diff, HBASE-13012-V6.diff, > HBASE-13012-V7.diff, HBASE-13012-V9.diff, HBASE-13012.v8.patch, > hbase-13012-V1.patch, hbase-13012-V2.diff, hbase-13012-V3.diff, > hbase-13012-V4.diff > > > Currently the MobFileCompactor is run by HMaster periodically, we need to add > a shell to trigger the compactor by commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13012) Add shell commands to trigger the mob file compactor
[ https://issues.apache.org/jira/browse/HBASE-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13012: --- Release Note: This adds two new shell commands -- compact_mob and major_compact_mob to the hbase shell. Run compaction on a mob enabled column family or all mob enabled column families within a table Examples: Compact a column family within a table: hbase> compact_mob 't1', 'c1' Compact all mob enabled column families hbase> compact_mob 't1' Run major compaction on a mob enabled column family or all mob enabled column families within a table Examples: Compact a column family within a table: hbase> major_compact_mob 't1', 'c1' Compact all mob enabled column families within a table hbase> major_compact_mob 't1' > Add shell commands to trigger the mob file compactor > - > > Key: HBASE-13012 > URL: https://issues.apache.org/jira/browse/HBASE-13012 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13012-V5.diff, HBASE-13012-V6.diff, > HBASE-13012-V7.diff, HBASE-13012-V9.diff, HBASE-13012.v8.patch, > hbase-13012-V1.patch, hbase-13012-V2.diff, hbase-13012-V3.diff, > hbase-13012-V4.diff > > > Currently the MobFileCompactor is run by HMaster periodically, we need to add > a shell to trigger the compactor by commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11339) HBase MOB
[ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348357#comment-14348357 ] Wilm Schumacher commented on HBASE-11339: - love that feature! But I have two newbie feature requests: 1.) When I run my hbase with the mob feature everything works fine. Except that I get java.lang.illegalargumentexception key value size too large when the data i load up gets to large. One solution would be to set the limit to 0, but I think that the limitation is kind of a usefull feature. Perhaps it would be a nice feature to ignore the limit for the families which have the is_mob => 'true'. 2.) in the documentation there is "MOB" and "LOB" defined. However, in the corpus of the text only MOBs are discussed. By the design explaination I cannot see why LOBs would be more problematic (except the client upload) to save than MOBs. Are there any reasons to avoid "LOBs" (up to several hundreds of MBs) from the database site of view? > HBase MOB > - > > Key: HBASE-11339 > URL: https://issues.apache.org/jira/browse/HBASE-11339 > Project: HBase > Issue Type: Umbrella > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBase MOB Design-v2.pdf, HBase MOB Design-v3.pdf, HBase > MOB Design-v4.pdf, HBase MOB Design.pdf, MOB user guide.docx, MOB user > guide_v2.docx, MOB user guide_v3.docx, MOB user guide_v4.docx, MOB user > guide_v5.docx, hbase-11339-in-dev.patch, merge-150212.patch, > merge.150212b.patch, merge.150212c.patch > > > It's quite useful to save the medium binary data like images, documents > into Apache HBase. Unfortunately directly saving the binary MOB(medium > object) to HBase leads to a worse performance since the frequent split and > compaction. > In this design, the MOB data are stored in an more efficient way, which > keeps a high write/read performance and guarantees the data consistency in > Apache HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13150) TestMasterObserver failing disable table at end of test
[ https://issues.apache.org/jira/browse/HBASE-13150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348358#comment-14348358 ] zhangduo commented on HBASE-13150: -- Yes, I think remove the code above in assign method could solve the problem of this test. But we need to confirm that removing it is safe? > TestMasterObserver failing disable table at end of test > --- > > Key: HBASE-13150 > URL: https://issues.apache.org/jira/browse/HBASE-13150 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > > I see in > https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6202/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestMasterObserver/testRegionTransitionOperations/ > , now we have added in timeouts, that we are failing to disable a table. It > looks like table is disabled but regions are being opened on the disabled > table still, like HBASE-6537 > Let me see if can figure why this happening. Will be back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13154) Add support for mob in TestAcidGuarantees and IntegrationTestAcidGuarantees
[ https://issues.apache.org/jira/browse/HBASE-13154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13154: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Add support for mob in TestAcidGuarantees and IntegrationTestAcidGuarantees > --- > > Key: HBASE-13154 > URL: https://issues.apache.org/jira/browse/HBASE-13154 > Project: HBase > Issue Type: Sub-task > Components: integration tests, mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: hbase-13154.patch > > > Adding support to run acid guarantees tests on data that is primarily in the > mob path. > We essentially set the mob size to be very small (4) and run the same test. > This is activated using the -DuseMob=true tool style command line arg. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11339) HBase MOB
[ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348362#comment-14348362 ] Jonathan Hsieh commented on HBASE-11339: Hi [~Wilm]. This feature is called MOB -- focused on cells that are 100k-10MB in size (possibly slightly larger than than). Currently large objects (we'll define those to be >10MB) are problematic because we lack a streaming api to handle breaking rpc requests up to efficiently ship data from the server side to the client side. While they are hypothetically possible with the current api, the will cause large memory allocations that will stress the memory systems of both the servers and the clients. We may try to address cases with larger blobs in the future, but for now we're limiting our scope. > HBase MOB > - > > Key: HBASE-11339 > URL: https://issues.apache.org/jira/browse/HBASE-11339 > Project: HBase > Issue Type: Umbrella > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBase MOB Design-v2.pdf, HBase MOB Design-v3.pdf, HBase > MOB Design-v4.pdf, HBase MOB Design.pdf, MOB user guide.docx, MOB user > guide_v2.docx, MOB user guide_v3.docx, MOB user guide_v4.docx, MOB user > guide_v5.docx, hbase-11339-in-dev.patch, merge-150212.patch, > merge.150212b.patch, merge.150212c.patch > > > It's quite useful to save the medium binary data like images, documents > into Apache HBase. Unfortunately directly saving the binary MOB(medium > object) to HBase leads to a worse performance since the frequent split and > compaction. > In this design, the MOB data are stored in an more efficient way, which > keeps a high write/read performance and guarantees the data consistency in > Apache HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13154) Add support for mob in TestAcidGuarantees and IntegrationTestAcidGuarantees
[ https://issues.apache.org/jira/browse/HBASE-13154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348365#comment-14348365 ] ramkrishna.s.vasudevan commented on HBASE-13154: Belated +1. > Add support for mob in TestAcidGuarantees and IntegrationTestAcidGuarantees > --- > > Key: HBASE-13154 > URL: https://issues.apache.org/jira/browse/HBASE-13154 > Project: HBase > Issue Type: Sub-task > Components: integration tests, mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: hbase-13154.patch > > > Adding support to run acid guarantees tests on data that is primarily in the > mob path. > We essentially set the mob size to be very small (4) and run the same test. > This is activated using the -DuseMob=true tool style command line arg. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13157) Add mob compaction actions and monkeys to Chaos Monkey
[ https://issues.apache.org/jira/browse/HBASE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13157: --- Attachment: hbase-13157.patch patch attached. I've tested this on a 5 node testing rig cluster running a suite of IT tests. > Add mob compaction actions and monkeys to Chaos Monkey > -- > > Key: HBASE-13157 > URL: https://issues.apache.org/jira/browse/HBASE-13157 > Project: HBase > Issue Type: Sub-task > Components: integration tests, mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: hbase-13157.patch > > > After HBASE-13012 (mob compaction shell and admin api) we can now trigger mob > compaction. This adds an action to chaos monkey so we can randomly trigger > these compactions during integration test runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13157) Add mob compaction actions and monkeys to Chaos Monkey
[ https://issues.apache.org/jira/browse/HBASE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13157: --- Status: Patch Available (was: Open) > Add mob compaction actions and monkeys to Chaos Monkey > -- > > Key: HBASE-13157 > URL: https://issues.apache.org/jira/browse/HBASE-13157 > Project: HBase > Issue Type: Sub-task > Components: integration tests, mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: hbase-13157.patch > > > After HBASE-13012 (mob compaction shell and admin api) we can now trigger mob > compaction. This adds an action to chaos monkey so we can randomly trigger > these compactions during integration test runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13151) IllegalArgumentException in compaction when table has a namespace
[ https://issues.apache.org/jira/browse/HBASE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348369#comment-14348369 ] ramkrishna.s.vasudevan commented on HBASE-13151: +1 on patch > IllegalArgumentException in compaction when table has a namespace > - > > Key: HBASE-13151 > URL: https://issues.apache.org/jira/browse/HBASE-13151 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13151-V2.diff, HBASE-13151-V3.diff, > HBASE-13151.diff > > > In the PartitionedMobFileCompactor we have a bulk load path, currently it > directly use table.getNameAsString as part of the path, if the table has a > namespace, the path contains character ":" which is regarded as a illegal > character in file path. As fixing, we need to divide the namespace and > qualifier into directories like the way in table path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348380#comment-14348380 ] Hudson commented on HBASE-13142: FAILURE: Integrated in HBase-TRUNK #6209 (See [https://builds.apache.org/job/HBase-TRUNK/6209/]) HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer (stack: rev 55f8f56ad28f71a893acb1e5993689499134a018) * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferOutputStream.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferResevoir.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferOutputStream.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/BoundedByteBufferPool.java Revert "HBASE-13142 [PERF] Reuse the IPCUtil#buildCellBlock buffer" (stack: rev 74837d9b1f13b42415395e08ff168aeed0c21676) * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferOutputStream.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/BoundedByteBufferPool.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferOutputStream.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferResevoir.java > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, traces.2.svg, traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13156) Fix minor rat violation recently introduced (asciidoctor.css).
[ https://issues.apache.org/jira/browse/HBASE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348379#comment-14348379 ] Hudson commented on HBASE-13156: FAILURE: Integrated in HBase-TRUNK #6209 (See [https://builds.apache.org/job/HBase-TRUNK/6209/]) HBASE-13156 Fix minor rat violation recently introduced (asciidoctor.css) (stack: rev d259bd402a507a2a81908f2acbd759d25e612de5) * pom.xml > Fix minor rat violation recently introduced (asciidoctor.css). > -- > > Key: HBASE-13156 > URL: https://issues.apache.org/jira/browse/HBASE-13156 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Attachments: 13156.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13155) Fix TestPrefixTree
[
https://issues.apache.org/jira/browse/HBASE-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348392#comment-14348392
]
Hadoop QA commented on HBASE-13155:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702726/HBASE-13155.patch
against master branch at commit 5bd27af8b035eaf1ffa3d1e5e4a6cf35324ef8f7.
ATTACHMENT ID: 12702726
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/patchReleaseAuditWarnings.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13095//console
This message is automatically generated.
> Fix TestPrefixTree
> --
>
> Key: HBASE-13155
> URL: https://issues.apache.org/jira/browse/HBASE-13155
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0, 1.1.0
>Reporter: zhangduo
>Assignee: zhangduo
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-13155-branch-1.patch, HBASE-13155.patch
>
>
> It should be my fault when addressing HBASE-12817, I commented a line in
> master and not in branch-1. This breaks the branch-1 build after HBASE-11544
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13155) Fix TestPrefixTree
[
https://issues.apache.org/jira/browse/HBASE-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348393#comment-14348393
]
Hadoop QA commented on HBASE-13155:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702728/HBASE-13155-branch-1.patch
against branch-1 branch at commit 5bd27af8b035eaf1ffa3d1e5e4a6cf35324ef8f7.
ATTACHMENT ID: 12702728
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportExport
org.apache.hadoop.hbase.util.TestProcessBasedCluster
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hdfs.TestAppendSnapshotTruncate.testAST(TestAppendSnapshotTruncate.java:128)
at
org.apache.hadoop.hdfs.TestEncryptionZones.testBasicOperations(TestEncryptionZones.java:372)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13096//console
This message is automatically generated.
> Fix TestPrefixTree
> --
>
> Key: HBASE-13155
> URL: https://issues.apache.org/jira/browse/HBASE-13155
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0, 1.1.0
>Reporter: zhangduo
>Assignee: zhangduo
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-13155-branch-1.patch, HBASE-13155.patch
>
>
> It should be my fault when addressing HBASE-12817, I commented a line in
> master and not in branch-1. This breaks the branch-1 build after HBASE-11544
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13157) Add mob compaction actions and monkeys to Chaos Monkey
[
https://issues.apache.org/jira/browse/HBASE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348394#comment-14348394
]
Hadoop QA commented on HBASE-13157:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702745/hbase-13157.patch
against master branch at commit 74837d9b1f13b42415395e08ff168aeed0c21676.
ATTACHMENT ID: 12702745
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail with Hadoop version 2.4.1.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/CompactMobAction.java:[54,14]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/CompactMobAction.java:[56,14]
cannot find symbol
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:testCompile
(default-testCompile) on project hbase-it: Compilation failure: Compilation
failure:
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/CompactMobAction.java:[54,14]
cannot find symbol
[ERROR] symbol: method majorCompactMob(org.apache.hadoop.hbase.TableName)
[ERROR] location: variable admin of type org.apache.hadoop.hbase.client.Admin
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/CompactMobAction.java:[56,14]
cannot find symbol
[ERROR] symbol: method compactMob(org.apache.hadoop.hbase.TableName)
[ERROR] location: variable admin of type org.apache.hadoop.hbase.client.Admin
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hbase-it
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13099//console
This message is automatically generated.
> Add mob compaction actions and monkeys to Chaos Monkey
> --
>
> Key: HBASE-13157
> URL: https://issues.apache.org/jira/browse/HBASE-13157
> Project: HBase
> Issue Type: Sub-task
> Components: integration tests, mob
>Affects Versions: hbase-11339
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-11339
>
> Attachments: hbase-13157.patch
>
>
> After HBASE-13012 (mob compaction shell and admin api) we can now trigger mob
> compaction. This adds an action to chaos monkey so we can randomly trigger
> these compactions during integration test runs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13151) IllegalArgumentException in compaction when table has a namespace
[ https://issues.apache.org/jira/browse/HBASE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348418#comment-14348418 ] Jonathan Hsieh commented on HBASE-13151: I committed HBASE-13012 and this patch doesn't apply anymore. Let's get another rev. > IllegalArgumentException in compaction when table has a namespace > - > > Key: HBASE-13151 > URL: https://issues.apache.org/jira/browse/HBASE-13151 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13151-V2.diff, HBASE-13151-V3.diff, > HBASE-13151.diff > > > In the PartitionedMobFileCompactor we have a bulk load path, currently it > directly use table.getNameAsString as part of the path, if the table has a > namespace, the path contains character ":" which is regarded as a illegal > character in file path. As fixing, we need to divide the namespace and > qualifier into directories like the way in table path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
Anoop Sam John created HBASE-13158: -- Summary: When client supports CellBlock, return the result Cells as controller payload for get(Get) API also Key: HBASE-13158 URL: https://issues.apache.org/jira/browse/HBASE-13158 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instead of retry when resolving mobfiles
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348428#comment-14348428 ] Jonathan Hsieh commented on HBASE-12332: Sorry for being slow on this one. This patch doesn't apply after HBASE-13012. Can we get a new rev of this? I'll review within a day. > [mob] use filelink instead of retry when resolving mobfiles > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jiajia Li > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff, HBASE-12332-V2.patch, > HBASE-12332-V3.patch, HBASE-12332-V5.patch, hbase-12332.link.v4.patch, > hbase-12332.patch > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13158: --- Attachment: HBASE-13158.patch Not compatible with old version of client and new version of server so only a 2.0 improvement. > When client supports CellBlock, return the result Cells as controller payload > for get(Get) API also > --- > > Key: HBASE-13158 > URL: https://issues.apache.org/jira/browse/HBASE-13158 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-13158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13158: --- Status: Patch Available (was: Open) > When client supports CellBlock, return the result Cells as controller payload > for get(Get) API also > --- > > Key: HBASE-13158 > URL: https://issues.apache.org/jira/browse/HBASE-13158 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-13158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13151) IllegalArgumentException in compaction when table has a namespace
[ https://issues.apache.org/jira/browse/HBASE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348441#comment-14348441 ] Jingcheng Du commented on HBASE-13151: -- Let me pull the latest code and re-submit the patch. > IllegalArgumentException in compaction when table has a namespace > - > > Key: HBASE-13151 > URL: https://issues.apache.org/jira/browse/HBASE-13151 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13151-V2.diff, HBASE-13151-V3.diff, > HBASE-13151.diff > > > In the PartitionedMobFileCompactor we have a bulk load path, currently it > directly use table.getNameAsString as part of the path, if the table has a > namespace, the path contains character ":" which is regarded as a illegal > character in file path. As fixing, we need to divide the namespace and > qualifier into directories like the way in table path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instead of retry when resolving mobfiles
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348442#comment-14348442 ] Jingcheng Du commented on HBASE-12332: -- I'll update the patch against the latest code in hbase-11339, will upload it later. Thanks. > [mob] use filelink instead of retry when resolving mobfiles > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jiajia Li > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff, HBASE-12332-V2.patch, > HBASE-12332-V3.patch, HBASE-12332-V5.patch, hbase-12332.link.v4.patch, > hbase-12332.patch > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13135) Move replication ops mgmt stuff from Javadoc to Ref Guide
[
https://issues.apache.org/jira/browse/HBASE-13135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348446#comment-14348446
]
Hadoop QA commented on HBASE-13135:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702736/HBASE-13135-v2.patch
against master branch at commit 74837d9b1f13b42415395e08ff168aeed0c21676.
ATTACHMENT ID: 12702736
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+Cluster replication documentation has been moved to the
link:http://hbase.apache.org/book.html#_cluster_replication[Cluster
Replication] section of the link:http://hbase.apache.org/book.html[Apache HBase
Reference Guide].
+. If both clusters use the same ZooKeeper cluster, you must use a different
`zookeeper.znode.parent`, because they cannot write in the same folder.
+. On the source cluster, enable replication by setting `hbase.replication` to
`true` in _hbase-site.xml_.
+. On the source cluster, in HBase Shell, add the destination cluster as a
peer, using the `add_peer` command.
+ * CLUSTER_KEY: composed using the following template, with appropriate
place-holders:
`hbase.zookeeper.quorum:hbase.zookeeper.property.clientPort:zookeeper.znode.parent`
+ Disable a replication relationship. HBase will no longer send edits to that
peer cluster, but it still keeps track of all the new WALs that it will need to
replicate if and when it is re-enabled.
+ Disable and remove a replication relationship. HBase will no longer send
edits to that peer cluster or keep track of WALs.
+The `VerifyReplication` MapReduce job, which is included in HBase, performs a
systematic comparison of replicated data between two different clusters. Run
the VerifyReplication job on the master cluster, supplying it with the peer ID
and table name to use for validation. You can limit the verification further by
specifying a time range or specific families. The job's short name is
`verifyrep`. To run the job, use a command like the following:
+$ HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath`
"${HADOOP_HOME}/bin/hadoop" jar "${HBASE_HOME}/hbase-server-VERSION.jar"
verifyrep --starttime= --stoptime= --families=
+The `VerifyReplication` command prints out `GOODROWS` and `BADROWS` counters
to indicate rows that did and did not replicate correctly.
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13098//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13098//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13098//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13098//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13098//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13098//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13098//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13098//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13098//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs w
[jira] [Updated] (HBASE-13151) IllegalArgumentException in compaction when table has a namespace
[ https://issues.apache.org/jira/browse/HBASE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-13151: - Attachment: HBASE-13151-V4.diff Pull the latest code from hbase-11339 and re-generate the patch, now upload it (V4) again. > IllegalArgumentException in compaction when table has a namespace > - > > Key: HBASE-13151 > URL: https://issues.apache.org/jira/browse/HBASE-13151 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13151-V2.diff, HBASE-13151-V3.diff, > HBASE-13151-V4.diff, HBASE-13151.diff > > > In the PartitionedMobFileCompactor we have a bulk load path, currently it > directly use table.getNameAsString as part of the path, if the table has a > namespace, the path contains character ":" which is regarded as a illegal > character in file path. As fixing, we need to divide the namespace and > qualifier into directories like the way in table path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12332) [mob] use filelink instead of retry when resolving mobfiles
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-12332: - Attachment: HBASE-12332-V6.diff Hi [~jmhsieh], I pulled the latest code from hbase-11339 and re-generated the patch, upload it again! Thanks. > [mob] use filelink instead of retry when resolving mobfiles > --- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jiajia Li > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff, HBASE-12332-V2.patch, > HBASE-12332-V3.patch, HBASE-12332-V5.patch, HBASE-12332-V6.diff, > hbase-12332.link.v4.patch, hbase-12332.patch > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[
https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348603#comment-14348603
]
Hadoop QA commented on HBASE-13158:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702753/HBASE-13158.patch
against master branch at commit 74837d9b1f13b42415395e08ff168aeed0c21676.
ATTACHMENT ID: 12702753
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13100//console
This message is automatically generated.
> When client supports CellBlock, return the result Cells as controller payload
> for get(Get) API also
> ---
>
> Key: HBASE-13158
> URL: https://issues.apache.org/jira/browse/HBASE-13158
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-13158.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13149) HBase server MR tools are broken on Hadoop 2.5+ Yarn
[
https://issues.apache.org/jira/browse/HBASE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348759#comment-14348759
]
Sean Busbey commented on HBASE-13149:
-
{quote}
bq. Have we confirmed that application classpaths don't mitigate this?
I missed this part. Can you elaborate a little? See if I can do any testing.
{quote}
Sure thing. There's a feature in Hadoop 2 to use a different classloader for
user submitted code. Unfortunately, it isn't really documented. The relevant
jiras are YARN-286, MAPREDUCE-1700, and as of Hadoop 2.6 HADOOP-10893.
> HBase server MR tools are broken on Hadoop 2.5+ Yarn
>
>
> Key: HBASE-13149
> URL: https://issues.apache.org/jira/browse/HBASE-13149
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0, 0.98.10.1
>Reporter: Jerry He
>Priority: Critical
> Attachments: HBASE-13149-0.98.patch,
> jackson-core-asl-compat_report.html, jackson-jaxrs-compat_report.html,
> jackson-mapper-asl-compat_report.html, jackson-xc-compat_report.html
>
>
> Running on the server MR tools is not working on Yarn version 2.5+.
> Running org.apache.hadoop.hbase.mapreduce.Export:
> {noformat}
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
> at
> org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:59)
> at
> org.apache.hadoop.yarn.util.timeline.TimelineUtils.(TimelineUtils.java:47)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:96)
> at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:112)
> at
> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75)
> at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
> at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
> at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:189)
> {noformat}
> The problem seems to be the jackson jar version. HADOOP-10104 updated
> jackson version to 1.9.13. YARN-2092 reported a problem as well.
> HBase is using jackson 1.8.8. This version of the jar in the classpath seem
> to cause the problem.
> Should we upgrade to jackson 1.9.13?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-13023) Document multiWAL
[ https://issues.apache.org/jira/browse/HBASE-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-13023: --- Assignee: Misty Stanley-Jones (was: Sean Busbey) +1 The multiwal section looks good. the patch has a bunch of unrelated changes for coprocessors. fine to fix on commit. > Document multiWAL > - > > Key: HBASE-13023 > URL: https://issues.apache.org/jira/browse/HBASE-13023 > Project: HBase > Issue Type: Bug > Components: documentation, wal >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-13023-v1.patch, HBASE-13023.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13155) Fix TestPrefixTree
[ https://issues.apache.org/jira/browse/HBASE-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13155: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the patch, Duo. > Fix TestPrefixTree > -- > > Key: HBASE-13155 > URL: https://issues.apache.org/jira/browse/HBASE-13155 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.1.0 >Reporter: zhangduo >Assignee: zhangduo > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-13155-branch-1.patch, HBASE-13155.patch > > > It should be my fault when addressing HBASE-12817, I commented a line in > master and not in branch-1. This breaks the branch-1 build after HBASE-11544 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348830#comment-14348830 ] Ted Yu commented on HBASE-13158: +1 > When client supports CellBlock, return the result Cells as controller payload > for get(Get) API also > --- > > Key: HBASE-13158 > URL: https://issues.apache.org/jira/browse/HBASE-13158 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-13158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13157) Add mob compaction actions and monkeys to Chaos Monkey
[ https://issues.apache.org/jira/browse/HBASE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348834#comment-14348834 ] Ted Yu commented on HBASE-13157: lgtm Please add javadoc for MobSlowDeterministicMonkeyFactory class. > Add mob compaction actions and monkeys to Chaos Monkey > -- > > Key: HBASE-13157 > URL: https://issues.apache.org/jira/browse/HBASE-13157 > Project: HBase > Issue Type: Sub-task > Components: integration tests, mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: hbase-13157.patch > > > After HBASE-13012 (mob compaction shell and admin api) we can now trigger mob > compaction. This adds an action to chaos monkey so we can randomly trigger > these compactions during integration test runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12931:
--
Attachment: (was: HBASE-12931.patch)
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12931:
--
Status: Open (was: Patch Available)
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12931:
--
Status: Patch Available (was: Open)
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12931:
--
Attachment: HBASE-12931
resubmit
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348854#comment-14348854
]
Ted Yu commented on HBASE-12931:
Please use file extension of either .patch or .txt so that QA bot can pick up.
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348859#comment-14348859
]
Hadoop QA commented on HBASE-12931:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against master branch at commit c15e684bf91fffee4fb43e5a1440154a6db2d3db.
ATTACHMENT ID: http:
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13102//console
This message is automatically generated.
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13128) Make HBCK's lock file retry creation and deletion
[ https://issues.apache.org/jira/browse/HBASE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13128: Status: Open (was: Patch Available) > Make HBCK's lock file retry creation and deletion > - > > Key: HBASE-13128 > URL: https://issues.apache.org/jira/browse/HBASE-13128 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Victoria >Assignee: Victoria >Priority: Minor > Attachments: hbck_files_fix-v1.patch, hbck_files_fix-v2.patch, > hbck_files_fix-v3.patch, hbck_files_fix.patch > > > When hbck runs it creates a lock file to ensure that no two hbck instances > are running. We've been seeing creating and removing that file fail sometimes. > This improvement should make the creation, closing of the file, and the > deletion retry multiple times. This should allow our alerting which uses this > command to be more reliable and have fewer false positives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13128) Make HBCK's lock file retry creation and deletion
[
https://issues.apache.org/jira/browse/HBASE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348867#comment-14348867
]
Sean Busbey commented on HBASE-13128:
-
please fix these two checkstyle errors:
{noformat}
{noformat}
The tab warning only prints for the first occurrence, so you should check for
more tabs elsewhere in the patch.
> Make HBCK's lock file retry creation and deletion
> -
>
> Key: HBASE-13128
> URL: https://issues.apache.org/jira/browse/HBASE-13128
> Project: HBase
> Issue Type: Improvement
> Components: hbck
>Reporter: Victoria
>Assignee: Victoria
>Priority: Minor
> Attachments: hbck_files_fix-v1.patch, hbck_files_fix-v2.patch,
> hbck_files_fix-v3.patch, hbck_files_fix.patch
>
>
> When hbck runs it creates a lock file to ensure that no two hbck instances
> are running. We've been seeing creating and removing that file fail sometimes.
> This improvement should make the creation, closing of the file, and the
> deletion retry multiple times. This should allow our alerting which uses this
> command to be more reliable and have fewer false positives.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12931:
--
Attachment: HBASE-12931.patch
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931.patch
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12931:
--
Status: Patch Available (was: Open)
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931.patch
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12931:
--
Status: Open (was: Patch Available)
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931.patch
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12931:
--
Attachment: (was: HBASE-12931)
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931.patch
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348932#comment-14348932
]
ChiaPing Tsai commented on HBASE-12931:
---
Thanks for reminding me.
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931.patch
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348970#comment-14348970 ] stack commented on HBASE-13158: --- Looks good. What is the benefit? > When client supports CellBlock, return the result Cells as controller payload > for get(Get) API also > --- > > Key: HBASE-13158 > URL: https://issues.apache.org/jira/browse/HBASE-13158 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-13158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-9066) TestReplicationTrackerZKImpl fails
[ https://issues.apache.org/jira/browse/HBASE-9066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-9066. -- Resolution: Cannot Reproduce Resolving as cannot reproduce > TestReplicationTrackerZKImpl fails > -- > > Key: HBASE-9066 > URL: https://issues.apache.org/jira/browse/HBASE-9066 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack > > This new test has failed a few times over last day or so. > Here is a sample w/ two fails in it. > http://54.241.6.143/job/HBase-0.95/707/ > Hope you don't mind me assigning it to you Chris. > I'm going to disable these two tests for now so can get some good test runs > in over the w/e. Thanks boss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348984#comment-14348984 ] stack commented on HBASE-13071: --- bq. Makes sense? Yes. Thank you. The test 'reads' the result but does not processing, true. Yes, 30 per batch. What test would you like me to run that makes this feature shine? Thanks [~eshcar] > Hbase Streaming Scan Feature > > > Key: HBASE-13071 > URL: https://issues.apache.org/jira/browse/HBASE-13071 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.11 >Reporter: Eshcar Hillel > Attachments: 99.eshcar.png, HBASE-13071_98_1.patch, > HBASE-13071_trunk_1.patch, HBASE-13071_trunk_2.patch, > HBASE-13071_trunk_3.patch, HBASE-13071_trunk_4.patch, > HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf, > gc.eshcar.png, hits.eshcar.png, network.png > > > A scan operation iterates over all rows of a table or a subrange of the > table. The synchronous nature in which the data is served at the client side > hinders the speed the application traverses the data: it increases the > overall processing time, and may cause a great variance in the times the > application waits for the next piece of data. > The scanner next() method at the client side invokes an RPC to the > regionserver and then stores the results in a cache. The application can > specify how many rows will be transmitted per RPC; by default this is set to > 100 rows. > The cache can be considered as a producer-consumer queue, where the hbase > client pushes the data to the queue and the application consumes it. > Currently this queue is synchronous, i.e., blocking. More specifically, when > the application consumed all the data from the cache --- so the cache is > empty --- the hbase client retrieves additional data from the server and > re-fills the cache with new data. During this time the application is blocked. > Under the assumption that the application processing time can be balanced by > the time it takes to retrieve the data, an asynchronous approach can reduce > the time the application is waiting for data. > We attach a design document. > We also have a patch that is based on a private branch, and some evaluation > results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13142: -- Attachment: 13142v3.txt Add categorization (I heart unit tests) > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, > traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13122) Improve efficiency for return codes of some filters
[
https://issues.apache.org/jira/browse/HBASE-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349033#comment-14349033
]
Hadoop QA commented on HBASE-13122:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702709/13122-master.patch
against master branch at commit 74837d9b1f13b42415395e08ff168aeed0c21676.
ATTACHMENT ID: 12702709
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13101//console
This message is automatically generated.
> Improve efficiency for return codes of some filters
> ---
>
> Key: HBASE-13122
> URL: https://issues.apache.org/jira/browse/HBASE-13122
> Project: HBase
> Issue Type: Improvement
> Components: Filters
>Affects Versions: 0.94.24, 1.0.1, 0.98.10.1
>Reporter: Shuaifeng Zhou
> Fix For: 2.0.0, 1.1.0
>
> Attachments: 13122-master.patch, 13122-master.patch,
> 13122-master.patch, 13122.patch
>
>
> ColumnRangeFilter:
> when minColumnInclusive is false, it means all the cells at the current
> row&column not fit the condition, so it should skip to next column, return
> code should be NEXT_COL, not SKIP.
> FamilyFilter is the similar sitution.
> Currently, SKIP will not causing error, but not efficent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13155) Fix TestPrefixTree
[ https://issues.apache.org/jira/browse/HBASE-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349046#comment-14349046 ] Hudson commented on HBASE-13155: SUCCESS: Integrated in HBase-TRUNK #6210 (See [https://builds.apache.org/job/HBase-TRUNK/6210/]) HBASE-13155 Fix TestPrefixTree (Duo Zhang) (tedyu: rev c15e684bf91fffee4fb43e5a1440154a6db2d3db) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTree.java > Fix TestPrefixTree > -- > > Key: HBASE-13155 > URL: https://issues.apache.org/jira/browse/HBASE-13155 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.1.0 >Reporter: zhangduo >Assignee: zhangduo > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-13155-branch-1.patch, HBASE-13155.patch > > > It should be my fault when addressing HBASE-12817, I commented a line in > master and not in branch-1. This breaks the branch-1 build after HBASE-11544 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13155) Fix TestPrefixTree
[ https://issues.apache.org/jira/browse/HBASE-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349086#comment-14349086 ] Hudson commented on HBASE-13155: FAILURE: Integrated in HBase-1.1 #250 (See [https://builds.apache.org/job/HBase-1.1/250/]) HBASE-13155 Fix TestPrefixTree (Duo Zhang) (tedyu: rev efb6e7005fa5d8939312b796778961550e759253) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTree.java > Fix TestPrefixTree > -- > > Key: HBASE-13155 > URL: https://issues.apache.org/jira/browse/HBASE-13155 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.1.0 >Reporter: zhangduo >Assignee: zhangduo > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-13155-branch-1.patch, HBASE-13155.patch > > > It should be my fault when addressing HBASE-12817, I commented a line in > master and not in branch-1. This breaks the branch-1 build after HBASE-11544 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13151) IllegalArgumentException in compaction when table has a namespace
[
https://issues.apache.org/jira/browse/HBASE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349093#comment-14349093
]
Jonathan Hsieh commented on HBASE-13151:
With V4, I'm getting some compile errors aroudn TestMobFileCompactor
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:testCompile
(default-testCompile) on project hbase-server: Compilation failure: Compilation
failure:
[ERROR]
/home/jon/proj/hbase-11339/hbase-server/src/test/java/org/apache/hadoop/hbase/mob/filecompactions/TestMobFileCompactor.java:[464,5]
method loadData in class
org.apache.hadoop.hbase.mob.filecompactions.TestMobFileCompactor cannot be
applied to given types;
[ERROR] required:
org.apache.hadoop.hbase.client.Admin,org.apache.hadoop.hbase.client.HTable,org.apache.hadoop.hbase.TableName,int,int
[ERROR] found: int,int
[ERROR] reason: actual and formal argument lists differ in length
...
{code}
> IllegalArgumentException in compaction when table has a namespace
> -
>
> Key: HBASE-13151
> URL: https://issues.apache.org/jira/browse/HBASE-13151
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
>Affects Versions: hbase-11339
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: hbase-11339
>
> Attachments: HBASE-13151-V2.diff, HBASE-13151-V3.diff,
> HBASE-13151-V4.diff, HBASE-13151.diff
>
>
> In the PartitionedMobFileCompactor we have a bulk load path, currently it
> directly use table.getNameAsString as part of the path, if the table has a
> namespace, the path contains character ":" which is regarded as a illegal
> character in file path. As fixing, we need to divide the namespace and
> qualifier into directories like the way in table path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12969) Parameter Validation is not there for shell script, local-master-backup.sh and local-regionservers.sh
[ https://issues.apache.org/jira/browse/HBASE-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349157#comment-14349157 ] Andrew Purtell commented on HBASE-12969: +1 Will commit shortly > Parameter Validation is not there for shell script, local-master-backup.sh > and local-regionservers.sh > - > > Key: HBASE-12969 > URL: https://issues.apache.org/jira/browse/HBASE-12969 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 0.98.9 >Reporter: Y. SREENIVASULU REDDY >Assignee: Y. SREENIVASULU REDDY >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-12969.patch > > > while executing local-regionservers.sh or local-master-backup.sh in > $HBASE_HOME/bin > if parameter is non numeric value then scripts are throwing failures. > we need to handle the validation also for those scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349162#comment-14349162
]
Hadoop QA commented on HBASE-12931:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702836/HBASE-12931.patch
against master branch at commit c15e684bf91fffee4fb43e5a1440154a6db2d3db.
ATTACHMENT ID: 12702836
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13103//console
This message is automatically generated.
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931.patch
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
>
[jira] [Commented] (HBASE-13135) Move replication ops mgmt stuff from Javadoc to Ref Guide
[
https://issues.apache.org/jira/browse/HBASE-13135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349168#comment-14349168
]
Jean-Daniel Cryans commented on HBASE-13135:
Good stuff Misty, a few more things while we're here.
I think we can remove this line since it's on by default, maybe say that the
operator should check it wasn't disabled:
bq. On the source cluster, enable replication by setting `hbase.replication` to
`true` in _hbase-site.xml_.
The following can be removed, we don't print this out at least since the
endpoint changes:
{quote}
Considering 1 rs, with ratio 0.1
Getting 1 rs from peer cluster # 0
Choosing peer 10.10.1.49:62020
{quote}
This is a more reliable thing to point out instead, coming from
ReplicationSource:
bq. LOG.info("Replicating "+clusterId + " -> " + peerClusterId);
> Move replication ops mgmt stuff from Javadoc to Ref Guide
> -
>
> Key: HBASE-13135
> URL: https://issues.apache.org/jira/browse/HBASE-13135
> Project: HBase
> Issue Type: Bug
> Components: documentation, Replication
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Attachments: HBASE-13135-v1.patch, HBASE-13135-v2.patch,
> HBASE-13135.patch
>
>
> As per discussion with [~jmhsieh] and [~saint@gmail.com]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12332) [mob] improve how we resolve mobfiles in reads
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-12332: --- Summary: [mob] improve how we resolve mobfiles in reads (was: [mob] use filelink instead of retry when resolving mobfiles) > [mob] improve how we resolve mobfiles in reads > -- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jiajia Li > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff, HBASE-12332-V2.patch, > HBASE-12332-V3.patch, HBASE-12332-V5.patch, HBASE-12332-V6.diff, > hbase-12332.link.v4.patch, hbase-12332.patch > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12931) The existing KeyValues in memstroe are not removed completely after inserting cell into memStore
[
https://issues.apache.org/jira/browse/HBASE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349185#comment-14349185
]
stack commented on HBASE-12931:
---
Patch lgtm.
> The existing KeyValues in memstroe are not removed completely after inserting
> cell into memStore
> -
>
> Key: HBASE-12931
> URL: https://issues.apache.org/jira/browse/HBASE-12931
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Minor
> Attachments: HBASE-12931.patch
>
>
> If I'm not wrong, the UPSERT method of memStore should remove all existing
> KeyValues except the newer version.
> In memStore,
> {code:title=DefaultMemStore.java|borderStyle=solid}
> int versIionsVisible = 0;
> ...
> if (cur.getTypeByte() == KeyValue.Type.Put.getCode() &&
> cur.getSequenceId() <= readpoint) {
> if (versionsVisible > 1) {
> // if we get here we have seen at least one version visible to
> the oldest scanner,
> // which means we can prove that no scanner will see this version
> // false means there was a change, so give us the size.
> long delta = heapSizeChange(cur, true);
> addedSize -= delta;
> this.size.addAndGet(-delta);
> it.remove();
> setOldestEditTimeToNow();
> } else {
> versionsVisible++;
> }
> {code}
> Does "versionsVisible > 1" should be changed to "versionsVisible >= 1" ?
> thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-12332) [mob] improve how we resolve mobfiles in reads
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh resolved HBASE-12332. Resolution: Fixed Hadoop Flags: Reviewed Thanks for the patience Jiajia, I've committed the patch. > [mob] improve how we resolve mobfiles in reads > -- > > Key: HBASE-12332 > URL: https://issues.apache.org/jira/browse/HBASE-12332 > Project: HBase > Issue Type: Sub-task > Components: mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jiajia Li > Fix For: hbase-11339 > > Attachments: HBASE-12332-V1.diff, HBASE-12332-V2.patch, > HBASE-12332-V3.patch, HBASE-12332-V5.patch, HBASE-12332-V6.diff, > hbase-12332.link.v4.patch, hbase-12332.patch > > > in the snapshot code, hmobstore was modified to traverse an hfile link to a > mob. Ideally this should use the transparent filelink code to read the data. > Also there will likely be some issues with the mob file cache with these > links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12468) AUTHORIZATIONS should be part of Visibility Label Docs
[ https://issues.apache.org/jira/browse/HBASE-12468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349208#comment-14349208 ] Jerry He commented on HBASE-12468: -- Hi, [~misty] bq. link:http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/security/visibility/VisibilityClient.htmlsgetAuths%28org.apache.hadoop.conf.Configuration,%20java.lang.String%29[setAuths()] method Inside the link, the method should be VisibilityClient.htmls *setAuths*, with the url for *setAuths* Same problem for HBASE-12466 > AUTHORIZATIONS should be part of Visibility Label Docs > -- > > Key: HBASE-12468 > URL: https://issues.apache.org/jira/browse/HBASE-12468 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 0.98.6.1 >Reporter: Kevin Odell >Assignee: Misty Stanley-Jones > Attachments: HBASE-12468-v1.patch, HBASE-12468.patch, > HBASE_12468-v2.patch > > > Per https://issues.apache.org/jira/browse/HBASE-12346 you need to use > AUTHORIZATIONS or setAuthorizations to see your labels. We may want to update > http://hbase.apache.org/book/ch08s03.html with that information -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13157) Add mob compaction actions and monkeys to Chaos Monkey
[ https://issues.apache.org/jira/browse/HBASE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349211#comment-14349211 ] Jonathan Hsieh commented on HBASE-13157: thanks for the review ted. I've updated the comment and committed to hbase-11339. > Add mob compaction actions and monkeys to Chaos Monkey > -- > > Key: HBASE-13157 > URL: https://issues.apache.org/jira/browse/HBASE-13157 > Project: HBase > Issue Type: Sub-task > Components: integration tests, mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: hbase-13157.patch > > > After HBASE-13012 (mob compaction shell and admin api) we can now trigger mob > compaction. This adds an action to chaos monkey so we can randomly trigger > these compactions during integration test runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[
https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349212#comment-14349212
]
Hadoop QA commented on HBASE-13142:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702842/13142v3.txt
against master branch at commit c15e684bf91fffee4fb43e5a1440154a6db2d3db.
ATTACHMENT ID: 12702842
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13104//console
This message is automatically generated.
> [PERF] Reuse the IPCUtil#buildCellBlock buffer
> --
>
> Key: HBASE-13142
> URL: https://issues.apache.org/jira/browse/HBASE-13142
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Reporter: stack
>Assignee: stack
> Labels: beginner
> Fix For: 2.0.0, 1.1.0
>
> Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg,
> traces.svg
>
>
> Running some scan profiling, flight recorder was mildly fingering resize of
> the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It
> was half-hearted blaming it for a few hundreds of ms over a five minute
> sampling with a few tens of instances showing.
> I tried then w/ flamegraph/lightweight profiler and this reported the buffer
> allocations taking 22% of our total CPU. See attachment trace.svg.
> I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and
> indeed every allocation was doing a resize from initial allocation of 16k --
> the default up to 220k (this test returns ten randomly sized rows zipfian
> sized between 0 and 8k).
> Upping the allocation to 220k meant we now avoided the resize but the initial
> allo
[jira] [Updated] (HBASE-13157) Add mob compaction actions and monkeys to Chaos Monkey
[ https://issues.apache.org/jira/browse/HBASE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13157: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Add mob compaction actions and monkeys to Chaos Monkey > -- > > Key: HBASE-13157 > URL: https://issues.apache.org/jira/browse/HBASE-13157 > Project: HBase > Issue Type: Sub-task > Components: integration tests, mob >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-11339 > > Attachments: hbase-13157.patch > > > After HBASE-13012 (mob compaction shell and admin api) we can now trigger mob > compaction. This adds an action to chaos monkey so we can randomly trigger > these compactions during integration test runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349214#comment-14349214 ] Nick Dimiduk commented on HBASE-13158: -- This should be really helpful for avoiding KV rewriting on the RS side. Any chance to implementing in a backward compatible way so we can get this in for 1.x (and not necessitate downtime)? See also Matteo's recent thread on the dev list. > When client supports CellBlock, return the result Cells as controller payload > for get(Get) API also > --- > > Key: HBASE-13158 > URL: https://issues.apache.org/jira/browse/HBASE-13158 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-13158.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13090) Progress heartbeats for long running scanners
[ https://issues.apache.org/jira/browse/HBASE-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349215#comment-14349215 ] Jonathan Lawlor commented on HBASE-13090: - With HBASE-11544 now in, I was thinking of tackling this one next and was looking for some feedback on the thought process: Implementing the timeout server side would involve changes at three different levels: * RSRpcServers * RegionScannerImpl/ReversedRegionScannerImpl * StoreScanner The RSRpcServices could maintain a variable; something along the lines of remainingScanTime. This value could be initialized to be some fraction of the scanner timeout (maybe half would be good enough?). On each call to RegionScanner#nextRaw, RSRpcServices would communicate that the RegionScanner can take at most remainingScanTime to retrieve a Result -- if a Result cannot be formed in that time, a timeout occurs. The RegionScanner would communicate this same remainingScanTime to the StoreScanner so that calls to InternalScanner#next() may also timeout if they are taking too long. Note that if partial Results are NOT supported by the scan configuration (as is the case for small scans, and scans with a filter that requires whole rows to be read before a filtering decision can be made) then the timeout would not be enforceable within StoreScanner but only within RegionScannerImpl and RSRpcServices. This means that it would still be possible to timeout due to a single long running StoreScanner#next() call in the event that partial Results are not supported. If a timeout does occur on the server, we would have to decide how this should be communicated back to the Client. I was thinking it would be most appropriate to communicate this back to the client via fields in the ScanResponse rather than flags on the Results in the ScanResponse (there is already a lot of state information implied through the contents of the Results in the ScanResponse and adding more seems like it would complicate things). Something along the lines of a timeoutOccurred boolean flag may be sufficient. Then on the Client side we could decide if enough Results were accumulated prior to the timeout to service the application request or if we must make another RPC to enough Results. If anyone else has been thinking about how to approach the solution to this issue or has any other ideas please chime in. Any feedback would be much appreciated. > Progress heartbeats for long running scanners > - > > Key: HBASE-13090 > URL: https://issues.apache.org/jira/browse/HBASE-13090 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Purtell > > It can be necessary to set very long timeouts for clients that issue scans > over large regions when all data in the region might be filtered out > depending on scan criteria. This is a usability concern because it can be > hard to identify what worst case timeout to use until scans are > occasionally/intermittently failing in production, depending on variable scan > criteria. It would be better if the client-server scan protocol can send back > periodic progress heartbeats to clients as long as server scanners are alive > and making progress. > This is related but orthogonal to streaming scan (HBASE-13071). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12180) Fix and reenable TestRegionReplicaReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-12180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349226#comment-14349226
]
Enis Soztutar commented on HBASE-12180:
---
Not sure where this comes from:
{code}
Unapproved licenses:
src/main/asciidoc/asciidoctor.css
{code}
> Fix and reenable TestRegionReplicaReplicationEndpoint
> --
>
> Key: HBASE-12180
> URL: https://issues.apache.org/jira/browse/HBASE-12180
> Project: HBase
> Issue Type: Task
> Components: test
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.1.0
>
> Attachments: hbase-12180_v1.patch
>
>
> HBASE-12172 disabled this test because it was flakey. This issue is about
> reenabling it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12180) Fix and reenable TestRegionReplicaReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-12180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349231#comment-14349231 ] Devaraj Das commented on HBASE-12180: - +1 > Fix and reenable TestRegionReplicaReplicationEndpoint > -- > > Key: HBASE-12180 > URL: https://issues.apache.org/jira/browse/HBASE-12180 > Project: HBase > Issue Type: Task > Components: test >Affects Versions: 2.0.0 >Reporter: stack >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-12180_v1.patch > > > HBASE-12172 disabled this test because it was flakey. This issue is about > reenabling it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13128) Make HBCK's lock file retry creation and deletion
[ https://issues.apache.org/jira/browse/HBASE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria updated HBASE-13128: - Attachment: hbck_files_fix-v4.patch Fixed the tab character style error. Regarding the long line - its long because parameter name is long - I think splitting the statement to multi-line will make it less straightforward. I can assign variable to the value of the parameter and then use it - let me know if this is really needed. > Make HBCK's lock file retry creation and deletion > - > > Key: HBASE-13128 > URL: https://issues.apache.org/jira/browse/HBASE-13128 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Victoria >Assignee: Victoria >Priority: Minor > Attachments: hbck_files_fix-v1.patch, hbck_files_fix-v2.patch, > hbck_files_fix-v3.patch, hbck_files_fix-v4.patch, hbck_files_fix.patch > > > When hbck runs it creates a lock file to ensure that no two hbck instances > are running. We've been seeing creating and removing that file fail sometimes. > This improvement should make the creation, closing of the file, and the > deletion retry multiple times. This should allow our alerting which uses this > command to be more reliable and have fewer false positives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13107) Refactor MOB Snapshot logic to reduce code duplication.
[ https://issues.apache.org/jira/browse/HBASE-13107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349244#comment-14349244 ] Jonathan Hsieh commented on HBASE-13107: [~jingchengdu], looks good. after testing will commit. > Refactor MOB Snapshot logic to reduce code duplication. > --- > > Key: HBASE-13107 > URL: https://issues.apache.org/jira/browse/HBASE-13107 > Project: HBase > Issue Type: Sub-task > Components: mob, snapshots >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13107.diff > > > The MOB Snapshot code contains a lot of code duplication with the normal > snapshot code path. We should do some refactoring to clean this up before > merging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13128) Make HBCK's lock file retry creation and deletion
[ https://issues.apache.org/jira/browse/HBASE-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349291#comment-14349291 ] Sean Busbey commented on HBASE-13128: - that sounds like a reasonable approach. > Make HBCK's lock file retry creation and deletion > - > > Key: HBASE-13128 > URL: https://issues.apache.org/jira/browse/HBASE-13128 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Victoria >Assignee: Victoria >Priority: Minor > Attachments: hbck_files_fix-v1.patch, hbck_files_fix-v2.patch, > hbck_files_fix-v3.patch, hbck_files_fix-v4.patch, hbck_files_fix.patch > > > When hbck runs it creates a lock file to ensure that no two hbck instances > are running. We've been seeing creating and removing that file fail sometimes. > This improvement should make the creation, closing of the file, and the > deletion retry multiple times. This should allow our alerting which uses this > command to be more reliable and have fewer false positives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11670) Build PDF of Ref Guide
[ https://issues.apache.org/jira/browse/HBASE-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349316#comment-14349316 ] Nick Dimiduk commented on HBASE-11670: -- Hi [~misty]. You and [~stack] work through the issues? Looks like something was committed. How can I help? > Build PDF of Ref Guide > -- > > Key: HBASE-11670 > URL: https://issues.apache.org/jira/browse/HBASE-11670 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Minor > Attachments: HBASE-11670-v1.patch, HBASE-11670.patch, > hbase_logo_with_orca.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13076) Table can be forcibly enabled in AssignmentManager during table disabling.
[ https://issues.apache.org/jira/browse/HBASE-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349313#comment-14349313 ] Jimmy Xiang commented on HBASE-13076: - I remember that the code removed in this patch is initially introduced in HBASE-5155 (later on changed a little in HBASE-6229). At that time, we had a some problem to tell if a table is enabled if it is already enabled. For this issue, I think we can remove the code, or fail the assignment if the table is not enabled/enabling. I prefer to remove the code since the table state is checked later anyway (and the change is simpler/safer). (Note: If we fail the assignment now, it is good, but we need to update the state accordingly. That's some enhancement. If this doesn't happen a lot, we may not need the enhancement.) > Table can be forcibly enabled in AssignmentManager during table disabling. > -- > > Key: HBASE-13076 > URL: https://issues.apache.org/jira/browse/HBASE-13076 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Affects Versions: 2.0.0 >Reporter: Andrey Stepachev >Assignee: Andrey Stepachev > Attachments: 23757f039d83f4f17ca18815eae70b28.log, HBASE-13076.patch > > > Got situation where region can be opened while table is disabling by > DisableTableHandler. Here is relevant log for such situation. There is no > clues who issued OPEN to region. > Log file attached. > UPD: A bit more details. It seems that even in case of new state put into > meta, it still possible to get previous state. > That leads to one more round of assignment invoked in > AssignmentManager#onRegionClosed. > UPD: Table become ENABLED, thats leads to regions instructed to assign > immediately on onRegionClosed. BulkDisabler will not know about that and will > wait indefinitely, because it will not issue unassign for newly opened > regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13150) TestMasterObserver failing disable table at end of test
[ https://issues.apache.org/jira/browse/HBASE-13150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349319#comment-14349319 ] Jimmy Xiang commented on HBASE-13150: - Good analysing. I think we are good to remove that part as Andrey did in HBASE-13076. We should not change (write) a table state in assigning any region. We only need to check (read) the state instead. > TestMasterObserver failing disable table at end of test > --- > > Key: HBASE-13150 > URL: https://issues.apache.org/jira/browse/HBASE-13150 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > > I see in > https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6202/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestMasterObserver/testRegionTransitionOperations/ > , now we have added in timeouts, that we are failing to disable a table. It > looks like table is disabled but regions are being opened on the disabled > table still, like HBASE-6537 > Let me see if can figure why this happening. Will be back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349320#comment-14349320 ] stack commented on HBASE-13142: --- Any chance of a review here? Let me try on cluster and see if can get pictures of benefit. > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, > traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-13107) Refactor MOB Snapshot logic to reduce code duplication.
[ https://issues.apache.org/jira/browse/HBASE-13107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh resolved HBASE-13107. Resolution: Fixed Hadoop Flags: Reviewed lgtm. thanks jingcheng! committed to hbase-11339 > Refactor MOB Snapshot logic to reduce code duplication. > --- > > Key: HBASE-13107 > URL: https://issues.apache.org/jira/browse/HBASE-13107 > Project: HBase > Issue Type: Sub-task > Components: mob, snapshots >Affects Versions: hbase-11339 >Reporter: Jonathan Hsieh >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13107.diff > > > The MOB Snapshot code contains a lot of code duplication with the normal > snapshot code path. We should do some refactoring to clean this up before > merging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13150) TestMasterObserver failing disable table at end of test
[ https://issues.apache.org/jira/browse/HBASE-13150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349370#comment-14349370 ] stack commented on HBASE-13150: --- Thanks [~jxiang] I committed HBASE-13076. Lets see if it helps here. Will leave the issue open a while as we watch it. > TestMasterObserver failing disable table at end of test > --- > > Key: HBASE-13150 > URL: https://issues.apache.org/jira/browse/HBASE-13150 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > > I see in > https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/6202/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestMasterObserver/testRegionTransitionOperations/ > , now we have added in timeouts, that we are failing to disable a table. It > looks like table is disabled but regions are being opened on the disabled > table still, like HBASE-6537 > Let me see if can figure why this happening. Will be back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13076) Table can be forcibly enabled in AssignmentManager during table disabling.
[ https://issues.apache.org/jira/browse/HBASE-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349376#comment-14349376 ] Rajeshbabu Chintaguntla commented on HBASE-13076: - Earlier there was a check whether table is disabling or disabled state and skip assignment before the code of setting it to ENABLED. Some how it was removed. > Table can be forcibly enabled in AssignmentManager during table disabling. > -- > > Key: HBASE-13076 > URL: https://issues.apache.org/jira/browse/HBASE-13076 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Affects Versions: 2.0.0 >Reporter: Andrey Stepachev >Assignee: Andrey Stepachev > Fix For: 2.0.0 > > Attachments: 23757f039d83f4f17ca18815eae70b28.log, HBASE-13076.patch > > > Got situation where region can be opened while table is disabling by > DisableTableHandler. Here is relevant log for such situation. There is no > clues who issued OPEN to region. > Log file attached. > UPD: A bit more details. It seems that even in case of new state put into > meta, it still possible to get previous state. > That leads to one more round of assignment invoked in > AssignmentManager#onRegionClosed. > UPD: Table become ENABLED, thats leads to regions instructed to assign > immediately on onRegionClosed. BulkDisabler will not know about that and will > wait indefinitely, because it will not issue unassign for newly opened > regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13076) Table can be forcibly enabled in AssignmentManager during table disabling.
[ https://issues.apache.org/jira/browse/HBASE-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13076: -- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master only. Lets see if it fixes HBASE-13150 > Table can be forcibly enabled in AssignmentManager during table disabling. > -- > > Key: HBASE-13076 > URL: https://issues.apache.org/jira/browse/HBASE-13076 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Affects Versions: 2.0.0 >Reporter: Andrey Stepachev >Assignee: Andrey Stepachev > Fix For: 2.0.0 > > Attachments: 23757f039d83f4f17ca18815eae70b28.log, HBASE-13076.patch > > > Got situation where region can be opened while table is disabling by > DisableTableHandler. Here is relevant log for such situation. There is no > clues who issued OPEN to region. > Log file attached. > UPD: A bit more details. It seems that even in case of new state put into > meta, it still possible to get previous state. > That leads to one more round of assignment invoked in > AssignmentManager#onRegionClosed. > UPD: Table become ENABLED, thats leads to regions instructed to assign > immediately on onRegionClosed. BulkDisabler will not know about that and will > wait indefinitely, because it will not issue unassign for newly opened > regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13151) IllegalArgumentException in compaction when table has a namespace
[ https://issues.apache.org/jira/browse/HBASE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13151: --- Attachment: HBASE-13151-V5.diff jingcheng, I added few tweaks to handle the functions added by other patches. Other than that lgtm, and I've committed to hbase-11339 > IllegalArgumentException in compaction when table has a namespace > - > > Key: HBASE-13151 > URL: https://issues.apache.org/jira/browse/HBASE-13151 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13151-V2.diff, HBASE-13151-V3.diff, > HBASE-13151-V4.diff, HBASE-13151-V5.diff, HBASE-13151.diff > > > In the PartitionedMobFileCompactor we have a bulk load path, currently it > directly use table.getNameAsString as part of the path, if the table has a > namespace, the path contains character ":" which is regarded as a illegal > character in file path. As fixing, we need to divide the namespace and > qualifier into directories like the way in table path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13076) Table can be forcibly enabled in AssignmentManager during table disabling.
[ https://issues.apache.org/jira/browse/HBASE-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349386#comment-14349386 ] stack commented on HBASE-13076: --- [~rajesh23] Is it the code that @zhangduo references over in HBASE-13150 that is in branch-1? > Table can be forcibly enabled in AssignmentManager during table disabling. > -- > > Key: HBASE-13076 > URL: https://issues.apache.org/jira/browse/HBASE-13076 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Affects Versions: 2.0.0 >Reporter: Andrey Stepachev >Assignee: Andrey Stepachev > Fix For: 2.0.0 > > Attachments: 23757f039d83f4f17ca18815eae70b28.log, HBASE-13076.patch > > > Got situation where region can be opened while table is disabling by > DisableTableHandler. Here is relevant log for such situation. There is no > clues who issued OPEN to region. > Log file attached. > UPD: A bit more details. It seems that even in case of new state put into > meta, it still possible to get previous state. > That leads to one more round of assignment invoked in > AssignmentManager#onRegionClosed. > UPD: Table become ENABLED, thats leads to regions instructed to assign > immediately on onRegionClosed. BulkDisabler will not know about that and will > wait indefinitely, because it will not issue unassign for newly opened > regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-13151) IllegalArgumentException in compaction when table has a namespace
[ https://issues.apache.org/jira/browse/HBASE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh resolved HBASE-13151. Resolution: Fixed Hadoop Flags: Reviewed > IllegalArgumentException in compaction when table has a namespace > - > > Key: HBASE-13151 > URL: https://issues.apache.org/jira/browse/HBASE-13151 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: hbase-11339 >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: hbase-11339 > > Attachments: HBASE-13151-V2.diff, HBASE-13151-V3.diff, > HBASE-13151-V4.diff, HBASE-13151-V5.diff, HBASE-13151.diff > > > In the PartitionedMobFileCompactor we have a bulk load path, currently it > directly use table.getNameAsString as part of the path, if the table has a > namespace, the path contains character ":" which is regarded as a illegal > character in file path. As fixing, we need to divide the namespace and > qualifier into directories like the way in table path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349400#comment-14349400 ] Elliott Clark commented on HBASE-13142: --- Looking now. > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, > traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13076) Table can be forcibly enabled in AssignmentManager during table disabling.
[ https://issues.apache.org/jira/browse/HBASE-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349413#comment-14349413 ] Rajeshbabu Chintaguntla commented on HBASE-13076: - Yes [~stack]. HBASE-13150 is implication of removing the check. Before starting assignment we can check the table is disabling/disabled if yes then we can skip it. Any way after this fix the region should be unassigned at the end of assignment if the table in disabling/disabled state. > Table can be forcibly enabled in AssignmentManager during table disabling. > -- > > Key: HBASE-13076 > URL: https://issues.apache.org/jira/browse/HBASE-13076 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Affects Versions: 2.0.0 >Reporter: Andrey Stepachev >Assignee: Andrey Stepachev > Fix For: 2.0.0 > > Attachments: 23757f039d83f4f17ca18815eae70b28.log, HBASE-13076.patch > > > Got situation where region can be opened while table is disabling by > DisableTableHandler. Here is relevant log for such situation. There is no > clues who issued OPEN to region. > Log file attached. > UPD: A bit more details. It seems that even in case of new state put into > meta, it still possible to get previous state. > That leads to one more round of assignment invoked in > AssignmentManager#onRegionClosed. > UPD: Table become ENABLED, thats leads to regions instructed to assign > immediately on onRegionClosed. BulkDisabler will not know about that and will > wait indefinitely, because it will not issue unassign for newly opened > regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13076) Table can be forcibly enabled in AssignmentManager during table disabling.
[ https://issues.apache.org/jira/browse/HBASE-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349418#comment-14349418 ] stack commented on HBASE-13076: --- Ok. Thanks for speaking up... [~rajesh23] > Table can be forcibly enabled in AssignmentManager during table disabling. > -- > > Key: HBASE-13076 > URL: https://issues.apache.org/jira/browse/HBASE-13076 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Affects Versions: 2.0.0 >Reporter: Andrey Stepachev >Assignee: Andrey Stepachev > Fix For: 2.0.0 > > Attachments: 23757f039d83f4f17ca18815eae70b28.log, HBASE-13076.patch > > > Got situation where region can be opened while table is disabling by > DisableTableHandler. Here is relevant log for such situation. There is no > clues who issued OPEN to region. > Log file attached. > UPD: A bit more details. It seems that even in case of new state put into > meta, it still possible to get previous state. > That leads to one more round of assignment invoked in > AssignmentManager#onRegionClosed. > UPD: Table become ENABLED, thats leads to regions instructed to assign > immediately on onRegionClosed. BulkDisabler will not know about that and will > wait indefinitely, because it will not issue unassign for newly opened > regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349423#comment-14349423 ] Elliott Clark commented on HBASE-13142: --- There's going to be a lot of contention on getting a bb and putting a bb. Do you think the functionality of returning buffers in strictly sorted order is worth that much locking? Seems like this could be done with a pretty light-weight queue. Additionally shrinking the amount of code under lock could help a lot here. > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, > traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349426#comment-14349426 ] stack commented on HBASE-13142: --- [~eclark] bq, Do you think the functionality of returning buffers in strictly sorted order is worth that much locking? Not that important I'd say. BBs would be less likely to be same general size but can't see a real downside. Let me get graphs on current state then play with something dumber. > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, > traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13159) Consider RangeReferenceFiles with transformations
[ https://issues.apache.org/jira/browse/HBASE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-13159: -- Description: Currently we have References used by HalfStoreReaders and HFileLinks. For various use cases we have here we have need for a RangeReferences with simple transformation of the keys. That would allow us to map HFiles between regions or even tables without copying any data. We can probably combine HalfStores, HFileLinks, and RangeReferences into a single concept: * RangeReference = arbitrary start and stop row, arbitrary key transformation * HFileLink = start and stop keys set to the linked file's start/stop key, transformation = identity * (HalfStore) References = start/stop key set according to top or bottom reference, transformation = identity Note this is a *brainstorming* issue. :) (Could start with just references with arbitrary start/stop keys, and do transformations later) was: Currently we have References used by HalfStoreReaders and HFileLinks. For various use case we have here we have need for a RangeReferences with simple transformation of the keys. That would allow us to map HFiles between region or even tables without copying any data. We can probably combine HalfStores, HFileLinks, and RangeReferences into a single concept: * RangeReference = arbitrary start and stop row, arbitrary key transformation * HFileLink = start and stop keys set to the linked file's start/stop key, transformation = identity * (HalfStore) References = start/stop key set according to top or bottom reference, transformation = identity Note this is a *brainstorming* issue. :) (Could start with just references with arbitrary start/stop keys, and do transformations later) > Consider RangeReferenceFiles with transformations > - > > Key: HBASE-13159 > URL: https://issues.apache.org/jira/browse/HBASE-13159 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl > > Currently we have References used by HalfStoreReaders and HFileLinks. > For various use cases we have here we have need for a RangeReferences with > simple transformation of the keys. > That would allow us to map HFiles between regions or even tables without > copying any data. > We can probably combine HalfStores, HFileLinks, and RangeReferences into a > single concept: > * RangeReference = arbitrary start and stop row, arbitrary key transformation > * HFileLink = start and stop keys set to the linked file's start/stop key, > transformation = identity > * (HalfStore) References = start/stop key set according to top or bottom > reference, transformation = identity > Note this is a *brainstorming* issue. :) > (Could start with just references with arbitrary start/stop keys, and do > transformations later) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13159) Consider RangeReferences with transformations
Lars Hofhansl created HBASE-13159: - Summary: Consider RangeReferences with transformations Key: HBASE-13159 URL: https://issues.apache.org/jira/browse/HBASE-13159 Project: HBase Issue Type: Brainstorming Reporter: Lars Hofhansl Currently we have References used by HalfStoreReaders and HFileLinks. For various use case we have here we have need for a RangeReferences with simple transformation of the keys. That would allow us to map HFiles between region or even tables without copying any data. We can probably combine HalfStores, HFileLinks, and RangeReferences into a single concept: * RangeReference = arbitrary start and stop row, arbitrary key transformation * HFileLink = start and stop keys set to the linked file's start/stop key, transformation = identity * (HalfStore) References = start/stop key set according to top or bottom reference, transformation = identity Note this is a *brainstorming* issue. :) (Could start with just references with arbitrary start/stop keys, and do transformations later) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13159) Consider RangeReferenceFiles with transformations
[ https://issues.apache.org/jira/browse/HBASE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-13159: -- Summary: Consider RangeReferenceFiles with transformations (was: Consider RangeReferences with transformations) > Consider RangeReferenceFiles with transformations > - > > Key: HBASE-13159 > URL: https://issues.apache.org/jira/browse/HBASE-13159 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl > > Currently we have References used by HalfStoreReaders and HFileLinks. > For various use case we have here we have need for a RangeReferences with > simple transformation of the keys. > That would allow us to map HFiles between region or even tables without > copying any data. > We can probably combine HalfStores, HFileLinks, and RangeReferences into a > single concept: > * RangeReference = arbitrary start and stop row, arbitrary key transformation > * HFileLink = start and stop keys set to the linked file's start/stop key, > transformation = identity > * (HalfStore) References = start/stop key set according to top or bottom > reference, transformation = identity > Note this is a *brainstorming* issue. :) > (Could start with just references with arbitrary start/stop keys, and do > transformations later) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349455#comment-14349455
]
Andrew Purtell commented on HBASE-12972:
A quick update. I have been working on a patch for master. It's become a multi
day effort as where drawing the line on what should be in Region vs HRegion is
a trade off between getting good encapsulation in Region and a bunch of casting
where code wants HRegion internals. I'm working on keeping casts to a minimum.
I should have something today.
> Region, a supportable public/evolving subset of HRegion
> ---
>
> Key: HBASE-12972
> URL: https://issues.apache.org/jira/browse/HBASE-12972
> Project: HBase
> Issue Type: New Feature
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-12972-0.98.patch
>
>
> On HBASE-12566, [~lhofhansl] proposed:
> {quote}
> Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
> {{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
> but used in some coprocessor hooks.
> {quote}
> By example, now coprocessors have to reach into HRegion in order to
> participate in row and region locking protocols, this is one area where the
> functionality is legitimate for coprocessors but not for users, so an
> in-between interface make sense.
> In addition we should promote {{Store}}'s interface audience to
> LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13159) Consider RangeReferenceFiles with transformations
[ https://issues.apache.org/jira/browse/HBASE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349459#comment-14349459 ] Vladimir Rodionov commented on HBASE-13159: --- What is the purpose of sharing one HFile between different regions? The file will be split during next major compaction anyway and this should happen immediately after region's split. > Consider RangeReferenceFiles with transformations > - > > Key: HBASE-13159 > URL: https://issues.apache.org/jira/browse/HBASE-13159 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl > > Currently we have References used by HalfStoreReaders and HFileLinks. > For various use cases we have here we have need for a RangeReferences with > simple transformation of the keys. > That would allow us to map HFiles between regions or even tables without > copying any data. > We can probably combine HalfStores, HFileLinks, and RangeReferences into a > single concept: > * RangeReference = arbitrary start and stop row, arbitrary key transformation > * HFileLink = start and stop keys set to the linked file's start/stop key, > transformation = identity > * (HalfStore) References = start/stop key set according to top or bottom > reference, transformation = identity > Note this is a *brainstorming* issue. :) > (Could start with just references with arbitrary start/stop keys, and do > transformations later) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13159) Consider RangeReferenceFiles with transformations
[ https://issues.apache.org/jira/browse/HBASE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349470#comment-14349470 ] Vladimir Rodionov commented on HBASE-13159: --- Transformation idea is interesting. I did suggest something similar while ago (do not remember JIRA) to support bulk delete and update efficiently , but on table/cf level - not a hfile level. If this is what I think you are talking about, [~lhofhansl]. > Consider RangeReferenceFiles with transformations > - > > Key: HBASE-13159 > URL: https://issues.apache.org/jira/browse/HBASE-13159 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl > > Currently we have References used by HalfStoreReaders and HFileLinks. > For various use cases we have here we have need for a RangeReferences with > simple transformation of the keys. > That would allow us to map HFiles between regions or even tables without > copying any data. > We can probably combine HalfStores, HFileLinks, and RangeReferences into a > single concept: > * RangeReference = arbitrary start and stop row, arbitrary key transformation > * HFileLink = start and stop keys set to the linked file's start/stop key, > transformation = identity > * (HalfStore) References = start/stop key set according to top or bottom > reference, transformation = identity > Note this is a *brainstorming* issue. :) > (Could start with just references with arbitrary start/stop keys, and do > transformations later) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13076) Table can be forcibly enabled in AssignmentManager during table disabling.
[ https://issues.apache.org/jira/browse/HBASE-13076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349495#comment-14349495 ] Hudson commented on HBASE-13076: SUCCESS: Integrated in HBase-TRUNK #6211 (See [https://builds.apache.org/job/HBase-TRUNK/6211/]) HBASE-13076 Table can be forcibly enabled in AssignmentManager during table disabling (Andrey Stepachev) (stack: rev 4570451f7cf2dba2b22664b470b73547e2496a88) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Table can be forcibly enabled in AssignmentManager during table disabling. > -- > > Key: HBASE-13076 > URL: https://issues.apache.org/jira/browse/HBASE-13076 > Project: HBase > Issue Type: Bug > Components: master, Region Assignment >Affects Versions: 2.0.0 >Reporter: Andrey Stepachev >Assignee: Andrey Stepachev > Fix For: 2.0.0 > > Attachments: 23757f039d83f4f17ca18815eae70b28.log, HBASE-13076.patch > > > Got situation where region can be opened while table is disabling by > DisableTableHandler. Here is relevant log for such situation. There is no > clues who issued OPEN to region. > Log file attached. > UPD: A bit more details. It seems that even in case of new state put into > meta, it still possible to get previous state. > That leads to one more round of assignment invoked in > AssignmentManager#onRegionClosed. > UPD: Table become ENABLED, thats leads to regions instructed to assign > immediately on onRegionClosed. BulkDisabler will not know about that and will > wait indefinitely, because it will not issue unassign for newly opened > regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13142) [PERF] Reuse the IPCUtil#buildCellBlock buffer
[ https://issues.apache.org/jira/browse/HBASE-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349510#comment-14349510 ] stack commented on HBASE-13142: --- Hmm... I love testing. My key implementation is actually broken. Am allocating every time. Even with this breakage, the fact that we are getting an average size makes it so we skip a bunch of resizing and this alone is making for increased throughput (in spite of upped GC). Let me fix or rather, do the @eclark suggestion > [PERF] Reuse the IPCUtil#buildCellBlock buffer > -- > > Key: HBASE-13142 > URL: https://issues.apache.org/jira/browse/HBASE-13142 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: stack > Labels: beginner > Fix For: 2.0.0, 1.1.0 > > Attachments: 13142.txt, 13142v2.txt, 13142v3.txt, traces.2.svg, > traces.svg > > > Running some scan profiling, flight recorder was mildly fingering resize of > the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It > was half-hearted blaming it for a few hundreds of ms over a five minute > sampling with a few tens of instances showing. > I tried then w/ flamegraph/lightweight profiler and this reported the buffer > allocations taking 22% of our total CPU. See attachment trace.svg. > I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and > indeed every allocation was doing a resize from initial allocation of 16k -- > the default up to 220k (this test returns ten randomly sized rows zipfian > sized between 0 and 8k). > Upping the allocation to 220k meant we now avoided the resize but the initial > allocation was now blamed for 10% of allocations (see trace.2.svg attached). > Lets do buffer reuse. Will save a bunch of allocation and CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13159) Consider RangeReferenceFiles with transformations
[ https://issues.apache.org/jira/browse/HBASE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349519#comment-14349519 ] Lars Hofhansl commented on HBASE-13159: --- In our case we'd want to share *initial* data between tenants. The next compaction (if the other tenant changes data) would cause the data to be rewritten, that's how we'd want it. (or maybe we'd want to optionally block these reference files from being compacted). Might be very specific to our use case. Bulk delete and update by declaring a transformation is also very interesting. > Consider RangeReferenceFiles with transformations > - > > Key: HBASE-13159 > URL: https://issues.apache.org/jira/browse/HBASE-13159 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl > > Currently we have References used by HalfStoreReaders and HFileLinks. > For various use cases we have here we have need for a RangeReferences with > simple transformation of the keys. > That would allow us to map HFiles between regions or even tables without > copying any data. > We can probably combine HalfStores, HFileLinks, and RangeReferences into a > single concept: > * RangeReference = arbitrary start and stop row, arbitrary key transformation > * HFileLink = start and stop keys set to the linked file's start/stop key, > transformation = identity > * (HalfStore) References = start/stop key set according to top or bottom > reference, transformation = identity > Note this is a *brainstorming* issue. :) > (Could start with just references with arbitrary start/stop keys, and do > transformations later) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13103:
-
Description:
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
"reshape" or "balance" a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
> reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above "reshape"
operation on an interval. That way, the cluster will automatically self-correct
toward a desirable state.
was:
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
"reshape" or "balance" a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
> reshape TABLE, numSplits
{noformat}
Summary: [ergonomics] add region size balancing as a feature of master
(was: [ergonomics] add shell,API to "reshape" a table)
> [ergonomics] add region size balancing as a feature of master
> -
>
> Key: HBASE-13103
> URL: https://issues.apache.org/jira/browse/HBASE-13103
> Project: HBase
> Issue Type: Brainstorming
> Components: Usability
>Reporter: Nick Dimiduk
>
> Often enough, folks miss-judge split points or otherwise end up with a
> suboptimal number of regions. We should have an automated, reliable way to
> "reshape" or "balance" a table's region boundaries. This would be for tables
> that contain existing data. This might look like:
> {noformat}
> Admin#reshapeTable(TableName, int numSplits);
> {noformat}
> or from the shell:
> {noformat}
> > reshape TABLE, numSplits
> {noformat}
> Better still would be to have a maintenance process, similar to the existing
> Balancer that runs AssignmentManager on an interval, to run the above
> "reshape" operation on an interval. That way, the cluster will automatically
> self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13160) SplitLogWorker does not pick up the task immediately
Enis Soztutar created HBASE-13160:
-
Summary: SplitLogWorker does not pick up the task immediately
Key: HBASE-13160
URL: https://issues.apache.org/jira/browse/HBASE-13160
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.1.0
We were reading some code with Jeffrey, and we realized that the
SplitLogWorker's internal task loop is weird. It does {{ls}} every second and
sleeps, but have another mechanism to learn about new tasks, but does not make
affective use of the zk notification.
I have a simple patch which might improve this area.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13160) SplitLogWorker does not pick up the task immediately
[
https://issues.apache.org/jira/browse/HBASE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13160:
--
Status: Patch Available (was: Open)
> SplitLogWorker does not pick up the task immediately
>
>
> Key: HBASE-13160
> URL: https://issues.apache.org/jira/browse/HBASE-13160
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.1.0
>
> Attachments: hbase-13160_v1.patch
>
>
> We were reading some code with Jeffrey, and we realized that the
> SplitLogWorker's internal task loop is weird. It does {{ls}} every second and
> sleeps, but have another mechanism to learn about new tasks, but does not
> make affective use of the zk notification.
> I have a simple patch which might improve this area.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
