[jira] [Updated] (HBASE-16611) Flakey org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet
[
https://issues.apache.org/jira/browse/HBASE-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-16611:
--
Fix Version/s: 2.0.0
> Flakey org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet
> -
>
> Key: HBASE-16611
> URL: https://issues.apache.org/jira/browse/HBASE-16611
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-16611.patch, HBASE-16611.v1.patch,
> HBASE-16611.v1.patch, HBASE-16611.v2.patch
>
>
> see
> https://builds.apache.org/job/PreCommit-HBASE-Build/3494/artifact/patchprocess/patch-unit-hbase-server.txt
> {code}
> testCancelOfMultiGet(org.apache.hadoop.hbase.client.TestReplicasClient) Time
> elapsed: 4.026 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet(TestReplicasClient.java:579)
> Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 94.401 sec -
> in org.apache.hadoop.hbase.client.TestAdmin2
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.861 sec -
> in org.apache.hadoop.hbase.client.TestClientScannerRPCTimeout
> Running
> org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 261.925 sec
> <<< FAILURE! - in org.apache.hadoop.hbase.client.TestReplicasClient
> testCancelOfMultiGet(org.apache.hadoop.hbase.client.TestReplicasClient) Time
> elapsed: 4.522 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet(TestReplicasClient.java:581)
> Running org.apache.hadoop.hbase.client.TestFastFail
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.648 sec -
> in org.apache.hadoop.hbase.client.TestFastFail
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 277.894 sec
> <<< FAILURE! - in org.apache.hadoop.hbase.client.TestReplicasClient
> testCancelOfMultiGet(org.apache.hadoop.hbase.client.TestReplicasClient) Time
> elapsed: 5.359 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet(TestReplicasClient.java:579)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16611) Flakey org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet
[
https://issues.apache.org/jira/browse/HBASE-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-16611:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Flakey org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet
> -
>
> Key: HBASE-16611
> URL: https://issues.apache.org/jira/browse/HBASE-16611
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-16611.patch, HBASE-16611.v1.patch,
> HBASE-16611.v1.patch, HBASE-16611.v2.patch
>
>
> see
> https://builds.apache.org/job/PreCommit-HBASE-Build/3494/artifact/patchprocess/patch-unit-hbase-server.txt
> {code}
> testCancelOfMultiGet(org.apache.hadoop.hbase.client.TestReplicasClient) Time
> elapsed: 4.026 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet(TestReplicasClient.java:579)
> Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 94.401 sec -
> in org.apache.hadoop.hbase.client.TestAdmin2
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.861 sec -
> in org.apache.hadoop.hbase.client.TestClientScannerRPCTimeout
> Running
> org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 261.925 sec
> <<< FAILURE! - in org.apache.hadoop.hbase.client.TestReplicasClient
> testCancelOfMultiGet(org.apache.hadoop.hbase.client.TestReplicasClient) Time
> elapsed: 4.522 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet(TestReplicasClient.java:581)
> Running org.apache.hadoop.hbase.client.TestFastFail
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.648 sec -
> in org.apache.hadoop.hbase.client.TestFastFail
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 277.894 sec
> <<< FAILURE! - in org.apache.hadoop.hbase.client.TestReplicasClient
> testCancelOfMultiGet(org.apache.hadoop.hbase.client.TestReplicasClient) Time
> elapsed: 5.359 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet(TestReplicasClient.java:579)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16540) Scan should do additional validation on start and stop row
[ https://issues.apache.org/jira/browse/HBASE-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486509#comment-15486509 ] Sean Busbey commented on HBASE-16540: - I'm confused, because looking at master I see a properly labeled commit for this issue but not the incorrectly labeled commit that Duo pointed out originally. > Scan should do additional validation on start and stop row > -- > > Key: HBASE-16540 > URL: https://issues.apache.org/jira/browse/HBASE-16540 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Dustin Pho > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16540.002.patch, HBASE-16540.patch > > > Scan.setStartRow() and setStopRow() should validate the byte[] passed to > ensure it meets the criteria for a row key. If the byte[] length is greater > that Short.MAX_VALUE, we should throw an IllegalArgumentException in order to > fast fail and prevent server-side errors being thrown and retried. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16616) Rpc handlers stuck on ThreadLocalMap.expungeStaleEntry
[
https://issues.apache.org/jira/browse/HBASE-16616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486513#comment-15486513
]
Hiroshi Ikeda commented on HBASE-16616:
---
It might be good to change indexHolderThreadLocal to be static. It is temporary
which CPU executes which thread, so that the index to avoid cache-line
contention per thread will be changing, after all. LongAdder does in the
similar manner; LongAdder uses a newly added field into Thread, sharing between
all instances per Thread. Adding the new method requires for every counter to
call.
> Rpc handlers stuck on ThreadLocalMap.expungeStaleEntry
> --
>
> Key: HBASE-16616
> URL: https://issues.apache.org/jira/browse/HBASE-16616
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.2
>Reporter: Tomu Tsuruhara
>Assignee: Tomu Tsuruhara
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16616.branch-1.v2.txt, HBASE-16616.master.001.patch,
> HBASE-16616.master.002.patch, ScreenShot 2016-09-09 14.17.53.png
>
>
> In our HBase 1.2.2 cluster, some regionserver showed too bad
> "QueueCallTime_99th_percentile" exceeding 10 seconds.
> Most rpc handler threads stuck on ThreadLocalMap.expungeStaleEntry call at
> that time.
> {noformat}
> "PriorityRpcServer.handler=18,queue=0,port=16020" #322 daemon prio=5
> os_prio=0 tid=0x7fd422062800 nid=0x19b89 runnable [0x7fcb8a821000]
>java.lang.Thread.State: RUNNABLE
> at
> java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(ThreadLocal.java:617)
> at java.lang.ThreadLocal$ThreadLocalMap.remove(ThreadLocal.java:499)
> at
> java.lang.ThreadLocal$ThreadLocalMap.access$200(ThreadLocal.java:298)
> at java.lang.ThreadLocal.remove(ThreadLocal.java:222)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:426)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
> at
> com.yammer.metrics.stats.ExponentiallyDecayingSample.unlockForRegularUsage(ExponentiallyDecayingSample.java:196)
> at
> com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:113)
> at
> com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:81)
> at
> org.apache.hadoop.metrics2.lib.MutableHistogram.add(MutableHistogram.java:81)
> at
> org.apache.hadoop.metrics2.lib.MutableRangeHistogram.add(MutableRangeHistogram.java:59)
> at
> org.apache.hadoop.hbase.ipc.MetricsHBaseServerSourceImpl.dequeuedCall(MetricsHBaseServerSourceImpl.java:194)
> at
> org.apache.hadoop.hbase.ipc.MetricsHBaseServer.dequeuedCall(MetricsHBaseServer.java:76)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2192)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We were using jdk 1.8.0_92 and here is a snippet from ThreadLocal.java.
> {code}
> 616:while (tab[h] != null)
> 617:h = nextIndex(h, len);
> {code}
> So I hypothesized that there're too many consecutive entries in {{tab}} array
> and actually I found them in the heapdump.
> !ScreenShot 2016-09-09 14.17.53.png|width=50%!
> Most of these entries pointed at instance of
> {{org.apache.hadoop.hbase.util.Counter$1}}
> which is equivarent to {{indexHolderThreadLocal}} instance-variable in the
> {{Counter}} class.
> Because {{RpcServer$Connection}} class creates a {{Counter}} instance
> {{rpcCount}} for every connections,
> it is possible to have lots of {{Counter#indexHolderThreadLocal}} instances
> in RegionServer process
> when we repeat connect-and-close from client. As a result, a ThreadLocalMap
> can have lots of consecutive
> entires.
> Usually, since each entry is a {{WeakReference}}, these entries are collected
> and removed
> by garbage-collector soon after connection closed.
> But if connection's life-time was long enough to survive youngGC, it wouldn't
> be collected until old-gen collector runs.
> Furthermore, under G1GC deployment, it is possible not to be collected even
> by old-gen GC(mixed GC)
> if entries sit in a region which doesn't have much garbages.
> Actually we used G1GC when we encountered this problem.
> We should remove the entry from ThreadLocalMap by calling ThreadLocal#remove
> explicitly.
--
Thi
[jira] [Updated] (HBASE-16610) Unify append, increment with AP
[ https://issues.apache.org/jira/browse/HBASE-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16610: -- Attachment: HBASE-16610.v1.patch retry > Unify append, increment with AP > --- > > Key: HBASE-16610 > URL: https://issues.apache.org/jira/browse/HBASE-16610 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16610.patch, HBASE-16610.v1.patch, > HBASE-16610.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486541#comment-15486541 ] Duo Zhang commented on HBASE-15921: --- I think the work here is related to [~chenheng]'s work in HBASE-16593. If we can unify all operations with AsyncProcess, then change AsyncProcess to use async rpc stub and expose CompletableFuture as return value, then the implementation of an async table is already there. A simple wrapper can give us a AsyncTable. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16540) Scan should do additional validation on start and stop row
[ https://issues.apache.org/jira/browse/HBASE-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486542#comment-15486542 ] Gary Helmling commented on HBASE-16540: --- I did a rebase/reword with push --force-with-leases to fix the commit message. [~anoop.hbase] I saw your commit in the list. Did it screw up your commit history? > Scan should do additional validation on start and stop row > -- > > Key: HBASE-16540 > URL: https://issues.apache.org/jira/browse/HBASE-16540 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Dustin Pho > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16540.002.patch, HBASE-16540.patch > > > Scan.setStartRow() and setStopRow() should validate the byte[] passed to > ensure it meets the criteria for a row key. If the byte[] length is greater > that Short.MAX_VALUE, we should throw an IllegalArgumentException in order to > fast fail and prevent server-side errors being thrown and retried. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16540) Scan should do additional validation on start and stop row
[ https://issues.apache.org/jira/browse/HBASE-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486555#comment-15486555 ] Duo Zhang commented on HBASE-16540: --- The commits on git server are correct I think https://git-wip-us.apache.org/repos/asf?p=hbase.git But seems the fix push also involved two later commits. [~anoopsamjohn] Maybe you should reset your local repo to this one first https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=commit;h=8855670cd701fdf9c2ab41907f9525d122608e6d And then try a git pull? > Scan should do additional validation on start and stop row > -- > > Key: HBASE-16540 > URL: https://issues.apache.org/jira/browse/HBASE-16540 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Dustin Pho > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16540.002.patch, HBASE-16540.patch > > > Scan.setStartRow() and setStopRow() should validate the byte[] passed to > ensure it meets the criteria for a row key. If the byte[] length is greater > that Short.MAX_VALUE, we should throw an IllegalArgumentException in order to > fast fail and prevent server-side errors being thrown and retried. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486558#comment-15486558 ] Heng Chen commented on HBASE-15921: --- Awkward! Just notice the issue here. As [~Apache9] said, yes, that is my plan, AsyncProcess is a good choice for AsyncTable, it has lots of optimizations and has been tested during all release versions. BTW. Very sorry to find this issue so late. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16592) Unify Delete request with AP
[ https://issues.apache.org/jira/browse/HBASE-16592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486608#comment-15486608 ] Hudson commented on HBASE-16592: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1592 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1592/]) HBASE-16592 Unify Delete request with AP (chenheng: rev 831fb3ccb8a0ba449d249962379afd268e8fe032) * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractResponse.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/SingleResponse.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java > Unify Delete request with AP > > > Key: HBASE-16592 > URL: https://issues.apache.org/jira/browse/HBASE-16592 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16592.patch, HBASE-16592.v1.patch, > HBASE-16592.v1.patch, HBASE-16592.v2.patch, HBASE-16592.v3.patch > > > This is the first step try to unify the HTable with AP only, to extend AP > could process single action, i introduced AbstractResponse, multiResponse > and singleResponse (introduced to deal with single result) will extend this > class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486617#comment-15486617 ] Duo Zhang commented on HBASE-15921: --- But to be honest, the AsyncProcess is too complicated... Maybe we'd better write a new class to replace AsyncProcess? > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486626#comment-15486626 ] Heng Chen commented on HBASE-15921: --- yeah, AP is complicated but powerful, just need some refactor job. And after HBASE-16593, we could remove some classes in client to make the logical be more clear. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[
https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486656#comment-15486656
]
Duo Zhang commented on HBASE-15921:
---
In fact, we do not need to thread pool to execute the request with async rpc
stub(maybe we still need a thread to trigger retry), so the implementation will
be greatly different... The whole {{RetryingCallable}} should be redesigned and
reimplemented...
> Add first AsyncTable impl and create TableImpl based on it
> --
>
> Key: HBASE-15921
> URL: https://issues.apache.org/jira/browse/HBASE-15921
> Project: HBase
> Issue Type: Improvement
>Reporter: Jurriaan Mous
>Assignee: Jurriaan Mous
> Attachments: HBASE-15921.patch, HBASE-15921.v1.patch
>
>
> First we create an AsyncTable interface with implementation without the Scan
> functionality. Those will land in a separate patch since they need a refactor
> of existing scans.
> Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl
> internally and should be a bit faster because it does jump through less hoops
> to do ProtoBuf transportation. This way we can run all existing tests on the
> AsyncTableImpl to guarantee its quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[
https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486656#comment-15486656
]
Duo Zhang edited comment on HBASE-15921 at 9/13/16 8:36 AM:
In fact, we do not need a thread pool to execute the request with async rpc
stub(maybe we still need a thread to trigger retry), so the implementation will
be greatly different... The whole {{RetryingCallable}} should be redesigned and
reimplemented...
was (Author: apache9):
In fact, we do not need to thread pool to execute the request with async rpc
stub(maybe we still need a thread to trigger retry), so the implementation will
be greatly different... The whole {{RetryingCallable}} should be redesigned and
reimplemented...
> Add first AsyncTable impl and create TableImpl based on it
> --
>
> Key: HBASE-15921
> URL: https://issues.apache.org/jira/browse/HBASE-15921
> Project: HBase
> Issue Type: Improvement
>Reporter: Jurriaan Mous
>Assignee: Jurriaan Mous
> Attachments: HBASE-15921.patch, HBASE-15921.v1.patch
>
>
> First we create an AsyncTable interface with implementation without the Scan
> functionality. Those will land in a separate patch since they need a refactor
> of existing scans.
> Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl
> internally and should be a bit faster because it does jump through less hoops
> to do ProtoBuf transportation. This way we can run all existing tests on the
> AsyncTableImpl to guarantee its quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486670#comment-15486670 ] Heng Chen commented on HBASE-15921: --- Yeah, actually ap not use async rpc stub because it has it's own threadPool. That's due to some logical such as replica call schedule will still pend the thread. So IMO async rpc stub is not enough to implement async table. But AP is async in natural, and it has been used to implements async interface for multi actions. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16624) MVCC DeSerialization bug in the HFileScannerImpl
[ https://issues.apache.org/jira/browse/HBASE-16624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486672#comment-15486672 ] Anoop Sam John commented on HBASE-16624: +1. Lets get this in. Will see how/whether we can make this as a util kind of method or so, so that we can UT this. Lets get this in first. This is an imp fix. Similar read of mvcc vlong is done in 1.x versions also I believe. May be 1.2+ versions. cc [~saint@gmail.com] > MVCC DeSerialization bug in the HFileScannerImpl > > > Key: HBASE-16624 > URL: https://issues.apache.org/jira/browse/HBASE-16624 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Blocker > Attachments: HBASE-16624.patch > > > My colleague [~naggarwal] found a bug in the deserialization of mvcc from > HFile, As a part of the optimization of deserialization of VLong, we read a > int at once but we forgot to convert it to unsigned one. > This would cause issues because once we cross the integer threshold in > sequenceId and a compaction happens we would write MAX_MEMSTORE_TS in the > trailer as 0 (because we will be reading negative values from the file that > got flushed with sequenceId > Integer.MAX_VALUE). And once we have > MAX_MEMSTORE_TS as 0, and there are sequenceId values present alongside with > KeyValues the regionserver will now start failing to read the compacted file > and thus corruption. > Interestingly this would happen only on the tables that don't have > DataBlockEncoding enabled and unfortunately in our case that turned out to be > META and a another small table. > Fix is small (~20 chars) and attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16624) MVCC DeSerialization bug in the HFileScannerImpl
[ https://issues.apache.org/jira/browse/HBASE-16624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486672#comment-15486672 ] Anoop Sam John edited comment on HBASE-16624 at 9/13/16 8:40 AM: - +1 Lets get this in. Will see how/whether we can make this as a util kind of method or so, so that we can UT this. Lets get this in first. This is an imp fix. Similar read of mvcc vlong is done in 1.x versions also I believe. May be 1.2+ versions. cc [~saint@gmail.com] was (Author: anoop.hbase): +1. Lets get this in. Will see how/whether we can make this as a util kind of method or so, so that we can UT this. Lets get this in first. This is an imp fix. Similar read of mvcc vlong is done in 1.x versions also I believe. May be 1.2+ versions. cc [~saint@gmail.com] > MVCC DeSerialization bug in the HFileScannerImpl > > > Key: HBASE-16624 > URL: https://issues.apache.org/jira/browse/HBASE-16624 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Blocker > Attachments: HBASE-16624.patch > > > My colleague [~naggarwal] found a bug in the deserialization of mvcc from > HFile, As a part of the optimization of deserialization of VLong, we read a > int at once but we forgot to convert it to unsigned one. > This would cause issues because once we cross the integer threshold in > sequenceId and a compaction happens we would write MAX_MEMSTORE_TS in the > trailer as 0 (because we will be reading negative values from the file that > got flushed with sequenceId > Integer.MAX_VALUE). And once we have > MAX_MEMSTORE_TS as 0, and there are sequenceId values present alongside with > KeyValues the regionserver will now start failing to read the compacted file > and thus corruption. > Interestingly this would happen only on the tables that don't have > DataBlockEncoding enabled and unfortunately in our case that turned out to be > META and a another small table. > Fix is small (~20 chars) and attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16463) Improve transparent table/CF encryption with Commons Crypto
[ https://issues.apache.org/jira/browse/HBASE-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486688#comment-15486688 ] Dapeng Sun commented on HBASE-16463: Hi [~apurtell], [~anoopsamjohn] and [~ram_krish], thank all for the previous work on Transparent table/CF encryption (HBASE-7544), this patch added a new provider with Crypto and didn't modify current source code, do you have any comments about the patch? > Improve transparent table/CF encryption with Commons Crypto > --- > > Key: HBASE-16463 > URL: https://issues.apache.org/jira/browse/HBASE-16463 > Project: HBase > Issue Type: New Feature > Components: encryption >Affects Versions: 2.0.0 >Reporter: Dapeng Sun > Attachments: HBASE-16463.001.patch, HBASE-16463.002.patch, > HBASE-16463.003.patch > > > Apache Commons Crypto > (https://commons.apache.org/proper/commons-crypto/index.html) is a > cryptographic library optimized with AES-NI. > HBASE-7544 introduces a framework for transparent encryption feature for > protecting HFile and WAL data at rest. Currently JCE cipher is used bu > default, the improvement will use Commons Crypto to accelerate the > transparent encryption of HBase. new crypto provider with Commons CRYPTO will > be provided for Transparent encryption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486708#comment-15486708 ] Duo Zhang commented on HBASE-15921: --- But we should not use another thread pool to implement AsyncProcess if we already have async stub... We do not need to block, we have callback now... > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16610) Unify append, increment with AP
[ https://issues.apache.org/jira/browse/HBASE-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486751#comment-15486751 ] Heng Chen commented on HBASE-16610: --- I will make a performance test for the new interface. Because it introduces thread context changed with AP, i suspect the performance will be downgrade. > Unify append, increment with AP > --- > > Key: HBASE-16610 > URL: https://issues.apache.org/jira/browse/HBASE-16610 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16610.patch, HBASE-16610.v1.patch, > HBASE-16610.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16610) Unify append, increment with AP
[
https://issues.apache.org/jira/browse/HBASE-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486781#comment-15486781
]
Hadoop QA commented on HBASE-16610:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 42s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 56s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 2s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestMultiParallel |
| | hadoop.hbase.client.TestHCM |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.TestReplicationStateHBaseImpl |
| | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.master.TestDistributedLogSplitting |
| | org.apache.hadoop.hbase.master.TestMasterShutdown |
| |
org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController
|
| | org.apache.hadoop.hbase.TestIOFencing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828188/HBASE-16610.v1.patch |
| JIRA Issue | HBASE-16610 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4ce474996ffe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_6
[jira] [Commented] (HBASE-16594) ROW_INDEX_V2 DBE
[ https://issues.apache.org/jira/browse/HBASE-16594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486789#comment-15486789 ] Yu Li commented on HBASE-16594: --- +1 on the idea, but I think it might be better to supply data of EncodedSeekPerformanceTest and E2E testing just like you did in V1, rather than using a special case. Wdyt? [~aoxiang] Let's also wait for others' thoughts. > ROW_INDEX_V2 DBE > > > Key: HBASE-16594 > URL: https://issues.apache.org/jira/browse/HBASE-16594 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16594-master_v1.patch, HBASE-16594-master_v2.patch > > > See HBASE-16213, ROW_INDEX_V1 DataBlockEncoding. > ROW_INDEX_V1 is the first version which have no storage optimization, > ROW_INDEX_V2 do storage optimization: store every row only once, store column > family only once in a HFileBlock. > ROW_INDEX_V1 is : > /** > * Store cells following every row's start offset, so we can binary search to > a row's cells. > * > * Format: > * flat cells > * integer: number of rows > * integer: row0's offset > * integer: row1's offset > * > * integer: dataSize > * > */ > ROW_INDEX_V2 is : > * row1 qualifier timestamp type value tag > * qualifier timestamp type value tag > * qualifier timestamp type value tag > * row2 qualifier timestamp type value tag > * row3 qualifier timestamp type value tag > * qualifier timestamp type value tag > * > * integer: number of rows > * integer: row0's offset > * integer: row1's offset > * > * column family > * integer: dataSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16625) Performance test for interface unified with AP
Heng Chen created HBASE-16625: - Summary: Performance test for interface unified with AP Key: HBASE-16625 URL: https://issues.apache.org/jira/browse/HBASE-16625 Project: HBase Issue Type: Sub-task Reporter: Heng Chen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16625) Performance test for interface unified with AP
[ https://issues.apache.org/jira/browse/HBASE-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16625: -- Description: Due to one thread context changed in AP, the performance should be tested. Let me do it in this issue. > Performance test for interface unified with AP > -- > > Key: HBASE-16625 > URL: https://issues.apache.org/jira/browse/HBASE-16625 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen > > Due to one thread context changed in AP, the performance should be tested. > Let me do it in this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16610) Unify append, increment with AP
[ https://issues.apache.org/jira/browse/HBASE-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486855#comment-15486855 ] Heng Chen commented on HBASE-16610: --- see HBASE-16625 > Unify append, increment with AP > --- > > Key: HBASE-16610 > URL: https://issues.apache.org/jira/browse/HBASE-16610 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16610.patch, HBASE-16610.v1.patch, > HBASE-16610.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16625) Performance test for interface unified with AP
[ https://issues.apache.org/jira/browse/HBASE-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16625: -- Description: Due to one more thread context changed in AP, the performance should be tested. Let me do it in this issue. (was: Due to one thread context changed in AP, the performance should be tested. Let me do it in this issue.) > Performance test for interface unified with AP > -- > > Key: HBASE-16625 > URL: https://issues.apache.org/jira/browse/HBASE-16625 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen > > Due to one more thread context changed in AP, the performance should be > tested. Let me do it in this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486867#comment-15486867
]
Phil Yang commented on HBASE-15968:
---
After reading code introduced by HBASE-7663 and HBASE-10885, if I am not wrong,
visibility labels will not change the logic of versions. If the maxVersion is
3, we can only read first three versions and check if it can be seen according
to tags. For delete, we can set tags for it and it only mask Put with same
tags(no tag is also a kind of tag).
For mvcc-sensitive semantics, we have VERSION_MASKED which means this Put is
masked by enough number of versions and can never be read. So for
mvcc-sensitive with visibility labels. If we have 4 Puts whose tags are a, a,
b, a. The first one will never be read no mater what labels the reader has. But
if we delete the third Put after we put it. The order is Put(a), Put(a),
Put(b), Delete(b), Put(a). What should be the result of reader if its label is
a? The latest two Put(a), or all three of them?
[~anoop.hbase][~ram_krish][~Apache9] What do you think? Thanks.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
Th
[jira] [Comment Edited] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486867#comment-15486867
]
Phil Yang edited comment on HBASE-15968 at 9/13/16 10:24 AM:
-
After reading code introduced by HBASE-7663 and HBASE-10885, if I am not wrong,
visibility labels will not change the logic of versions. If the maxVersion is
3, we can only read first three versions and check if it can be seen according
to tags. For delete, we can set tags for it and it only mask Put with same
tags(no tag is also a kind of tag).
For mvcc-sensitive semantics, we have VERSION_MASKED which means this Put is
masked by enough number of versions and can never be read. So for
mvcc-sensitive with visibility labels. If we have 4 Puts whose tags are a, a,
b, a. The first one will never be read no mater what labels the reader has. But
if we delete the third Put after we put it. The order is Put(a), Put(a),
Put(b), Delete(b), Put(a). What should be the result of reader if its label is
a? The latest two Put(a) because we can not see Delete(b), or all three of them
because the Put(b) has been deleted although we can not see them?
[~anoop.hbase][~ram_krish][~Apache9] What do you think? Thanks.
was (Author: yangzhe1991):
After reading code introduced by HBASE-7663 and HBASE-10885, if I am not wrong,
visibility labels will not change the logic of versions. If the maxVersion is
3, we can only read first three versions and check if it can be seen according
to tags. For delete, we can set tags for it and it only mask Put with same
tags(no tag is also a kind of tag).
For mvcc-sensitive semantics, we have VERSION_MASKED which means this Put is
masked by enough number of versions and can never be read. So for
mvcc-sensitive with visibility labels. If we have 4 Puts whose tags are a, a,
b, a. The first one will never be read no mater what labels the reader has. But
if we delete the third Put after we put it. The order is Put(a), Put(a),
Put(b), Delete(b), Put(a). What should be the result of reader if its label is
a? The latest two Put(a), or all three of them?
[~anoop.hbase][~ram_krish][~Apache9] What do you think? Thanks.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t
[jira] [Assigned] (HBASE-16625) Performance test for interface unified with AP
[ https://issues.apache.org/jira/browse/HBASE-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen reassigned HBASE-16625: - Assignee: Heng Chen > Performance test for interface unified with AP > -- > > Key: HBASE-16625 > URL: https://issues.apache.org/jira/browse/HBASE-16625 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > > Due to one more thread context changed in AP, the performance should be > tested. Let me do it in this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16615) Fix flaky TestScannerHeartbeatMessages
[ https://issues.apache.org/jira/browse/HBASE-16615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16615: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~stack] for reviewing. > Fix flaky TestScannerHeartbeatMessages > -- > > Key: HBASE-16615 > URL: https://issues.apache.org/jira/browse/HBASE-16615 > Project: HBase > Issue Type: Bug > Components: Client, Scanners >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16615.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486867#comment-15486867
]
Phil Yang edited comment on HBASE-15968 at 9/13/16 10:41 AM:
-
After reading code introduced by HBASE-7663 and HBASE-10885, if I am not wrong,
visibility labels will not change the logic of versions. If the maxVersion is
3, we can only read first three versions and check if it can be seen according
to tags. For delete, we can set tags for it and it only mask Put with same
tags(no tag is also a kind of tag). For example, if we have Put(b), Put(b),
Put(a), Delete(a), Put(b), Put(a), for reader(a) we can read the lated Put(a)
and for reader(b) we can read two Puts and the oldest one is masked, right?
For mvcc-sensitive semantics, we have VERSION_MASKED which means this Put is
masked by enough number of versions and can never be read. So for
mvcc-sensitive with visibility labels. If the order is Put(b), Put(b), Put(a),
Delete(a), Put(b), Put(a). What should be the result of reader(b)? The latest
two Put(b) because the third put is deleted so the second will not be masked,
or only the latest Put(b) because we can not see the Delete(a)?
[~anoop.hbase] [~ram_krish] [~Apache9] What do you think? Thanks.
was (Author: yangzhe1991):
After reading code introduced by HBASE-7663 and HBASE-10885, if I am not wrong,
visibility labels will not change the logic of versions. If the maxVersion is
3, we can only read first three versions and check if it can be seen according
to tags. For delete, we can set tags for it and it only mask Put with same
tags(no tag is also a kind of tag). For example, if we have Put(b), Put(b),
Put(a), Delete(a), Put(b), Put(a), for reader(a) we can read the lated Put(a)
and for reader(b) we can read two Puts and the oldest one is masked, right?
For mvcc-sensitive semantics, we have VERSION_MASKED which means this Put is
masked by enough number of versions and can never be read. So for
mvcc-sensitive with visibility labels. If the order is Put(b), Put(b), Put(a),
Delete(a), Put(b), Put(a). What should be the result of reader(b)? The latest
two Put(b) because the third put is deleted so the second will not be masked,
or only the latest Put(b) because we can not see the Delete(a)?
[~anoop.hbase][~ram_krish][~Apache9] What do you think? Thanks.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version who
[jira] [Comment Edited] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486867#comment-15486867
]
Phil Yang edited comment on HBASE-15968 at 9/13/16 10:41 AM:
-
After reading code introduced by HBASE-7663 and HBASE-10885, if I am not wrong,
visibility labels will not change the logic of versions. If the maxVersion is
3, we can only read first three versions and check if it can be seen according
to tags. For delete, we can set tags for it and it only mask Put with same
tags(no tag is also a kind of tag). For example, if we have Put(b), Put(b),
Put(a), Delete(a), Put(b), Put(a), for reader(a) we can read the lated Put(a)
and for reader(b) we can read two Puts and the oldest one is masked, right?
For mvcc-sensitive semantics, we have VERSION_MASKED which means this Put is
masked by enough number of versions and can never be read. So for
mvcc-sensitive with visibility labels. If the order is Put(b), Put(b), Put(a),
Delete(a), Put(b), Put(a). What should be the result of reader(b)? The latest
two Put(b) because the third put is deleted so the second will not be masked,
or only the latest Put(b) because we can not see the Delete(a)?
[~anoop.hbase][~ram_krish][~Apache9] What do you think? Thanks.
was (Author: yangzhe1991):
After reading code introduced by HBASE-7663 and HBASE-10885, if I am not wrong,
visibility labels will not change the logic of versions. If the maxVersion is
3, we can only read first three versions and check if it can be seen according
to tags. For delete, we can set tags for it and it only mask Put with same
tags(no tag is also a kind of tag).
For mvcc-sensitive semantics, we have VERSION_MASKED which means this Put is
masked by enough number of versions and can never be read. So for
mvcc-sensitive with visibility labels. If we have 4 Puts whose tags are a, a,
b, a. The first one will never be read no mater what labels the reader has. But
if we delete the third Put after we put it. The order is Put(a), Put(a),
Put(b), Delete(b), Put(a). What should be the result of reader if its label is
a? The latest two Put(a) because we can not see Delete(b), or all three of them
because the Put(b) has been deleted although we can not see them?
[~anoop.hbase][~ram_krish][~Apache9] What do you think? Thanks.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> de
[jira] [Issue Comment Deleted] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Comment: was deleted
(was: After reading code introduced by HBASE-7663 and HBASE-10885, if I am not
wrong, visibility labels will not change the logic of versions. If the
maxVersion is 3, we can only read first three versions and check if it can be
seen according to tags. For delete, we can set tags for it and it only mask Put
with same tags(no tag is also a kind of tag). For example, if we have Put(b),
Put(b), Put(a), Delete(a), Put(b), Put(a), for reader(a) we can read the lated
Put(a) and for reader(b) we can read two Puts and the oldest one is masked,
right?
For mvcc-sensitive semantics, we have VERSION_MASKED which means this Put is
masked by enough number of versions and can never be read. So for
mvcc-sensitive with visibility labels. If the order is Put(b), Put(b), Put(a),
Delete(a), Put(b), Put(a). What should be the result of reader(b)? The latest
two Put(b) because the third put is deleted so the second will not be masked,
or only the latest Put(b) because we can not see the Delete(a)?
[~anoop.hbase] [~ram_krish] [~Apache9] What do you think? Thanks.)
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486939#comment-15486939
]
Phil Yang commented on HBASE-15968:
---
(delete the previous comments which is not a question now..)
For visibility labels I think we have to keep all labels in
MvccSensitiveTracker. Label is like timestamp that we should consider if the
previous passed delete marker can mask this Put.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15297) error message is wrong when a wrong namspace is specified in grant in hbase shell
[
https://issues.apache.org/jira/browse/HBASE-15297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486960#comment-15486960
]
Hudson commented on HBASE-15297:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #15 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/15/])
HBASE-15297 Correct handling of namespace existence checks in shell. (busbey:
rev 05d8b248f81440d9974613b1a6e5abaa4cbc2f16)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* (edit) hbase-shell/src/main/ruby/hbase/security.rb
> error message is wrong when a wrong namspace is specified in grant in hbase
> shell
> -
>
> Key: HBASE-15297
> URL: https://issues.apache.org/jira/browse/HBASE-15297
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Xiang Li
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-15297.v1.patch
>
>
> In HBase shell, specify a non-existing namespace in "grant" command, such as
> {code}
> hbase(main):001:0> grant 'a1', 'R', '@aaa'<--- there is no namespace
> called "aaa"
> {code}
> The error message issued is not correct
> {code}
> ERROR: Unknown namespace a1!
> {code}
> a1 is the user name, not the namespace.
> The following error message would be better
> {code}
> ERROR: Unknown namespace aaa!
> {code}
> or
> {code}
> Can't find a namespace: aaa
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16626) User customized RegionScanner from 1.X is incompatible in 2.0.0's off heap part
Charlie Qiangeng Xu created HBASE-16626: --- Summary: User customized RegionScanner from 1.X is incompatible in 2.0.0's off heap part Key: HBASE-16626 URL: https://issues.apache.org/jira/browse/HBASE-16626 Project: HBase Issue Type: Sub-task Components: Offheaping Affects Versions: 1.1.6, 1.2.2 Reporter: Charlie Qiangeng Xu Fix For: 2.0.0 Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a new interface Shipper which contains a "shipped()" method. In our case, some application user defined customized scanner that implements Interface RegionScanner in 1.X . After we back ported the Off Heap feature of 2.0.0, RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when executing scanner.shipped(); Instead of forcing every one of our users to add a empty implementation(if they don't really need to scan the file or the RS don't use L2 cache) , adding a default method of shipped might be a better way.(since HBASE-15624 we decide to use JAVA8 only in 2.0.0) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16626) User customized RegionScanner from 1.X is incompatible in 2.0.0's off heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charlie Qiangeng Xu updated HBASE-16626: Remaining Estimate: (was: 48h) Original Estimate: (was: 48h) > User customized RegionScanner from 1.X is incompatible in 2.0.0's off heap > part > --- > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that > implements Interface RegionScanner in 1.X . After we back ported the Off Heap > feature of 2.0.0, RegionScannerShippedCallBack throws a > "java.lang.AbstractMethodError" when executing scanner.shipped(); > Instead of forcing every one of our users to add a empty implementation(if > they don't really need to scan the file or the RS don't use L2 cache) , > adding a default method of shipped might be a better way.(since HBASE-15624 > we decide to use JAVA8 only in 2.0.0) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16626) User customized RegionScanner from 1.X is incompatible in 2.0.0's off heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charlie Qiangeng Xu updated HBASE-16626: Description: Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a new interface Shipper which contains a "shipped()" method. In our case, some application user defined customized scanner that implements Interface RegionScanner in 1.X . After we back ported the Off Heap feature of 2.0.0, RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when executing scanner.shipped(). It is because the customized scanner didn't override the shipped method yet. Instead of forcing every user to add a empty implementation(if they don't really need to scan the file or the RS don't use L2 cache, they don't need to do anything in shipped method) , adding a default method of shipped in Interface RegionScanner might be a better way.(since HBASE-15624 we decide to use JAVA8 only in 2.0.0) was: Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a new interface Shipper which contains a "shipped()" method. In our case, some application user defined customized scanner that implements Interface RegionScanner in 1.X . After we back ported the Off Heap feature of 2.0.0, RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when executing scanner.shipped(); Instead of forcing every one of our users to add a empty implementation(if they don't really need to scan the file or the RS don't use L2 cache) , adding a default method of shipped might be a better way.(since HBASE-15624 we decide to use JAVA8 only in 2.0.0) > User customized RegionScanner from 1.X is incompatible in 2.0.0's off heap > part > --- > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that implements > Interface RegionScanner in 1.X . > After we back ported the Off Heap feature of 2.0.0, > RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when > executing scanner.shipped(). It is because the customized scanner didn't > override the shipped method yet. > Instead of forcing every user to add a empty implementation(if they don't > really need to scan the file or the RS don't use L2 cache, they don't need to > do anything in shipped method) , adding a default method of shipped in > Interface RegionScanner might be a better way.(since HBASE-15624 we decide > to use JAVA8 only in 2.0.0) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charlie Qiangeng Xu updated HBASE-16626: Summary: User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part (was: User customized RegionScanner from 1.X is incompatible in 2.0.0's off heap part) > User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap > part > - > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that implements > Interface RegionScanner in 1.X . > After we back ported the Off Heap feature of 2.0.0, > RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when > executing scanner.shipped(). It is because the customized scanner didn't > override the shipped method yet. > Instead of forcing every user to add a empty implementation(if they don't > really need to scan the file or the RS don't use L2 cache, they don't need to > do anything in shipped method) , adding a default method of shipped in > Interface RegionScanner might be a better way.(since HBASE-15624 we decide > to use JAVA8 only in 2.0.0) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charlie Qiangeng Xu updated HBASE-16626: Description: Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a new interface Shipper which contains a "shipped()" method. In our case, some application user defined customized scanner that implements Interface RegionScanner in 1.X . After we back ported the Off Heap feature of 2.0.0, RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when executing scanner.shipped(). It is because the customized scanner didn't override the shipped method yet. Instead of forcing every user to add a empty implementation(if they don't really need to scan the file or the RS don't use L2 cache, they don't need to do anything in shipped method) , adding a default method of shipped in Interface RegionScanner might be a better way. was: Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a new interface Shipper which contains a "shipped()" method. In our case, some application user defined customized scanner that implements Interface RegionScanner in 1.X . After we back ported the Off Heap feature of 2.0.0, RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when executing scanner.shipped(). It is because the customized scanner didn't override the shipped method yet. Instead of forcing every user to add a empty implementation(if they don't really need to scan the file or the RS don't use L2 cache, they don't need to do anything in shipped method) , adding a default method of shipped in Interface RegionScanner might be a better way.(since HBASE-15624 we decide to use JAVA8 only in 2.0.0) > User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap > part > - > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that implements > Interface RegionScanner in 1.X . > After we back ported the Off Heap feature of 2.0.0, > RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when > executing scanner.shipped(). It is because the customized scanner didn't > override the shipped method yet. > Instead of forcing every user to add a empty implementation(if they don't > really need to scan the file or the RS don't use L2 cache, they don't need to > do anything in shipped method) , adding a default method of shipped in > Interface RegionScanner might be a better way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15297) error message is wrong when a wrong namspace is specified in grant in hbase shell
[
https://issues.apache.org/jira/browse/HBASE-15297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487030#comment-15487030
]
Hudson commented on HBASE-15297:
FAILURE: Integrated in Jenkins build HBase-1.4 #411 (See
[https://builds.apache.org/job/HBase-1.4/411/])
HBASE-15297 Correct handling of namespace existence checks in shell. (busbey:
rev 059a169d3a22cc2788ff0f4ea46dd03015338cfc)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* (edit) hbase-shell/src/main/ruby/hbase/security.rb
> error message is wrong when a wrong namspace is specified in grant in hbase
> shell
> -
>
> Key: HBASE-15297
> URL: https://issues.apache.org/jira/browse/HBASE-15297
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Xiang Li
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-15297.v1.patch
>
>
> In HBase shell, specify a non-existing namespace in "grant" command, such as
> {code}
> hbase(main):001:0> grant 'a1', 'R', '@aaa'<--- there is no namespace
> called "aaa"
> {code}
> The error message issued is not correct
> {code}
> ERROR: Unknown namespace a1!
> {code}
> a1 is the user name, not the namespace.
> The following error message would be better
> {code}
> ERROR: Unknown namespace aaa!
> {code}
> or
> {code}
> Can't find a namespace: aaa
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16611) Flakey org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet
[
https://issues.apache.org/jira/browse/HBASE-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487060#comment-15487060
]
Hudson commented on HBASE-16611:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1593 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1593/])
HBASE-16611 Flakey (chenheng: rev cd9f42237344756a7db395bd8241f41b00e359a2)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Flakey org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet
> -
>
> Key: HBASE-16611
> URL: https://issues.apache.org/jira/browse/HBASE-16611
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-16611.patch, HBASE-16611.v1.patch,
> HBASE-16611.v1.patch, HBASE-16611.v2.patch
>
>
> see
> https://builds.apache.org/job/PreCommit-HBASE-Build/3494/artifact/patchprocess/patch-unit-hbase-server.txt
> {code}
> testCancelOfMultiGet(org.apache.hadoop.hbase.client.TestReplicasClient) Time
> elapsed: 4.026 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet(TestReplicasClient.java:579)
> Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 94.401 sec -
> in org.apache.hadoop.hbase.client.TestAdmin2
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.861 sec -
> in org.apache.hadoop.hbase.client.TestClientScannerRPCTimeout
> Running
> org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 261.925 sec
> <<< FAILURE! - in org.apache.hadoop.hbase.client.TestReplicasClient
> testCancelOfMultiGet(org.apache.hadoop.hbase.client.TestReplicasClient) Time
> elapsed: 4.522 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet(TestReplicasClient.java:581)
> Running org.apache.hadoop.hbase.client.TestFastFail
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.648 sec -
> in org.apache.hadoop.hbase.client.TestFastFail
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 277.894 sec
> <<< FAILURE! - in org.apache.hadoop.hbase.client.TestReplicasClient
> testCancelOfMultiGet(org.apache.hadoop.hbase.client.TestReplicasClient) Time
> elapsed: 5.359 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.client.TestReplicasClient.testCancelOfMultiGet(TestReplicasClient.java:579)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16592) Unify Delete request with AP
[ https://issues.apache.org/jira/browse/HBASE-16592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487058#comment-15487058 ] Hudson commented on HBASE-16592: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1593 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1593/]) HBASE-16592 Unify Delete request with AP (garyh: rev 2566cfeb60de644f287ac192d360f3fc15376c8f) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/SingleResponse.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractResponse.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java > Unify Delete request with AP > > > Key: HBASE-16592 > URL: https://issues.apache.org/jira/browse/HBASE-16592 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16592.patch, HBASE-16592.v1.patch, > HBASE-16592.v1.patch, HBASE-16592.v2.patch, HBASE-16592.v3.patch > > > This is the first step try to unify the HTable with AP only, to extend AP > could process single action, i introduced AbstractResponse, multiResponse > and singleResponse (introduced to deal with single result) will extend this > class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16229) Cleaning up size and heapSize calculation
[
https://issues.apache.org/jira/browse/HBASE-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487059#comment-15487059
]
Hudson commented on HBASE-16229:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1593 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1593/])
HBASE-16229 Cleaning up size and heapSize calculation. (garyh: rev
77b327320a72ca01b35f655c42f8c13f659dff31)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MutableSegment.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMemStore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellArrayMapMemStore.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/ClassSize.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactor.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPerColumnFamilyFlush.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentFactory.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWalAndCompactingMemStoreFlush.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
> Cleaning up size and heapSize calculation
> -
>
> Key: HBASE-16229
> URL: https://issues.apache.org/jira/browse/HBASE-16229
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16229.patch, HBASE-16229_V2.patch,
> HBASE-16229_V3.patch, HBASE-16229_V4.patch, HBASE-16229_V5.patch,
> HBASE-16229_V5.patch, HBASE-16229_V6.patch
>
>
> It is bit ugly now. For eg:
> AbstractMemStore
> {code}
> public final static long FIXED_OVERHEAD = ClassSize.align(
> ClassSize.OBJECT +
> (4 * ClassSize.REFERENCE) +
> (2 * Bytes.SIZEOF_LONG));
> public final static long DEEP_OVERHEAD = ClassSize.align(FIXED_OVERHEAD +
> (ClassSize.ATOMIC_LONG + ClassSize.TIMERANGE_TRACKER +
> ClassSize.CELL_SKIPLIST_SET + ClassSize.CONCURRENT_SKIPLISTMAP));
> {code}
> We include the heap overhead of Segment also here. It will be better the
> Segment contains its overhead part and the Memstore impl uses the heap size
> of all of its segments to calculate its size.
> Also this
> {code}
> public long heapSize() {
> return getActive().getSize();
> }
> {code}
> HeapSize to consider all segment's size not just active's. I am not able to
> see an override method in CompactingMemstore.
> This jira tries to solve some of these.
> When we create a Segment, we seems pass some initial heap size value to it.
> Why? The segment object internally has to know what is its heap size not
> like some one else dictate it.
> More to add when doing this cleanup
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15297) error message is wrong when a wrong namspace is specified in grant in hbase shell
[
https://issues.apache.org/jira/browse/HBASE-15297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487061#comment-15487061
]
Hudson commented on HBASE-15297:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1593 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1593/])
HBASE-15297 Correct handling of namespace existence checks in shell. (busbey:
rev 422734e73d8846e4a357178cf665220d689e2e6e)
* (edit) hbase-shell/src/main/ruby/hbase/security.rb
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
> error message is wrong when a wrong namspace is specified in grant in hbase
> shell
> -
>
> Key: HBASE-15297
> URL: https://issues.apache.org/jira/browse/HBASE-15297
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Xiang Li
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-15297.v1.patch
>
>
> In HBase shell, specify a non-existing namespace in "grant" command, such as
> {code}
> hbase(main):001:0> grant 'a1', 'R', '@aaa'<--- there is no namespace
> called "aaa"
> {code}
> The error message issued is not correct
> {code}
> ERROR: Unknown namespace a1!
> {code}
> a1 is the user name, not the namespace.
> The following error message would be better
> {code}
> ERROR: Unknown namespace aaa!
> {code}
> or
> {code}
> Can't find a namespace: aaa
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16540) Scan should do additional validation on start and stop row
[ https://issues.apache.org/jira/browse/HBASE-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487062#comment-15487062 ] Hudson commented on HBASE-16540: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1593 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1593/]) HBASE-16540 Adding checks in Scanner's setStartRow and setStopRow for (garyh: rev c57acf28e7cabcfcbce8ae0006080088cdc47f50) * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestScan.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java > Scan should do additional validation on start and stop row > -- > > Key: HBASE-16540 > URL: https://issues.apache.org/jira/browse/HBASE-16540 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Dustin Pho > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16540.002.patch, HBASE-16540.patch > > > Scan.setStartRow() and setStopRow() should validate the byte[] passed to > ensure it meets the criteria for a row key. If the byte[] length is greater > that Short.MAX_VALUE, we should throw an IllegalArgumentException in order to > fast fail and prevent server-side errors being thrown and retried. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16447) Replication by namespaces config in peer
[ https://issues.apache.org/jira/browse/HBASE-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-16447: --- Attachment: HBASE-16447-v5.patch Rebase master. > Replication by namespaces config in peer > > > Key: HBASE-16447 > URL: https://issues.apache.org/jira/browse/HBASE-16447 > Project: HBase > Issue Type: New Feature > Components: Replication >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-16447-v1.patch, HBASE-16447-v2.patch, > HBASE-16447-v3.patch, HBASE-16447-v4.patch, HBASE-16447-v5.patch > > > Now we only config table cfs in peer. But in our production cluster, there > are a dozen of namespace and every namespace has dozens of tables. It was > complicated to config all table cfs in peer. For some namespace, it need > replication all tables to other slave cluster. It will be easy to config if > we support replication by namespace. Suggestions and discussions are welcomed. > Review board: https://reviews.apache.org/r/51521/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15297) error message is wrong when a wrong namspace is specified in grant in hbase shell
[
https://issues.apache.org/jira/browse/HBASE-15297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-15297:
Resolution: Fixed
Hadoop Flags: Incompatible change
Fix Version/s: (was: 1.2.4)
Release Note:
The security admin instance available within the HBase shell now returns
"false" from the namespace_exists? method for non-existent namespaces rather
than raising a wrapped NamespaceNotFoundException.
As a side effect, when the "grant" and "revoke" commands in the HBase shell are
invoked with a non-existent namespace the resulting error message now properly
refers to said namespace rather than to the user.
Status: Resolved (was: Patch Available)
I only brought this back to the branches for 1.3+, because the
{{namespace_exists?}} method is exposed in the shell. That means altering it
breaks operational compatibility, which we don't do in maintenance releases.
here's an example of the behavior before the change:
{code}
hbase(main):001:0> @shell.hbase_security_admin.namespace_exists? "hbase"
=> true
hbase(main):002:0> @shell.hbase_security_admin.namespace_exists? "foobar"
NativeException: org.apache.hadoop.hbase.NamespaceNotFoundException:
org.apache.hadoop.hbase.NamespaceNotFoundException: foobar
at
org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:2549)
at
org.apache.hadoop.hbase.master.MasterRpcServices.getNamespaceDescriptor(MasterRpcServices.java:817)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55732)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
{code}
Now, naturally, the second call returns a proper {{false}} which will break any
downstream folks expecting an exception. Let me know if the release note needs
more details to properly warn folks who upgrade.
> error message is wrong when a wrong namspace is specified in grant in hbase
> shell
> -
>
> Key: HBASE-15297
> URL: https://issues.apache.org/jira/browse/HBASE-15297
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Xiang Li
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15297.v1.patch
>
>
> In HBase shell, specify a non-existing namespace in "grant" command, such as
> {code}
> hbase(main):001:0> grant 'a1', 'R', '@aaa'<--- there is no namespace
> called "aaa"
> {code}
> The error message issued is not correct
> {code}
> ERROR: Unknown namespace a1!
> {code}
> a1 is the user name, not the namespace.
> The following error message would be better
> {code}
> ERROR: Unknown namespace aaa!
> {code}
> or
> {code}
> Can't find a namespace: aaa
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487161#comment-15487161 ] stack commented on HBASE-16388: --- Ok. Expert config. If you are seeing SBE, then you have set hbase.client.perserver.requests.threshold down from its default value so you know about it. The patch has rotted because of HBASE-16445 "Refactor and reimplement RpcClient" . Please redo, change ServerBusyException to ServerTooBusyException just so it has same form as RegionTooBusyException and I'm +. Thanks [~yangzhe1991] > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16388-branch-1-v1.patch, HBASE-16388-v1.patch, > HBASE-16388-v2.patch, HBASE-16388-v2.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15297) error message is wrong when a wrong namspace is specified in grant in hbase shell
[
https://issues.apache.org/jira/browse/HBASE-15297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487223#comment-15487223
]
Hudson commented on HBASE-15297:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #15 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/15/])
HBASE-15297 Correct handling of namespace existence checks in shell. (busbey:
rev 05d8b248f81440d9974613b1a6e5abaa4cbc2f16)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
* (edit) hbase-shell/src/main/ruby/hbase/security.rb
> error message is wrong when a wrong namspace is specified in grant in hbase
> shell
> -
>
> Key: HBASE-15297
> URL: https://issues.apache.org/jira/browse/HBASE-15297
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Xiang Li
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15297.v1.patch
>
>
> In HBase shell, specify a non-existing namespace in "grant" command, such as
> {code}
> hbase(main):001:0> grant 'a1', 'R', '@aaa'<--- there is no namespace
> called "aaa"
> {code}
> The error message issued is not correct
> {code}
> ERROR: Unknown namespace a1!
> {code}
> a1 is the user name, not the namespace.
> The following error message would be better
> {code}
> ERROR: Unknown namespace aaa!
> {code}
> or
> {code}
> Can't find a namespace: aaa
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487230#comment-15487230
]
stack commented on HBASE-15968:
---
This 'fixed' behavior should be default in 2.0.0. Rather than call the config
'MVCC_SENSITIVE', the config should be inverted. An operator should explicitly
ask for the OLD_BROKEN_DELETE_BEHAVIOR (smile). They way you have it makes the
patch awkward (Not your fault. You are trying to do the minimal disruption
which is good usually, but here you are addressing a long-standing, ugly bug).
Yeah, its an outstanding question as to when it is safe to set sequenceid/mvcc
== 0. We had an heuristic of N days after which it was thought there could be
no possible outstanding Scanner so then it was safe to remove consideration of
sequenceid from the Cell.
I scanned the patch. Is that all it takes to fix this long-standing issue? That
is great. Will do a closer review soon.
This is for new tables only? I cannot turn this on on an existing table? It
will change results? Might be fine by some operators.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message w
[jira] [Commented] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487246#comment-15487246 ] ramkrishna.s.vasudevan commented on HBASE-16414: [~apurtell] Your feedback/review would be much appreicated on the updated patch from [~colin_mjj]. A gentle reminder. > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch, > HBASE-16414.003.patch, HBASE-16414.004.patch, > HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487266#comment-15487266
]
stack commented on HBASE-15968:
---
mvcc-sensitive is not a good name because the whole system is already mvcc
sensitive. Not joking, the flag should be inverted and be called
oldBrokenDeleteBehavior rather than pass and check mvccSensitive.
CompoundTracker for the new Column+Delete combined Tracker... or
CombinedTracker rather than MvccSensitive...
With this patch, when do we now set mvcc/sequenceid to zero, if at all?
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487269#comment-15487269
]
stack commented on HBASE-15968:
---
Oh, finally, this work is great.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487290#comment-15487290 ] Anoop Sam John commented on HBASE-16626: bq.adding a default method of shipped in Interface RegionScanner might be a better way. +1 > User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap > part > - > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that implements > Interface RegionScanner in 1.X . > After we back ported the Off Heap feature of 2.0.0, > RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when > executing scanner.shipped(). It is because the customized scanner didn't > override the shipped method yet. > Instead of forcing every user to add a empty implementation(if they don't > really need to scan the file or the RS don't use L2 cache, they don't need to > do anything in shipped method) , adding a default method of shipped in > Interface RegionScanner might be a better way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15570) renewable delegation tokens for long-lived spark applications
[ https://issues.apache.org/jira/browse/HBASE-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487438#comment-15487438 ] Sean Busbey commented on HBASE-15570: - Maybe superceded by SPARK-14743? > renewable delegation tokens for long-lived spark applications > - > > Key: HBASE-15570 > URL: https://issues.apache.org/jira/browse/HBASE-15570 > Project: HBase > Issue Type: Improvement > Components: spark >Reporter: Sean Busbey >Assignee: Sean Busbey > > Right now our spark integration works on secure clusters by getting > delegation tokens and sending them to the executors. Unfortunately, > applications that need to run for longer than the delegation token lifetime > (by default 7 days) will fail. > In particular, this is an issue for Spark Streaming applications. Since they > expect to run indefinitely, we should have a means for renewing the > delegation tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16615) Fix flaky TestScannerHeartbeatMessages
[ https://issues.apache.org/jira/browse/HBASE-16615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487499#comment-15487499 ] Hudson commented on HBASE-16615: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1594 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1594/]) HBASE-16615 Fix flaky TestScannerHeartbeatMessages (zhangduo: rev a602aaf9baae779ac654fcb0fcedfdc9f8acc6ce) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScannerHeartbeatMessages.java > Fix flaky TestScannerHeartbeatMessages > -- > > Key: HBASE-16615 > URL: https://issues.apache.org/jira/browse/HBASE-16615 > Project: HBase > Issue Type: Bug > Components: Client, Scanners >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16615.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16447) Replication by namespaces config in peer
[
https://issues.apache.org/jira/browse/HBASE-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487538#comment-15487538
]
Hadoop QA commented on HBASE-16447:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 1s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 1s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 8s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 50s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 28s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 165m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService
|
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithCustomVisLabService
|
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828215/HBASE-16447-v5.patch |
| JIRA Issue | HBA
[jira] [Commented] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487589#comment-15487589 ] stack commented on HBASE-16626: --- Yes. Master is jdk8 so you could make use of the new language feature in master. What will you do for branch 1.x? > User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap > part > - > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that implements > Interface RegionScanner in 1.X . > After we back ported the Off Heap feature of 2.0.0, > RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when > executing scanner.shipped(). It is because the customized scanner didn't > override the shipped method yet. > Instead of forcing every user to add a empty implementation(if they don't > really need to scan the file or the RS don't use L2 cache, they don't need to > do anything in shipped method) , adding a default method of shipped in > Interface RegionScanner might be a better way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16624) MVCC DeSerialization bug in the HFileScannerImpl
[ https://issues.apache.org/jira/browse/HBASE-16624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487595#comment-15487595 ] stack commented on HBASE-16624: --- I think this a good supposition but method looks too big to inline... > MVCC DeSerialization bug in the HFileScannerImpl > > > Key: HBASE-16624 > URL: https://issues.apache.org/jira/browse/HBASE-16624 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Blocker > Attachments: HBASE-16624.patch > > > My colleague [~naggarwal] found a bug in the deserialization of mvcc from > HFile, As a part of the optimization of deserialization of VLong, we read a > int at once but we forgot to convert it to unsigned one. > This would cause issues because once we cross the integer threshold in > sequenceId and a compaction happens we would write MAX_MEMSTORE_TS in the > trailer as 0 (because we will be reading negative values from the file that > got flushed with sequenceId > Integer.MAX_VALUE). And once we have > MAX_MEMSTORE_TS as 0, and there are sequenceId values present alongside with > KeyValues the regionserver will now start failing to read the compacted file > and thus corruption. > Interestingly this would happen only on the tables that don't have > DataBlockEncoding enabled and unfortunately in our case that turned out to be > META and a another small table. > Fix is small (~20 chars) and attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16624) MVCC DeSerialization bug in the HFileScannerImpl
[ https://issues.apache.org/jira/browse/HBASE-16624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487602#comment-15487602 ] stack commented on HBASE-16624: --- +1 on the fix. Good find. Agree a test would be good but rather, we should have a bigger project that looks for all places where we do this sort of dodgy convertion ...I'd think this sort of practice, from int to long, would flag in findbugs or other static analysis tools. > MVCC DeSerialization bug in the HFileScannerImpl > > > Key: HBASE-16624 > URL: https://issues.apache.org/jira/browse/HBASE-16624 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Blocker > Attachments: HBASE-16624.patch > > > My colleague [~naggarwal] found a bug in the deserialization of mvcc from > HFile, As a part of the optimization of deserialization of VLong, we read a > int at once but we forgot to convert it to unsigned one. > This would cause issues because once we cross the integer threshold in > sequenceId and a compaction happens we would write MAX_MEMSTORE_TS in the > trailer as 0 (because we will be reading negative values from the file that > got flushed with sequenceId > Integer.MAX_VALUE). And once we have > MAX_MEMSTORE_TS as 0, and there are sequenceId values present alongside with > KeyValues the regionserver will now start failing to read the compacted file > and thus corruption. > Interestingly this would happen only on the tables that don't have > DataBlockEncoding enabled and unfortunately in our case that turned out to be > META and a another small table. > Fix is small (~20 chars) and attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16613) Return the unused ByteBuffer to BoundedByteBufferPool when no cell is got from the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487608#comment-15487608
]
Ted Yu commented on HBASE-16613:
+1
> Return the unused ByteBuffer to BoundedByteBufferPool when no cell is got
> from the CellScanner
> --
>
> Key: HBASE-16613
> URL: https://issues.apache.org/jira/browse/HBASE-16613
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16613.branch-1.v0.patch,
> HBASE-16613.branch-1.v1.patch, HBASE-16613.branch-1.v1.patch
>
>
> The critical code is shown below:
> {code:title=IPCUtil.java|borderStyle=solid}
> // We should put the ByteBuffer into pool before return null
> public ByteBuffer buildCellBlock(final Codec codec, final CompressionCodec
> compressor,
> final CellScanner cellScanner, final BoundedByteBufferPool pool) {
> ...
> if (pool != null) {
> ByteBuffer bb = pool.getBuffer();
> bufferSize = bb.capacity();
> baos = new ByteBufferOutputStream(bb);
> }
> ...
> int count = 0;
> while (cellScanner.advance()) {
> encoder.write(cellScanner.current());
> count++;
> }
> encoder.flush();
> if (count == 0) return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487623#comment-15487623 ] stack commented on HBASE-15921: --- I've been in here over the last week or so. The variety of Callables is stunning (Nonced, Cancellable, Retrying, Master, Abstract, Base) and then the various hierarchies with duplicated logic in each branch adds bloat and incomprehensibility. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487625#comment-15487625 ] stack commented on HBASE-15921: --- Sorry, this remark was supposed to go w/ the suggestion that we need to redo RetryingCallable. Agree AsyncProcess is hard to grok. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16620) Command-line tool usability issues
[
https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487664#comment-15487664
]
Ted Yu commented on HBASE-16620:
{code}
87+ " -t table name. If specified, only backup
images, which contain this table\n "
{code}
The above should start with '-t table' to match synopsis.
{code}
+if (tables == null) {
+ System.out.println("ERROR: Backup set '" + setName+ "' is either
empty or does not exist");
{code}
System.err should be used when exit code is negative, right ?
{code}
+ } catch(IllegalArgumentException e) {
+System.out.println("ERROR: Illegal argument for backup root path: "+
value);
{code}
Add space between catch and (.
Please use System.err
> Command-line tool usability issues
> --
>
> Key: HBASE-16620
> URL: https://issues.apache.org/jira/browse/HBASE-16620
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16620-v1.patch
>
>
> We need to address issues found by [~saint@gmail.com]
> https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487675#comment-15487675 ] Heng Chen commented on HBASE-15921: --- After unified, lots of callables could be removed. We could control retry in AsyncProcess. The total logical should be clear. As for AsyncProcess, it is complicated but also powerful, all replica logic, traffic control, retry logic etc. has been implemented in it, and the AP has been existed for all released versions, we could say it is stable. What we need is to do some refactor for it. It is not a good idea to just abandon it. Something i worried about with unification is performance, i will do it in HBASE-16625 > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16627) AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table exists
Ted Yu created HBASE-16627:
--
Summary: AssignmentManager#isDisabledorDisablingRegionInRIT should
check whether table exists
Key: HBASE-16627
URL: https://issues.apache.org/jira/browse/HBASE-16627
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
[~stack] first reported this issue when he played with backup feature.
The following exception can be observed in backup unit tests:
{code}
2016-09-13 16:21:57,661 ERROR [ProcedureExecutor-3]
master.TableStateManager(134): Unable to get table hbase:backup state
org.apache.hadoop.hbase.TableNotFoundException: hbase:backup
at
org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
at
org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
at
org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546)
at
org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254)
at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430)
at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127)
at
org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808)
{code}
AssignmentManager#isDisabledorDisablingRegionInRIT should take table existence
into account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487718#comment-15487718
]
Ted Yu commented on HBASE-14123:
No backup / restore tests failed.
Tried a few failed tests from above list with patch v19:
{code}
Running org.apache.hadoop.hbase.client.TestFromClientSide
Tests run: 78, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 219.012 sec -
in org.apache.hadoop.hbase.client.TestFromClientSide
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.client.TestMultiRespectsLimits
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.427 sec - in
org.apache.hadoop.hbase.client.TestMultiRespectsLimits
{code}
The increase of test failures seems to be related to the switch to Java 8 in
master branch.
> HBase Backup/Restore Phase 2
>
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
> Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: 14123-master.v14.txt, 14123-master.v15.txt,
> 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt,
> 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v3.txt,
> 14123-master.v5.txt, 14123-master.v6.txt, 14123-master.v7.txt,
> 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt,
> HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch,
> HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch,
> HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch,
> HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch,
> HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch,
> HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16613) Return the unused ByteBuffer to BoundedByteBufferPool when no cell is retrieved from the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16613:
---
Summary: Return the unused ByteBuffer to BoundedByteBufferPool when no cell
is retrieved from the CellScanner (was: Return the unused ByteBuffer to
BoundedByteBufferPool when no cell is got from the CellScanner)
> Return the unused ByteBuffer to BoundedByteBufferPool when no cell is
> retrieved from the CellScanner
>
>
> Key: HBASE-16613
> URL: https://issues.apache.org/jira/browse/HBASE-16613
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16613.branch-1.v0.patch,
> HBASE-16613.branch-1.v1.patch, HBASE-16613.branch-1.v1.patch
>
>
> The critical code is shown below:
> {code:title=IPCUtil.java|borderStyle=solid}
> // We should put the ByteBuffer into pool before return null
> public ByteBuffer buildCellBlock(final Codec codec, final CompressionCodec
> compressor,
> final CellScanner cellScanner, final BoundedByteBufferPool pool) {
> ...
> if (pool != null) {
> ByteBuffer bb = pool.getBuffer();
> bufferSize = bb.capacity();
> baos = new ByteBufferOutputStream(bb);
> }
> ...
> int count = 0;
> while (cellScanner.advance()) {
> encoder.write(cellScanner.current());
> count++;
> }
> encoder.flush();
> if (count == 0) return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16613) Return the unused ByteBuffer to BoundedByteBufferPool when no cell is retrieved from the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487728#comment-15487728
]
Ted Yu commented on HBASE-16613:
Integrated to branch-1.
[~mantonov]:
Do you want this fix in branch-1.3 ?
> Return the unused ByteBuffer to BoundedByteBufferPool when no cell is
> retrieved from the CellScanner
>
>
> Key: HBASE-16613
> URL: https://issues.apache.org/jira/browse/HBASE-16613
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16613.branch-1.v0.patch,
> HBASE-16613.branch-1.v1.patch, HBASE-16613.branch-1.v1.patch
>
>
> The critical code is shown below:
> {code:title=IPCUtil.java|borderStyle=solid}
> // We should put the ByteBuffer into pool before return null
> public ByteBuffer buildCellBlock(final Codec codec, final CompressionCodec
> compressor,
> final CellScanner cellScanner, final BoundedByteBufferPool pool) {
> ...
> if (pool != null) {
> ByteBuffer bb = pool.getBuffer();
> bufferSize = bb.capacity();
> baos = new ByteBufferOutputStream(bb);
> }
> ...
> int count = 0;
> while (cellScanner.advance()) {
> encoder.write(cellScanner.current());
> count++;
> }
> encoder.flush();
> if (count == 0) return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487753#comment-15487753
]
Phil Yang commented on HBASE-15968:
---
[~stack] Thanks for your reply. This patch has some bugs, I fixed them locally
and not already upload a new patch because it does not support visibility
labels. I can upload a new patch with the fixes (although visibility labels is
not done) if needed and upload it to review board to help you review.
{quote}
This 'fixed' behavior should be default in 2.0.0
{quote}
A concern is performance. In current "broken" behavior, we read cell by the
order of timestamp desc and each cell we have an O(1) time complexity. But in
the new behavior, we have to save some info from all cells whose ts is higher
than this cell(and all cells for family-delete marker) and check if we can see
this cell according to their ts and mvcc, which is not O(1). I am not very
sure, complexity may be O(N*logM) where N is number of delete markers whose ts
is higher but mvcc is lower, and M is the maxversion of the cf's conf. I
implement the data structure by current design because I think N will not be
very high even if we have many Puts and Deletes because in the most case we
will not have a Cell with higher mvcc but lower timestamp, and M is usually
only 1,2, 3 or some small number.
{quote}
Yeah, its an outstanding question as to when it is safe to set sequenceid/mvcc
== 0.
This is for new tables only?
{quote}
In the patch I disable this feature, we always save mvcc. So if we alter a
table into new behavior, we should handle Cells whose mvcc is in HFile's
header. Many Cells will have same mvcc, which is not a very difficult issue but
we need prove there is no bug for this situation. And we have to define the
order with same mvcc, just like we define the order of Type.
{quote}
mvcc-sensitive is not a good name because the whole system is already mvcc
sensitive.
{quote}
To be honest, I spend some time on naming this issue but I have no idea what is
the best Just call it "fix the bug" is very exciting for me :)
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is maske
[jira] [Commented] (HBASE-16625) Performance test for interface unified with AP
[
https://issues.apache.org/jira/browse/HBASE-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487768#comment-15487768
]
Heng Chen commented on HBASE-16625:
---
Test for Increment w/ or w/o patch in HBASE-16610, in standalone mode, do
three tests with PE
{code}
bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=1
--nomapred increment 10
{code}
without the patch, results:
{code}
2016-09-14 00:39:29,190 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Summary of timings (ms): [25339, 25285, 25288, 25344, 25331,
25272, 25304, 25340, 25323, 25294]
2016-09-14 00:39:29,191 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Min: 25272msMax: 25344msAvg: 25312ms
2016-09-14 00:46:06,282 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Summary of timings (ms): [27515, 27547, 27386, 27547, 27542,
27362, 27529, 27377, 27527, 27537]
2016-09-14 00:46:06,283 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Min: 27362msMax: 27547msAvg: 27486ms
2016-09-14 00:49:05,878 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Summary of timings (ms): [28079, 28076, 28055, 27834, 28096,
28090, 27758, 28078, 28046, 28094]
2016-09-14 00:49:05,878 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Min: 27758msMax: 28096msAvg: 28020ms
{code}
with the pach:
{code}
2016-09-14 00:56:05,546 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Summary of timings (ms): [26391, 26562, 26553, 26617, 26434,
26616, 26486, 26622, 26337, 26529]
2016-09-14 00:56:05,547 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Min: 26337msMax: 26622msAvg: 26514ms
2016-09-14 00:57:00,886 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Summary of timings (ms): [25914, 25981, 25996, 25899, 26026,
26067, 26060, 25944, 26033, 25921]
2016-09-14 00:57:00,887 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Min: 25899msMax: 26067msAvg: 25984ms
2016-09-14 00:57:46,107 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Summary of timings (ms): [26405, 26258, 26299, 26420, 26355,
26159, 26123, 26387, 26409, 26370]
2016-09-14 00:57:46,107 INFO [main] hbase.PerformanceEvaluation:
[IncrementTest] Min: 26123msMax: 26420msAvg: 26318ms
{code}
It seems there is no big difference w/ or w/o patch in HBASE-16610
> Performance test for interface unified with AP
> --
>
> Key: HBASE-16625
> URL: https://issues.apache.org/jira/browse/HBASE-16625
> Project: HBase
> Issue Type: Sub-task
>Reporter: Heng Chen
>Assignee: Heng Chen
>
> Due to one more thread context changed in AP, the performance should be
> tested. Let me do it in this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487753#comment-15487753
]
Phil Yang edited comment on HBASE-15968 at 9/13/16 5:04 PM:
[~stack] Thanks for your reply. This patch has some bugs, I fixed them locally
and not already upload a new patch because it does not support visibility
labels. I can upload a new patch with the fixes (although visibility labels is
not done) if needed and upload it to review board to help you review.
{quote}
This 'fixed' behavior should be default in 2.0.0
{quote}
A concern is performance. In current "broken" behavior, we read cell by the
order of timestamp desc and each cell we have an O(1) time complexity. But in
the new behavior, we have to save some info from all cells whose ts is higher
than this cell(and all cells for family-delete marker) and check if we can see
this cell according to their ts and mvcc, which is not O(1). I am not very
sure, complexity may be O(N*logM) where N is number of delete markers whose ts
is higher but mvcc is lower, and M is the number of Puts with higher timestamp
and can not be seen for users. I implement the data structure by current design
because I think N will not be very high even if we have many Puts and Deletes
because in the most case we will not have a Cell with higher mvcc but lower
timestamp, and M equals to maxversion if there is no Delete.
{quote}
Yeah, its an outstanding question as to when it is safe to set sequenceid/mvcc
== 0.
This is for new tables only?
{quote}
In the patch I disable this feature, we always save mvcc. So if we alter a
table into new behavior, we should handle Cells whose mvcc is in HFile's
header. Many Cells will have same mvcc, which is not a very difficult issue but
we need prove there is no bug for this situation. And we have to define the
order with same mvcc, just like we define the order of Type.
{quote}
mvcc-sensitive is not a good name because the whole system is already mvcc
sensitive.
{quote}
To be honest, I spend some time on naming this issue but I have no idea what is
the best Just call it "fix the bug" is very exciting for me :)
was (Author: yangzhe1991):
[~stack] Thanks for your reply. This patch has some bugs, I fixed them locally
and not already upload a new patch because it does not support visibility
labels. I can upload a new patch with the fixes (although visibility labels is
not done) if needed and upload it to review board to help you review.
{quote}
This 'fixed' behavior should be default in 2.0.0
{quote}
A concern is performance. In current "broken" behavior, we read cell by the
order of timestamp desc and each cell we have an O(1) time complexity. But in
the new behavior, we have to save some info from all cells whose ts is higher
than this cell(and all cells for family-delete marker) and check if we can see
this cell according to their ts and mvcc, which is not O(1). I am not very
sure, complexity may be O(N*logM) where N is number of delete markers whose ts
is higher but mvcc is lower, and M is the maxversion of the cf's conf. I
implement the data structure by current design because I think N will not be
very high even if we have many Puts and Deletes because in the most case we
will not have a Cell with higher mvcc but lower timestamp, and M is usually
only 1,2, 3 or some small number.
{quote}
Yeah, its an outstanding question as to when it is safe to set sequenceid/mvcc
== 0.
This is for new tables only?
{quote}
In the patch I disable this feature, we always save mvcc. So if we alter a
table into new behavior, we should handle Cells whose mvcc is in HFile's
header. Many Cells will have same mvcc, which is not a very difficult issue but
we need prove there is no bug for this situation. And we have to define the
order with same mvcc, just like we define the order of Type.
{quote}
mvcc-sensitive is not a good name because the whole system is already mvcc
sensitive.
{quote}
To be honest, I spend some time on naming this issue but I have no idea what is
the best Just call it "fix the bug" is very exciting for me :)
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major com
[jira] [Assigned] (HBASE-16627) AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table exists
[
https://issues.apache.org/jira/browse/HBASE-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Yuan Jiang reassigned HBASE-16627:
--
Assignee: Stephen Yuan Jiang
> AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table
> exists
>
>
> Key: HBASE-16627
> URL: https://issues.apache.org/jira/browse/HBASE-16627
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
>Priority: Minor
>
> [~stack] first reported this issue when he played with backup feature.
> The following exception can be observed in backup unit tests:
> {code}
> 2016-09-13 16:21:57,661 ERROR [ProcedureExecutor-3]
> master.TableStateManager(134): Unable to get table hbase:backup state
> org.apache.hadoop.hbase.TableNotFoundException: hbase:backup
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808)
> {code}
> AssignmentManager#isDisabledorDisablingRegionInRIT should take table
> existence into account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16627) AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table exists
[
https://issues.apache.org/jira/browse/HBASE-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487795#comment-15487795
]
Stephen Yuan Jiang commented on HBASE-16627:
It is interesting. This is common code path for creating a table. We should
not throw exception (otherwise, all table creation would fail). [~tedyu] and
[~stack], can this be easily repro by just creating backup table?
Assignment manager code is in the middle of updating. Unless this is a easy
repro and block your testing, let me hold this bug for now and I will do more
investigation later. If this is a blocker, I will spend some time to unblock
you in the backup branch.
> AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table
> exists
>
>
> Key: HBASE-16627
> URL: https://issues.apache.org/jira/browse/HBASE-16627
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
>Priority: Minor
>
> [~stack] first reported this issue when he played with backup feature.
> The following exception can be observed in backup unit tests:
> {code}
> 2016-09-13 16:21:57,661 ERROR [ProcedureExecutor-3]
> master.TableStateManager(134): Unable to get table hbase:backup state
> org.apache.hadoop.hbase.TableNotFoundException: hbase:backup
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808)
> {code}
> AssignmentManager#isDisabledorDisablingRegionInRIT should take table
> existence into account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16627) AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table exists
[
https://issues.apache.org/jira/browse/HBASE-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487806#comment-15487806
]
Ted Yu commented on HBASE-16627:
Backup tests can pass.
This is not blocker for backup / restore.
Thanks, Stephen.
> AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table
> exists
>
>
> Key: HBASE-16627
> URL: https://issues.apache.org/jira/browse/HBASE-16627
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
>Priority: Minor
>
> [~stack] first reported this issue when he played with backup feature.
> The following exception can be observed in backup unit tests:
> {code}
> 2016-09-13 16:21:57,661 ERROR [ProcedureExecutor-3]
> master.TableStateManager(134): Unable to get table hbase:backup state
> org.apache.hadoop.hbase.TableNotFoundException: hbase:backup
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808)
> {code}
> AssignmentManager#isDisabledorDisablingRegionInRIT should take table
> existence into account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16627) AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table exists
[
https://issues.apache.org/jira/browse/HBASE-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487812#comment-15487812
]
Matteo Bertozzi commented on HBASE-16627:
-
we don't throw exception up, TableStateManager try to ask for the state of the
table, and table is not found (because we are creating it), so we get that the
table is not in that state.
{code}
public boolean isTableState(TableName tableName, TableState.State... states) {
try {
TableState.State tableState = getTableState(tableName);
return TableState.isInStates(tableState, states);
} catch (IOException e) {
LOG.error("Unable to get table " + tableName + " state", e);
return false;
}
}
{code}
> AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table
> exists
>
>
> Key: HBASE-16627
> URL: https://issues.apache.org/jira/browse/HBASE-16627
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
>Priority: Minor
>
> [~stack] first reported this issue when he played with backup feature.
> The following exception can be observed in backup unit tests:
> {code}
> 2016-09-13 16:21:57,661 ERROR [ProcedureExecutor-3]
> master.TableStateManager(134): Unable to get table hbase:backup state
> org.apache.hadoop.hbase.TableNotFoundException: hbase:backup
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808)
> {code}
> AssignmentManager#isDisabledorDisablingRegionInRIT should take table
> existence into account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16627) AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table exists
[
https://issues.apache.org/jira/browse/HBASE-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487829#comment-15487829
]
Stephen Yuan Jiang commented on HBASE-16627:
That is true, {{isTableState()}} 'eats' the TableNotFoundException exception
from {{getTableState()}} and log an ERROR.
{noformat}
@NonNull
public TableState.State getTableState(TableName tableName) throws IOException
{
TableState currentState = readMetaState(tableName);
if (currentState == null) {
throw new TableNotFoundException(tableName);
}
return currentState.getState();
}
{noformat}
Now, I think we should lower the log level to WARN - this is not error, as
caller could legally ask for a non-existing table.
> AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table
> exists
>
>
> Key: HBASE-16627
> URL: https://issues.apache.org/jira/browse/HBASE-16627
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
>Priority: Minor
>
> [~stack] first reported this issue when he played with backup feature.
> The following exception can be observed in backup unit tests:
> {code}
> 2016-09-13 16:21:57,661 ERROR [ProcedureExecutor-3]
> master.TableStateManager(134): Unable to get table hbase:backup state
> org.apache.hadoop.hbase.TableNotFoundException: hbase:backup
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808)
> {code}
> AssignmentManager#isDisabledorDisablingRegionInRIT should take table
> existence into account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16628) Apache HBase ™ Reference Guide: 143.1.1. Code Formatting: miss the location of preference item
alexxiyang created HBASE-16628:
--
Summary: Apache HBase ™ Reference Guide: 143.1.1. Code Formatting:
miss the location of preference item
Key: HBASE-16628
URL: https://issues.apache.org/jira/browse/HBASE-16628
Project: HBase
Issue Type: Bug
Reporter: alexxiyang
In 143.1.1. Code Formatting
it just said
{code}
Still in Preferences, click . Be sure the following options are selected:Apache
HBase ™ Reference Guide
{code}
But nothing after click. It should be Java->Editor->Save Actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16627) AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table exists
[
https://issues.apache.org/jira/browse/HBASE-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487858#comment-15487858
]
stack commented on HBASE-16627:
---
Not a blocker. Warn sounds better but if not an issue can we save spew in log.
Am Fi e waiting till am further along
> AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table
> exists
>
>
> Key: HBASE-16627
> URL: https://issues.apache.org/jira/browse/HBASE-16627
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
>Priority: Minor
>
> [~stack] first reported this issue when he played with backup feature.
> The following exception can be observed in backup unit tests:
> {code}
> 2016-09-13 16:21:57,661 ERROR [ProcedureExecutor-3]
> master.TableStateManager(134): Unable to get table hbase:backup state
> org.apache.hadoop.hbase.TableNotFoundException: hbase:backup
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808)
> {code}
> AssignmentManager#isDisabledorDisablingRegionInRIT should take table
> existence into account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16628) Apache HBase ™ Reference Guide: 143.1.1. Code Formatting: miss the location of preference item
[
https://issues.apache.org/jira/browse/HBASE-16628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487864#comment-15487864
]
Dima Spivak commented on HBASE-16628:
-
Hey [~alexxiyang], I'm loving that you're helping us find issues with our ref
guide. What if instead of opening separate JIRAs, we close this one and rename
HBASE-16622 to something like "Fix some issues with the HBase reference guide,"
where you can continue to post problems you've encountered? If we continue the
conversation over there, we'd be happy to help you put together a patch for
these, as well, to fix them.
> Apache HBase ™ Reference Guide: 143.1.1. Code Formatting: miss the location
> of preference item
> --
>
> Key: HBASE-16628
> URL: https://issues.apache.org/jira/browse/HBASE-16628
> Project: HBase
> Issue Type: Bug
>Reporter: alexxiyang
>
> In 143.1.1. Code Formatting
> it just said
> {code}
> Still in Preferences, click . Be sure the following options are
> selected:Apache HBase ™ Reference Guide
> {code}
> But nothing after click. It should be Java->Editor->Save Actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487896#comment-15487896
]
stack commented on HBASE-15968:
---
How should visibility labels be done? In the compound tracker if visibility is
enabled?
On perf., Yeah let's take a look but can do stuff like take a flag w the type
of deletes allowed in a table. This would not be only locale that could benefit
if a table marked no_delete. And of course perf is nice but wonky behavior
costs too. Would be good to get a handle on it though for sure. By your
reasoning which seems sound, this approach shouldn't be that much more
expensive; usually no extra cost.
As anoop reminds us, our mvcc in the cell is an expensive paste (vint and we
have to skip past value and tags iirc -- our kV layout is suboptimal). But
having mvcc always available will help elsewhere, in replay and replication.
We can do naming later but the bigger point that this is how it should have
been all along and that this should be default should change the patch approach
some. Thanks.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent
[jira] [Updated] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[
https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16604:
--
Status: Patch Available (was: Open)
> Scanner retries on IOException can cause the scans to miss data
>
>
> Key: HBASE-16604
> URL: https://issues.apache.org/jira/browse/HBASE-16604
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16604_v1.patch
>
>
> Debugging an ITBLL failure, where the Verify did not "see" all the data in
> the cluster, I've noticed that if we end up getting a generic IOException
> from the HFileReader level, we may end up missing the rest of the data in the
> region. I was able to manually test this, and this stack trace helps to
> understand what is going on:
> {code}
> 2016-09-09 16:27:15,633 INFO [hconnection-0x71ad3d8a-shared--pool21-t9]
> client.ScannerCallable(376): Open scanner=1 for
> scan={"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":2097152,"families":{"testFamily":["testFamily"]},"caching":100,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> on region
> region=testScanThrowsException,,1473463632707.b2adfb618e5d0fe225c1dc40c0eabfee.,
> hostname=hw10676,51833,1473463626529, seqNum=2
> 2016-09-09 16:27:15,634 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2196): scan request:scanner_id: 1 number_of_rows:
> 100 close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true renew: false
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2510): Rolling back next call seqId
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2565): Throwing new
> ServiceExceptionjava.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866998,
> cur=/testFamily:/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0] to key
> /testFamily:testFamily/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> 2016-09-09 16:27:15,635 DEBUG
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833] ipc.CallRunner(110):
> B.fifo.QRpcServer.handler=5,queue=0,port=51833: callId: 26 service:
> ClientService methodName: Scan size: 26 connection: 192.168.42.75:51903
> java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for
> reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866998,
> cur=/testFamily:/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0] to key
> /testFamily:testFamily/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:224)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:312)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.jav
[jira] [Commented] (HBASE-16618) Procedure v2 - Add base class for table and ns procedures
[ https://issues.apache.org/jira/browse/HBASE-16618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487953#comment-15487953 ] Stephen Yuan Jiang commented on HBASE-16618: +1 good idea to reduce dupe code and for better code maintenance > Procedure v2 - Add base class for table and ns procedures > - > > Key: HBASE-16618 > URL: https://issues.apache.org/jira/browse/HBASE-16618 > Project: HBase > Issue Type: Sub-task > Components: master, proc-v2 >Affects Versions: 2.0.0, 1.4.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16618-v0.patch > > > Now that we have a bunch of procedures implemented, we can add a base class > for the Table and Namespace procedure with a couple of the common pattern > used (e.g. basic locking, toString, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16622) Apache HBase ™ Reference Guide: HBase Java API example has several errors
[
https://issues.apache.org/jira/browse/HBASE-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
alexxiyang updated HBASE-16622:
---
Attachment: HBASE-16622-v0.diff
> Apache HBase ™ Reference Guide: HBase Java API example has several errors
> -
>
> Key: HBASE-16622
> URL: https://issues.apache.org/jira/browse/HBASE-16622
> Project: HBase
> Issue Type: Bug
>Reporter: alexxiyang
> Attachments: HBASE-16622-v0.diff
>
>
> 1.
> {code}
> if (admin.tableExists(tableName)) {
> System.out.println("Table does not exist.");
> System.exit(-1);
> }
> {code}
> This should be
> {code}
> if (!admin.tableExists(tableName)) {
> {code}
> 2.
> SNAPPY is not suitable for begginer. They may get exceptions like
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException):
> org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm
> 'snappy' previously failed test. Set hbase.table.sanity.checks to false at
> conf or table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1701)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1569)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1491)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:462)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2178)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> So the code below
> {code}
> table.addFamily(new
> HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.SNAPPY));
> {code}
> it better to change into
> {code}
> table.addFamily(new
> HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.NONE));
> {code}
> 3.
> Before modify column family , get the table from connection
> Change
> {code}
> HTableDescriptor table = new HTableDescriptor(tableName);
> {code}
> into
> {code}
> Table table = connection.getTable(TableName.valueOf(tablename));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16628) Apache HBase ™ Reference Guide: 143.1.1. Code Formatting: miss the location of preference item
[
https://issues.apache.org/jira/browse/HBASE-16628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487987#comment-15487987
]
alexxiyang commented on HBASE-16628:
Hi [~dimaspivak], that's a good idea. I will attach the patch of HBASE-16628 to
HBASE-16622. You can feel free to close this issue.
Thanks,
Alex
> Apache HBase ™ Reference Guide: 143.1.1. Code Formatting: miss the location
> of preference item
> --
>
> Key: HBASE-16628
> URL: https://issues.apache.org/jira/browse/HBASE-16628
> Project: HBase
> Issue Type: Bug
>Reporter: alexxiyang
>
> In 143.1.1. Code Formatting
> it just said
> {code}
> Still in Preferences, click . Be sure the following options are
> selected:Apache HBase ™ Reference Guide
> {code}
> But nothing after click. It should be Java->Editor->Save Actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16622) Fix some issues with the HBase reference guide
[
https://issues.apache.org/jira/browse/HBASE-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
alexxiyang updated HBASE-16622:
---
Summary: Fix some issues with the HBase reference guide (was: Apache HBase
™ Reference Guide: HBase Java API example has several errors)
> Fix some issues with the HBase reference guide
> --
>
> Key: HBASE-16622
> URL: https://issues.apache.org/jira/browse/HBASE-16622
> Project: HBase
> Issue Type: Bug
>Reporter: alexxiyang
> Attachments: HBASE-16622-v0.diff
>
>
> 1.
> {code}
> if (admin.tableExists(tableName)) {
> System.out.println("Table does not exist.");
> System.exit(-1);
> }
> {code}
> This should be
> {code}
> if (!admin.tableExists(tableName)) {
> {code}
> 2.
> SNAPPY is not suitable for begginer. They may get exceptions like
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException):
> org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm
> 'snappy' previously failed test. Set hbase.table.sanity.checks to false at
> conf or table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1701)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1569)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1491)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:462)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2178)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> So the code below
> {code}
> table.addFamily(new
> HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.SNAPPY));
> {code}
> it better to change into
> {code}
> table.addFamily(new
> HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.NONE));
> {code}
> 3.
> Before modify column family , get the table from connection
> Change
> {code}
> HTableDescriptor table = new HTableDescriptor(tableName);
> {code}
> into
> {code}
> Table table = connection.getTable(TableName.valueOf(tablename));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16622) Fix some issues with the HBase reference guide
[
https://issues.apache.org/jira/browse/HBASE-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
alexxiyang updated HBASE-16622:
---
Description:
1.
{code}
if (admin.tableExists(tableName)) {
System.out.println("Table does not exist.");
System.exit(-1);
}
{code}
This should be
{code}
if (!admin.tableExists(tableName)) {
{code}
2.
SNAPPY is not suitable for begginer. They may get exceptions like
{code}
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException):
org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'snappy'
previously failed test. Set hbase.table.sanity.checks to false at conf or table
descriptor if you want to bypass sanity checks
at
org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1701)
at
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1569)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1491)
at
org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:462)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2178)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
{code}
So the code below
{code}
table.addFamily(new
HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.SNAPPY));
{code}
it better to change into
{code}
table.addFamily(new
HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.NONE));
{code}
3.
Before modify column family , get the table from connection
Change
{code}
HTableDescriptor table = new HTableDescriptor(tableName);
{code}
into
{code}
Table table = connection.getTable(TableName.valueOf(tablename));
{code}
4.
In 143.1.1. Code Formatting
it just said
{code}
Still in Preferences, click . Be sure the following options are selected:Apache
HBase ™ Reference Guide
{code}
But nothing after click. It should be Java->Editor->Save Actions
was:
1.
{code}
if (admin.tableExists(tableName)) {
System.out.println("Table does not exist.");
System.exit(-1);
}
{code}
This should be
{code}
if (!admin.tableExists(tableName)) {
{code}
2.
SNAPPY is not suitable for begginer. They may get exceptions like
{code}
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException):
org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'snappy'
previously failed test. Set hbase.table.sanity.checks to false at conf or table
descriptor if you want to bypass sanity checks
at
org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1701)
at
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1569)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1491)
at
org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:462)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2178)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
{code}
So the code below
{code}
table.addFamily(new
HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.SNAPPY));
{code}
it better to change into
{code}
table.addFamily(new
HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.NONE));
{code}
3.
Before modify column family , get the table from connection
Change
{code}
HTableDescriptor table = new HTableDescriptor(tableName);
{code}
into
{code}
Table table = connection.getTable(TableName.valueOf(tablename));
{code}
> Fix some issues with the HBase reference guide
> --
>
> Key: HBASE-16622
> URL: https://issues.apache.org/jira/browse/HBASE-16622
> Project: HBase
> Issue Type: Bug
>Reporter: alexxiyang
> Attachments: HBASE-16622-v0.diff
>
>
> 1.
> {code}
> if (admin.tableExists(tableName)) {
> System.out.println("Table does not exist.");
> System.exit(-1);
> }
> {code}
> This should be
> {code}
> if (!admin.tableExists(tableName)) {
> {code}
> 2.
> SNAPPY is not suitable for begginer. T
[jira] [Updated] (HBASE-16618) Procedure v2 - Add base class for table and ns procedures
[ https://issues.apache.org/jira/browse/HBASE-16618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16618: Resolution: Fixed Fix Version/s: (was: 1.4.0) Status: Resolved (was: Patch Available) > Procedure v2 - Add base class for table and ns procedures > - > > Key: HBASE-16618 > URL: https://issues.apache.org/jira/browse/HBASE-16618 > Project: HBase > Issue Type: Sub-task > Components: master, proc-v2 >Affects Versions: 2.0.0, 1.4.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16618-v0.patch > > > Now that we have a bunch of procedures implemented, we can add a base class > for the Table and Namespace procedure with a couple of the common pattern > used (e.g. basic locking, toString, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488180#comment-15488180 ] Yu Li commented on HBASE-16626: --- For branch-1 it's a little bit awkward, since for jdk7 or lower we have to change RegionScanner to abstract class to add a default behavior, which also requires customer code to change (implements RegionScanner -> extends RegionScanner)... So we decided to change the customer coprocessor code directly (and luckily our customer accept this way)... Just to make it clear to others, for branch-1 the problem only exists after we did a backport of the read-path offheap work, no need to worry if using official released versions. :-) > User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap > part > - > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that implements > Interface RegionScanner in 1.X . > After we back ported the Off Heap feature of 2.0.0, > RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when > executing scanner.shipped(). It is because the customized scanner didn't > override the shipped method yet. > Instead of forcing every user to add a empty implementation(if they don't > really need to scan the file or the RS don't use L2 cache, they don't need to > do anything in shipped method) , adding a default method of shipped in > Interface RegionScanner might be a better way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16612) Use array to cache Types for KeyValue.Type.codeToType
[ https://issues.apache.org/jira/browse/HBASE-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488185#comment-15488185 ] Hudson commented on HBASE-16612: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1595 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1595/]) HBASE-16612 Use array to cache Types for KeyValue.Type.codeToType (Phil (tedyu: rev 981200bf1344e2c58559874cb7a66132f703efd6) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java > Use array to cache Types for KeyValue.Type.codeToType > - > > Key: HBASE-16612 > URL: https://issues.apache.org/jira/browse/HBASE-16612 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.6, 1.2.3, 0.98.22 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16612-v1.patch > > > We don't rely on enum ordinals in KeyValye.Type. We have own code in it. In > codeToType, we use a loop to find the Type which is not a good idea. We can > just use an arryay[256] to cache all types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488196#comment-15488196 ] Yu Li commented on HBASE-16626: --- We also encountered HBASE-16609, and I guess more will expose if there's any since we already start A/B testing for the offheap backpoted version, JFYI. > User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap > part > - > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that implements > Interface RegionScanner in 1.X . > After we back ported the Off Heap feature of 2.0.0, > RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when > executing scanner.shipped(). It is because the customized scanner didn't > override the shipped method yet. > Instead of forcing every user to add a empty implementation(if they don't > really need to scan the file or the RS don't use L2 cache, they don't need to > do anything in shipped method) , adding a default method of shipped in > Interface RegionScanner might be a better way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488200#comment-15488200 ] Yu Li commented on HBASE-16626: --- [~stack], [~busbey], [~tedyu], [~xharlie] is my workmate and could you help add him into the contributor list, so he could assign this JIRA to himself? Many thanks. > User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap > part > - > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that implements > Interface RegionScanner in 1.X . > After we back ported the Off Heap feature of 2.0.0, > RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when > executing scanner.shipped(). It is because the customized scanner didn't > override the shipped method yet. > Instead of forcing every user to add a empty implementation(if they don't > really need to scan the file or the RS don't use L2 cache, they don't need to > do anything in shipped method) , adding a default method of shipped in > Interface RegionScanner might be a better way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16612) Use array to cache Types for KeyValue.Type.codeToType
[ https://issues.apache.org/jira/browse/HBASE-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16612: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 Status: Resolved (was: Patch Available) Thanks for the patch, Phil. Thanks for the reviews. > Use array to cache Types for KeyValue.Type.codeToType > - > > Key: HBASE-16612 > URL: https://issues.apache.org/jira/browse/HBASE-16612 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.6, 1.2.3, 0.98.22 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16612-v1.patch > > > We don't rely on enum ordinals in KeyValye.Type. We have own code in it. In > codeToType, we use a loop to find the Type which is not a good idea. We can > just use an arryay[256] to cache all types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488205#comment-15488205 ] Ted Yu commented on HBASE-16626: Added Charlie as contributor. > User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap > part > - > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that implements > Interface RegionScanner in 1.X . > After we back ported the Off Heap feature of 2.0.0, > RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when > executing scanner.shipped(). It is because the customized scanner didn't > override the shipped method yet. > Instead of forcing every user to add a empty implementation(if they don't > really need to scan the file or the RS don't use L2 cache, they don't need to > do anything in shipped method) , adding a default method of shipped in > Interface RegionScanner might be a better way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16613) Return the unused ByteBuffer to BoundedByteBufferPool when no cell is retrieved from the CellScanner
[
https://issues.apache.org/jira/browse/HBASE-16613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488258#comment-15488258
]
Hudson commented on HBASE-16613:
FAILURE: Integrated in Jenkins build HBase-1.4 #412 (See
[https://builds.apache.org/job/HBase-1.4/412/])
HBASE-16613 Return the unused ByteBuffer to BoundedByteBufferPool when (tedyu:
rev 8e25ea536aab69ed07de862583d4d081e10dd2ed)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java
> Return the unused ByteBuffer to BoundedByteBufferPool when no cell is
> retrieved from the CellScanner
>
>
> Key: HBASE-16613
> URL: https://issues.apache.org/jira/browse/HBASE-16613
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16613.branch-1.v0.patch,
> HBASE-16613.branch-1.v1.patch, HBASE-16613.branch-1.v1.patch
>
>
> The critical code is shown below:
> {code:title=IPCUtil.java|borderStyle=solid}
> // We should put the ByteBuffer into pool before return null
> public ByteBuffer buildCellBlock(final Codec codec, final CompressionCodec
> compressor,
> final CellScanner cellScanner, final BoundedByteBufferPool pool) {
> ...
> if (pool != null) {
> ByteBuffer bb = pool.getBuffer();
> bufferSize = bb.capacity();
> baos = new ByteBufferOutputStream(bb);
> }
> ...
> int count = 0;
> while (cellScanner.advance()) {
> encoder.write(cellScanner.current());
> count++;
> }
> encoder.flush();
> if (count == 0) return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16612) Use array to cache Types for KeyValue.Type.codeToType
[ https://issues.apache.org/jira/browse/HBASE-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488260#comment-15488260 ] Hudson commented on HBASE-16612: FAILURE: Integrated in Jenkins build HBase-1.4 #412 (See [https://builds.apache.org/job/HBase-1.4/412/]) HBASE-16612 Use array to cache Types for KeyValue.Type.codeToType (Phil (tedyu: rev ac1ee77f40a542a596b992fbc7f00a833f89d7a6) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java > Use array to cache Types for KeyValue.Type.codeToType > - > > Key: HBASE-16612 > URL: https://issues.apache.org/jira/browse/HBASE-16612 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.6, 1.2.3, 0.98.22 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16612-v1.patch > > > We don't rely on enum ordinals in KeyValye.Type. We have own code in it. In > codeToType, we use a loop to find the Type which is not a good idea. We can > just use an arryay[256] to cache all types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16381) Shell deleteall command should support row key prefixes
[ https://issues.apache.org/jira/browse/HBASE-16381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488278#comment-15488278 ] Yi Liang commented on HBASE-16381: -- yes, i will change it to reuse this list > Shell deleteall command should support row key prefixes > --- > > Key: HBASE-16381 > URL: https://issues.apache.org/jira/browse/HBASE-16381 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Andrew Purtell >Assignee: Yi Liang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16381-V1.patch, HBASE-16381-V2.patch, > HBASE-16381-V3.patch, HBASE-16381-V4.patch > > > The shell's deleteall command should support deleting a row range using a row > key prefix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16622) Fix some issues with the HBase reference guide
[
https://issues.apache.org/jira/browse/HBASE-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak updated HBASE-16622:
Status: Patch Available (was: Open)
> Fix some issues with the HBase reference guide
> --
>
> Key: HBASE-16622
> URL: https://issues.apache.org/jira/browse/HBASE-16622
> Project: HBase
> Issue Type: Bug
>Reporter: alexxiyang
> Attachments: HBASE-16622-v0.diff
>
>
> 1.
> {code}
> if (admin.tableExists(tableName)) {
> System.out.println("Table does not exist.");
> System.exit(-1);
> }
> {code}
> This should be
> {code}
> if (!admin.tableExists(tableName)) {
> {code}
> 2.
> SNAPPY is not suitable for begginer. They may get exceptions like
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException):
> org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm
> 'snappy' previously failed test. Set hbase.table.sanity.checks to false at
> conf or table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1701)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1569)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1491)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:462)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2178)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> So the code below
> {code}
> table.addFamily(new
> HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.SNAPPY));
> {code}
> it better to change into
> {code}
> table.addFamily(new
> HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.NONE));
> {code}
> 3.
> Before modify column family , get the table from connection
> Change
> {code}
> HTableDescriptor table = new HTableDescriptor(tableName);
> {code}
> into
> {code}
> Table table = connection.getTable(TableName.valueOf(tablename));
> {code}
> 4.
> In 143.1.1. Code Formatting
> it just said
> {code}
> Still in Preferences, click . Be sure the following options are
> selected:Apache HBase ™ Reference Guide
> {code}
> But nothing after click. It should be Java->Editor->Save Actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part
[ https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-16626: -- Assignee: Charlie Qiangeng Xu Thanks [~tedyu] for the quick help! Assigning JIRA to [~xharlie]. > User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap > part > - > > Key: HBASE-16626 > URL: https://issues.apache.org/jira/browse/HBASE-16626 > Project: HBase > Issue Type: Sub-task > Components: Offheaping >Affects Versions: 1.2.2, 1.1.6 >Reporter: Charlie Qiangeng Xu >Assignee: Charlie Qiangeng Xu > Fix For: 2.0.0 > > > Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a > new interface Shipper which contains a "shipped()" method. > In our case, some application user defined customized scanner that implements > Interface RegionScanner in 1.X . > After we back ported the Off Heap feature of 2.0.0, > RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when > executing scanner.shipped(). It is because the customized scanner didn't > override the shipped method yet. > Instead of forcing every user to add a empty implementation(if they don't > really need to scan the file or the RS don't use L2 cache, they don't need to > do anything in shipped method) , adding a default method of shipped in > Interface RegionScanner might be a better way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
