[jira] [Commented] (HBASE-17893) Allow HBase to build against Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HBASE-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16036162#comment-16036162
]
Lars Hofhansl commented on HBASE-17893:
---
Comments?
[~busbey], [~mantonov], fine to commit to branch-1 (and perhaps branch-1.3)?
> Allow HBase to build against Hadoop 2.8.0
> -
>
> Key: HBASE-17893
> URL: https://issues.apache.org/jira/browse/HBASE-17893
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.5
>Reporter: Lars Hofhansl
> Attachments: 17883-1.2-BROKEN.txt, 17893-1.3-backport.txt
>
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hbase-assembly: Error rendering velocity resource. Error invoking
> method 'get(java.lang.Integer)' in java.util.ArrayList at
> META-INF/LICENSE.vm[line 1671, column 8]: InvocationTargetException: Index:
> 0, Size: 0 -> [Help 1]
> {code}
> The in the generated LICENSE.
> {code}
> This product includes Nimbus JOSE+JWT licensed under the The Apache Software
> License, Version 2.0.
> ${dep.licenses[0].comments}
> Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> com.nimbusds
> nimbus-jose-jwt
> 3.9
> maven central search
> g:com.nimbusds AND a:nimbus-jose-jwt AND v:3.9
> project website
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> project source
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> {code}
> Maybe the problem is just that it says: Apache _Software_ License
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030628#comment-16030628 ] Lars Hofhansl commented on HBASE-18095: --- I do like dropping the ZK dependencies for regular clients, and make ZK more of an HBase-internal communication service. > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Purtell > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16027200#comment-16027200 ] Lars Hofhansl commented on HBASE-18027: --- Looks good. +1 > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027-branch-1.patch, HBASE-18027-branch-1.patch, > HBASE-18027-branch-1.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18127) Allow regionobserver to optionally skip postPut/postDelete when postBatchMutate was called
[ https://issues.apache.org/jira/browse/HBASE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026889#comment-16026889 ] Lars Hofhansl commented on HBASE-18127: --- OK... Moved to an HBase issue. > Allow regionobserver to optionally skip postPut/postDelete when > postBatchMutate was called > -- > > Key: HBASE-18127 > URL: https://issues.apache.org/jira/browse/HBASE-18127 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl > > Right now a RegionObserver can only statically implement one or the other. In > scenarios where we need to work sometimes on the single postPut and > postDelete hooks and sometimes on the batchMutate hooks, there is currently > no place to convey this information to the single hooks. I.e. the work has > been done in the batch, skip the single hooks. > There are various solutions: > 1. Allow some state to be passed _per operation_. > 2. Remove the single hooks and always only call batch hooks (with a default > wrapper for the single hooks). > 3. more? > [~apurtell], what we had discussed a few days back. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Moved] (HBASE-18127) Allow regionobserver to optionally skip postPut/postDelete when postBatchMutate was called
[ https://issues.apache.org/jira/browse/HBASE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl moved PHOENIX-3892 to HBASE-18127: Issue Type: New Feature (was: Bug) Key: HBASE-18127 (was: PHOENIX-3892) Project: HBase (was: Phoenix) > Allow regionobserver to optionally skip postPut/postDelete when > postBatchMutate was called > -- > > Key: HBASE-18127 > URL: https://issues.apache.org/jira/browse/HBASE-18127 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl > > Right now a RegionObserver can only statically implement one or the other. In > scenarios where we need to work sometimes on the single postPut and > postDelete hooks and sometimes on the batchMutate hooks, there is currently > no place to convey this information to the single hooks. I.e. the work has > been done in the batch, skip the single hooks. > There are various solutions: > 1. Allow some state to be passed _per operation_. > 2. Remove the single hooks and always only call batch hooks (with a default > wrapper for the single hooks). > 3. more? > [~apurtell], what we had discussed a few days back. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17893) Allow HBase to build against Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HBASE-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-17893:
--
Attachment: 17893-1.3-backport.txt
This is against branch 1.3. Can be a first step. Maybe we just leave 1.2 and
earlier.
> Allow HBase to build against Hadoop 2.8.0
> -
>
> Key: HBASE-17893
> URL: https://issues.apache.org/jira/browse/HBASE-17893
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.5
>Reporter: Lars Hofhansl
> Attachments: 17883-1.2-BROKEN.txt, 17893-1.3-backport.txt
>
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hbase-assembly: Error rendering velocity resource. Error invoking
> method 'get(java.lang.Integer)' in java.util.ArrayList at
> META-INF/LICENSE.vm[line 1671, column 8]: InvocationTargetException: Index:
> 0, Size: 0 -> [Help 1]
> {code}
> The in the generated LICENSE.
> {code}
> This product includes Nimbus JOSE+JWT licensed under the The Apache Software
> License, Version 2.0.
> ${dep.licenses[0].comments}
> Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> com.nimbusds
> nimbus-jose-jwt
> 3.9
> maven central search
> g:com.nimbusds AND a:nimbus-jose-jwt AND v:3.9
> project website
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> project source
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> {code}
> Maybe the problem is just that it says: Apache _Software_ License
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17893) Allow HBase to build against Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HBASE-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026863#comment-16026863
]
Lars Hofhansl commented on HBASE-17893:
---
If now I just backport HBASE-16712, I can actually get it to compile.
> Allow HBase to build against Hadoop 2.8.0
> -
>
> Key: HBASE-17893
> URL: https://issues.apache.org/jira/browse/HBASE-17893
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.5
>Reporter: Lars Hofhansl
> Attachments: 17883-1.2-BROKEN.txt
>
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hbase-assembly: Error rendering velocity resource. Error invoking
> method 'get(java.lang.Integer)' in java.util.ArrayList at
> META-INF/LICENSE.vm[line 1671, column 8]: InvocationTargetException: Index:
> 0, Size: 0 -> [Help 1]
> {code}
> The in the generated LICENSE.
> {code}
> This product includes Nimbus JOSE+JWT licensed under the The Apache Software
> License, Version 2.0.
> ${dep.licenses[0].comments}
> Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> com.nimbusds
> nimbus-jose-jwt
> 3.9
> maven central search
> g:com.nimbusds AND a:nimbus-jose-jwt AND v:3.9
> project website
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> project source
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> {code}
> Maybe the problem is just that it says: Apache _Software_ License
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17893) Allow HBase to build against Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HBASE-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026853#comment-16026853
]
Lars Hofhansl commented on HBASE-17893:
---
Confirmed that LICENSE seems fine now. Failing on the NOTICE file now (exactly
where you say it does [~busbey])
> Allow HBase to build against Hadoop 2.8.0
> -
>
> Key: HBASE-17893
> URL: https://issues.apache.org/jira/browse/HBASE-17893
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.5
>Reporter: Lars Hofhansl
> Attachments: 17883-1.2-BROKEN.txt
>
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hbase-assembly: Error rendering velocity resource. Error invoking
> method 'get(java.lang.Integer)' in java.util.ArrayList at
> META-INF/LICENSE.vm[line 1671, column 8]: InvocationTargetException: Index:
> 0, Size: 0 -> [Help 1]
> {code}
> The in the generated LICENSE.
> {code}
> This product includes Nimbus JOSE+JWT licensed under the The Apache Software
> License, Version 2.0.
> ${dep.licenses[0].comments}
> Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> com.nimbusds
> nimbus-jose-jwt
> 3.9
> maven central search
> g:com.nimbusds AND a:nimbus-jose-jwt AND v:3.9
> project website
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> project source
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> {code}
> Maybe the problem is just that it says: Apache _Software_ License
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026547#comment-16026547 ] Lars Hofhansl commented on HBASE-18027: --- [~apurtell], [~THEcreationist], let's go with the initial approach then. Sorry for sending you on this detour! > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027-branch-1.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023218#comment-16023218 ] Lars Hofhansl commented on HBASE-18027: --- +1 > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011277#comment-16011277
]
Lars Hofhansl commented on HBASE-18042:
---
So just to be clear: The HBase 1.2.x client will successfully work with a 1.3.x
cluster, right? Including no leaked scanners that have to wait for their leases
to expire... (Just making sure)
[~enis] can you confirm?
If 1.2.x clients work fine then it seems we should update asynchbase with this
new behavior. [~tsuna], in case you have an opinion.
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011270#comment-16011270 ] Lars Hofhansl commented on HBASE-17489: --- Thanks [~tsuna] (has been a while) :) All good now. For correct asynchronous behavior it seems generally we do not want to impede our ability to link different callbacks back to their originators - even when this is not needed in this particular case. (We fixed it in HBase ... but ... if you had an opinion on HBASE-18042 that would be great!) > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17887) Row-level consistency is broken for read
[ https://issues.apache.org/jira/browse/HBASE-17887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011062#comment-16011062 ] Lars Hofhansl commented on HBASE-17887: --- bq. I guess Lars say HBASE-14970 and this one should not ideally be committed to minor versions. So both should go have been going into 2.0 only. Correct Lars? That is what I was saying. But now that's water under the bridge. Let's move forward. Not trying to be an a** :) Thanks for finishing my idea there [~ram_krish]! > Row-level consistency is broken for read > > > Key: HBASE-17887 > URL: https://issues.apache.org/jira/browse/HBASE-17887 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0, 1.3.0 >Reporter: Umesh Agashe >Assignee: Chia-Ping Tsai >Priority: Blocker > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-17887.branch-1.v0.patch, > HBASE-17887.branch-1.v1.patch, HBASE-17887.branch-1.v1.patch, > HBASE-17887.branch-1.v2.patch, HBASE-17887.branch-1.v2.patch, > HBASE-17887.branch-1.v3.patch, HBASE-17887.branch-1.v4.patch, > HBASE-17887.branch-1.v4.patch, HBASE-17887.branch-1.v4.patch, > HBASE-17887.branch-1.v5.patch, HBASE-17887.branch-1.v6.patch, > HBASE-17887.ut.patch, HBASE-17887.v0.patch, HBASE-17887.v1.patch, > HBASE-17887.v2.patch, HBASE-17887.v3.patch, HBASE-17887.v4.patch, > HBASE-17887.v5.patch, HBASE-17887.v5.patch > > > The scanner of latest memstore may be lost if we make quick flushes. The > following step may help explain this issue. > # put data_A (seq id = 10, active store data_A and snapshots is empty) > # snapshot of 1st flush (active is empty and snapshot stores data_A) > # put data_B (seq id = 11, active store data_B and snapshot store data_A) > # create user scanner (read point = 11, so It should see the data_B) > # commit of 1st flush > #* clear snapshot ((hfile_A has data_A, active store data_B, and snapshot is > empty) > #* update the reader (the user scanner receives the hfile_A) > # snapshot of 2st flush (active is empty and snapshot store data_B) > # commit of 2st flush > #* clear snapshot (hfile_A has data_A, hfile_B has data_B, active is empty, > and snapshot is empty) – this is critical piece. > #* -update the reader- (haven't happen) > # user scanner update the kv scanners (it creates scanner of hfile_A but > nothing of memstore) > # user see the older data A – wrong result -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits
[
https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009513#comment-16009513
]
Lars Hofhansl commented on HBASE-18027:
---
I also notice that that readAllEntriesToReplicateOrNextFile calculates the size
this way (in 1.3.x):
{code}
WAL.Entry entry = ...;
...
currentSize += entry.getEdit().heapSize();
currentSize += calculateTotalSizeOfStoreFiles(edit);
{code}
Perhaps that may be the discrepancy...? (and the fact that we check after we
added the entry - as you point out)
We can do this patch of course. But I do think it'd be simpler and easier to
follow/change later if we fix it in the caller and don't introduce another loop
inside the sending code.
> HBaseInterClusterReplicationEndpoint should respect RPC size limits when
> batching edits
> ---
>
> Key: HBASE-18027
> URL: https://issues.apache.org/jira/browse/HBASE-18027
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-18027-branch-1.patch, HBASE-18027-branch-1.patch,
> HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch
>
>
> In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in
> batches. We create N lists. N is the minimum of configured replicator
> threads, number of 100-waledit batches, or number of current sinks. Every
> pending entry in the replication context is then placed in order by hash of
> encoded region name into one of these N lists. Each of the N lists is then
> sent all at once in one replication RPC. We do not test if the sum of data in
> each N list will exceed RPC size limits. This code presumes each individual
> edit is reasonably small. Not checking for aggregate size while assembling
> the lists into RPCs is an oversight and can lead to replication failure when
> that assumption is violated.
> We can fix this by generating as many replication RPC calls as we need to
> drain a list, keeping each RPC under limit, instead of assuming the whole
> list will fit in one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18027) HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009511#comment-16009511 ] Lars Hofhansl commented on HBASE-18027: --- bq. The worklist handed to HICRE#Replicator can exceed the RPC limit so we break it into separate RPCs But how would it do that? That can only happen when the last edit in the batch is huge (so it would be batch-size-limit + size-of-last-edit). The initial design was for simplicity, so that only the outer code needs to do any bookkeeping (like progress in ZK). Lemme look at the patch a little closer. Offhand I'd be more a fan of handling this in the calling code. > HBaseInterClusterReplicationEndpoint should respect RPC size limits when > batching edits > --- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027-branch-1.patch, HBASE-18027-branch-1.patch, > HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[ https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009510#comment-16009510 ] Lars Hofhansl commented on HBASE-18043: --- Still +1 :) > Institute a hard limit for individual cell size that cannot be overridden by > clients > > > Key: HBASE-18043 > URL: https://issues.apache.org/jira/browse/HBASE-18043 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, > HBASE-18043-branch-1.patch, HBASE-18043.patch, HBASE-18043.patch, > HBASE-18043.patch, HBASE-18043.patch > > > For sake of service protection we should not give absolute trust to clients > regarding resource limits that can impact stability, like cell size limits. > We should add a server side configuration that sets a hard limit for > individual cell size that cannot be overridden by the client. We can keep the > client side check, because it's expensive to reject a RPC that has already > come in. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009136#comment-16009136 ] Lars Hofhansl commented on HBASE-18027: --- (And hence perhaps this is just checking the the replication batch size limit is <= the RPC size limit) > HBaseInterClusterReplicationEndpoint should respect RPC size limits when > batching edits > --- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027-branch-1.patch, HBASE-18027.patch, > HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[ https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009135#comment-16009135 ] Lars Hofhansl commented on HBASE-18043: --- Thanks! +1 > Institute a hard limit for individual cell size that cannot be overridden by > clients > > > Key: HBASE-18043 > URL: https://issues.apache.org/jira/browse/HBASE-18043 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, > HBASE-18043.patch, HBASE-18043.patch > > > For sake of service protection we should not give absolute trust to clients > regarding resource limits that can impact stability, like cell size limits. > We should add a server side configuration that sets a hard limit for > individual cell size that cannot be overridden by the client. We can keep the > client side check, because it's expensive to reject a RPC that has already > come in. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18027) HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits
[
https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009134#comment-16009134
]
Lars Hofhansl edited comment on HBASE-18027 at 5/13/17 4:47 AM:
So looking at the code... In the original code I assume that the caller do the
size enforcement.
And indeed I see that happening in the code.
{{HBaseInterClusterReplicationEndpoint.replicate}} is called from
{{ReplicationSourceWorkerThread.shipEdits}}, which is called from
{{ReplicationSourceWorkerThread.run}} after the call to
{{ReplicationSourceWorkerThread.readAllEntriesToReplicateOrNextFile}} which
reads the next batch _and_ - crucially - enforces the replication batch size
limit. So any single batch issued from within {{replicate}} cannot be larger
than the overall batch size enforced (which defaults to 64MB).
So I don't seen how this cause a problem (but as usually, it is entirely
possible that I missed a piece of the puzzle here)
was (Author: lhofhansl):
So looking at the code... In the original code I assume that the caller do the
size enforcement.
And indeed I see that happening in the code.
{{HBaseInterClusterReplicationEndpoint.replicate}} is called from
{{ReplicationSourceWorkerThread.shipEdits}}, which is called from
{{ReplicationSourceWorkerThread.run}} after the call to
{{ReplicationSourceWorkerThread.readAllEntriesToReplicateOrNextFile}} which
reads the next batch _and_ crucially enforces the replication batch size limit.
So any single batch issued from within {{replicate}} can be larger than the
overall batch size enforced (which defaults to 64MB).
So I don't seen how this cause a problem (but as usually, it is entirely
possible that I missed a piece of the puzzle here)
> HBaseInterClusterReplicationEndpoint should respect RPC size limits when
> batching edits
> ---
>
> Key: HBASE-18027
> URL: https://issues.apache.org/jira/browse/HBASE-18027
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-18027-branch-1.patch, HBASE-18027.patch,
> HBASE-18027.patch
>
>
> In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in
> batches. We create N lists. N is the minimum of configured replicator
> threads, number of 100-waledit batches, or number of current sinks. Every
> pending entry in the replication context is then placed in order by hash of
> encoded region name into one of these N lists. Each of the N lists is then
> sent all at once in one replication RPC. We do not test if the sum of data in
> each N list will exceed RPC size limits. This code presumes each individual
> edit is reasonably small. Not checking for aggregate size while assembling
> the lists into RPCs is an oversight and can lead to replication failure when
> that assumption is violated.
> We can fix this by generating as many replication RPC calls as we need to
> drain a list, keeping each RPC under limit, instead of assuming the whole
> list will fit in one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18027) HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits
[
https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009134#comment-16009134
]
Lars Hofhansl commented on HBASE-18027:
---
So looking at the code... In the original code I assume that the caller do the
size enforcement.
And indeed I see that happening in the code.
{{HBaseInterClusterReplicationEndpoint.replicate}} is called from
{{ReplicationSourceWorkerThread.shipEdits}}, which is called from
{{ReplicationSourceWorkerThread.run}} after the call to
{{ReplicationSourceWorkerThread.readAllEntriesToReplicateOrNextFile}} which
reads the next batch _and_ crucially enforces the replication batch size limit.
So any single batch issued from within {{replicate}} can be larger than the
overall batch size enforced (which defaults to 64MB).
So I don't seen how this cause a problem (but as usually, it is entirely
possible that I missed a piece of the puzzle here)
> HBaseInterClusterReplicationEndpoint should respect RPC size limits when
> batching edits
> ---
>
> Key: HBASE-18027
> URL: https://issues.apache.org/jira/browse/HBASE-18027
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 2.0.0, 1.4.0, 1.3.2
>
> Attachments: HBASE-18027-branch-1.patch, HBASE-18027.patch,
> HBASE-18027.patch
>
>
> In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in
> batches. We create N lists. N is the minimum of configured replicator
> threads, number of 100-waledit batches, or number of current sinks. Every
> pending entry in the replication context is then placed in order by hash of
> encoded region name into one of these N lists. Each of the N lists is then
> sent all at once in one replication RPC. We do not test if the sum of data in
> each N list will exceed RPC size limits. This code presumes each individual
> edit is reasonably small. Not checking for aggregate size while assembling
> the lists into RPCs is an oversight and can lead to replication failure when
> that assumption is violated.
> We can fix this by generating as many replication RPC calls as we need to
> drain a list, keeping each RPC under limit, instead of assuming the whole
> list will fit in one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18027) HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009126#comment-16009126 ] Lars Hofhansl commented on HBASE-18027: --- Looking > HBaseInterClusterReplicationEndpoint should respect RPC size limits when > batching edits > --- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027-branch-1.patch, HBASE-18027.patch, > HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17887) Row-level consistency is broken for read
[ https://issues.apache.org/jira/browse/HBASE-17887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009074#comment-16009074 ] Lars Hofhansl commented on HBASE-17887: --- Wow. Changes like HBASE-14970 and this one really do not belong into a minor release of HBase. This is scary stuff. I cannot convince myself from just looking at the code that these two do not introduce more subtle issues. Sorry for the whining. :) Of course this is great stuff. Just that in hindsight it might have been better to be contained in 2.x. [~apurtell], for your consideration w.r.t. to our planned upgrade to 1.3.x. > Row-level consistency is broken for read > > > Key: HBASE-17887 > URL: https://issues.apache.org/jira/browse/HBASE-17887 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0, 1.3.0 >Reporter: Umesh Agashe >Assignee: Chia-Ping Tsai >Priority: Blocker > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-17887.branch-1.v0.patch, > HBASE-17887.branch-1.v1.patch, HBASE-17887.branch-1.v1.patch, > HBASE-17887.branch-1.v2.patch, HBASE-17887.branch-1.v2.patch, > HBASE-17887.branch-1.v3.patch, HBASE-17887.branch-1.v4.patch, > HBASE-17887.branch-1.v4.patch, HBASE-17887.branch-1.v4.patch, > HBASE-17887.branch-1.v5.patch, HBASE-17887.branch-1.v6.patch, > HBASE-17887.ut.patch, HBASE-17887.v0.patch, HBASE-17887.v1.patch, > HBASE-17887.v2.patch, HBASE-17887.v3.patch, HBASE-17887.v4.patch, > HBASE-17887.v5.patch, HBASE-17887.v5.patch > > > The scanner of latest memstore may be lost if we make quick flushes. The > following step may help explain this issue. > # put data_A (seq id = 10, active store data_A and snapshots is empty) > # snapshot of 1st flush (active is empty and snapshot stores data_A) > # put data_B (seq id = 11, active store data_B and snapshot store data_A) > # create user scanner (read point = 11, so It should see the data_B) > # commit of 1st flush > #* clear snapshot ((hfile_A has data_A, active store data_B, and snapshot is > empty) > #* update the reader (the user scanner receives the hfile_A) > # snapshot of 2st flush (active is empty and snapshot store data_B) > # commit of 2st flush > #* clear snapshot (hfile_A has data_A, hfile_B has data_B, active is empty, > and snapshot is empty) – this is critical piece. > #* -update the reader- (haven't happen) > # user scanner update the kv scanners (it creates scanner of hfile_A but > nothing of memstore) > # user see the older data A – wrong result -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[ https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008936#comment-16008936 ] Lars Hofhansl commented on HBASE-18043: --- Looks good. Should we add a test with Put size of 9K, or 9.5K, to make sure the size estimation is not completely off? > Institute a hard limit for individual cell size that cannot be overridden by > clients > > > Key: HBASE-18043 > URL: https://issues.apache.org/jira/browse/HBASE-18043 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18043-branch-1.patch, HBASE-18043.patch > > > For sake of service protection we should not give absolute trust to clients > regarding resource limits that can impact stability, like cell size limits. > We should add a server side configuration that sets a hard limit for > individual cell size that cannot be overridden by the client. We can keep the > client side check, because it's expensive to reject a RPC that has already > come in. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18000) Make sure we always return the scanner id with ScanResponse
[ https://issues.apache.org/jira/browse/HBASE-18000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998561#comment-15998561 ] Lars Hofhansl commented on HBASE-18000: --- [~karanmehta93], can't retroactively change a release :) We could do a 1.3.1.1 or 1.3.2. [~Apache9], thanks for looking at it quickly! I think that's all it takes. +1 [~apurtell], FYI. Needed for AsyncHBase/OpenTSDB with HBase 1.3.1. > Make sure we always return the scanner id with ScanResponse > --- > > Key: HBASE-18000 > URL: https://issues.apache.org/jira/browse/HBASE-18000 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Lars Hofhansl >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18000.patch > > > Some external tooling (like OpenTSDB) relies on the scanner id to tie > asynchronous responses back to their requests. > (see comments on HBASE-17489) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18000) Make sure we always return the scanner id with ScanResponse
[ https://issues.apache.org/jira/browse/HBASE-18000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-18000: -- Affects Version/s: 1.3.1 > Make sure we always return the scanner id with ScanResponse > --- > > Key: HBASE-18000 > URL: https://issues.apache.org/jira/browse/HBASE-18000 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1 >Reporter: Lars Hofhansl > > Some external tooling (like OpenTSDB) relies on the scanner id to tie > asynchronous responses back to their requests. > (see comments on HBASE-17489) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18000) Make sure we always return the scanner id with ScanResponse
[ https://issues.apache.org/jira/browse/HBASE-18000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997824#comment-15997824 ] Lars Hofhansl commented on HBASE-18000: --- Perhaps it's as simple to just builder.setScannerId back into the Scan methods (RSRcpServices). > Make sure we always return the scanner id with ScanResponse > --- > > Key: HBASE-18000 > URL: https://issues.apache.org/jira/browse/HBASE-18000 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > > Some external tooling (like OpenTSDB) relies on the scanner id to tie > asynchronous responses back to their requests. > (see comments on HBASE-17489) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997822#comment-15997822 ] Lars Hofhansl commented on HBASE-17489: --- Thanks [~Apache9]. Filed HBASE-18000 (wow 18k jiras!!) > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18000) Make sure we always return the scanner id with ScanResponse
Lars Hofhansl created HBASE-18000: - Summary: Make sure we always return the scanner id with ScanResponse Key: HBASE-18000 URL: https://issues.apache.org/jira/browse/HBASE-18000 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Some external tooling (like OpenTSDB) relies on the scanner id to tie asynchronous responses back to their requests. (see comments on HBASE-17489) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997677#comment-15997677 ] Lars Hofhansl commented on HBASE-17489: --- Hey [~Apache9]. Quick question: It seems that now we only set the scanner id in the ScanResponse when we actually created a new region scanner (but not on next or close). Is that by design? There is a problem with this now with OpenTSBD (well AsyncHBase) which ties requests and responses together using the scanner id. > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989763#comment-15989763 ] Lars Hofhansl commented on HBASE-17917: --- The title does not quite reflect the patch, since it does try to switch to streaming reads. Couple of question: # did you try with many store files and with many column families. preads even when followed by a switch to streaming involve a lot of seeking. # which version of HDFS did you test this against? Does that version have HDFS-6735 fixed? # what's the load on the data nodes that you observe? > Use pread by default for all user scan > -- > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17917.patch, HBASE-17917-v1.patch, > HBASE-17917-v2.patch, HBASE-17917-v2.patch, HBASE-17917-v3.patch, > HBASE-17917-v4.patch, HBASE-17917-v5.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987599#comment-15987599 ] Lars Hofhansl commented on HBASE-17877: --- Thanks for the patches [~vik.karma]! > Improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.branch-1.3.003.patch, HBASE-17877.master.001.patch, > HBASE-17877.master.002.patch, HBASE-17877.master.003.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17877) Improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-17877: -- Resolution: Fixed Fix Version/s: 1.3.2 1.4.0 2.0.0 Status: Resolved (was: Patch Available) Committed to branch-1.3, branch-1, and master. > Improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.branch-1.3.003.patch, HBASE-17877.master.001.patch, > HBASE-17877.master.002.patch, HBASE-17877.master.003.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17877) Improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-17877: -- Summary: Improve HBase's byte[] comparator (was: Replace/improve HBase's byte[] comparator) > Improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.branch-1.3.003.patch, HBASE-17877.master.001.patch, > HBASE-17877.master.002.patch, HBASE-17877.master.003.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986033#comment-15986033 ] Lars Hofhansl commented on HBASE-17877: --- +1 Unless I hear objections I will commit to branch-1.3, branch-1, and master. (assuming branch-2 is not real) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.branch-1.3.003.patch, HBASE-17877.master.001.patch, > HBASE-17877.master.002.patch, HBASE-17877.master.003.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17958) Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
[
https://issues.apache.org/jira/browse/HBASE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986024#comment-15986024
]
Lars Hofhansl commented on HBASE-17958:
---
bq. Can you imagine that, you call a seek on an InputStream, and the
InputStream may do nothing and still return the next byte to you and you need
to guess whether it really done the seek?
This is different. What you get from an InputStream does not have an id. Here
we do (the key of the Cell). An id that the ColumnTrackers later verifies again
anyway.
Anyway. The patch looks good, except for the potential extra compare for SKIPed
columns and/or rows.
> Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
>
>
> Key: HBASE-17958
> URL: https://issues.apache.org/jira/browse/HBASE-17958
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: 0001-add-one-ut-testWithColumnCountGetFilter.patch,
> HBASE-17958-v1.patch, HBASE-17958-v2.patch, HBASE-17958-v3.patch
>
>
> {code}
> ScanQueryMatcher.MatchCode qcode = matcher.match(cell);
> qcode = optimize(qcode, cell);
> {code}
> The optimize method may change the MatchCode from SEEK_NEXT_COL/SEEK_NEXT_ROW
> to SKIP. But it still pass the next cell to ScanQueryMatcher. It will get
> wrong result when use some filter, etc. ColumnCountGetFilter. It just count
> the columns's number. If pass a same column to this filter, the count result
> will be wrong. So we should avoid passing cell to ScanQueryMatcher when
> optimize SEEK to SKIP.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17958) Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
[
https://issues.apache.org/jira/browse/HBASE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986019#comment-15986019
]
Lars Hofhansl commented on HBASE-17958:
---
Looks good generally.
Are you sure we're not introducing at least one extra compare for each SKIP?
We're comparing now in optimizeSeekToSkip and the ColumnTrackers will compare
again before declaring a column/row as done.
> Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
>
>
> Key: HBASE-17958
> URL: https://issues.apache.org/jira/browse/HBASE-17958
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: 0001-add-one-ut-testWithColumnCountGetFilter.patch,
> HBASE-17958-v1.patch, HBASE-17958-v2.patch, HBASE-17958-v3.patch
>
>
> {code}
> ScanQueryMatcher.MatchCode qcode = matcher.match(cell);
> qcode = optimize(qcode, cell);
> {code}
> The optimize method may change the MatchCode from SEEK_NEXT_COL/SEEK_NEXT_ROW
> to SKIP. But it still pass the next cell to ScanQueryMatcher. It will get
> wrong result when use some filter, etc. ColumnCountGetFilter. It just count
> the columns's number. If pass a same column to this filter, the count result
> will be wrong. So we should avoid passing cell to ScanQueryMatcher when
> optimize SEEK to SKIP.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986000#comment-15986000 ] Lars Hofhansl commented on HBASE-17917: --- Nice > Use pread by default for all user scan > -- > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17917.patch, HBASE-17917-v1.patch, > HBASE-17917-v2.patch, HBASE-17917-v2.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17958) Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
[
https://issues.apache.org/jira/browse/HBASE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983966#comment-15983966
]
Lars Hofhansl commented on HBASE-17958:
---
The problem is the order in which the filters and column trackers are executed.
The column trackers do their own comparisons and do the right thing if a SKIP
bubbles up (checked that carefully) - unless a filter interferes.
On the other hand there _is no_ right order between the trackers and the
filters. Sometimes you want one order, sometimes the other.
If in the SKIP case we call next until we hit the next row or column, we're
doing the trackers' job.
Hmm... I guess there has to be some acceptance that filters are a fairly low
level concept and expose you to some of the HBase inerts.
Especially counting filters are problematic since rows may have different
number of columns and columns have varying numbers of versions.
> Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
>
>
> Key: HBASE-17958
> URL: https://issues.apache.org/jira/browse/HBASE-17958
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>
> {code}
> ScanQueryMatcher.MatchCode qcode = matcher.match(cell);
> qcode = optimize(qcode, cell);
> {code}
> The optimize method may change the MatchCode from SEEK_NEXT_COL/SEEK_NEXT_ROW
> to SKIP. But it still pass the next cell to ScanQueryMatcher. It will get
> wrong result when use some filter, etc. ColumnCountGetFilter. It just count
> the columns's number. If pass a same column to this filter, the count result
> will be wrong. So we should avoid passing cell to ScanQueryMatcher when
> optimize SEEK to SKIP.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17958) Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
[
https://issues.apache.org/jira/browse/HBASE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983320#comment-15983320
]
Lars Hofhansl commented on HBASE-17958:
---
It is expected that the column trackers so the right thing. Filters that count
the columns might be indeed be a problem, though. We should look at the column
trackers in b this case.
(I do object slightly to "this is really a bad practise", though. Seeking is
unduly expensive in HBase, especially when the seek just lands is on the same
block anyway.)
> Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
>
>
> Key: HBASE-17958
> URL: https://issues.apache.org/jira/browse/HBASE-17958
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>
> {code}
> ScanQueryMatcher.MatchCode qcode = matcher.match(cell);
> qcode = optimize(qcode, cell);
> {code}
> The optimize method may change the MatchCode from SEEK_NEXT_COL/SEEK_NEXT_ROW
> to SKIP. But it still pass the next cell to ScanQueryMatcher. It will get
> wrong result when use some filter, etc. ColumnCountGetFilter. It just count
> the columns's number. If pass a same column to this filter, the count result
> will be wrong. So we should avoid passing cell to ScanQueryMatcher when
> optimize SEEK to SKIP.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17958) Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
[
https://issues.apache.org/jira/browse/HBASE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983308#comment-15983308
]
Lars Hofhansl commented on HBASE-17958:
---
Do you guys have a unit test to show this?
> Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
>
>
> Key: HBASE-17958
> URL: https://issues.apache.org/jira/browse/HBASE-17958
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>
> {code}
> ScanQueryMatcher.MatchCode qcode = matcher.match(cell);
> qcode = optimize(qcode, cell);
> {code}
> The optimize method may change the MatchCode from SEEK_NEXT_COL/SEEK_NEXT_ROW
> to SKIP. But it still pass the next cell to ScanQueryMatcher. It will get
> wrong result when use some filter, etc. ColumnCountGetFilter. It just count
> the columns's number. If pass a same column to this filter, the count result
> will be wrong. So we should avoid passing cell to ScanQueryMatcher when
> optimize SEEK to SKIP.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982345#comment-15982345 ] Lars Hofhansl commented on HBASE-17877: --- [~vik.karma] if you have time, sure. Happy to do it too. (credit still all goes to you :) ) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982114#comment-15982114 ] Lars Hofhansl commented on HBASE-17877: --- [~busbey], you are right. We looked at Guava's master branch, but it's like this in v21 as well. Let's change that and the NOTICE file. I can do that and then commit to 1.3.x and following. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17448) Export metrics from RecoverableZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-17448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980275#comment-15980275 ] Lars Hofhansl commented on HBASE-17448: --- lgtm > Export metrics from RecoverableZooKeeper > > > Key: HBASE-17448 > URL: https://issues.apache.org/jira/browse/HBASE-17448 > Project: HBase > Issue Type: Improvement > Components: Zookeeper >Affects Versions: 1.3.1 >Reporter: Andrew Purtell >Assignee: Chinmay Kulkarni > Labels: patch > Fix For: 1.3.2 > > Attachments: HBASE-17448.patch > > > Consider adding instrumentation to RecoverableZooKeeper that exposes metrics > on the performance and health of the embedded ZooKeeper client: latency > histograms for each op type, number of reconnections, number of ops where a > reconnection was necessary to proceed, number of failed ops due to > CONNECTIONLOSS, number of failed ops due to SESSIONEXIPRED, number of failed > ops due to OPERATIONTIMEOUT. > RecoverableZooKeeper is a class in hbase-client so we can hook up the new > metrics to both client- and server-side metrics reporters. Probably this > metrics source should be a process singleton. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980272#comment-15980272 ] Lars Hofhansl commented on HBASE-17877: --- If there're no further comments... Good to commit? All branches? [~busbey], [~mantonov]? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973407#comment-15973407 ] Lars Hofhansl commented on HBASE-17937: --- +1 on branch-1 patch. (nice to get rid of the call in the finally block!) For the master patch, shouldn't we also move the size update after the sync of the wal? [~Apache9], [~abhishek.chouhan] > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, > HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973189#comment-15973189 ] Lars Hofhansl edited comment on HBASE-17937 at 4/18/17 6:06 PM: Agreed. I think it's just [~abhishek.chouhan]'s fix, with the first size update moved right after the logsync. was (Author: lhofhansl): Agreed. I think it's just [~abhishek.chouhan]'s fix, which the first size update move right after the logsync. > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.master.001.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973189#comment-15973189 ] Lars Hofhansl commented on HBASE-17937: --- Agreed. I think it's just [~abhishek.chouhan]'s fix, which the first size update move right after the logsync. > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.master.001.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973163#comment-15973163 ] Lars Hofhansl commented on HBASE-17937: --- So we looked at the code here. I think this is only a temporary condition like this: 1. doMiniBatchMutation starts and writes things into the memstore 2. the background flusher flushes the memstore. Decrementing the memstore size. 3. doMiniBatchMutation finishes. Incrementing the memstore size. So we only have an issue between #2 and #3. When doMiniBatchMutation finishes the size will be correct. I do _not_ think it's a big problem, just a spurious log warning. If we did not want that, we should move the incrementing of the memstore size right after the sync of the wal in doMiniBatchMutation (inside the MVCC transaction). > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.master.001.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972129#comment-15972129 ] Lars Hofhansl commented on HBASE-17877: --- Should we add a little comment about how most of this code was lifted from Guava? (and perhaps state the version, in case Guava ever changes it's license - unlikely, but whatever) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.master.001.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972127#comment-15972127 ] Lars Hofhansl commented on HBASE-17877: --- I like -v4! > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.master.001.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965441#comment-15965441 ] Lars Hofhansl edited comment on HBASE-17877 at 4/12/17 6:21 AM: Also looking at the code (with offset, as in Hadoop and HBase) I wonder how efficient it is to call unsafe.getLong on an unaligned address (i.e. not passing multiples of 8 as offset). Would it perhaps be better to read the first few bytes (if not aligned) and then switch to 8 bytes processing once we reach an aligned address? was (Author: lhofhansl): Also looking at the code (with offset, as in Hadoop and HBase) I wonder how efficient it is to call unsafe.getLong on an unaligned address (i.e. passing 1-7 as offset). Would it perhaps be better to read the first few bytes (if not aligned) and then switch to 8 bytes processing once we reach an aligned address? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965441#comment-15965441 ] Lars Hofhansl commented on HBASE-17877: --- Also looking at the code (with offset, as in Hadoop and HBase) I wonder how efficient it is to call unsafe.getLong on an unaligned address (i.e. passing 1-7 as offset). Would it perhaps be better to read the first few bytes (if not aligned) and then switch to 8 bytes processing once we reach an aligned address? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965426#comment-15965426 ] Lars Hofhansl commented on HBASE-17877: --- Hey [~vik.karma], I notice the Guava version does not allow passing in offset and length. Did you adapt that, or did you test against the API that just takes two byte[]'s? (I'd assume that'd be slightly faster) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965409#comment-15965409 ] Lars Hofhansl commented on HBASE-17877: --- I like the latest Guava code best anyway. Easier to understand. We can pull that into HBase (but remove Guava dependencies :) ) Want me to do that [~vik.karma] (or wanna do a patch yourself)? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17893) Allow HBase to build against Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HBASE-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965403#comment-15965403
]
Lars Hofhansl commented on HBASE-17893:
---
I'll defer to your judgment. :)
Thanks for taking a look!
Note that I tried the debug stuff from HBASE-16712 (i.e. having the NOTICE.vm
template fail on unknown licenses). Still fails in LICENSE, and still without
any indication as what went wrong.
> Allow HBase to build against Hadoop 2.8.0
> -
>
> Key: HBASE-17893
> URL: https://issues.apache.org/jira/browse/HBASE-17893
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.5
>Reporter: Lars Hofhansl
> Attachments: 17883-1.2-BROKEN.txt
>
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hbase-assembly: Error rendering velocity resource. Error invoking
> method 'get(java.lang.Integer)' in java.util.ArrayList at
> META-INF/LICENSE.vm[line 1671, column 8]: InvocationTargetException: Index:
> 0, Size: 0 -> [Help 1]
> {code}
> The in the generated LICENSE.
> {code}
> This product includes Nimbus JOSE+JWT licensed under the The Apache Software
> License, Version 2.0.
> ${dep.licenses[0].comments}
> Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> com.nimbusds
> nimbus-jose-jwt
> 3.9
> maven central search
> g:com.nimbusds AND a:nimbus-jose-jwt AND v:3.9
> project website
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> project source
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> {code}
> Maybe the problem is just that it says: Apache _Software_ License
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17893) Allow HBase to build against Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HBASE-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-17893:
--
Attachment: 17883-1.2-BROKEN.txt
This adds the 3 missing dependencies.
But... Build still fails:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on
project hbase-assembly: Error rendering velocity resource. Error invoking
method 'get(java.lang.Integer)' in java.util.ArrayList at
META-INF/LICENSE.vm[line 1623, column 22]: InvocationTargetException: Index: 0,
Size: 0 -> [Help 1]
{code}
That gets me past this problem.
But ... now there's some other failure and the generated LICENSE file shows no
errors/messages anymore.
> Allow HBase to build against Hadoop 2.8.0
> -
>
> Key: HBASE-17893
> URL: https://issues.apache.org/jira/browse/HBASE-17893
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.5
>Reporter: Lars Hofhansl
> Attachments: 17883-1.2-BROKEN.txt
>
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hbase-assembly: Error rendering velocity resource. Error invoking
> method 'get(java.lang.Integer)' in java.util.ArrayList at
> META-INF/LICENSE.vm[line 1671, column 8]: InvocationTargetException: Index:
> 0, Size: 0 -> [Help 1]
> {code}
> The in the generated LICENSE.
> {code}
> This product includes Nimbus JOSE+JWT licensed under the The Apache Software
> License, Version 2.0.
> ${dep.licenses[0].comments}
> Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> com.nimbusds
> nimbus-jose-jwt
> 3.9
> maven central search
> g:com.nimbusds AND a:nimbus-jose-jwt AND v:3.9
> project website
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> project source
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> {code}
> Maybe the problem is just that it says: Apache _Software_ License
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962420#comment-15962420 ] Lars Hofhansl commented on HBASE-17877: --- Hi [~vik.karma], none of the scenarios in the table seems 30-40% slower. Is that a different scenario you are referring to in the above text? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17893) Allow HBase to build against Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HBASE-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-17893:
--
Affects Version/s: 1.2.5
> Allow HBase to build against Hadoop 2.8.0
> -
>
> Key: HBASE-17893
> URL: https://issues.apache.org/jira/browse/HBASE-17893
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.5
>Reporter: Lars Hofhansl
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hbase-assembly: Error rendering velocity resource. Error invoking
> method 'get(java.lang.Integer)' in java.util.ArrayList at
> META-INF/LICENSE.vm[line 1671, column 8]: InvocationTargetException: Index:
> 0, Size: 0 -> [Help 1]
> {code}
> The in the generated LICENSE.
> {code}
> This product includes Nimbus JOSE+JWT licensed under the The Apache Software
> License, Version 2.0.
> ${dep.licenses[0].comments}
> Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> com.nimbusds
> nimbus-jose-jwt
> 3.9
> maven central search
> g:com.nimbusds AND a:nimbus-jose-jwt AND v:3.9
> project website
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> project source
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> {code}
> Maybe the problem is just that it says: Apache _Software_ License
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17893) Allow HBase to build against Hadoop 2.8.0
[
https://issues.apache.org/jira/browse/HBASE-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961659#comment-15961659
]
Lars Hofhansl commented on HBASE-17893:
---
[~apurtell], [~busbey]. The fix is probably simple, but all that license stuff
is a big jumble to me.
> Allow HBase to build against Hadoop 2.8.0
> -
>
> Key: HBASE-17893
> URL: https://issues.apache.org/jira/browse/HBASE-17893
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.5
>Reporter: Lars Hofhansl
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hbase-assembly: Error rendering velocity resource. Error invoking
> method 'get(java.lang.Integer)' in java.util.ArrayList at
> META-INF/LICENSE.vm[line 1671, column 8]: InvocationTargetException: Index:
> 0, Size: 0 -> [Help 1]
> {code}
> The in the generated LICENSE.
> {code}
> This product includes Nimbus JOSE+JWT licensed under the The Apache Software
> License, Version 2.0.
> ${dep.licenses[0].comments}
> Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> com.nimbusds
> nimbus-jose-jwt
> 3.9
> maven central search
> g:com.nimbusds AND a:nimbus-jose-jwt AND v:3.9
> project website
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> project source
> https://bitbucket.org/connect2id/nimbus-jose-jwt
> {code}
> Maybe the problem is just that it says: Apache _Software_ License
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17893) Allow HBase to build against Hadoop 2.8.0
Lars Hofhansl created HBASE-17893:
-
Summary: Allow HBase to build against Hadoop 2.8.0
Key: HBASE-17893
URL: https://issues.apache.org/jira/browse/HBASE-17893
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on
project hbase-assembly: Error rendering velocity resource. Error invoking
method 'get(java.lang.Integer)' in java.util.ArrayList at
META-INF/LICENSE.vm[line 1671, column 8]: InvocationTargetException: Index: 0,
Size: 0 -> [Help 1]
{code}
The in the generated LICENSE.
{code}
This product includes Nimbus JOSE+JWT licensed under the The Apache Software
License, Version 2.0.
${dep.licenses[0].comments}
Please check this License for acceptability here:
https://www.apache.org/legal/resolved
If it is okay, then update the list named 'non_aggregate_fine' in the
LICENSE.vm file.
If it isn't okay, then revert the change that added the dependency.
More info on the dependency:
com.nimbusds
nimbus-jose-jwt
3.9
maven central search
g:com.nimbusds AND a:nimbus-jose-jwt AND v:3.9
project website
https://bitbucket.org/connect2id/nimbus-jose-jwt
project source
https://bitbucket.org/connect2id/nimbus-jose-jwt
{code}
Maybe the problem is just that it says: Apache _Software_ License
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961633#comment-15961633 ] Lars Hofhansl commented on HBASE-17877: --- OK... Waiting for one more test from [~vik.karma] (updated patch and a few million iteration to get the volatility of the tests down) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961631#comment-15961631 ] Lars Hofhansl commented on HBASE-17877: --- Just be complete. Here's the plain Gauva version: https://github.com/google/guava/blob/master/guava/src/com/google/common/primitives/UnsignedBytes.java#L362 > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-9739) HBaseClient does not behave nicely when the called thread is interrupted
[
https://issues.apache.org/jira/browse/HBASE-9739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl resolved HBASE-9739.
--
Resolution: Won't Fix
Old issue. No update. Closing.
> HBaseClient does not behave nicely when the called thread is interrupted
>
>
> Key: HBASE-9739
> URL: https://issues.apache.org/jira/browse/HBASE-9739
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>
> Just ran into a scenario where HBaseClient became permanently useless after
> we interrupted the using thread.
> The problem is here:
> {code}
> } catch(IOException e) {
> markClosed(e);
> {code}
> In sendParam(...).
> If the IOException is caused by an interrupt we should not close the
> connection.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-10944) Remove all kv.getBuffer() and kv.getRow() references existing in the code
[ https://issues.apache.org/jira/browse/HBASE-10944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961229#comment-15961229 ] Lars Hofhansl commented on HBASE-10944: --- How about we get rid of ensureKeyValue? Right now we're in a bad mess: We have Cells, but not quite, and in important places we still assume Cells backed by a single byte[], and the we force the single byte[] all over the place. I'll still vote for getting rid of getBuffer() there should be absolutely no requirement to have a Cell backed by a single byte[]. > Remove all kv.getBuffer() and kv.getRow() references existing in the code > - > > Key: HBASE-10944 > URL: https://issues.apache.org/jira/browse/HBASE-10944 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.5.0 > > > kv.getRow() and kv.getBuffers() are still used in places to form key byte[] > and row byte[]. Removing all such instances including testcases will make > the usage of Cell complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-10145) Table creation should proceed in the presence of a stale znode
[ https://issues.apache.org/jira/browse/HBASE-10145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-10145. --- Resolution: Won't Fix Closing old issue. > Table creation should proceed in the presence of a stale znode > -- > > Key: HBASE-10145 > URL: https://issues.apache.org/jira/browse/HBASE-10145 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Minor > > HBASE-7600 fixed a race condition where concurrent attempts to create the > same table could succeed. > An unfortunate side effect is that it is now impossible to create a table as > long as the table's znode is around, which is an issue when a cluster was > wiped at the HDFS level. > Minor issue as we have discussed this many times before, but it ought to be > possible to check whether the table directory exists and if not either create > it or remove the corresponding znode. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-10028) Cleanup metrics documentation
[ https://issues.apache.org/jira/browse/HBASE-10028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-10028. --- Resolution: Won't Fix Closing old issue. > Cleanup metrics documentation > - > > Key: HBASE-10028 > URL: https://issues.apache.org/jira/browse/HBASE-10028 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > > The current documentation of the metrics is incomplete and at point incorrect > (HDFS latencies are in ns rather than ms for example). > We should clean this up and add other related metrics as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-6492) Remove Reflection based Hadoop abstractions
[ https://issues.apache.org/jira/browse/HBASE-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-6492. -- Resolution: Won't Fix Closing old issue. > Remove Reflection based Hadoop abstractions > --- > > Key: HBASE-6492 > URL: https://issues.apache.org/jira/browse/HBASE-6492 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl > > In 0.96 we now have the Hadoop1-compat and Hadoop2-compat projects. > The reflection we're using to deal with different versions of Hadoop should > be removed in favour of using the compact projects. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-5475) Allow importtsv and Import to work truly offline when using bulk import option
[ https://issues.apache.org/jira/browse/HBASE-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-5475. -- Resolution: Won't Fix Closing old issue. > Allow importtsv and Import to work truly offline when using bulk import option > -- > > Key: HBASE-5475 > URL: https://issues.apache.org/jira/browse/HBASE-5475 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Lars Hofhansl >Priority: Minor > > Currently importtsv (and now also Import with HBASE-5440) support using > HFileOutputFormat for later bulk loading. > However, currently that cannot be without having access to the table we're > going to import to, because both importtsv and Import need to lookup the > split points, and find the compression setting. > It would be nice if there would be an offline way to provide the split point > and compression setting. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-5311) Allow inmemory Memstore compactions
[ https://issues.apache.org/jira/browse/HBASE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-5311. -- Resolution: Won't Fix Closing old issue. > Allow inmemory Memstore compactions > --- > > Key: HBASE-5311 > URL: https://issues.apache.org/jira/browse/HBASE-5311 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl > Attachments: InternallyLayeredMap.java > > > Just like we periodically compact the StoreFiles we should also periodically > compact the MemStore. > During these compactions we eliminate deleted cells, expired cells, cells to > removed because of version count, etc, before we even do a memstore flush. > Besides the optimization that we could get from this, it should also allow us > to remove the special handling of ICV, Increment, and Append (all of which > use upsert logic to avoid accumulating excessive cells in the Memstore). > Not targeting this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15453) [Performance] Considering reverting HBASE-10015 - reinstate synchronized in StoreScanner
[ https://issues.apache.org/jira/browse/HBASE-15453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959998#comment-15959998 ] Lars Hofhansl commented on HBASE-15453: --- I guess I dropped the ball on this one. If there's still interest, this can go to all branches but master. > [Performance] Considering reverting HBASE-10015 - reinstate synchronized in > StoreScanner > > > Key: HBASE-15453 > URL: https://issues.apache.org/jira/browse/HBASE-15453 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Attachments: 15453-0.98.txt > > > In HBASE-10015 back then I found that intrinsic locks (synchronized) in > StoreScanner are slower that explicit locks. > I was surprised by this. To make sure I added a simple perf test and many > folks ran it on their machines. All found that explicit locks were faster. > Now... I just ran that test again. On the latest JDK8 I find that now the > intrinsic locks are significantly faster: > (OpenJDK Runtime Environment (build 1.8.0_72-b15)) > Explicit locks: > 10 runs mean:2223.6 sigma:72.29412147609237 > Intrinsic locks: > 10 runs mean:1865.3 sigma:32.63755505548784 > I confirmed the same with timing some Phoenix scans. We can save a bunch of > time by changing this back > Arrghhh... So maybe it's time to revert this now...? > (Note that in trunk due to [~ram_krish]'s work, we do not lock in > StoreScanner anymore) > I'll attach the perf test and a patch that changes lock to synchronized, if > some folks could run this on 0.98, that'd be great. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17521) Avoid stopping the load balancer in graceful stop
[ https://issues.apache.org/jira/browse/HBASE-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959996#comment-15959996 ] Lars Hofhansl commented on HBASE-17521: --- [~sandeep.guggilam], [~mnpoonia], any update on this? > Avoid stopping the load balancer in graceful stop > - > > Key: HBASE-17521 > URL: https://issues.apache.org/jira/browse/HBASE-17521 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > > ... instead setting the regionserver in question to draining. > [~sandeep.guggilam], FYI -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17785) RSGroupBasedLoadBalancer fails to assign new table regions when cloning snapshot
[
https://issues.apache.org/jira/browse/HBASE-17785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957663#comment-15957663
]
Lars Hofhansl commented on HBASE-17785:
---
So all we do is also assign the table for a snapshot.
+1
> RSGroupBasedLoadBalancer fails to assign new table regions when cloning
> snapshot
>
>
> Key: HBASE-17785
> URL: https://issues.apache.org/jira/browse/HBASE-17785
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 2.0.0
>
> Attachments: HBASE-17785.patch
>
>
> A novice starting out with RSGroupBasedLoadBalancer will want to enable it
> and, before assigning tables to groups, may want to create some test tables.
> Currently that does not work when creating a table by cloning a snapshot, in
> a surprising way. All regions of the table fail to open yet it is moved into
> ENABLED state. The client hangs indefinitely.
> {noformat}
> 2017-03-14 19:25:49,833 INFO [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> snapshot.CloneSnapshotHandler: Clone snapshot=seed on table=test_1 completed!
> 2017-03-14 19:25:49,871 INFO [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> hbase.MetaTableAccessor: Added 25
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table
[jira] [Updated] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-17877:
--
Attachment: 17877-v2-1.3.patch
Slightly different patch. Just retains some working variables as final.
If it wasn't before now {{i < minWords * SIZEOF_LONG}} is definitely optimized
out of the loop, since minWords is final.
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch, 17877-v2-1.3.patch,
> ByteComparatorJiraHBASE-17877.pdf
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is
> faster - dramatically faster in some cases.
> Patch forthcoming.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957181#comment-15957181 ] Lars Hofhansl commented on HBASE-17877: --- We could also flat-out use the Hadoop comparator - or for that matter even the plain Guava comparator. I do think, though, since this is do core to HBase that code should be part of HBase and be changed by us, and not by a project we depend on. Lemme make a new patch bringing back the change [~Apache9] points out - I wonder whether that perhaps is the part that makes it slower in the one case that [~vik.karma] found; and also leave the variables as final. The goal with this was to simply pull the Hadoop code into HBase (and remove the extra Guava dependency). > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956274#comment-15956274 ] Lars Hofhansl commented on HBASE-17877: --- But it's certainly a change we can undo and leave it the way it was. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956266#comment-15956266 ] Lars Hofhansl edited comment on HBASE-17877 at 4/5/17 4:03 AM: --- [~Apache9] that's what the Hadoop comparator does, which tested faster. :) (Here: https://github.com/hanborq/hadoop/blob/master/src/core/org/apache/hadoop/io/FastByteComparisons.java#L161) As tests one can use PE and YCSB. (The test that I used was a simple scan test with 10-100m keys.) was (Author: lhofhansl): [~Apache9] that's what the Hadoop comparator does, which tested faster. :) As tests one can use PE and YCSB. (The test that I used was a simple scan test with 10-100m keys.) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956266#comment-15956266 ] Lars Hofhansl edited comment on HBASE-17877 at 4/5/17 4:01 AM: --- [~Apache9] that's what the Hadoop comparator does, which tested faster. :) As tests one can use PE and YCSB. (The test that I used was a simple scan test with 10-100m keys.) was (Author: lhofhansl): [~Apache9] that's what the Hadoop comparator does, which tested faster. :) As tests one can use PE and YCSB. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956266#comment-15956266 ] Lars Hofhansl commented on HBASE-17877: --- [~Apache9] that's what the Hadoop comparator does, which tested faster. :) As tests one can use PE and YCSB. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956065#comment-15956065 ] Lars Hofhansl commented on HBASE-17877: --- [~vik.karma], can you attach your performance findings? (micro benchmarks and end-to-end) In my own test I use to vet perf changes I found a 20-30% performance improvement - but I am using short key. The effect will be much less pronounced for longer keys (where memory pure bandwidth becomes the dominating factor) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956060#comment-15956060 ] Lars Hofhansl edited comment on HBASE-17877 at 4/4/17 11:41 PM: Here's a patch I made that avoids any additional Guava dependencies. All the credit goes to [~vik.karma]. He did the testing. Found the better Hadoop implementation, etc. I assigned the issue to Vikas. was (Author: lhofhansl): Here's a patch I made that avoid any additional Guava dependencies. All the credit goes to [~vik.karma]. He did the testing. Found the better Hadoop implementation, etc. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-17877: -- Attachment: 17877-1.2.patch Here's a patch I made that avoid any additional Guava dependencies. All the credit goes to [~vik.karma]. He did the testing. Found the better Hadoop implementation, etc. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-17877: - Assignee: Lars Hofhansl > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-17877: - Assignee: Vikas Vishwakarma (was: Lars Hofhansl) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17877) Replace/improve HBase's byte[] comparator
Lars Hofhansl created HBASE-17877: - Summary: Replace/improve HBase's byte[] comparator Key: HBASE-17877 URL: https://issues.apache.org/jira/browse/HBASE-17877 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl [~vik.karma] did some extensive tests and found that Hadoop's version is faster - dramatically faster in some cases. Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895552#comment-15895552 ] Lars Hofhansl commented on HBASE-12790: --- bq. RpcScheduler is pluggable. You need more than that? Is that all we need [~jamestaylor]? Let me summarize how the Phoenix folks got here: # HBase scan contract is serial per scan. I.e. a scan will already return all keys in order whether the client needs it that way or not. Hence no parallel execution on behalf of a single scan (both [~stack] and I had made attempts to improve that but did not finish) # Scans cannot easily be broken down to units smaller than a region (it's certainly possible to do that, but there's no information about internal data skew inside a region) # For this Phoenix adds "guideposts". These are equidistant markers, so that Phoenix can know about the key distribution inside a region. # Phoenix uses guideposts to schedule many small scans. The units are fairly small (100MB-1GB worth of cells) to allow for fairness between queries. # If many query-chunks - a.k.a. scans - of a large query can hog the RPC queues than much of the advantage is lost. # Hence the desire for a this type of "group based" scheduling so that small queries can finish before all large queries in the queue need to finish. The group is a Phoenix query. So it is simply the desire to extend the fair queuing that HBase already has (HBASE-10993) to a query in Phoenix which may issue 1000's of scans to as many region servers. That just for the history... I do agree that the patch proposed here is too complex and perhaps wants to do too much. Now perhaps hbase.ipc.server.callqueue.scan.ratio from HBASE-11355 and HBASE-11724 gives us what we need _if_ we can use this for small scans, so that small scans can land on the "Get" queue. That way we can reserve that queue for small scans and Gets, and other queues for large scans. It's not ideal, though. The best is to allow somehow to round-robin between the queries on behave which the scans are operating. That abstraction is not available in HBase. > Support fairness across parallelized scans > -- > > Key: HBASE-12790 > URL: https://issues.apache.org/jira/browse/HBASE-12790 > Project: HBase > Issue Type: New Feature >Reporter: James Taylor >Assignee: ramkrishna.s.vasudevan > Labels: Phoenix > Attachments: AbstractRoundRobinQueue.java, HBASE-12790_1.patch, > HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, HBASE-12790.patch, > HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip > > > Some HBase clients parallelize the execution of a scan to reduce latency in > getting back results. This can lead to starvation with a loaded cluster and > interleaved scans, since the RPC queue will be ordered and processed on a > FIFO basis. For example, if there are two clients, A & B that submit largish > scans at the same time. Say each scan is broken down into 100 scans by the > client (broken down into equal depth chunks along the row key), and the 100 > scans of client A are queued first, followed immediately by the 100 scans of > client B. In this case, client B will be starved out of getting any results > back until the scans for client A complete. > One solution to this is to use the attached AbstractRoundRobinQueue instead > of the standard FIFO queue. The queue to be used could be (maybe it already > is) configurable based on a new config parameter. Using this queue would > require the client to have the same identifier for all of the 100 parallel > scans that represent a single logical scan from the clients point of view. > With this information, the round robin queue would pick off a task from the > queue in a round robin fashion (instead of a strictly FIFO manner) to prevent > starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895299#comment-15895299 ] Lars Hofhansl commented on HBASE-12790: --- Can we restart this discussion? [~jamestaylor] suggests to simply make the queue class pluggable. Phoenix can then inject its own implementation. > Support fairness across parallelized scans > -- > > Key: HBASE-12790 > URL: https://issues.apache.org/jira/browse/HBASE-12790 > Project: HBase > Issue Type: New Feature >Reporter: James Taylor >Assignee: ramkrishna.s.vasudevan > Labels: Phoenix > Attachments: AbstractRoundRobinQueue.java, HBASE-12790_1.patch, > HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, HBASE-12790.patch, > HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip > > > Some HBase clients parallelize the execution of a scan to reduce latency in > getting back results. This can lead to starvation with a loaded cluster and > interleaved scans, since the RPC queue will be ordered and processed on a > FIFO basis. For example, if there are two clients, A & B that submit largish > scans at the same time. Say each scan is broken down into 100 scans by the > client (broken down into equal depth chunks along the row key), and the 100 > scans of client A are queued first, followed immediately by the 100 scans of > client B. In this case, client B will be starved out of getting any results > back until the scans for client A complete. > One solution to this is to use the attached AbstractRoundRobinQueue instead > of the standard FIFO queue. The queue to be used could be (maybe it already > is) configurable based on a new config parameter. Using this queue would > require the client to have the same identifier for all of the 100 parallel > scans that represent a single logical scan from the clients point of view. > With this information, the round robin queue would pick off a task from the > queue in a round robin fashion (instead of a strictly FIFO manner) to prevent > starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888542#comment-15888542
]
Lars Hofhansl commented on HBASE-17069:
---
Seems to me we should change the signature of the WAL interface and add a
version of append that defaults {{inMemstore}} to true. False seems to be the
rare exception.
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>Reporter: Andrew Purtell
>Assignee: Abhishek Singh Chouhan
>Priority: Blocker
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5
>
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log,
> HBASE-17069-addendum.branch-1.3.001.patch, HBASE-17069.branch-1.3.001.patch,
> HBASE-17069.branch-1.3.002.patch, HBASE-17069.master.001.patch, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63,
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320,
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWr
[jira] [Commented] (HBASE-17609) Allow for region merging in the UI
[ https://issues.apache.org/jira/browse/HBASE-17609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15857125#comment-15857125 ] Lars Hofhansl commented on HBASE-17609: --- Let's do one thing at a time. The basic functionality, then we can make it look nice (unless the making-it-look-nice is simple). > Allow for region merging in the UI > --- > > Key: HBASE-17609 > URL: https://issues.apache.org/jira/browse/HBASE-17609 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0, 1.4.0 >Reporter: churro morales >Assignee: churro morales > Attachments: HBASE-17609-branch-1.3.patch, HBASE-17609.patch > > > HBASE-49 discussed having the ability to merge regions through the HBase UI, > but online region merging wasn't around back then. > I have created additional form fields for the table.jsp where you can pass in > two encoded region names (must be adjacent regions) and a merge can be called > through the UI. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17521) Avoid stopping the load balancer in graceful stop
[ https://issues.apache.org/jira/browse/HBASE-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855446#comment-15855446 ] Lars Hofhansl commented on HBASE-17521: --- Mr. [~stack], any comments? > Avoid stopping the load balancer in graceful stop > - > > Key: HBASE-17521 > URL: https://issues.apache.org/jira/browse/HBASE-17521 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > > ... instead setting the regionserver in question to draining. > [~sandeep.guggilam], FYI -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17448) Export metrics from RecoverableZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-17448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819503#comment-15819503 ] Lars Hofhansl commented on HBASE-17448: --- There're also improvements that we can do to the logging of ZK exception. For example ConnectionLoss is thrown under a variety of conditions and right now we cannot tell why it is being thrown. > Export metrics from RecoverableZooKeeper > > > Key: HBASE-17448 > URL: https://issues.apache.org/jira/browse/HBASE-17448 > Project: HBase > Issue Type: Improvement > Components: Zookeeper >Reporter: Andrew Purtell > > Consider adding instrumentation to RecoverableZooKeeper that exposes metrics > on the performance and health of the embedded ZooKeeper client: latency > histograms for each op type, number of reconnections, number of ops where a > reconnection was necessary to proceed, number of failed ops due to > CONNECTIONLOSS, number of failed ops due to SESSIONEXIPRED, number of failed > ops due to OPERATIONTIMEOUT. > RecoverableZooKeeper is a class in hbase-client so we can hook up the new > metrics to both client- and server-side metrics reporters. Probably this > metrics source should be a process singleton. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-17440. --- Resolution: Fixed Fix Version/s: 0.98.25 Done. > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.25 > > Attachments: 17440.txt > > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813406#comment-15813406 ] Lars Hofhansl commented on HBASE-17440: --- OK... Going to commit in a few. (since there won't be another 0.98 release likely, it's just for completeness) > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 17440.txt > > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-17440: -- Attachment: 17440.txt Here's the trivial fix, just trigger the chore immediately if the connection is stopped, the chore will then determine that its connection is shutdown and end itself. I tested this, I was not able to build up 1000's of DelayedClosing threads in a tight loop. (as said, this is not the main issue we've been seeing, but something I noticed on the way) Please have a look. [~apurtell], fyi. > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 17440.txt > > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-17440: -- Affects Version/s: 0.98.24 > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
Lars Hofhansl created HBASE-17440: - Summary: [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed Key: HBASE-17440 URL: https://issues.apache.org/jira/browse/HBASE-17440 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl We're seeing many issue with run-away ZK client connection in long running app servers. 10k or more send or event threads are happening frequently. While I looked around in the code I noticed that DelayedClosing closing is not immediately ended when an HConnection is closed, when there's an issue with HBase or ZK and client reconnect in a tight loop, this can lead temporarily to very many threads running. These will all get cleaned out after at most 60s, but during that time a lot of threads can be created. The fix is a one-liner. We'll likely file other issues soon. Interestingly branch-1 and beyond do not have this chore anymore, although - at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13094) Consider Filters that are evaluated before deletes and see delete markers
[ https://issues.apache.org/jira/browse/HBASE-13094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757947#comment-15757947 ] Lars Hofhansl commented on HBASE-13094: --- So the trick is to make a scan raw and then filter the KVs you do not want. Unfortunately you'd have to deal with column filtering as well, that'd the only thing that this could do. > Consider Filters that are evaluated before deletes and see delete markers > - > > Key: HBASE-13094 > URL: https://issues.apache.org/jira/browse/HBASE-13094 > Project: HBase > Issue Type: Brainstorming > Components: regionserver, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 13094-0.98.txt > > > That would be good for full control filtering of all cells, such as needed > for some transaction implementations. > [~ghelmling] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13094) Consider Filters that are evaluated before deletes and see delete markers
[ https://issues.apache.org/jira/browse/HBASE-13094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757926#comment-15757926 ] Lars Hofhansl commented on HBASE-13094: --- Wait. Isn't this all doable with just a raw-scan? [~jongray], [~jamestaylor], [~ghelmling]? I was just about to make a patch against trunk and found that this would lead to behavior as a raw scan. A raw scan can be handed a filter and that filter would always see all Cells (deleted and all columns as well). > Consider Filters that are evaluated before deletes and see delete markers > - > > Key: HBASE-13094 > URL: https://issues.apache.org/jira/browse/HBASE-13094 > Project: HBase > Issue Type: Brainstorming > Components: regionserver, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 13094-0.98.txt > > > That would be good for full control filtering of all cells, such as needed > for some transaction implementations. > [~ghelmling] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-13094) Consider Filters that are evaluated before deletes and see delete markers
[ https://issues.apache.org/jira/browse/HBASE-13094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757926#comment-15757926 ] Lars Hofhansl edited comment on HBASE-13094 at 12/18/16 1:05 AM: - Wait. Isn't this all doable with just a raw-scan? [~jongray], [~jamestaylor], [~ghelmling], [~apurtell]? I was just about to make a patch against trunk and found that this would lead to behavior as a raw scan. A raw scan can be handed a filter and that filter would always see all Cells (deleted and all columns as well). was (Author: lhofhansl): Wait. Isn't this all doable with just a raw-scan? [~jongray], [~jamestaylor], [~ghelmling]? I was just about to make a patch against trunk and found that this would lead to behavior as a raw scan. A raw scan can be handed a filter and that filter would always see all Cells (deleted and all columns as well). > Consider Filters that are evaluated before deletes and see delete markers > - > > Key: HBASE-13094 > URL: https://issues.apache.org/jira/browse/HBASE-13094 > Project: HBase > Issue Type: Brainstorming > Components: regionserver, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 13094-0.98.txt > > > That would be good for full control filtering of all cells, such as needed > for some transaction implementations. > [~ghelmling] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16115) Missing security context in RegionObserver coprocessor when a compaction/split is triggered manually
[ https://issues.apache.org/jira/browse/HBASE-16115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-16115. --- Resolution: Won't Fix > Missing security context in RegionObserver coprocessor when a > compaction/split is triggered manually > > > Key: HBASE-16115 > URL: https://issues.apache.org/jira/browse/HBASE-16115 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.20 >Reporter: Lars Hofhansl > > We ran into an interesting phenomenon which can easily render a cluster > unusable. > We loaded some tests data into a test table and forced a manual compaction > through the UI. We have some compaction hooks implemented in a region > observer, which writes back to another HBase table when the compaction > finishes. We noticed that this coprocessor is not setup correctly, it seems > the security context is missing. > The interesting part is that this _only_ happens when the compaction is > triggere through the UI. Automatic compactions (major or minor) or when > triggered via the HBase shell (folling a kinit) work fine. Only the > UI-triggered compactions cause this issues and lead to essentially > neverending compactions, immovable regions, etc. > Not sure what exactly the issue is, but I wanted to make sure I capture this. > [~apurtell], [~ghelmling], FYI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15700978#comment-15700978
]
Lars Hofhansl edited comment on HBASE-17125 at 11/28/16 6:00 AM:
-
Simply put: There _is_ no right order in which to evaluate filters, versions,
column tracking, and deletes. Regardless of which order you pick someone will
complain about it. :)
This has changed various times in the past. I think we should stick with how it
is. If filters can see versions they can act accordingly (and filter extra
versions if needed). If filters do not see all version, but you happen to need
to look at all versions... Well then you'd be out of luck if we make this
change.
See also HBASE-13094. That one is about the order of filters w.r.t. delete
markers.
I guess, in hindsight for maximum flexibility filters should have been placed
before everything else so that they can be used to intervene into the entire
pipeline.
was (Author: lhofhansl):
Simply put: There _is_ no right order in which to evaluate filters, versions,
column tracking, and deletes. Regardless of which order you pick someone will
complain about it. :)
This has changed various times in the past. I think we should stick with how it
is. If filter can see versions they can act accordingly (and filter extra
versions if needed). If filters do not see all version, but you need to look at
all version, well then you'd be out of luck.
See also HBASE-13094. That one is about the order of filters w.r.t. delete
markers.
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
> Attachments: example.diff
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
