[jira] [Commented] (HBASE-12431) Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread Safe
[
https://issues.apache.org/jira/browse/HBASE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234004#comment-14234004
]
Hudson commented on HBASE-12431:
SUCCESS: Integrated in HBase-TRUNK #5865 (See
[https://builds.apache.org/job/HBase-TRUNK/5865/])
HBASE-12431 Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is
Not Thread Safe (Jingcheng Du) (stack: rev
c45772e76a581838ea89ff303d0ff67ceaa82845)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread
Safe
-
Key: HBASE-12431
URL: https://issues.apache.org/jira/browse/HBASE-12431
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.1
Reporter: Jonathan Jarvis
Assignee: Jingcheng Du
Fix For: 1.0.0, 2.0.0, 0.98.9
Attachments: HBASE-12431-V2.diff, HBASE-12431-V3.diff,
HBASE-12431.diff
Result declares that it is NOT THREAD SAFE at the top of the source code, but
one would assume that refers to many different threads accessing the same
Result object. I've run into an issue when I have several different threads
accessing their own Result object that runs into an issue because of use of
common static member variable.
I noticed the problem when I switched from:
getColumnLatestCell(byte[], byte[]) to
getColumnLatestCell(byte[], int, int, byte[], int, int)
These methods call different binarySearch methods, the latter invoking:
protected int binarySearch(final Cell [] kvs,
309 final byte [] family, final int foffset, final int flength,
310 final byte [] qualifier, final int qoffset, final int qlength) {
This method utilizes a private static member variable called buffer
If more than one thread is utilizing buffer you'll see unpredictable
behavior unless you synchronize(Result.class) {}.
If buffer is to remain a static variable, I would recommend changing it to a
ThreadLocalbyte[] instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12431) Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread Safe
[
https://issues.apache.org/jira/browse/HBASE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234043#comment-14234043
]
Hudson commented on HBASE-12431:
FAILURE: Integrated in HBase-0.98 #715 (See
[https://builds.apache.org/job/HBase-0.98/715/])
HBASE-12431 Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is
Not Thread Safe (Jingcheng Du) (stack: rev
769f93ff07940e92ff6b6653fb53357bc99c8966)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread
Safe
-
Key: HBASE-12431
URL: https://issues.apache.org/jira/browse/HBASE-12431
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.1
Reporter: Jonathan Jarvis
Assignee: Jingcheng Du
Fix For: 1.0.0, 2.0.0, 0.98.9
Attachments: HBASE-12431-V2.diff, HBASE-12431-V3.diff,
HBASE-12431.diff
Result declares that it is NOT THREAD SAFE at the top of the source code, but
one would assume that refers to many different threads accessing the same
Result object. I've run into an issue when I have several different threads
accessing their own Result object that runs into an issue because of use of
common static member variable.
I noticed the problem when I switched from:
getColumnLatestCell(byte[], byte[]) to
getColumnLatestCell(byte[], int, int, byte[], int, int)
These methods call different binarySearch methods, the latter invoking:
protected int binarySearch(final Cell [] kvs,
309 final byte [] family, final int foffset, final int flength,
310 final byte [] qualifier, final int qoffset, final int qlength) {
This method utilizes a private static member variable called buffer
If more than one thread is utilizing buffer you'll see unpredictable
behavior unless you synchronize(Result.class) {}.
If buffer is to remain a static variable, I would recommend changing it to a
ThreadLocalbyte[] instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11861) Native MOB Compaction mechanisms.
[ https://issues.apache.org/jira/browse/HBASE-11861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11861: - Attachment: mob compaction.pdf Upload the high level design for the mob compaction. Hi all, please review and comment. Thanks a lot! Native MOB Compaction mechanisms. - Key: HBASE-11861 URL: https://issues.apache.org/jira/browse/HBASE-11861 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Affects Versions: 2.0.0 Reporter: Jonathan Hsieh Attachments: 141030-mob-compaction.pdf, mob compaction.pdf Currently, the first cut of mob will have external processes to age off old mob data (the ttl cleaner), and to compact away deleted or over written data (the sweep tool). From an operational point of view, having two external tools, especially one that relies on MapReduce is undesirable. In this issue we'll tackle integrating these into hbase without requiring external processes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12611) Create autoCommit() method and remove clearBufferOnFail
[
https://issues.apache.org/jira/browse/HBASE-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234055#comment-14234055
]
Nicolas Liochon commented on HBASE-12611:
-
bq. stack and Nick Dimiduk came to the conclusion that the flush method should
be called autoCommit() similar to the SQL APIs.
Sorry for beeing late in the game. The meaning in SQL is slightly different. In
jdbc, whatever the value for autoCommit, the query will be sent to be server
and executed. autocommit is set to false if the client application wants to
send multiple queries within a single transaction (and then it will do a
begin/commit explicitly). I haven't double checked if it's the standard or an
implementation detail (the docs are not very clear), but it's unlikely to
change anyway: there is another set of methods for batches in jdbc.
Our old autoFlush is different as it impacts the client behavior. I think we're
creating a confusion here. Moreover, if we add transactions between rows in the
future, then may be we will want to use autoCommit for what it really is. As
I'm very late here I leave the decision to you, but we should at least be clear
in the javadoc imho.
bq. Do we also want to change the default with this patch?
I like the fact that HBase its secure by default and it would be very confusing
for the users as well imho.
{code}
- public boolean isAutoFlush() {
-return autoFlush;
+ public boolean getAutoCommit() {
+return autoCommit;
{code}
If I'm not wrong we use 'is' for getters on boolean?
Create autoCommit() method and remove clearBufferOnFail
---
Key: HBASE-12611
URL: https://issues.apache.org/jira/browse/HBASE-12611
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0, 0.99.2
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Fix For: 1.0.0
Attachments: HBASE-12611.patch
There was quite a bit of good discussion on HBASE-12490 about this topic.
[~stack] and [~ndimiduk] came to the conclusion that the flush method should
be called autoCommit() similar to the SQL APIs. [~ndimiduk] also suggested
that clearBufferOnFail should be removed from HTable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12431) Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread Safe
[
https://issues.apache.org/jira/browse/HBASE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234060#comment-14234060
]
Hudson commented on HBASE-12431:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #682 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/682/])
HBASE-12431 Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is
Not Thread Safe (Jingcheng Du) (stack: rev
769f93ff07940e92ff6b6653fb53357bc99c8966)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread
Safe
-
Key: HBASE-12431
URL: https://issues.apache.org/jira/browse/HBASE-12431
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.1
Reporter: Jonathan Jarvis
Assignee: Jingcheng Du
Fix For: 1.0.0, 2.0.0, 0.98.9
Attachments: HBASE-12431-V2.diff, HBASE-12431-V3.diff,
HBASE-12431.diff
Result declares that it is NOT THREAD SAFE at the top of the source code, but
one would assume that refers to many different threads accessing the same
Result object. I've run into an issue when I have several different threads
accessing their own Result object that runs into an issue because of use of
common static member variable.
I noticed the problem when I switched from:
getColumnLatestCell(byte[], byte[]) to
getColumnLatestCell(byte[], int, int, byte[], int, int)
These methods call different binarySearch methods, the latter invoking:
protected int binarySearch(final Cell [] kvs,
309 final byte [] family, final int foffset, final int flength,
310 final byte [] qualifier, final int qoffset, final int qlength) {
This method utilizes a private static member variable called buffer
If more than one thread is utilizing buffer you'll see unpredictable
behavior unless you synchronize(Result.class) {}.
If buffer is to remain a static variable, I would recommend changing it to a
ThreadLocalbyte[] instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12597) Add RpcClient interface and enable changing of RpcClient implementation
[ https://issues.apache.org/jira/browse/HBASE-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12597: -- Attachment: HBASE-12597-v4.patch New patch - Moved the static constants to RpcClient interface. They are indeed pretty generic and I use them too in my implementation. - RpcClient extends Closeable now and stop method is now called close() - Moved out Call class from RpcClientImpl. I didn't use it in my async implementation, but maybe others can use it in another sync implementation. bq. Ditto Connection. If so, np. I suppose an RpcClient has a Connection? A single Connection? Or this is implementation detail? Or I supopse it is not... protocol is all around Connections. If so, how you doing this over in your client? Connection is not a static class and is tightly linked into RpcClientImpl. It uses the socket factory and creates sockets which are pretty much tied into RpcClientImpl its implementation. In many ways it is the implementation of RpcClient. I had to create something new in my implementation based on Netty Channels. Maybe something reusable can be abstracted out of it to also simplify how I implemented it but I think that falls outside the scope of this issue. CallFuture is pretty small and a private static class. It is used in the internal queue. I couldn't use it for my implementation. BlockingRpcChannelImplementation links to the call method in RpcClientImpl which is currently not in the RpcClient interface. And I would not like it to be in there since PayloadCarryingRpcController is currently not reusable in async context. Could be with some changes but I think they need some discussion, but somehow I think it is better to leave RpcClient as clean as possible so RpcClients can try out new things like doing without an PayloadCarryingRpcController. bq. I beileve Anoop Sam John is refering to the 'release note' attribute of this here JIRA. Ah I see. Add RpcClient interface and enable changing of RpcClient implementation --- Key: HBASE-12597 URL: https://issues.apache.org/jira/browse/HBASE-12597 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Attachments: HBASE-12597-v1.patch, HBASE-12597-v2.patch, HBASE-12597-v3.patch, HBASE-12597-v4.patch, HBASE-12597.patch Currently HConnectionImplementation works with the included RpcClient which is a direct implementation and not defined by an interface. It would be great to be able to swap out the default RpcClient with another implementation which can also be controlled by the default HConnectionImplementation. Suggested changes: - Create a RpcClient interface which defines all the ways HConnectionImplementation interacts with an RPC client. Like getting a blocking protobuf service interface or closing the client. - Define which RpcClient implementation to construct by setting a configuration variable which defaults to the current RpcClient. - Possibly create an abstract RpcClient class to only load all the basic Rpc layer configurations to be used in an implementation. Why? It enables experimentation with RpcClients which could enable new features or could be more performant than the included client. I created a new RpcClient implementation based on Netty which can also be called asynchronously. It would be great to also be able to use this RpcClient in all the default ways and tests to see if there are any issues with it. https://github.com/jurmous/async-hbase-client/ https://github.com/jurmous/async-hbase-client/blob/master/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-12634: -- Attachment: HBASE-12634-v3.patch Changed the authorization api in preCreateNamespace api in v3 patch. As authorizing a user with admin rights for a non existing namespace was not correct. Fix the AccessController#requireGlobalPermission(ns) with NS - Key: HBASE-12634 URL: https://issues.apache.org/jira/browse/HBASE-12634 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.8 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, HBASE-12634.patch The namespace argument passed to AccessController#requireGlobalPermission API to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234105#comment-14234105 ] Ashish Singhi commented on HBASE-12634: --- Please review the v3 patch and share your comments. Fix the AccessController#requireGlobalPermission(ns) with NS - Key: HBASE-12634 URL: https://issues.apache.org/jira/browse/HBASE-12634 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.8 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, HBASE-12634.patch The namespace argument passed to AccessController#requireGlobalPermission API to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234111#comment-14234111 ] Matteo Bertozzi commented on HBASE-12634: - +1 v3 looks good to me Fix the AccessController#requireGlobalPermission(ns) with NS - Key: HBASE-12634 URL: https://issues.apache.org/jira/browse/HBASE-12634 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.8 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, HBASE-12634.patch The namespace argument passed to AccessController#requireGlobalPermission API to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangduo updated HBASE-10201: - Attachment: HBASE-10201_13.patch Use a FlushPolicy instead of hbase.hregion.memstore.percolumnfamilyflush.enabled config. I'm not good at naming things, and may break some rules when add the policy. So just point out if you have a better name of the policy and anything that you think is wrong in this patch. I will fix it as soon as possible. Thanks. Port 'Make flush decisions per column family' to trunk -- Key: HBASE-10201 URL: https://issues.apache.org/jira/browse/HBASE-10201 Project: HBase Issue Type: Improvement Components: wal Reporter: Ted Yu Assignee: zhangduo Priority: Critical Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, HBASE-10201_2.patch, HBASE-10201_3.patch, HBASE-10201_4.patch, HBASE-10201_5.patch, HBASE-10201_6.patch, HBASE-10201_7.patch, HBASE-10201_8.patch, HBASE-10201_9.patch Currently the flush decision is made using the aggregate size of all column families. When large and small column families co-exist, this causes many small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12635) Delete acl notify znode of table after the table is deleted
Liu Shaohui created HBASE-12635: --- Summary: Delete acl notify znode of table after the table is deleted Key: HBASE-12635 URL: https://issues.apache.org/jira/browse/HBASE-12635 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor In our multi-tenant hbase cluster, we found that there are over 1M znodes under the acl node. The reason is that users create and delete tables with different names frequently. The acl notify znode are left there after the tables are deleted. Simple solution is that deleting acl notify znode of table in AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-12635: Attachment: HBASE-12635-v1.diff Patch for trunk. Delete acl notify znode of table after the table is deleted --- Key: HBASE-12635 URL: https://issues.apache.org/jira/browse/HBASE-12635 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-12635-v1.diff In our multi-tenant hbase cluster, we found that there are over 1M znodes under the acl node. The reason is that users create and delete tables with different names frequently. The acl notify znode are left there after the tables are deleted. Simple solution is that deleting acl notify znode of table in AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-12635: Fix Version/s: 2.0.0 Affects Version/s: 0.94.11 0.98.6 Status: Patch Available (was: Open) Delete acl notify znode of table after the table is deleted --- Key: HBASE-12635 URL: https://issues.apache.org/jira/browse/HBASE-12635 Project: HBase Issue Type: Bug Affects Versions: 0.98.6, 0.94.11 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Attachments: HBASE-12635-v1.diff In our multi-tenant hbase cluster, we found that there are over 1M znodes under the acl node. The reason is that users create and delete tables with different names frequently. The acl notify znode are left there after the tables are deleted. Simple solution is that deleting acl notify znode of table in AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234166#comment-14234166 ] Anoop Sam John commented on HBASE-12634: +1 for V3 Fix the AccessController#requireGlobalPermission(ns) with NS - Key: HBASE-12634 URL: https://issues.apache.org/jira/browse/HBASE-12634 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.8 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, HBASE-12634.patch The namespace argument passed to AccessController#requireGlobalPermission API to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12622) user_permission should require global admin to display global and ns permissions
[ https://issues.apache.org/jira/browse/HBASE-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234167#comment-14234167 ] Anoop Sam John commented on HBASE-12622: +1 for this patch after HBASE-12634 goes in. user_permission should require global admin to display global and ns permissions Key: HBASE-12622 URL: https://issues.apache.org/jira/browse/HBASE-12622 Project: HBase Issue Type: Bug Components: security Affects Versions: 2.0.0, 0.98.8, 0.99.2 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12622-v0.patch user_permission check the user permission only on the table level (requiring at least a table-level admin) global and namespace permission listing is done without checking anything. but only a global admins should be able to perform this operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12622) user_permission should require global admin to display global and ns permissions
[ https://issues.apache.org/jira/browse/HBASE-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234170#comment-14234170 ] Ashish Singhi commented on HBASE-12622: --- [~mbertozzi] can we move the test to TestNamespaceCommands, as all the namespace related test are available there and also we can validate for the user with admin rights on namespace is allowed to execute user_permission.? user_permission should require global admin to display global and ns permissions Key: HBASE-12622 URL: https://issues.apache.org/jira/browse/HBASE-12622 Project: HBase Issue Type: Bug Components: security Affects Versions: 2.0.0, 0.98.8, 0.99.2 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12622-v0.patch user_permission check the user permission only on the table level (requiring at least a table-level admin) global and namespace permission listing is done without checking anything. but only a global admins should be able to perform this operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12636) Avoid too many write operations on zookeeper in replication
Liu Shaohui created HBASE-12636: --- Summary: Avoid too many write operations on zookeeper in replication Key: HBASE-12636 URL: https://issues.apache.org/jira/browse/HBASE-12636 Project: HBase Issue Type: Improvement Affects Versions: 0.94.11 Reporter: Liu Shaohui Assignee: Liu Shaohui Fix For: 1.0.0 In our production cluster, we found there are about over 1k write operations per second on zookeeper from hbase replication. The reason is that the replication source will write the log position to zookeeper for every edit shipping. If the current replicating WAL is just the WAL that regionserver is writing to, each skipping will be very small but the frequency is very high, which causes many write operations on zookeeper. A simple solution is that writing log position to zookeeper when position diff or skipped edit number is larger than a threshold, not every edit shipping. Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12564) consolidate the getTableDescriptors() semantic
[ https://issues.apache.org/jira/browse/HBASE-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12564: Attachment: HBASE-12564-v2.patch consolidate the getTableDescriptors() semantic -- Key: HBASE-12564 URL: https://issues.apache.org/jira/browse/HBASE-12564 Project: HBase Issue Type: Bug Components: Client, master Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0 Attachments: HBASE-12564-v0.patch, HBASE-12564-v1.patch, HBASE-12564-v2.patch Master getTableDescriptors() which is called by Admin.listTables() has a couple of different behaviors depending on how it is called. after HBASE-12073 with the AccessController enabled, we now get a global admin required if listTables() is called without a regex otherwise we return only the table that the user can see (we show only the tables that the user have access to, which means or the user is a global admin or it has a table-level create/admin). We probably should have the second behavior even without regex, since I should able to see my own tables. getTableDescriptors() is returning only non system tables. Tools like user_permission that are doing for each listTable(): userPerm(table) are losing the system tables, so stuff like user_permission 'hbase:acls' will not result any result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234181#comment-14234181 ] ramkrishna.s.vasudevan commented on HBASE-12635: +1 Delete acl notify znode of table after the table is deleted --- Key: HBASE-12635 URL: https://issues.apache.org/jira/browse/HBASE-12635 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Attachments: HBASE-12635-v1.diff In our multi-tenant hbase cluster, we found that there are over 1M znodes under the acl node. The reason is that users create and delete tables with different names frequently. The acl notify znode are left there after the tables are deleted. Simple solution is that deleting acl notify znode of table in AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-12636: Attachment: HBASE-12636-v1.diff Patch for trunk Avoid too many write operations on zookeeper in replication --- Key: HBASE-12636 URL: https://issues.apache.org/jira/browse/HBASE-12636 Project: HBase Issue Type: Improvement Affects Versions: 0.94.11 Reporter: Liu Shaohui Assignee: Liu Shaohui Labels: replication Fix For: 1.0.0 Attachments: HBASE-12636-v1.diff In our production cluster, we found there are about over 1k write operations per second on zookeeper from hbase replication. The reason is that the replication source will write the log position to zookeeper for every edit shipping. If the current replicating WAL is just the WAL that regionserver is writing to, each skipping will be very small but the frequency is very high, which causes many write operations on zookeeper. A simple solution is that writing log position to zookeeper when position diff or skipped edit number is larger than a threshold, not every edit shipping. Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234270#comment-14234270 ] Ted Yu commented on HBASE-12204: Since hbase 1.0 RC is coming out soon, I think we should adopt the approach which is used in 0.98 That way, we can close the feature gap between 0.98 and 1.0 Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Andrey Stepachev Priority: Minor The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[
https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234302#comment-14234302
]
Hadoop QA commented on HBASE-12634:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685094/HBASE-12634-v3.patch
against master branch at commit c45772e76a581838ea89ff303d0ff67ceaa82845.
ATTACHMENT ID: 12685094
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11933//console
This message is automatically generated.
Fix the AccessController#requireGlobalPermission(ns) with NS
-
Key: HBASE-12634
URL: https://issues.apache.org/jira/browse/HBASE-12634
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.98.8
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0, 0.98.9
Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch,
HBASE-12634.patch
The namespace argument passed to AccessController#requireGlobalPermission API
to authorize namespace is actually not authorizing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12634: Resolution: Fixed Status: Resolved (was: Patch Available) Fix the AccessController#requireGlobalPermission(ns) with NS - Key: HBASE-12634 URL: https://issues.apache.org/jira/browse/HBASE-12634 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.8 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, HBASE-12634.patch The namespace argument passed to AccessController#requireGlobalPermission API to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12622) user_permission should require global admin to display global and ns permissions
[ https://issues.apache.org/jira/browse/HBASE-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12622: Attachment: HBASE-12622-v1.patch user_permission should require global admin to display global and ns permissions Key: HBASE-12622 URL: https://issues.apache.org/jira/browse/HBASE-12622 Project: HBase Issue Type: Bug Components: security Affects Versions: 2.0.0, 0.98.8, 0.99.2 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12622-v0.patch, HBASE-12622-v1.patch user_permission check the user permission only on the table level (requiring at least a table-level admin) global and namespace permission listing is done without checking anything. but only a global admins should be able to perform this operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234306#comment-14234306 ] Ashish Singhi commented on HBASE-12634: --- Thanks guys for the review and Matteo for the commit. Fix the AccessController#requireGlobalPermission(ns) with NS - Key: HBASE-12634 URL: https://issues.apache.org/jira/browse/HBASE-12634 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.8 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, HBASE-12634.patch The namespace argument passed to AccessController#requireGlobalPermission API to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12594) Remove the usage of deprecated HbaseAdmin#getTableNames from shell list command
[ https://issues.apache.org/jira/browse/HBASE-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-12594: -- Fix Version/s: 0.98.9 2.0.0 1.0.0 Remove the usage of deprecated HbaseAdmin#getTableNames from shell list command --- Key: HBASE-12594 URL: https://issues.apache.org/jira/browse/HBASE-12594 Project: HBase Issue Type: Bug Components: shell Affects Versions: 2.0.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12594.patch With this fix one behavior change a non-super user will notice is that he/she will be able to see only tables to which they have create/admin permissions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12553) request value is not consistent for restoreSnapshot in audit logs
[
https://issues.apache.org/jira/browse/HBASE-12553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-12553:
--
Fix Version/s: 2.0.0
request value is not consistent for restoreSnapshot in audit logs
-
Key: HBASE-12553
URL: https://issues.apache.org/jira/browse/HBASE-12553
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 2.0.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-12553.patch
{code}
requirePermission(restoreSnapshot, hTableDescriptor.getTableName(),
null, null,
Permission.Action.ADMIN);
} else {
requirePermission(restore, Action.ADMIN);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12338) Client side scanning prefetching.
[ https://issues.apache.org/jira/browse/HBASE-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Deng updated HBASE-12338: Status: Patch Available (was: Open) Client side scanning prefetching. - Key: HBASE-12338 URL: https://issues.apache.org/jira/browse/HBASE-12338 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 0.98.6.1, 1.0.0, 2.0.0 Reporter: Yi Deng Assignee: Yi Deng Labels: prefetch, results, scanner Attachments: 0001-Add-ScanPrefetcher-for-client-side-scanning-prefetch.patch, 0001-ScanPrefetcher.patch, 2.0-0001-Add-ScanPrefetcher-for-client-side-scanning-prefetch.patch, 2.0-0002-Add-ScanPrefetcher-for-client-side-scanning-prefetch.patch, 2.0-0003-Add-ScanPrefetcher-for-client-side-scanning-prefetch.patch Since server side prefetching was not proved to be a good way to prefetch, we need to do it on client side. This is a wrapper class that takes any instance of `ResultScanner` as the underneath scanning component. The class will schedule the scanning in a background thread. There is a buffering queue storing prefetched results, whose's length is configurable. The prefetcher will release the thread if the queue is full and wait for results to be consumed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12632) ThrottledInputStream/ExportSnapshot does not throttle
[
https://issues.apache.org/jira/browse/HBASE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234344#comment-14234344
]
Hadoop QA commented on HBASE-12632:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685038/12632-v1.txt
against master branch at commit c45772e76a581838ea89ff303d0ff67ceaa82845.
ATTACHMENT ID: 12685038
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11934//console
This message is automatically generated.
ThrottledInputStream/ExportSnapshot does not throttle
-
Key: HBASE-12632
URL: https://issues.apache.org/jira/browse/HBASE-12632
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.99.2
Reporter: Tobi Vollebregt
Attachments: 12632-v1.txt
I just transferred a ton of data using ExportSnapshot with bandwidth
throttling from one Hadoop cluster to another Hadoop cluster, and discovered
that ThrottledInputStream does not limit bandwidth.
The problem is that ThrottledInputStream sleeps once, for a fixed time (50
ms), at the start of each read call, disregarding the actual amount of data
read.
ExportSnapshot defaults to a buffer size as big as the block size of the
outputFs:
{code:java}
// Use the default block size of the outputFs if bigger
int defaultBlockSize = Math.max((int) outputFs.getDefaultBlockSize(),
BUFFER_SIZE);
bufferSize = conf.getInt(CONF_BUFFER_SIZE, defaultBlockSize);
LOG.info(Using bufferSize= +
StringUtils.humanReadableInt(bufferSize));
{code}
In my case, this was 256MB.
Hence, the ExportSnapshot
[jira] [Commented] (HBASE-12273) Generate .tabledesc file during upgrading if missing
[ https://issues.apache.org/jira/browse/HBASE-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234382#comment-14234382 ] Yi Deng commented on HBASE-12273: - Thanks [~stack]. But this diff is not supposed to work on master, it only works for branch-1 and before. Some legacy code is necessary for generating HRegionInfo from old format while that code fragment seems been removed on master. Generate .tabledesc file during upgrading if missing Key: HBASE-12273 URL: https://issues.apache.org/jira/browse/HBASE-12273 Project: HBase Issue Type: Sub-task Components: Admin Affects Versions: 1.0.0, 0.98.7 Reporter: Yi Deng Assignee: Yi Deng Labels: upgrade Fix For: 1.0.0, 0.98.7 Attachments: 1.0-0001-INTERNAL-Add-a-tool-for-fixing-missing-TableDescript.patch Generate .tabledesc file during upgrading if missing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12204: --- Attachment: 12204-1.0.txt All tests touched by the patch passes locally. Running full test suite on Linux now. Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Andrey Stepachev Priority: Minor Attachments: 12204-1.0.txt The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12545) Fix backwards compatibility issue introduced with HBASE-12363
[ https://issues.apache.org/jira/browse/HBASE-12545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234400#comment-14234400 ] Lars Hofhansl commented on HBASE-12545: --- Yeah, let's close this one. Fix backwards compatibility issue introduced with HBASE-12363 - Key: HBASE-12545 URL: https://issues.apache.org/jira/browse/HBASE-12545 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 1.0.0, 0.98.9 Attachments: 12545.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234396#comment-14234396 ] Ted Yu edited comment on HBASE-12204 at 12/4/14 5:50 PM: - All tests touched by the patch pass locally. Running full test suite on Linux now. was (Author: yuzhih...@gmail.com): All tests touched by the patch passes locally. Running full test suite on Linux now. Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Andrey Stepachev Priority: Minor Attachments: 12204-1.0.txt The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12173) Backport: [PE] Allow random value size
[ https://issues.apache.org/jira/browse/HBASE-12173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234402#comment-14234402 ] Lars Hofhansl commented on HBASE-12173: --- I should. :) Lemme finish this today. Backport: [PE] Allow random value size -- Key: HBASE-12173 URL: https://issues.apache.org/jira/browse/HBASE-12173 Project: HBase Issue Type: Sub-task Components: Performance Reporter: Lars Hofhansl Fix For: 0.94.26, 0.98.9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234411#comment-14234411 ] Hudson commented on HBASE-12634: SUCCESS: Integrated in HBase-1.0 #539 (See [https://builds.apache.org/job/HBase-1.0/539/]) HBASE-12634 Fix the AccessController#requireGlobalPermission(ns) with NS (matteo.bertozzi: rev ffbfe01bbeb00d91f8df1ce6ddc041af9887dca7) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java Fix the AccessController#requireGlobalPermission(ns) with NS - Key: HBASE-12634 URL: https://issues.apache.org/jira/browse/HBASE-12634 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.8 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, HBASE-12634.patch The namespace argument passed to AccessController#requireGlobalPermission API to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234418#comment-14234418 ] Lars Hofhansl commented on HBASE-12636: --- Means in the event of failure we could double replicate some entries, right? Avoid too many write operations on zookeeper in replication --- Key: HBASE-12636 URL: https://issues.apache.org/jira/browse/HBASE-12636 Project: HBase Issue Type: Improvement Affects Versions: 0.94.11 Reporter: Liu Shaohui Assignee: Liu Shaohui Labels: replication Fix For: 1.0.0 Attachments: HBASE-12636-v1.diff In our production cluster, we found there are about over 1k write operations per second on zookeeper from hbase replication. The reason is that the replication source will write the log position to zookeeper for every edit shipping. If the current replicating WAL is just the WAL that regionserver is writing to, each skipping will be very small but the frequency is very high, which causes many write operations on zookeeper. A simple solution is that writing log position to zookeeper when position diff or skipped edit number is larger than a threshold, not every edit shipping. Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-12204: -- Assignee: Ted Yu (was: Andrey Stepachev) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Attachments: 12204-1.0.txt The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12204: --- Status: Patch Available (was: Open) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Attachments: 12204-1.0.txt The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12204: --- Fix Version/s: 1.0.0 Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 1.0.0 Attachments: 12204-1.0.txt The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12597) Add RpcClient interface and enable changing of RpcClient implementation
[ https://issues.apache.org/jira/browse/HBASE-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12597: -- Fix Version/s: 2.0.0 1.0.0 Add RpcClient interface and enable changing of RpcClient implementation --- Key: HBASE-12597 URL: https://issues.apache.org/jira/browse/HBASE-12597 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12597-v1.patch, HBASE-12597-v2.patch, HBASE-12597-v3.patch, HBASE-12597-v4.patch, HBASE-12597.patch Currently HConnectionImplementation works with the included RpcClient which is a direct implementation and not defined by an interface. It would be great to be able to swap out the default RpcClient with another implementation which can also be controlled by the default HConnectionImplementation. Suggested changes: - Create a RpcClient interface which defines all the ways HConnectionImplementation interacts with an RPC client. Like getting a blocking protobuf service interface or closing the client. - Define which RpcClient implementation to construct by setting a configuration variable which defaults to the current RpcClient. - Possibly create an abstract RpcClient class to only load all the basic Rpc layer configurations to be used in an implementation. Why? It enables experimentation with RpcClients which could enable new features or could be more performant than the included client. I created a new RpcClient implementation based on Netty which can also be called asynchronously. It would be great to also be able to use this RpcClient in all the default ways and tests to see if there are any issues with it. https://github.com/jurmous/async-hbase-client/ https://github.com/jurmous/async-hbase-client/blob/master/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12545) Fix backwards compatibility issue introduced with HBASE-12363
[ https://issues.apache.org/jira/browse/HBASE-12545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-12545. Resolution: Not a Problem Fix Version/s: (was: 0.98.9) (was: 1.0.0) Assignee: (was: Lars Hofhansl) Resolving as Not A Problem Fix backwards compatibility issue introduced with HBASE-12363 - Key: HBASE-12545 URL: https://issues.apache.org/jira/browse/HBASE-12545 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Lars Hofhansl Attachments: 12545.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12597) Add RpcClient interface and enable changing of RpcClient implementation
[ https://issues.apache.org/jira/browse/HBASE-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234426#comment-14234426 ] stack commented on HBASE-12597: --- Skimmed. Will come back with a better review (should the defines in the Interface be public so folks don't have to be in same package to make an implementation?) Marked this as for 1.0. Would be good to get it in before 1.0 goes out. Lets see what hadoopqa thinks of this patch. Add RpcClient interface and enable changing of RpcClient implementation --- Key: HBASE-12597 URL: https://issues.apache.org/jira/browse/HBASE-12597 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12597-v1.patch, HBASE-12597-v2.patch, HBASE-12597-v3.patch, HBASE-12597-v4.patch, HBASE-12597.patch Currently HConnectionImplementation works with the included RpcClient which is a direct implementation and not defined by an interface. It would be great to be able to swap out the default RpcClient with another implementation which can also be controlled by the default HConnectionImplementation. Suggested changes: - Create a RpcClient interface which defines all the ways HConnectionImplementation interacts with an RPC client. Like getting a blocking protobuf service interface or closing the client. - Define which RpcClient implementation to construct by setting a configuration variable which defaults to the current RpcClient. - Possibly create an abstract RpcClient class to only load all the basic Rpc layer configurations to be used in an implementation. Why? It enables experimentation with RpcClients which could enable new features or could be more performant than the included client. I created a new RpcClient implementation based on Netty which can also be called asynchronously. It would be great to also be able to use this RpcClient in all the default ways and tests to see if there are any issues with it. https://github.com/jurmous/async-hbase-client/ https://github.com/jurmous/async-hbase-client/blob/master/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12597) Add RpcClient interface and enable changing of RpcClient implementation
[ https://issues.apache.org/jira/browse/HBASE-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12597: -- Status: Patch Available (was: Open) Add RpcClient interface and enable changing of RpcClient implementation --- Key: HBASE-12597 URL: https://issues.apache.org/jira/browse/HBASE-12597 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12597-v1.patch, HBASE-12597-v2.patch, HBASE-12597-v3.patch, HBASE-12597-v4.patch, HBASE-12597.patch Currently HConnectionImplementation works with the included RpcClient which is a direct implementation and not defined by an interface. It would be great to be able to swap out the default RpcClient with another implementation which can also be controlled by the default HConnectionImplementation. Suggested changes: - Create a RpcClient interface which defines all the ways HConnectionImplementation interacts with an RPC client. Like getting a blocking protobuf service interface or closing the client. - Define which RpcClient implementation to construct by setting a configuration variable which defaults to the current RpcClient. - Possibly create an abstract RpcClient class to only load all the basic Rpc layer configurations to be used in an implementation. Why? It enables experimentation with RpcClients which could enable new features or could be more performant than the included client. I created a new RpcClient implementation based on Netty which can also be called asynchronously. It would be great to also be able to use this RpcClient in all the default ways and tests to see if there are any issues with it. https://github.com/jurmous/async-hbase-client/ https://github.com/jurmous/async-hbase-client/blob/master/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234432#comment-14234432 ] Hudson commented on HBASE-12634: SUCCESS: Integrated in HBase-0.98 #716 (See [https://builds.apache.org/job/HBase-0.98/716/]) HBASE-12634 Fix the AccessController#requireGlobalPermission(ns) with NS (matteo.bertozzi: rev 6c927313c5ac7275bce3f9cdc4d215db28264601) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java Fix the AccessController#requireGlobalPermission(ns) with NS - Key: HBASE-12634 URL: https://issues.apache.org/jira/browse/HBASE-12634 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.8 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, HBASE-12634.patch The namespace argument passed to AccessController#requireGlobalPermission API to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234431#comment-14234431 ] Hudson commented on HBASE-12634: SUCCESS: Integrated in HBase-TRUNK #5866 (See [https://builds.apache.org/job/HBase-TRUNK/5866/]) HBASE-12634 Fix the AccessController#requireGlobalPermission(ns) with NS (matteo.bertozzi: rev 0299ab8bf69618d7e07a6ec7071ce9234d9d) * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java Fix the AccessController#requireGlobalPermission(ns) with NS - Key: HBASE-12634 URL: https://issues.apache.org/jira/browse/HBASE-12634 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.8 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, HBASE-12634.patch The namespace argument passed to AccessController#requireGlobalPermission API to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234440#comment-14234440 ] Andrew Purtell commented on HBASE-12636: What about recording this information into a system table instead? We can start the migration of replication state out of ZooKeeper. Avoid too many write operations on zookeeper in replication --- Key: HBASE-12636 URL: https://issues.apache.org/jira/browse/HBASE-12636 Project: HBase Issue Type: Improvement Affects Versions: 0.94.11 Reporter: Liu Shaohui Assignee: Liu Shaohui Labels: replication Fix For: 1.0.0 Attachments: HBASE-12636-v1.diff In our production cluster, we found there are about over 1k write operations per second on zookeeper from hbase replication. The reason is that the replication source will write the log position to zookeeper for every edit shipping. If the current replicating WAL is just the WAL that regionserver is writing to, each skipping will be very small but the frequency is very high, which causes many write operations on zookeeper. A simple solution is that writing log position to zookeeper when position diff or skipped edit number is larger than a threshold, not every edit shipping. Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12635: --- Fix Version/s: 0.98.9 1.0.0 Delete acl notify znode of table after the table is deleted --- Key: HBASE-12635 URL: https://issues.apache.org/jira/browse/HBASE-12635 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12635-v1.diff In our multi-tenant hbase cluster, we found that there are over 1M znodes under the acl node. The reason is that users create and delete tables with different names frequently. The acl notify znode are left there after the tables are deleted. Simple solution is that deleting acl notify znode of table in AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1423#comment-1423 ] Andrew Purtell commented on HBASE-12635: +1, I'll backport and commit to all branches shortly unless objection. Sorry [~liushaohui], we should have been doing this already and I'm surprised we are not. Delete acl notify znode of table after the table is deleted --- Key: HBASE-12635 URL: https://issues.apache.org/jira/browse/HBASE-12635 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12635-v1.diff In our multi-tenant hbase cluster, we found that there are over 1M znodes under the acl node. The reason is that users create and delete tables with different names frequently. The acl notify znode are left there after the tables are deleted. Simple solution is that deleting acl notify znode of table in AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12632) ThrottledInputStream/ExportSnapshot does not throttle
[
https://issues.apache.org/jira/browse/HBASE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234449#comment-14234449
]
Tobi Vollebregt commented on HBASE-12632:
-
Thanks Ted. Patch LGTM. I haven't tested the version with {{while}} yet though,
I will try to get around to that next week.
ThrottledInputStream/ExportSnapshot does not throttle
-
Key: HBASE-12632
URL: https://issues.apache.org/jira/browse/HBASE-12632
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.99.2
Reporter: Tobi Vollebregt
Attachments: 12632-v1.txt
I just transferred a ton of data using ExportSnapshot with bandwidth
throttling from one Hadoop cluster to another Hadoop cluster, and discovered
that ThrottledInputStream does not limit bandwidth.
The problem is that ThrottledInputStream sleeps once, for a fixed time (50
ms), at the start of each read call, disregarding the actual amount of data
read.
ExportSnapshot defaults to a buffer size as big as the block size of the
outputFs:
{code:java}
// Use the default block size of the outputFs if bigger
int defaultBlockSize = Math.max((int) outputFs.getDefaultBlockSize(),
BUFFER_SIZE);
bufferSize = conf.getInt(CONF_BUFFER_SIZE, defaultBlockSize);
LOG.info(Using bufferSize= +
StringUtils.humanReadableInt(bufferSize));
{code}
In my case, this was 256MB.
Hence, the ExportSnapshot mapper will attempt to read up to 256 MB at a time,
each time sleeping only 50ms. Thus, in the worst case where each call to read
fills the 256 MB buffer in negligible time, the ThrottledInputStream cannot
reduce the bandwidth to under (256 MB) / (5 ms) = 5 GB/s.
Even in a more realistic case where read returns about 1 MB per call, it
still cannot throttle the bandwidth to under 20 MB/s.
The issue is exacerbated by the fact that you need to set a low limit because
the total bandwidth per host depends on the number of mapper slots as well.
A simple solution would change the if in throttle to a while, so that it
keeps sleeping for 50 ms until the rate is finally low enough:
{code:java}
private void throttle() throws IOException {
while (getBytesPerSec() maxBytesPerSec) {
try {
Thread.sleep(SLEEP_DURATION_MS);
totalSleepTime += SLEEP_DURATION_MS;
} catch (InterruptedException e) {
throw new IOException(Thread aborted, e);
}
}
}
{code}
This issue affects the ThrottledInputStream in hadoop as well.
Another way to see this is that for big enough buffer sizes,
ThrottledInputStream will be throttling only the number of read calls to 20
per second, disregarding the number of bytes read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12358) Create ByteBuffer backed Cell
[ https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234450#comment-14234450 ] ramkrishna.s.vasudevan commented on HBASE-12358: Before we decide on BB or BR, we would like to highlight some points here so that we could decide on the behaviour of the new Cell APIs. Assume we will be working with BBs so we would introduce getXXXBuffer() APIs and also hasArray() in Cell itself directly. If we try to extend the cell or create a new Cell then everywhere we need to do instanceOf check or do type conversion and that is why adding new APIS to Cell interface itself makes sense. Plan is to use this getXXXBuffer API through out the read path. Now there are two ways to use it 1) Use getXXXBuffer along with getXXXOffset, getXXXLength like how we use now for getXXXArray APIs with the offset and length. Doing so would ensure that every where in the filters and CP one has to just replace the getXXXArray with getXXXBuffer and continue to use getXXXOffset and getXXXLength. We would do some wrapping of the byte[] with a BB incase of KeyValue type of cells so that getXXXBuffer along with offset and length holds true everywhere. Note that here if hasARray is false (for KV case) then getXXXArray would also work. 2)The other way of using this is that use only getXXXBuffer() API and ensure that the BB is always duplicated/sliced and only the portion of the total BB is returned which represents the individual component of the Cell. In this case there is no use of getXXXOffset (as it is going to be 0) and getXXXLength() is any way going to be the sliced BB's limit. But in the 2nd approach we may end up in creating lot of small objects even while doing comparison. Now the next problem that comes is what to do with the getXXXArray() APIs. If one sees hasArray() as false (a DBB backed Cell) and uses the getXXXArray() API along with offset and length - what should we do. Should we create a byte[] from the DBB and return it? Then in that case what would should the getXXXOffset() return for a getXXXBuffer or getXXXArray()? If we go with the 2nd approach then getXXXBuffer() should be clearly documented saying that it has to be used without Offset and length and use getXXXOFfset and getXXXLength only with getXXXArray(). Now if a Cell is backed by on heap BB then we could definitely return getXXXArray() also - but what to return in the getXXXOffset would be determined by what appraoch to use for getXXXBuffer. (based on (1) and (2)). We wanted to open up this topic now so that to get some feedback on what could be an option here. Since it is an user facing Interface we need to be careful with this. I would suggest that whenever a Cell is BB backed(Onheap or offheap) always hasARray would be false in that Cell impl. Every where we would use getXXXBuffer along with getXXXOffest and getXXXLength. Even in case of KV we could wrap the byte[] with BB so that we have uniformity through the read code and we don't have too many 'if' else conditions. When ever hasArray is false - using getXXXArray API would throw UnSupportedOperation Exception. AS said if we want getXXXArray to be supported as per the existing way then getXXXBuffer and getXXXOffset, getXXXLength should be clearly documented. Thoughts!!! Create ByteBuffer backed Cell - Key: HBASE-12358 URL: https://issues.apache.org/jira/browse/HBASE-12358 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12358.patch, HBASE-12358_1.patch, HBASE-12358_2.patch As part of HBASE-12224 and HBASE-12282 we wanted a Cell that is backed by BB. Changing the core Cell impl would not be needed as it is used in server only. So we will create a BB backed Cell and use it in the Server side read path. This JIRA just creates an interface that extends Cell and adds the needed API. The getTimeStamp and getTypebyte() can still refer to the original Cell API only. The getXXxOffset() and getXXXLength() can also refer to the original Cell only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12498) Integrate HydraBase RMap in RegionManager
[ https://issues.apache.org/jira/browse/HBASE-12498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amitanand Aiyer updated HBASE-12498: Description: We need to integrate the idea of RMap (Global Region Locator) into HBase. RMap: In Hydrabase, we wanted to avoid having special tables that require to be online and available, before other servers can start processing their regions. To this end, in hydrabase, we use a notion of RMap — that tells the servers which regions to open; and also servers as a look up place for the clients to discover the set of servers handling a particular region. This RMap is stored in a well known path, as specified in the configuration. Upon initialization, RegionServers read this file and determine the set of regions that it needs to open; and the set of peers that it expects to hear from for each of those regions. Since RMap is stored in HDFS, the only way to change it is to create a new HDFS file and update the servers about it. This is intended to be a not-so-common operation; however, this may be required when a new machine joins the service, or an existing machine needs to be taken away for a significant amount of time to perform maintenance. We run a daemon, every 15 minutes, (currently in the master) to determine the set of machines going to repair and vice versa; and update the RMap accordingly. 1. In the current hydraBase model, updating the RMap is done by a single daemon; while every region server individually reads and updates itself accordingly. 2. OSS HBase seems to be using a model where a dedicated master/server manager is responsible for coordinating the regions that should be opened/served by each region server. Region servers simply follow the instructions received from the server manager. With respect to integrating both. We plan to use the servermanager to read the RMap and coordinate the region opens on various regionservers. RMap will be still stored as a HDFS file, that will be read by the ServerManager/Master to assign regions. was: We need to integrate the idea of RMap (Global Region Locator) into HBase. Amit: Can you add more details about the design on this JIRA. Thanks, Rishit Integrate HydraBase RMap in RegionManager - Key: HBASE-12498 URL: https://issues.apache.org/jira/browse/HBASE-12498 Project: HBase Issue Type: Sub-task Components: master, Region Assignment Reporter: Rishit Shroff Assignee: Amitanand Aiyer We need to integrate the idea of RMap (Global Region Locator) into HBase. RMap: In Hydrabase, we wanted to avoid having special tables that require to be online and available, before other servers can start processing their regions. To this end, in hydrabase, we use a notion of RMap — that tells the servers which regions to open; and also servers as a look up place for the clients to discover the set of servers handling a particular region. This RMap is stored in a well known path, as specified in the configuration. Upon initialization, RegionServers read this file and determine the set of regions that it needs to open; and the set of peers that it expects to hear from for each of those regions. Since RMap is stored in HDFS, the only way to change it is to create a new HDFS file and update the servers about it. This is intended to be a not-so-common operation; however, this may be required when a new machine joins the service, or an existing machine needs to be taken away for a significant amount of time to perform maintenance. We run a daemon, every 15 minutes, (currently in the master) to determine the set of machines going to repair and vice versa; and update the RMap accordingly. 1. In the current hydraBase model, updating the RMap is done by a single daemon; while every region server individually reads and updates itself accordingly. 2. OSS HBase seems to be using a model where a dedicated master/server manager is responsible for coordinating the regions that should be opened/served by each region server. Region servers simply follow the instructions received from the server manager. With respect to integrating both. We plan to use the servermanager to read the RMap and coordinate the region opens on various regionservers. RMap will be still stored as a HDFS file, that will be read by the ServerManager/Master to assign regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12583) Allow creating reference files even the split row not lies in the storefile range if required
[
https://issues.apache.org/jira/browse/HBASE-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234455#comment-14234455
]
Andrew Purtell commented on HBASE-12583:
+1 for 0.98 if earlier reviewers are ok with the particulars. Skimmed the
patch, looks ok
Allow creating reference files even the split row not lies in the storefile
range if required
-
Key: HBASE-12583
URL: https://issues.apache.org/jira/browse/HBASE-12583
Project: HBase
Issue Type: Improvement
Reporter: rajeshbabu
Assignee: rajeshbabu
Labels: Phoenix
Fix For: 1.0.0, 2.0.0, 0.98.9
Attachments: HBASE-12583.patch, HBASE-12583_98.patch,
HBASE-12583_branch1.patch, HBASE-12583_v2.patch
Currently in HRegionFileSystem#splitStoreFile we are not creating reference
files if the split row not lies in the storefile range that means one of the
child region doesn't have any data.
{code}
// Check whether the split row lies in the range of the store file
// If it is outside the range, return directly.
if (top) {
//check if larger than last key.
KeyValue splitKey = KeyValueUtil.createFirstOnRow(splitRow);
byte[] lastKey = f.createReader().getLastKey();
// If lastKey is null means storefile is empty.
if (lastKey == null) return null;
if (f.getReader().getComparator().compareFlatKey(splitKey.getBuffer(),
splitKey.getKeyOffset(), splitKey.getKeyLength(), lastKey, 0,
lastKey.length) 0) {
return null;
}
} else {
//check if smaller than first key
KeyValue splitKey = KeyValueUtil.createLastOnRow(splitRow);
byte[] firstKey = f.createReader().getFirstKey();
// If firstKey is null means storefile is empty.
if (firstKey == null) return null;
if (f.getReader().getComparator().compareFlatKey(splitKey.getBuffer(),
splitKey.getKeyOffset(), splitKey.getKeyLength(), firstKey, 0,
firstKey.length) 0) {
return null;
}
}
{code}
In some cases when split row should be compared with part of rowkey(in
composite rowkey) mainly for secondary index tables we need to create
reference files even when split row not lies in the storefile range so that
they can be rewritten to it's child regions by some custom half store file
reader which compare the part of row key with split row.
The check of comparing split row with storefile range and returning directly
can be avoided by having special boolean attribute in table descriptor when
it set to true. Or else we can have coprocessor hooks so that in the hooks we
can create the references and bypass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234458#comment-14234458 ] Hudson commented on HBASE-12634: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #683 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/683/]) HBASE-12634 Fix the AccessController#requireGlobalPermission(ns) with NS (matteo.bertozzi: rev 6c927313c5ac7275bce3f9cdc4d215db28264601) * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java Fix the AccessController#requireGlobalPermission(ns) with NS - Key: HBASE-12634 URL: https://issues.apache.org/jira/browse/HBASE-12634 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.8 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, HBASE-12634.patch The namespace argument passed to AccessController#requireGlobalPermission API to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12622) user_permission should require global admin to display global and ns permissions
[ https://issues.apache.org/jira/browse/HBASE-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234464#comment-14234464 ] Andrew Purtell commented on HBASE-12622: Ok, thanks [~ashish singhi] user_permission should require global admin to display global and ns permissions Key: HBASE-12622 URL: https://issues.apache.org/jira/browse/HBASE-12622 Project: HBase Issue Type: Bug Components: security Affects Versions: 2.0.0, 0.98.8, 0.99.2 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12622-v0.patch, HBASE-12622-v1.patch user_permission check the user permission only on the table level (requiring at least a table-level admin) global and namespace permission listing is done without checking anything. but only a global admins should be able to perform this operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12358) Create ByteBuffer backed Cell
[ https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234468#comment-14234468 ] Anoop Sam John commented on HBASE-12358: bq.Note that here if hasARray is false (for KV case) then getXXXArray would also work. Correction : hasArray() is true Create ByteBuffer backed Cell - Key: HBASE-12358 URL: https://issues.apache.org/jira/browse/HBASE-12358 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12358.patch, HBASE-12358_1.patch, HBASE-12358_2.patch As part of HBASE-12224 and HBASE-12282 we wanted a Cell that is backed by BB. Changing the core Cell impl would not be needed as it is used in server only. So we will create a BB backed Cell and use it in the Server side read path. This JIRA just creates an interface that extends Cell and adds the needed API. The getTimeStamp and getTypebyte() can still refer to the original Cell API only. The getXXxOffset() and getXXXLength() can also refer to the original Cell only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12619) Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234480#comment-14234480 ] Andrew Purtell commented on HBASE-12619: I made a pass over this and it seems fine. Let me see about committing this today, it should be in place ahead of HBASE-11764 because there's a small conflict with tag type allocation. (I'll fix that in HBASE-11764). Will take a closer look then. One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be possible today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality. But that's not an issue for here, HBASE-11639 has already been reviewed and committed and this is a backport. Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98 --- Key: HBASE-12619 URL: https://issues.apache.org/jira/browse/HBASE-12619 Project: HBase Issue Type: Improvement Components: security Affects Versions: 0.98.8 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: visibility Fix For: 0.98.9 Attachments: HBASE-11639_0.98_1.patch Raising a back port issue for HBASE-11639. We have the patch ready just need to see how we are handling the new CP hook in RS Observer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234485#comment-14234485 ] Andrew Purtell commented on HBASE-11639: One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be possible today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality. Sorry I couldn't come here earlier, and anyway what has been committed is fine, but we should figure out what is the best next step here, if any. One option would be introducing a replication observer interface - ReplicationObserver - and coprocessor upcall chains out of replication code. Custom replication endpoint specification could be extended to allow specification of observer stacks. /cc [~lhofhansl] [~jesse_yates] [Visibility controller] Replicate the visibility of Cells as strings Key: HBASE-11639 URL: https://issues.apache.org/jira/browse/HBASE-11639 Project: HBase Issue Type: Improvement Components: Replication, security Affects Versions: 0.98.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: VisibilityLabels Fix For: 1.0.0, 2.0.0 Attachments: HBASE-11639_amend.patch, HBASE-11639_v11.patch, HBASE-11639_v13.patch, HBASE-11639_v13.patch, HBASE-11639_v14.patch, HBASE-11639_v15.patch, HBASE-11639_v16.patch, HBASE-11639_v17.patch, HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch, HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch, HBASE-11639_v9.patch This issue is aimed at persisting the visibility labels as strings in the WAL rather than Label ordinals. This would help in replicating the label ordinals to the replication cluster as strings directly and also that after HBASE-11553 would help because the replication cluster could have an implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234485#comment-14234485 ] Andrew Purtell edited comment on HBASE-11639 at 12/4/14 7:06 PM: - One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be guaranteed today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality, each implementation must take care to delegate. Sorry I couldn't come here earlier, and anyway what has been committed is fine, the VisibilityReplicationEndpoint delegates to whatever it finds in place, but we should figure out what is the best next step here, if any. One option would be introducing a replication observer interface - ReplicationObserver - and coprocessor upcall chains out of replication code. Custom replication endpoint specification could be extended to allow specification of observer stacks. /cc [~lhofhansl] [~jesse_yates] was (Author: apurtell): One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be possible today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality. Sorry I couldn't come here earlier, and anyway what has been committed is fine, but we should figure out what is the best next step here, if any. One option would be introducing a replication observer interface - ReplicationObserver - and coprocessor upcall chains out of replication code. Custom replication endpoint specification could be extended to allow specification of observer stacks. /cc [~lhofhansl] [~jesse_yates] [Visibility controller] Replicate the visibility of Cells as strings Key: HBASE-11639 URL: https://issues.apache.org/jira/browse/HBASE-11639 Project: HBase Issue Type: Improvement Components: Replication, security Affects Versions: 0.98.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: VisibilityLabels Fix For: 1.0.0, 2.0.0 Attachments: HBASE-11639_amend.patch, HBASE-11639_v11.patch, HBASE-11639_v13.patch, HBASE-11639_v13.patch, HBASE-11639_v14.patch, HBASE-11639_v15.patch, HBASE-11639_v16.patch, HBASE-11639_v17.patch, HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch, HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch, HBASE-11639_v9.patch This issue is aimed at persisting the visibility labels as strings in the WAL rather than Label ordinals. This would help in replicating the label ordinals to the replication cluster as strings directly and also that after HBASE-11553 would help because the replication cluster could have an implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12619) Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234480#comment-14234480 ] Andrew Purtell edited comment on HBASE-12619 at 12/4/14 7:06 PM: - I made a pass over this and it seems fine. Let me see about committing this today, it should be in place ahead of HBASE-11764 because there's a small conflict with tag type allocation. (I'll fix that in HBASE-11764). Will take a closer look then. One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be guaranteed today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality, each implementation must take care to delegate. But that's not an issue for here, HBASE-11639 has already been reviewed and committed and this is a backport. was (Author: apurtell): I made a pass over this and it seems fine. Let me see about committing this today, it should be in place ahead of HBASE-11764 because there's a small conflict with tag type allocation. (I'll fix that in HBASE-11764). Will take a closer look then. One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be possible today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality. But that's not an issue for here, HBASE-11639 has already been reviewed and committed and this is a backport. Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98 --- Key: HBASE-12619 URL: https://issues.apache.org/jira/browse/HBASE-12619 Project: HBase Issue Type: Improvement Components: security Affects Versions: 0.98.8 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: visibility Fix For: 0.98.9 Attachments: HBASE-11639_0.98_1.patch Raising a back port issue for HBASE-11639. We have the patch ready just need to see how we are handling the new CP hook in RS Observer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234491#comment-14234491 ] Anoop Sam John commented on HBASE-11639: I think there is no issue with custom replication endpoint. What we do is create a wrapper on top of the replication EP. This is done in VC new hook impl. So this will wrap the custom Replication EP. System calls the wrapper 1st and it convert visibility to String based and then the custom EP which can do its function. Does that sounds good enough Andy? [Visibility controller] Replicate the visibility of Cells as strings Key: HBASE-11639 URL: https://issues.apache.org/jira/browse/HBASE-11639 Project: HBase Issue Type: Improvement Components: Replication, security Affects Versions: 0.98.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: VisibilityLabels Fix For: 1.0.0, 2.0.0 Attachments: HBASE-11639_amend.patch, HBASE-11639_v11.patch, HBASE-11639_v13.patch, HBASE-11639_v13.patch, HBASE-11639_v14.patch, HBASE-11639_v15.patch, HBASE-11639_v16.patch, HBASE-11639_v17.patch, HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch, HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch, HBASE-11639_v9.patch This issue is aimed at persisting the visibility labels as strings in the WAL rather than Label ordinals. This would help in replicating the label ordinals to the replication cluster as strings directly and also that after HBASE-11553 would help because the replication cluster could have an implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12619) Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234490#comment-14234490 ] Andrew Purtell commented on HBASE-12619: /cc [~lhofhansl] [~jesse_yates], private replication API changes Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98 --- Key: HBASE-12619 URL: https://issues.apache.org/jira/browse/HBASE-12619 Project: HBase Issue Type: Improvement Components: security Affects Versions: 0.98.8 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: visibility Fix For: 0.98.9 Attachments: HBASE-11639_0.98_1.patch Raising a back port issue for HBASE-11639. We have the patch ready just need to see how we are handling the new CP hook in RS Observer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234494#comment-14234494 ] Ted Yu commented on HBASE-12204: Test suite passes on branch-1 Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 1.0.0 Attachments: 12204-1.0.txt The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12565) Race condition in HRegion.batchMutate() causes partial data to be written when region closes
[ https://issues.apache.org/jira/browse/HBASE-12565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith David Winkler updated HBASE-12565: Attachment: hbase-12565.patch Applies the two part recommended solution from the description. Divided the existing batchMutate() test into two tests, one that performs batchMutate while no row locks are held, one that performs batch mutate while multiple row locks are held and concurrently another thread calls region.close(). Race condition in HRegion.batchMutate() causes partial data to be written when region closes - Key: HBASE-12565 URL: https://issues.apache.org/jira/browse/HBASE-12565 Project: HBase Issue Type: Bug Components: Performance, regionserver Affects Versions: 0.98.6 Reporter: Scott Fines Attachments: hbase-12565.patch The following sequence of events is possible to occur in HRegion's batchMutate() call: 1. caller attempts to call HRegion.batchMutate() with a batch of N1 records 2. batchMutate acquires region lock in startRegionOperation, then calls doMiniBatchMutation() 3. doMiniBatchMutation acquires one row lock 4. Region closes 5. doMiniBatchMutation attempts to acquire second row lock. When this happens, the lock acquisition will also attempt to acquire the region lock, which fails (because the region is closing). At this stage, doMiniBatchMutation will stop writing further, BUT it WILL write data for the rows whose locks have already been acquired, and advance the index in MiniBatchOperationInProgress. Then, after it terminates successfully, batchMutate() will loop around a second time, and attempt AGAIN to acquire the region closing lock. When that happens, a NotServingRegionException is thrown back to the caller. Thus, we have a race condition where partial data can be written when a region server is closing. The main problem stems from the location of startRegionOperation() calls in batchMutate and doMiniBatchMutation(): 1. batchMutate() reacquires the region lock with each iteration of the loop, which can cause some successful writes to occur, but then fail on others 2. getRowLock() attempts to acquire the region lock once for each row, which allows doMiniBatchMutation to terminate early; this forces batchMutate() to use multiple iterations and results in condition 1 being hit. There appears to be two parts to the solution as well: 1. open an internal path so that doMiniBatchMutation() can acquire row locks without checking for region closure. This will have the added benefit of a significant performance improvement during large batch mutations. 2. move the startRegionOperation() out of the loop in batchMutate() so that multiple iterations of doMiniBatchMutation will not cause the operation to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11677) Make Logger instance modifiers consistent
[
https://issues.apache.org/jira/browse/HBASE-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234501#comment-14234501
]
Andrew Purtell commented on HBASE-11677:
Please make a patch for trunk [~Usha710] and then we can take these changes.
Make Logger instance modifiers consistent
-
Key: HBASE-11677
URL: https://issues.apache.org/jira/browse/HBASE-11677
Project: HBase
Issue Type: Task
Reporter: Sean Busbey
Priority: Minor
Labels: beginner, sonar
Attachments: HBASE-11677-0.98.patch
We have some instances of Logger that are missing one of being private,
static, and final.
ex from HealthChecker.java, missing final
{code}
private static Log LOG = LogFactory.getLog(HealthChecker.class);
{code}
* Clean up where possible by making {{private static final}}
* If we can't, add a non-javadoc note about why
One way to look for problematic instances is to grep for initial assignment
for the commonly used LOG member, e.g.
* missing final: {{grep -r LOG = * | grep -v final}}
* missing static: {{grep -r LOG = * | grep -v static}}
* missing private: {{grep -r LOG = * | grep -v private}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12564) consolidate the getTableDescriptors() semantic
[ https://issues.apache.org/jira/browse/HBASE-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234504#comment-14234504 ] Andrew Purtell commented on HBASE-12564: bq. I think the behavior should be give me all the names that I have access to so the AccessController should filter the tables like getTableDescriptors() but checking also Read/Write rights and not only Admin/Create. Andrew Purtell? No, users with read or write access should not be granted permission to potentially sensitive information in the schema. consolidate the getTableDescriptors() semantic -- Key: HBASE-12564 URL: https://issues.apache.org/jira/browse/HBASE-12564 Project: HBase Issue Type: Bug Components: Client, master Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0 Attachments: HBASE-12564-v0.patch, HBASE-12564-v1.patch, HBASE-12564-v2.patch Master getTableDescriptors() which is called by Admin.listTables() has a couple of different behaviors depending on how it is called. after HBASE-12073 with the AccessController enabled, we now get a global admin required if listTables() is called without a regex otherwise we return only the table that the user can see (we show only the tables that the user have access to, which means or the user is a global admin or it has a table-level create/admin). We probably should have the second behavior even without regex, since I should able to see my own tables. getTableDescriptors() is returning only non system tables. Tools like user_permission that are doing for each listTable(): userPerm(table) are losing the system tables, so stuff like user_permission 'hbase:acls' will not result any result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12565) Race condition in HRegion.batchMutate() causes partial data to be written when region closes
[ https://issues.apache.org/jira/browse/HBASE-12565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith David Winkler updated HBASE-12565: Affects Version/s: 2.0.0 Status: Patch Available (was: Open) Race condition in HRegion.batchMutate() causes partial data to be written when region closes - Key: HBASE-12565 URL: https://issues.apache.org/jira/browse/HBASE-12565 Project: HBase Issue Type: Bug Components: Performance, regionserver Affects Versions: 0.98.6, 2.0.0 Reporter: Scott Fines Attachments: hbase-12565.patch The following sequence of events is possible to occur in HRegion's batchMutate() call: 1. caller attempts to call HRegion.batchMutate() with a batch of N1 records 2. batchMutate acquires region lock in startRegionOperation, then calls doMiniBatchMutation() 3. doMiniBatchMutation acquires one row lock 4. Region closes 5. doMiniBatchMutation attempts to acquire second row lock. When this happens, the lock acquisition will also attempt to acquire the region lock, which fails (because the region is closing). At this stage, doMiniBatchMutation will stop writing further, BUT it WILL write data for the rows whose locks have already been acquired, and advance the index in MiniBatchOperationInProgress. Then, after it terminates successfully, batchMutate() will loop around a second time, and attempt AGAIN to acquire the region closing lock. When that happens, a NotServingRegionException is thrown back to the caller. Thus, we have a race condition where partial data can be written when a region server is closing. The main problem stems from the location of startRegionOperation() calls in batchMutate and doMiniBatchMutation(): 1. batchMutate() reacquires the region lock with each iteration of the loop, which can cause some successful writes to occur, but then fail on others 2. getRowLock() attempts to acquire the region lock once for each row, which allows doMiniBatchMutation to terminate early; this forces batchMutate() to use multiple iterations and results in condition 1 being hit. There appears to be two parts to the solution as well: 1. open an internal path so that doMiniBatchMutation() can acquire row locks without checking for region closure. This will have the added benefit of a significant performance improvement during large batch mutations. 2. move the startRegionOperation() out of the loop in batchMutate() so that multiple iterations of doMiniBatchMutation will not cause the operation to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12422) Use ConnectionFactory in HTable constructors
[ https://issues.apache.org/jira/browse/HBASE-12422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12422: -- Attachment: HBASE-12422.patch Agree, these should be fine if their close is going to clean up the connection. Retry hadoopqa. Use ConnectionFactory in HTable constructors Key: HBASE-12422 URL: https://issues.apache.org/jira/browse/HBASE-12422 Project: HBase Issue Type: Bug Components: Client Reporter: Solomon Duskis Assignee: Solomon Duskis Priority: Minor Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12422.patch, HBASE-12422.patch, HBASE-12422.patch Use ConnectionFactory in HTable instead of ConnectionManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12564) consolidate the getTableDescriptors() semantic
[ https://issues.apache.org/jira/browse/HBASE-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234516#comment-14234516 ] Andrew Purtell commented on HBASE-12564: Anyway I looked at the v2 patch and it seems to do the right thing, we won't return descriptors unless the user has ADMIN or CREATE permissions in dominant scope. In postGetTableNames we will return table _names_ if the user has any of the Action bits in dominant scope, but that doesn't seem to be a problem, it follows that a user with any access to a table must know its name to make use of it. consolidate the getTableDescriptors() semantic -- Key: HBASE-12564 URL: https://issues.apache.org/jira/browse/HBASE-12564 Project: HBase Issue Type: Bug Components: Client, master Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0 Attachments: HBASE-12564-v0.patch, HBASE-12564-v1.patch, HBASE-12564-v2.patch Master getTableDescriptors() which is called by Admin.listTables() has a couple of different behaviors depending on how it is called. after HBASE-12073 with the AccessController enabled, we now get a global admin required if listTables() is called without a regex otherwise we return only the table that the user can see (we show only the tables that the user have access to, which means or the user is a global admin or it has a table-level create/admin). We probably should have the second behavior even without regex, since I should able to see my own tables. getTableDescriptors() is returning only non system tables. Tools like user_permission that are doing for each listTable(): userPerm(table) are losing the system tables, so stuff like user_permission 'hbase:acls' will not result any result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12564) consolidate the getTableDescriptors() semantic
[
https://issues.apache.org/jira/browse/HBASE-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234525#comment-14234525
]
Matteo Bertozzi commented on HBASE-12564:
-
{quote}No, users with read or write access should not be granted permission to
potentially sensitive information in the schema.{quote}
yeah, sorry maybe I was not clear. I was talking about the getTableNames()
which just returns the table names. In that case if the user have access to
the table (by access I mean any RWXCA on any level of the table) the shell
list will display the names (only names).
while for the descriptor as you said only users granted with ADMIN/CREATE can
see the table schema.
consolidate the getTableDescriptors() semantic
--
Key: HBASE-12564
URL: https://issues.apache.org/jira/browse/HBASE-12564
Project: HBase
Issue Type: Bug
Components: Client, master
Affects Versions: 2.0.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-12564-v0.patch, HBASE-12564-v1.patch,
HBASE-12564-v2.patch
Master getTableDescriptors() which is called by Admin.listTables() has a
couple of different behaviors depending on how it is called.
after HBASE-12073 with the AccessController enabled, we now get a global
admin required if listTables() is called without a regex otherwise we return
only the table that the user can see (we show only the tables that the user
have access to, which means or the user is a global admin or it has a
table-level create/admin). We probably should have the second behavior even
without regex, since I should able to see my own tables.
getTableDescriptors() is returning only non system tables. Tools like
user_permission that are doing for each listTable(): userPerm(table) are
losing the system tables, so stuff like user_permission 'hbase:acls' will not
result any result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12622) user_permission should require global admin to display global and ns permissions
[ https://issues.apache.org/jira/browse/HBASE-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234533#comment-14234533 ] Srikanth Srungarapu commented on HBASE-12622: - +1 (non-binding). user_permission should require global admin to display global and ns permissions Key: HBASE-12622 URL: https://issues.apache.org/jira/browse/HBASE-12622 Project: HBase Issue Type: Bug Components: security Affects Versions: 2.0.0, 0.98.8, 0.99.2 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12622-v0.patch, HBASE-12622-v1.patch user_permission check the user permission only on the table level (requiring at least a table-level admin) global and namespace permission listing is done without checking anything. but only a global admins should be able to perform this operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12583) Allow creating reference files even the split row not lies in the storefile range if required
[
https://issues.apache.org/jira/browse/HBASE-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234555#comment-14234555
]
stack commented on HBASE-12583:
---
Would it make more sense and cleaner passing the splitPolicy in when you call
HRegionFS#splitStoreFile? Now you have amended code base to pass a boolean.
Later, splitting store files we might want to pass more info. We could pass it
as extra args or just pass the policy altogether now to future proof.
Otherwise, patch is fine by me.
Allow creating reference files even the split row not lies in the storefile
range if required
-
Key: HBASE-12583
URL: https://issues.apache.org/jira/browse/HBASE-12583
Project: HBase
Issue Type: Improvement
Reporter: rajeshbabu
Assignee: rajeshbabu
Labels: Phoenix
Fix For: 1.0.0, 2.0.0, 0.98.9
Attachments: HBASE-12583.patch, HBASE-12583_98.patch,
HBASE-12583_branch1.patch, HBASE-12583_v2.patch
Currently in HRegionFileSystem#splitStoreFile we are not creating reference
files if the split row not lies in the storefile range that means one of the
child region doesn't have any data.
{code}
// Check whether the split row lies in the range of the store file
// If it is outside the range, return directly.
if (top) {
//check if larger than last key.
KeyValue splitKey = KeyValueUtil.createFirstOnRow(splitRow);
byte[] lastKey = f.createReader().getLastKey();
// If lastKey is null means storefile is empty.
if (lastKey == null) return null;
if (f.getReader().getComparator().compareFlatKey(splitKey.getBuffer(),
splitKey.getKeyOffset(), splitKey.getKeyLength(), lastKey, 0,
lastKey.length) 0) {
return null;
}
} else {
//check if smaller than first key
KeyValue splitKey = KeyValueUtil.createLastOnRow(splitRow);
byte[] firstKey = f.createReader().getFirstKey();
// If firstKey is null means storefile is empty.
if (firstKey == null) return null;
if (f.getReader().getComparator().compareFlatKey(splitKey.getBuffer(),
splitKey.getKeyOffset(), splitKey.getKeyLength(), firstKey, 0,
firstKey.length) 0) {
return null;
}
}
{code}
In some cases when split row should be compared with part of rowkey(in
composite rowkey) mainly for secondary index tables we need to create
reference files even when split row not lies in the storefile range so that
they can be rewritten to it's child regions by some custom half store file
reader which compare the part of row key with split row.
The check of comparing split row with storefile range and returning directly
can be avoided by having special boolean attribute in table descriptor when
it set to true. Or else we can have coprocessor hooks so that in the hooks we
can create the references and bypass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12498) Integrate HydraBase RMap in RegionManager
[ https://issues.apache.org/jira/browse/HBASE-12498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234561#comment-14234561 ] stack commented on HBASE-12498: --- I like it that master continues to tell regionservers what they should serve. Thanks [~amitanand] Integrate HydraBase RMap in RegionManager - Key: HBASE-12498 URL: https://issues.apache.org/jira/browse/HBASE-12498 Project: HBase Issue Type: Sub-task Components: master, Region Assignment Reporter: Rishit Shroff Assignee: Amitanand Aiyer We need to integrate the idea of RMap (Global Region Locator) into HBase. RMap: In Hydrabase, we wanted to avoid having special tables that require to be online and available, before other servers can start processing their regions. To this end, in hydrabase, we use a notion of RMap — that tells the servers which regions to open; and also servers as a look up place for the clients to discover the set of servers handling a particular region. This RMap is stored in a well known path, as specified in the configuration. Upon initialization, RegionServers read this file and determine the set of regions that it needs to open; and the set of peers that it expects to hear from for each of those regions. Since RMap is stored in HDFS, the only way to change it is to create a new HDFS file and update the servers about it. This is intended to be a not-so-common operation; however, this may be required when a new machine joins the service, or an existing machine needs to be taken away for a significant amount of time to perform maintenance. We run a daemon, every 15 minutes, (currently in the master) to determine the set of machines going to repair and vice versa; and update the RMap accordingly. 1. In the current hydraBase model, updating the RMap is done by a single daemon; while every region server individually reads and updates itself accordingly. 2. OSS HBase seems to be using a model where a dedicated master/server manager is responsible for coordinating the regions that should be opened/served by each region server. Region servers simply follow the instructions received from the server manager. With respect to integrating both. We plan to use the servermanager to read the RMap and coordinate the region opens on various regionservers. RMap will be still stored as a HDFS file, that will be read by the ServerManager/Master to assign regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234569#comment-14234569 ] Enis Soztutar commented on HBASE-12204: --- I compared the patch with the 0.98 version. Seems good. +1. We should be more careful for this type of feature gaps. It seems that HBASE-12205 went into 0.98 without the branch-1 patch. Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 1.0.0 Attachments: 12204-1.0.txt The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12629) Remove hbase.regionsizecalculator.enable from RegionSizeCalculator
[ https://issues.apache.org/jira/browse/HBASE-12629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234572#comment-14234572 ] Enis Soztutar commented on HBASE-12629: --- I thought that it is a safeguard if you want to disable this. Why are we removing it? Remove hbase.regionsizecalculator.enable from RegionSizeCalculator -- Key: HBASE-12629 URL: https://issues.apache.org/jira/browse/HBASE-12629 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 1.0.0, 2.0.0 Attachments: 0001-HBASE-12629-Remove-hbase.regionsizecalculator.enable.patch The RegionSizeCalculator has a option to disable it. It is on by default and end-to-end use with it disabled is not tested or used anywhere except for a simple unit test. This removes it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12204: --- Hadoop Flags: Reviewed Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 1.0.0 Attachments: 12204-1.0.txt The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234580#comment-14234580 ] Enis Soztutar commented on HBASE-11639: --- While we are on the subject, the thing I did not fully understand in this patch and also in HBASE-11920 was that why do we do this way of getting a RE from config, then replacing that with a one coming from a CP. The RE is already pluggable, shouldn't the user configure RE accordingly, and then configure what RE to wrap in this new RE implementation rather. It seems very indirect. [Visibility controller] Replicate the visibility of Cells as strings Key: HBASE-11639 URL: https://issues.apache.org/jira/browse/HBASE-11639 Project: HBase Issue Type: Improvement Components: Replication, security Affects Versions: 0.98.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: VisibilityLabels Fix For: 1.0.0, 2.0.0 Attachments: HBASE-11639_amend.patch, HBASE-11639_v11.patch, HBASE-11639_v13.patch, HBASE-11639_v13.patch, HBASE-11639_v14.patch, HBASE-11639_v15.patch, HBASE-11639_v16.patch, HBASE-11639_v17.patch, HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch, HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch, HBASE-11639_v9.patch This issue is aimed at persisting the visibility labels as strings in the WAL rather than Label ordinals. This would help in replicating the label ordinals to the replication cluster as strings directly and also that after HBASE-11553 would help because the replication cluster could have an implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12493) User class should provide a way to re-use existing token
[ https://issues.apache.org/jira/browse/HBASE-12493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234584#comment-14234584 ] Yongzhi Chen commented on HBASE-12493: -- [~ghelmling], the hive HBaseStorageHandler does not have the info of the hbase connection, all your TokenUtil.addTokenForJob methods need connection, is that possible you add a method with addTokenForJob(Configuration conf, Job job, User user) Thanks User class should provide a way to re-use existing token Key: HBASE-12493 URL: https://issues.apache.org/jira/browse/HBASE-12493 Project: HBase Issue Type: Task Reporter: Brock Noland Assignee: Gary Helmling Fix For: 2.0.0 Attachments: HBASE-12493-v2.patch, HBASE-12493.patch In HIVE-8874 we had to re-use HBase classes market private to re-use using tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12485) Maintain SeqId monotonically increasing when Region Replica is on
[ https://issues.apache.org/jira/browse/HBASE-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12485: -- Fix Version/s: 2.0.0 1.0.0 Maintain SeqId monotonically increasing when Region Replica is on - Key: HBASE-12485 URL: https://issues.apache.org/jira/browse/HBASE-12485 Project: HBase Issue Type: Sub-task Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12485.patch We added FLUSH, REGION CLOSE events into WAL, for each those events the region SeqId is bumped. The issue comes from region close operation. Because when opening a region, we use flushed SeqId from store files while after store flush during region close we still write COMMIT_FLUSH, REGION_CLOSE events etc which respectively bump up SeqId. Therefore, the region opening SeqId is lower than it should be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12485) Maintain SeqId monotonically increasing
[ https://issues.apache.org/jira/browse/HBASE-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-12485: -- Summary: Maintain SeqId monotonically increasing (was: Maintain SeqId monotonically increasing when Region Replica is on) Maintain SeqId monotonically increasing --- Key: HBASE-12485 URL: https://issues.apache.org/jira/browse/HBASE-12485 Project: HBase Issue Type: Sub-task Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 1.0.0, 2.0.0 We added FLUSH, REGION CLOSE events into WAL, for each those events the region SeqId is bumped. The issue comes from region close operation. Because when opening a region, we use flushed SeqId from store files while after store flush during region close we still write COMMIT_FLUSH, REGION_CLOSE events etc which respectively bump up SeqId. Therefore, the region opening SeqId is lower than it should be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12485) Maintain SeqId monotonically increasing
[ https://issues.apache.org/jira/browse/HBASE-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-12485: -- Attachment: (was: HBASE-12485.patch) Maintain SeqId monotonically increasing --- Key: HBASE-12485 URL: https://issues.apache.org/jira/browse/HBASE-12485 Project: HBase Issue Type: Sub-task Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 1.0.0, 2.0.0 We added FLUSH, REGION CLOSE events into WAL, for each those events the region SeqId is bumped. The issue comes from region close operation. Because when opening a region, we use flushed SeqId from store files while after store flush during region close we still write COMMIT_FLUSH, REGION_CLOSE events etc which respectively bump up SeqId. Therefore, the region opening SeqId is lower than it should be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12485) Maintain SeqId monotonically increasing
[ https://issues.apache.org/jira/browse/HBASE-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-12485: -- Attachment: HBASE-12485.patch Resubmit for QA run Maintain SeqId monotonically increasing --- Key: HBASE-12485 URL: https://issues.apache.org/jira/browse/HBASE-12485 Project: HBase Issue Type: Sub-task Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12485.patch We added FLUSH, REGION CLOSE events into WAL, for each those events the region SeqId is bumped. The issue comes from region close operation. Because when opening a region, we use flushed SeqId from store files while after store flush during region close we still write COMMIT_FLUSH, REGION_CLOSE events etc which respectively bump up SeqId. Therefore, the region opening SeqId is lower than it should be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12493) User class should provide a way to re-use existing token
[
https://issues.apache.org/jira/browse/HBASE-12493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234613#comment-14234613
]
Gary Helmling commented on HBASE-12493:
---
[~ychena] The Hive HBaseStorageHandler should create and manage the connection,
doing something like:
{code}
Connection conn = ConnectionFactory.createConnection(job.getConfiguration());
try {
UserProvider userProvider = UserProvider.instantiate(job.getConfiguration());
TokenUtil.addTokenForJob(conn, userProvider.getCurrent(), job);
} finally {
conn.close();
}
{code}
Creating the connection and managing it should really be done explicitly. It's
clunky and inefficient for us to do this hidden from the client code, which
leads to nasty things like hidden connection caching that we've been trying to
remove from the HBase code for a long time. It's much simpler if we just make
the connection handling explicit. This is why I've deprecated all methods
which do _not_ take a {{Connection}} instance, and have to create one behind
the scenes.
Do you see any other issues?
User class should provide a way to re-use existing token
Key: HBASE-12493
URL: https://issues.apache.org/jira/browse/HBASE-12493
Project: HBase
Issue Type: Task
Reporter: Brock Noland
Assignee: Gary Helmling
Fix For: 2.0.0
Attachments: HBASE-12493-v2.patch, HBASE-12493.patch
In HIVE-8874 we had to re-use HBase classes market private to re-use using
tokens.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12637) Compilation with Hadoop-2.4- is broken
Enis Soztutar created HBASE-12637:
-
Summary: Compilation with Hadoop-2.4- is broken
Key: HBASE-12637
URL: https://issues.apache.org/jira/browse/HBASE-12637
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
HBASE-12554 introduced
{code}
public static class MockMapping extends ScriptBasedMapping {
{code}
which breaks compilation with earlier hadoop versions:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-server: Compilation failure: Compilation
failure:
[ERROR]
/Users/enis/projects/rc-test/hbase-0.99.2/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java:[77,42]
error: cannot inherit from final ScriptBasedMapping
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12637) Compilation with Hadoop-2.4- is broken
[
https://issues.apache.org/jira/browse/HBASE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12637:
--
Fix Version/s: 1.0.0
Compilation with Hadoop-2.4- is broken
---
Key: HBASE-12637
URL: https://issues.apache.org/jira/browse/HBASE-12637
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0
HBASE-12554 introduced
{code}
public static class MockMapping extends ScriptBasedMapping {
{code}
which breaks compilation with earlier hadoop versions:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-server: Compilation failure:
Compilation failure:
[ERROR]
/Users/enis/projects/rc-test/hbase-0.99.2/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java:[77,42]
error: cannot inherit from final ScriptBasedMapping
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12637) Compilation with Hadoop-2.4- is broken
[
https://issues.apache.org/jira/browse/HBASE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234626#comment-14234626
]
Ted Yu commented on HBASE-12637:
Thanks for catching this, Enis.
Compilation with Hadoop-2.4- is broken
---
Key: HBASE-12637
URL: https://issues.apache.org/jira/browse/HBASE-12637
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0
HBASE-12554 introduced
{code}
public static class MockMapping extends ScriptBasedMapping {
{code}
which breaks compilation with earlier hadoop versions:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-server: Compilation failure:
Compilation failure:
[ERROR]
/Users/enis/projects/rc-test/hbase-0.99.2/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java:[77,42]
error: cannot inherit from final ScriptBasedMapping
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12493) User class should provide a way to re-use existing token
[ https://issues.apache.org/jira/browse/HBASE-12493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234628#comment-14234628 ] Yongzhi Chen commented on HBASE-12493: -- [~ghelmling], I have no other issues, thanks User class should provide a way to re-use existing token Key: HBASE-12493 URL: https://issues.apache.org/jira/browse/HBASE-12493 Project: HBase Issue Type: Task Reporter: Brock Noland Assignee: Gary Helmling Fix For: 2.0.0 Attachments: HBASE-12493-v2.patch, HBASE-12493.patch In HIVE-8874 we had to re-use HBase classes market private to re-use using tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12637) Compilation with Hadoop-2.4- is broken
[
https://issues.apache.org/jira/browse/HBASE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12637:
--
Attachment: hbase-12637_v1.patch
Simple patch.
Compiled with 2.2.0, 2.3.0, 2.4.0, 2.5.0, and 2.6.0. And ran mvn test
-Dtest=TestBaseLoadBalancer with 2.6.0.
Compilation with Hadoop-2.4- is broken
---
Key: HBASE-12637
URL: https://issues.apache.org/jira/browse/HBASE-12637
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0
Attachments: hbase-12637_v1.patch
HBASE-12554 introduced
{code}
public static class MockMapping extends ScriptBasedMapping {
{code}
which breaks compilation with earlier hadoop versions:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-server: Compilation failure:
Compilation failure:
[ERROR]
/Users/enis/projects/rc-test/hbase-0.99.2/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java:[77,42]
error: cannot inherit from final ScriptBasedMapping
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12637) Compilation with Hadoop-2.4- is broken
[
https://issues.apache.org/jira/browse/HBASE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234638#comment-14234638
]
Ted Yu commented on HBASE-12637:
+1
Compilation with Hadoop-2.4- is broken
---
Key: HBASE-12637
URL: https://issues.apache.org/jira/browse/HBASE-12637
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0
Attachments: hbase-12637_v1.patch
HBASE-12554 introduced
{code}
public static class MockMapping extends ScriptBasedMapping {
{code}
which breaks compilation with earlier hadoop versions:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-server: Compilation failure:
Compilation failure:
[ERROR]
/Users/enis/projects/rc-test/hbase-0.99.2/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java:[77,42]
error: cannot inherit from final ScriptBasedMapping
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12625) Deprecate certain methods in classes annotated with InterfaceAudience.Public in branch-1
[
https://issues.apache.org/jira/browse/HBASE-12625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234643#comment-14234643
]
Andrew Purtell commented on HBASE-12625:
+1 for the patch on this issue
In general I wouldn't expect Phoenix to compile against branch-1, see
PHOENIX-1501. [~tedyu], if you have a list of compilation errors against
Private audience APIs handy perhaps you'd comment on the Phoenix JIRA?
Deprecate certain methods in classes annotated with InterfaceAudience.Public
in branch-1
Key: HBASE-12625
URL: https://issues.apache.org/jira/browse/HBASE-12625
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 1.0.0
Attachments: 12625-v1.txt
Compiling Phoenix against branch-1, I got compilation error below:
{code}
[ERROR]
/Users/tyu/phoenix/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java:[345,49]
cannot find symbol
[ERROR] symbol: method estimatedSizeOf(org.apache.hadoop.hbase.Cell)
[ERROR] location: class org.apache.hadoop.hbase.CellUtil
{code}
CellUtil is annotated with InterfaceAudience.Public
estimatedSizeOf() should be deprecated in branch-1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12204: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review, Enis. Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 Key: HBASE-12204 URL: https://issues.apache.org/jira/browse/HBASE-12204 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 1.0.0 Attachments: 12204-1.0.txt The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that HBASE-7767 was applied to master only. This JIRA is to backport HBASE-12016 to branch-1. This can be done when HBASE-12035 is resolved and branch-1 is brought to parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12625) Deprecate certain methods in classes annotated with InterfaceAudience.Public in branch-1
[
https://issues.apache.org/jira/browse/HBASE-12625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234648#comment-14234648
]
Andrew Purtell commented on HBASE-12625:
RpcScheduler, RpcSchedulerFactory, and friends is LimitedPrivate(PHOENIX) and
Evolving. My interpretation of what that means is we owe a deprecation cycle,
so should be deprecated in branch-1 and then we can do whatever on master. Or
revert the signature change in branch-1 and master.
Deprecate certain methods in classes annotated with InterfaceAudience.Public
in branch-1
Key: HBASE-12625
URL: https://issues.apache.org/jira/browse/HBASE-12625
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 1.0.0
Attachments: 12625-v1.txt
Compiling Phoenix against branch-1, I got compilation error below:
{code}
[ERROR]
/Users/tyu/phoenix/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java:[345,49]
cannot find symbol
[ERROR] symbol: method estimatedSizeOf(org.apache.hadoop.hbase.Cell)
[ERROR] location: class org.apache.hadoop.hbase.CellUtil
{code}
CellUtil is annotated with InterfaceAudience.Public
estimatedSizeOf() should be deprecated in branch-1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12485) Maintain SeqId monotonically increasing
[ https://issues.apache.org/jira/browse/HBASE-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234649#comment-14234649 ] stack commented on HBASE-12485: --- Throw exception rather than WARN. 3547LOG.warn(Found decreasing SeqId. PreId= + currentEditSeqId + key= + key 3548+ ; edit= + val); It will never happen, right? (smile) I did not realize we were writing a file into region with the sequenceid. Better than into zk I'd say. Should the special file instead be named with a dot prefix like other special files (see head of HRegionFileSystem for list -- .regioninfo, .tmp, etc.) Maintain SeqId monotonically increasing --- Key: HBASE-12485 URL: https://issues.apache.org/jira/browse/HBASE-12485 Project: HBase Issue Type: Sub-task Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12485.patch We added FLUSH, REGION CLOSE events into WAL, for each those events the region SeqId is bumped. The issue comes from region close operation. Because when opening a region, we use flushed SeqId from store files while after store flush during region close we still write COMMIT_FLUSH, REGION_CLOSE events etc which respectively bump up SeqId. Therefore, the region opening SeqId is lower than it should be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234652#comment-14234652 ] stack commented on HBASE-12635: --- FYI, hadoopqa is down currently Delete acl notify znode of table after the table is deleted --- Key: HBASE-12635 URL: https://issues.apache.org/jira/browse/HBASE-12635 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12635-v1.diff In our multi-tenant hbase cluster, we found that there are over 1M znodes under the acl node. The reason is that users create and delete tables with different names frequently. The acl notify znode are left there after the tables are deleted. Simple solution is that deleting acl notify znode of table in AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12625) Deprecate certain methods in classes annotated with InterfaceAudience.Public in branch-1
[
https://issues.apache.org/jira/browse/HBASE-12625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234656#comment-14234656
]
Andrew Purtell commented on HBASE-12625:
bq. return type short is not compatible with int
Yep, we deprecated that in 0.98 because we knew it would be an issue. The
original sin was selecting a {{short}} as return type. Apologies again for that.
Deprecate certain methods in classes annotated with InterfaceAudience.Public
in branch-1
Key: HBASE-12625
URL: https://issues.apache.org/jira/browse/HBASE-12625
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 1.0.0
Attachments: 12625-v1.txt
Compiling Phoenix against branch-1, I got compilation error below:
{code}
[ERROR]
/Users/tyu/phoenix/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java:[345,49]
cannot find symbol
[ERROR] symbol: method estimatedSizeOf(org.apache.hadoop.hbase.Cell)
[ERROR] location: class org.apache.hadoop.hbase.CellUtil
{code}
CellUtil is annotated with InterfaceAudience.Public
estimatedSizeOf() should be deprecated in branch-1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234662#comment-14234662 ] Andrew Purtell commented on HBASE-12635: I'll be testing locally. Or would you prefer this wait until Jenkins is back up [~stack]? Let me know. Delete acl notify znode of table after the table is deleted --- Key: HBASE-12635 URL: https://issues.apache.org/jira/browse/HBASE-12635 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.9 Attachments: HBASE-12635-v1.diff In our multi-tenant hbase cluster, we found that there are over 1M znodes under the acl node. The reason is that users create and delete tables with different names frequently. The acl notify znode are left there after the tables are deleted. Simple solution is that deleting acl notify znode of table in AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234663#comment-14234663 ] stack commented on HBASE-12636: --- bq. Means in the event of failure we could double replicate some entries, right? Yes. Thats ok though, right? bq. What about recording this information into a system table instead? We can start the migration of replication state out of ZooKeeper. Yes. In a separate JIRA? Patch LGTM (The background code looks to be duplicated) Avoid too many write operations on zookeeper in replication --- Key: HBASE-12636 URL: https://issues.apache.org/jira/browse/HBASE-12636 Project: HBase Issue Type: Improvement Affects Versions: 0.94.11 Reporter: Liu Shaohui Assignee: Liu Shaohui Labels: replication Fix For: 1.0.0 Attachments: HBASE-12636-v1.diff In our production cluster, we found there are about over 1k write operations per second on zookeeper from hbase replication. The reason is that the replication source will write the log position to zookeeper for every edit shipping. If the current replicating WAL is just the WAL that regionserver is writing to, each skipping will be very small but the frequency is very high, which causes many write operations on zookeeper. A simple solution is that writing log position to zookeeper when position diff or skipped edit number is larger than a threshold, not every edit shipping. Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12273) Generate .tabledesc file during upgrading if missing
[ https://issues.apache.org/jira/browse/HBASE-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234672#comment-14234672 ] stack commented on HBASE-12273: --- [~daviddengcn] pardon me. You want this in apache hbase? Generate .tabledesc file during upgrading if missing Key: HBASE-12273 URL: https://issues.apache.org/jira/browse/HBASE-12273 Project: HBase Issue Type: Sub-task Components: Admin Affects Versions: 1.0.0, 0.98.7 Reporter: Yi Deng Assignee: Yi Deng Labels: upgrade Fix For: 1.0.0, 0.98.7 Attachments: 1.0-0001-INTERNAL-Add-a-tool-for-fixing-missing-TableDescript.patch Generate .tabledesc file during upgrading if missing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234677#comment-14234677 ] Andrew Purtell commented on HBASE-12636: bq. Yes. In a separate JIRA? I'd say not ideally. Nobody's going to work on it. We keep putting it off. Now we're putting band aids on the problem. Avoid too many write operations on zookeeper in replication --- Key: HBASE-12636 URL: https://issues.apache.org/jira/browse/HBASE-12636 Project: HBase Issue Type: Improvement Affects Versions: 0.94.11 Reporter: Liu Shaohui Assignee: Liu Shaohui Labels: replication Fix For: 1.0.0 Attachments: HBASE-12636-v1.diff In our production cluster, we found there are about over 1k write operations per second on zookeeper from hbase replication. The reason is that the replication source will write the log position to zookeeper for every edit shipping. If the current replicating WAL is just the WAL that regionserver is writing to, each skipping will be very small but the frequency is very high, which causes many write operations on zookeeper. A simple solution is that writing log position to zookeeper when position diff or skipped edit number is larger than a threshold, not every edit shipping. Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12625) Deprecate certain methods in classes annotated with InterfaceAudience.Public in branch-1
[
https://issues.apache.org/jira/browse/HBASE-12625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234680#comment-14234680
]
Ted Yu commented on HBASE-12625:
w.r.t. RpcSchedulerFactory, I tried to add back the old method which accepts
RegionServerServices parameter.
It wasn't straight forward - AnnotationReadingPriorityFunction expects
parameter of type RSRpcServices.
Deprecate certain methods in classes annotated with InterfaceAudience.Public
in branch-1
Key: HBASE-12625
URL: https://issues.apache.org/jira/browse/HBASE-12625
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 1.0.0
Attachments: 12625-v1.txt
Compiling Phoenix against branch-1, I got compilation error below:
{code}
[ERROR]
/Users/tyu/phoenix/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java:[345,49]
cannot find symbol
[ERROR] symbol: method estimatedSizeOf(org.apache.hadoop.hbase.Cell)
[ERROR] location: class org.apache.hadoop.hbase.CellUtil
{code}
CellUtil is annotated with InterfaceAudience.Public
estimatedSizeOf() should be deprecated in branch-1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12625) Deprecate certain methods in classes annotated with InterfaceAudience.Public in branch-1
[
https://issues.apache.org/jira/browse/HBASE-12625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234688#comment-14234688
]
Andrew Purtell commented on HBASE-12625:
{quote}
w.r.t. RpcSchedulerFactory, I tried to add back the old method which accepts
RegionServerServices parameter.
It wasn't straight forward - AnnotationReadingPriorityFunction expects
parameter of type RSRpcServices.
{quote}
Ok, but it's incumbent on us to fix a LimitedPrivate interface we broke without
deprecation or warning.
/cc [~mbertozzi]
Deprecate certain methods in classes annotated with InterfaceAudience.Public
in branch-1
Key: HBASE-12625
URL: https://issues.apache.org/jira/browse/HBASE-12625
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 1.0.0
Attachments: 12625-v1.txt
Compiling Phoenix against branch-1, I got compilation error below:
{code}
[ERROR]
/Users/tyu/phoenix/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java:[345,49]
cannot find symbol
[ERROR] symbol: method estimatedSizeOf(org.apache.hadoop.hbase.Cell)
[ERROR] location: class org.apache.hadoop.hbase.CellUtil
{code}
CellUtil is annotated with InterfaceAudience.Public
estimatedSizeOf() should be deprecated in branch-1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
