[jira] [Commented] (HBASE-12431) Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread Safe
[ https://issues.apache.org/jira/browse/HBASE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234004#comment-14234004 ] Hudson commented on HBASE-12431: SUCCESS: Integrated in HBase-TRUNK #5865 (See [https://builds.apache.org/job/HBase-TRUNK/5865/]) HBASE-12431 Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread Safe (Jingcheng Du) (stack: rev c45772e76a581838ea89ff303d0ff67ceaa82845) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java > Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread > Safe > - > > Key: HBASE-12431 > URL: https://issues.apache.org/jira/browse/HBASE-12431 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.1 >Reporter: Jonathan Jarvis >Assignee: Jingcheng Du > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12431-V2.diff, HBASE-12431-V3.diff, > HBASE-12431.diff > > > Result declares that it is NOT THREAD SAFE at the top of the source code, but > one would assume that refers to many different threads accessing the same > Result object. I've run into an issue when I have several different threads > accessing their own Result object that runs into an issue because of use of > common static member variable. > I noticed the problem when I switched from: > getColumnLatestCell(byte[], byte[]) to > getColumnLatestCell(byte[], int, int, byte[], int, int) > These methods call different binarySearch methods, the latter invoking: > protected int binarySearch(final Cell [] kvs, > 309 final byte [] family, final int foffset, final int flength, > 310 final byte [] qualifier, final int qoffset, final int qlength) { > This method utilizes a private static member variable called "buffer" > If more than one thread is utilizing "buffer" you'll see unpredictable > behavior unless you synchronize(Result.class) {}. > If buffer is to remain a static variable, I would recommend changing it to a > ThreadLocal instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12431) Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread Safe
[ https://issues.apache.org/jira/browse/HBASE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234043#comment-14234043 ] Hudson commented on HBASE-12431: FAILURE: Integrated in HBase-0.98 #715 (See [https://builds.apache.org/job/HBase-0.98/715/]) HBASE-12431 Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread Safe (Jingcheng Du) (stack: rev 769f93ff07940e92ff6b6653fb53357bc99c8966) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java > Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread > Safe > - > > Key: HBASE-12431 > URL: https://issues.apache.org/jira/browse/HBASE-12431 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.1 >Reporter: Jonathan Jarvis >Assignee: Jingcheng Du > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12431-V2.diff, HBASE-12431-V3.diff, > HBASE-12431.diff > > > Result declares that it is NOT THREAD SAFE at the top of the source code, but > one would assume that refers to many different threads accessing the same > Result object. I've run into an issue when I have several different threads > accessing their own Result object that runs into an issue because of use of > common static member variable. > I noticed the problem when I switched from: > getColumnLatestCell(byte[], byte[]) to > getColumnLatestCell(byte[], int, int, byte[], int, int) > These methods call different binarySearch methods, the latter invoking: > protected int binarySearch(final Cell [] kvs, > 309 final byte [] family, final int foffset, final int flength, > 310 final byte [] qualifier, final int qoffset, final int qlength) { > This method utilizes a private static member variable called "buffer" > If more than one thread is utilizing "buffer" you'll see unpredictable > behavior unless you synchronize(Result.class) {}. > If buffer is to remain a static variable, I would recommend changing it to a > ThreadLocal instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11861) Native MOB Compaction mechanisms.
[ https://issues.apache.org/jira/browse/HBASE-11861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11861: - Attachment: mob compaction.pdf Upload the high level design for the mob compaction. Hi all, please review and comment. Thanks a lot! > Native MOB Compaction mechanisms. > - > > Key: HBASE-11861 > URL: https://issues.apache.org/jira/browse/HBASE-11861 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh > Attachments: 141030-mob-compaction.pdf, mob compaction.pdf > > > Currently, the first cut of mob will have external processes to age off old > mob data (the ttl cleaner), and to compact away deleted or over written data > (the sweep tool). > From an operational point of view, having two external tools, especially one > that relies on MapReduce is undesirable. In this issue we'll tackle > integrating these into hbase without requiring external processes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12611) Create autoCommit() method and remove clearBufferOnFail
[ https://issues.apache.org/jira/browse/HBASE-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234055#comment-14234055 ] Nicolas Liochon commented on HBASE-12611: - bq. stack and Nick Dimiduk came to the conclusion that the flush method should be called autoCommit() similar to the SQL APIs. Sorry for beeing late in the game. The meaning in SQL is slightly different. In jdbc, whatever the value for autoCommit, the query will be sent to be server and executed. autocommit is set to false if the client application wants to send multiple queries within a single transaction (and then it will do a begin/commit explicitly). I haven't double checked if it's the standard or an implementation detail (the docs are not very clear), but it's unlikely to change anyway: there is another set of methods for batches in jdbc. Our old autoFlush is different as it impacts the client behavior. I think we're creating a confusion here. Moreover, if we add transactions between rows in the future, then may be we will want to use autoCommit for what it really is. As I'm very late here I leave the decision to you, but we should at least be clear in the javadoc imho. bq. Do we also want to change the default with this patch? I like the fact that HBase its secure by default and it would be very confusing for the users as well imho. {code} - public boolean isAutoFlush() { -return autoFlush; + public boolean getAutoCommit() { +return autoCommit; {code} If I'm not wrong we use 'is' for getters on boolean? > Create autoCommit() method and remove clearBufferOnFail > --- > > Key: HBASE-12611 > URL: https://issues.apache.org/jira/browse/HBASE-12611 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.99.2 >Reporter: Solomon Duskis >Assignee: Solomon Duskis > Fix For: 1.0.0 > > Attachments: HBASE-12611.patch > > > There was quite a bit of good discussion on HBASE-12490 about this topic. > [~stack] and [~ndimiduk] came to the conclusion that the flush method should > be called autoCommit() similar to the SQL APIs. [~ndimiduk] also suggested > that clearBufferOnFail should be removed from HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12431) Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread Safe
[ https://issues.apache.org/jira/browse/HBASE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234060#comment-14234060 ] Hudson commented on HBASE-12431: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #682 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/682/]) HBASE-12431 Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread Safe (Jingcheng Du) (stack: rev 769f93ff07940e92ff6b6653fb53357bc99c8966) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java > Use of getColumnLatestCell(byte[], int, int, byte[], int, int) is Not Thread > Safe > - > > Key: HBASE-12431 > URL: https://issues.apache.org/jira/browse/HBASE-12431 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.1 >Reporter: Jonathan Jarvis >Assignee: Jingcheng Du > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12431-V2.diff, HBASE-12431-V3.diff, > HBASE-12431.diff > > > Result declares that it is NOT THREAD SAFE at the top of the source code, but > one would assume that refers to many different threads accessing the same > Result object. I've run into an issue when I have several different threads > accessing their own Result object that runs into an issue because of use of > common static member variable. > I noticed the problem when I switched from: > getColumnLatestCell(byte[], byte[]) to > getColumnLatestCell(byte[], int, int, byte[], int, int) > These methods call different binarySearch methods, the latter invoking: > protected int binarySearch(final Cell [] kvs, > 309 final byte [] family, final int foffset, final int flength, > 310 final byte [] qualifier, final int qoffset, final int qlength) { > This method utilizes a private static member variable called "buffer" > If more than one thread is utilizing "buffer" you'll see unpredictable > behavior unless you synchronize(Result.class) {}. > If buffer is to remain a static variable, I would recommend changing it to a > ThreadLocal instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12597) Add RpcClient interface and enable changing of RpcClient implementation
[ https://issues.apache.org/jira/browse/HBASE-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12597: -- Attachment: HBASE-12597-v4.patch New patch - Moved the static constants to RpcClient interface. They are indeed pretty generic and I use them too in my implementation. - RpcClient extends Closeable now and stop method is now called close() - Moved out Call class from RpcClientImpl. I didn't use it in my async implementation, but maybe others can use it in another sync implementation. bq. Ditto Connection. If so, np. I suppose an RpcClient has a Connection? A single Connection? Or this is implementation detail? Or I supopse it is not... protocol is all around Connections. If so, how you doing this over in your client? Connection is not a static class and is tightly linked into RpcClientImpl. It uses the socket factory and creates sockets which are pretty much tied into RpcClientImpl its implementation. In many ways it is the implementation of RpcClient. I had to create something new in my implementation based on Netty Channels. Maybe something reusable can be abstracted out of it to also simplify how I implemented it but I think that falls outside the scope of this issue. CallFuture is pretty small and a private static class. It is used in the internal queue. I couldn't use it for my implementation. BlockingRpcChannelImplementation links to the call method in RpcClientImpl which is currently not in the RpcClient interface. And I would not like it to be in there since PayloadCarryingRpcController is currently not reusable in async context. Could be with some changes but I think they need some discussion, but somehow I think it is better to leave RpcClient as clean as possible so RpcClients can try out new things like doing without an PayloadCarryingRpcController. bq. I beileve Anoop Sam John is refering to the 'release note' attribute of this here JIRA. Ah I see. > Add RpcClient interface and enable changing of RpcClient implementation > --- > > Key: HBASE-12597 > URL: https://issues.apache.org/jira/browse/HBASE-12597 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Jurriaan Mous > Attachments: HBASE-12597-v1.patch, HBASE-12597-v2.patch, > HBASE-12597-v3.patch, HBASE-12597-v4.patch, HBASE-12597.patch > > > Currently HConnectionImplementation works with the included RpcClient which > is a direct implementation and not defined by an interface. > It would be great to be able to swap out the default RpcClient with another > implementation which can also be controlled by the default > HConnectionImplementation. > Suggested changes: > - Create a RpcClient interface which defines all the ways > HConnectionImplementation interacts with an RPC client. Like getting a > blocking protobuf service interface or closing the client. > - Define which RpcClient implementation to construct by setting a > configuration variable which defaults to the current RpcClient. > - Possibly create an abstract RpcClient class to only load all the basic Rpc > layer configurations to be used in an implementation. > Why? It enables experimentation with RpcClients which could enable new > features or could be more performant than the included client. > I created a new RpcClient implementation based on Netty which can also be > called asynchronously. It would be great to also be able to use this > RpcClient in all the default ways and tests to see if there are any issues > with it. > https://github.com/jurmous/async-hbase-client/ > https://github.com/jurmous/async-hbase-client/blob/master/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-12634: -- Attachment: HBASE-12634-v3.patch Changed the authorization api in preCreateNamespace api in v3 patch. As authorizing a user with admin rights for a non existing namespace was not correct. > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234105#comment-14234105 ] Ashish Singhi commented on HBASE-12634: --- Please review the v3 patch and share your comments. > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234111#comment-14234111 ] Matteo Bertozzi commented on HBASE-12634: - +1 v3 looks good to me > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangduo updated HBASE-10201: - Attachment: HBASE-10201_13.patch Use a FlushPolicy instead of hbase.hregion.memstore.percolumnfamilyflush.enabled config. I'm not good at naming things, and may break some rules when add the policy. So just point out if you have a better name of the policy and anything that you think is wrong in this patch. I will fix it as soon as possible. Thanks. > Port 'Make flush decisions per column family' to trunk > -- > > Key: HBASE-10201 > URL: https://issues.apache.org/jira/browse/HBASE-10201 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Ted Yu >Assignee: zhangduo >Priority: Critical > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, > HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, > HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, > HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, > HBASE-10201_2.patch, HBASE-10201_3.patch, HBASE-10201_4.patch, > HBASE-10201_5.patch, HBASE-10201_6.patch, HBASE-10201_7.patch, > HBASE-10201_8.patch, HBASE-10201_9.patch > > > Currently the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12635) Delete acl notify znode of table after the table is deleted
Liu Shaohui created HBASE-12635: --- Summary: Delete acl notify znode of table after the table is deleted Key: HBASE-12635 URL: https://issues.apache.org/jira/browse/HBASE-12635 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor In our multi-tenant hbase cluster, we found that there are over 1M znodes under the acl node. The reason is that users create and delete tables with different names frequently. The acl notify znode are left there after the tables are deleted. Simple solution is that deleting acl notify znode of table in AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-12635: Attachment: HBASE-12635-v1.diff Patch for trunk. > Delete acl notify znode of table after the table is deleted > --- > > Key: HBASE-12635 > URL: https://issues.apache.org/jira/browse/HBASE-12635 > Project: HBase > Issue Type: Bug >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Attachments: HBASE-12635-v1.diff > > > In our multi-tenant hbase cluster, we found that there are over 1M znodes > under the acl node. The reason is that users create and delete tables with > different names frequently. The acl notify znode are left there after the > tables are deleted. > Simple solution is that deleting acl notify znode of table in > AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-12635: Fix Version/s: 2.0.0 Affects Version/s: 0.94.11 0.98.6 Status: Patch Available (was: Open) > Delete acl notify znode of table after the table is deleted > --- > > Key: HBASE-12635 > URL: https://issues.apache.org/jira/browse/HBASE-12635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.6, 0.94.11 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-12635-v1.diff > > > In our multi-tenant hbase cluster, we found that there are over 1M znodes > under the acl node. The reason is that users create and delete tables with > different names frequently. The acl notify znode are left there after the > tables are deleted. > Simple solution is that deleting acl notify znode of table in > AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234166#comment-14234166 ] Anoop Sam John commented on HBASE-12634: +1 for V3 > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12622) user_permission should require global admin to display global and ns permissions
[ https://issues.apache.org/jira/browse/HBASE-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234167#comment-14234167 ] Anoop Sam John commented on HBASE-12622: +1 for this patch after HBASE-12634 goes in. > user_permission should require global admin to display global and ns > permissions > > > Key: HBASE-12622 > URL: https://issues.apache.org/jira/browse/HBASE-12622 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 0.98.8, 0.99.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12622-v0.patch > > > user_permission check the user permission only on the table level (requiring > at least a table-level admin) > global and namespace permission listing is done without checking anything. > but only a global admins should be able to perform this operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12622) user_permission should require global admin to display global and ns permissions
[ https://issues.apache.org/jira/browse/HBASE-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234170#comment-14234170 ] Ashish Singhi commented on HBASE-12622: --- [~mbertozzi] can we move the test to TestNamespaceCommands, as all the namespace related test are available there and also we can validate for the user with admin rights on namespace is allowed to execute user_permission.? > user_permission should require global admin to display global and ns > permissions > > > Key: HBASE-12622 > URL: https://issues.apache.org/jira/browse/HBASE-12622 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 0.98.8, 0.99.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12622-v0.patch > > > user_permission check the user permission only on the table level (requiring > at least a table-level admin) > global and namespace permission listing is done without checking anything. > but only a global admins should be able to perform this operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12636) Avoid too many write operations on zookeeper in replication
Liu Shaohui created HBASE-12636: --- Summary: Avoid too many write operations on zookeeper in replication Key: HBASE-12636 URL: https://issues.apache.org/jira/browse/HBASE-12636 Project: HBase Issue Type: Improvement Affects Versions: 0.94.11 Reporter: Liu Shaohui Assignee: Liu Shaohui Fix For: 1.0.0 In our production cluster, we found there are about over 1k write operations per second on zookeeper from hbase replication. The reason is that the replication source will write the log position to zookeeper for every edit shipping. If the current replicating WAL is just the WAL that regionserver is writing to, each skipping will be very small but the frequency is very high, which causes many write operations on zookeeper. A simple solution is that writing log position to zookeeper when position diff or skipped edit number is larger than a threshold, not every edit shipping. Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12564) consolidate the getTableDescriptors() semantic
[ https://issues.apache.org/jira/browse/HBASE-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12564: Attachment: HBASE-12564-v2.patch > consolidate the getTableDescriptors() semantic > -- > > Key: HBASE-12564 > URL: https://issues.apache.org/jira/browse/HBASE-12564 > Project: HBase > Issue Type: Bug > Components: Client, master >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-12564-v0.patch, HBASE-12564-v1.patch, > HBASE-12564-v2.patch > > > Master getTableDescriptors() which is called by Admin.listTables() has a > couple of different behaviors depending on how it is called. > after HBASE-12073 with the AccessController enabled, we now get a "global > admin" required if listTables() is called without a regex otherwise we return > only the table that the user can see (we show only the tables that the user > have access to, which means or the user is a global admin or it has a > table-level create/admin). We probably should have the second behavior even > without regex, since I should able to see "my own tables". > getTableDescriptors() is returning only non system tables. Tools like > user_permission that are doing "for each listTable(): userPerm(table)" are > losing the system tables, so stuff like user_permission 'hbase:acls' will not > result any result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234181#comment-14234181 ] ramkrishna.s.vasudevan commented on HBASE-12635: +1 > Delete acl notify znode of table after the table is deleted > --- > > Key: HBASE-12635 > URL: https://issues.apache.org/jira/browse/HBASE-12635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.11, 0.98.6 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-12635-v1.diff > > > In our multi-tenant hbase cluster, we found that there are over 1M znodes > under the acl node. The reason is that users create and delete tables with > different names frequently. The acl notify znode are left there after the > tables are deleted. > Simple solution is that deleting acl notify znode of table in > AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-12636: Attachment: HBASE-12636-v1.diff Patch for trunk > Avoid too many write operations on zookeeper in replication > --- > > Key: HBASE-12636 > URL: https://issues.apache.org/jira/browse/HBASE-12636 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.11 >Reporter: Liu Shaohui >Assignee: Liu Shaohui > Labels: replication > Fix For: 1.0.0 > > Attachments: HBASE-12636-v1.diff > > > In our production cluster, we found there are about over 1k write operations > per second on zookeeper from hbase replication. The reason is that the > replication source will write the log position to zookeeper for every edit > shipping. If the current replicating WAL is just the WAL that regionserver is > writing to, each skipping will be very small but the frequency is very high, > which causes many write operations on zookeeper. > A simple solution is that writing log position to zookeeper when position > diff or skipped edit number is larger than a threshold, not every edit > shipping. > Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234270#comment-14234270 ] Ted Yu commented on HBASE-12204: Since hbase 1.0 RC is coming out soon, I think we should adopt the approach which is used in 0.98 That way, we can close the feature gap between 0.98 and 1.0 > Backport HBASE-12016 'Reduce number of versions in Meta table. Make it > configurable' to branch-1 > > > Key: HBASE-12204 > URL: https://issues.apache.org/jira/browse/HBASE-12204 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Andrey Stepachev >Priority: Minor > > The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that > HBASE-7767 was applied to master only. > This JIRA is to backport HBASE-12016 to branch-1. > This can be done when HBASE-12035 is resolved and branch-1 is brought to > parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234302#comment-14234302 ] Hadoop QA commented on HBASE-12634: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685094/HBASE-12634-v3.patch against master branch at commit c45772e76a581838ea89ff303d0ff67ceaa82845. ATTACHMENT ID: 12685094 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11933//console This message is automatically generated. > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12634: Resolution: Fixed Status: Resolved (was: Patch Available) > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12622) user_permission should require global admin to display global and ns permissions
[ https://issues.apache.org/jira/browse/HBASE-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12622: Attachment: HBASE-12622-v1.patch > user_permission should require global admin to display global and ns > permissions > > > Key: HBASE-12622 > URL: https://issues.apache.org/jira/browse/HBASE-12622 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 0.98.8, 0.99.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12622-v0.patch, HBASE-12622-v1.patch > > > user_permission check the user permission only on the table level (requiring > at least a table-level admin) > global and namespace permission listing is done without checking anything. > but only a global admins should be able to perform this operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234306#comment-14234306 ] Ashish Singhi commented on HBASE-12634: --- Thanks guys for the review and Matteo for the commit. > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12594) Remove the usage of deprecated HbaseAdmin#getTableNames from shell list command
[ https://issues.apache.org/jira/browse/HBASE-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-12594: -- Fix Version/s: 0.98.9 2.0.0 1.0.0 > Remove the usage of deprecated HbaseAdmin#getTableNames from shell list > command > --- > > Key: HBASE-12594 > URL: https://issues.apache.org/jira/browse/HBASE-12594 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12594.patch > > > With this fix one behavior change a non-super user will notice is that he/she > will be able to see only tables to which they have create/admin permissions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12553) request value is not consistent for restoreSnapshot in audit logs
[ https://issues.apache.org/jira/browse/HBASE-12553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-12553: -- Fix Version/s: 2.0.0 > request value is not consistent for restoreSnapshot in audit logs > - > > Key: HBASE-12553 > URL: https://issues.apache.org/jira/browse/HBASE-12553 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-12553.patch > > > {code} > requirePermission("restoreSnapshot", hTableDescriptor.getTableName(), > null, null, > Permission.Action.ADMIN); > } else { > requirePermission("restore", Action.ADMIN); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12338) Client side scanning prefetching.
[ https://issues.apache.org/jira/browse/HBASE-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Deng updated HBASE-12338: Status: Patch Available (was: Open) > Client side scanning prefetching. > - > > Key: HBASE-12338 > URL: https://issues.apache.org/jira/browse/HBASE-12338 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 0.98.6.1, 1.0.0, 2.0.0 >Reporter: Yi Deng >Assignee: Yi Deng > Labels: prefetch, results, scanner > Attachments: > 0001-Add-ScanPrefetcher-for-client-side-scanning-prefetch.patch, > 0001-ScanPrefetcher.patch, > 2.0-0001-Add-ScanPrefetcher-for-client-side-scanning-prefetch.patch, > 2.0-0002-Add-ScanPrefetcher-for-client-side-scanning-prefetch.patch, > 2.0-0003-Add-ScanPrefetcher-for-client-side-scanning-prefetch.patch > > > Since server side prefetching was not proved to be a good way to prefetch, we > need to do it on client side. > This is a wrapper class that takes any instance of `ResultScanner` as the > underneath scanning component. The class will schedule the scanning in a > background thread. There is a buffering queue storing prefetched results, > whose's length is configurable. The prefetcher will release the thread if the > queue is full and wait for results to be consumed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12632) ThrottledInputStream/ExportSnapshot does not throttle
[ https://issues.apache.org/jira/browse/HBASE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234344#comment-14234344 ] Hadoop QA commented on HBASE-12632: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685038/12632-v1.txt against master branch at commit c45772e76a581838ea89ff303d0ff67ceaa82845. ATTACHMENT ID: 12685038 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11934//console This message is automatically generated. > ThrottledInputStream/ExportSnapshot does not throttle > - > > Key: HBASE-12632 > URL: https://issues.apache.org/jira/browse/HBASE-12632 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.99.2 >Reporter: Tobi Vollebregt > Attachments: 12632-v1.txt > > > I just transferred a ton of data using ExportSnapshot with bandwidth > throttling from one Hadoop cluster to another Hadoop cluster, and discovered > that ThrottledInputStream does not limit bandwidth. > The problem is that ThrottledInputStream sleeps once, for a fixed time (50 > ms), at the start of each read call, disregarding the actual amount of data > read. > ExportSnapshot defaults to a buffer size as big as the block size of the > outputFs: > {code:java} > // Use the default block size of the outputFs if bigger > int defaultBlockSize = Math.max((int) outputFs.getDefaultBlockSize(), > BUFFER_SIZE); > bufferSize = conf.getInt(CONF_BUFFER_SIZE, defaultBlockSize); > LOG.info("Using bufferSize=" + > StringUtils.humanReadableInt(bufferSize)); > {code} > In my case, this was 256MB
[jira] [Commented] (HBASE-12273) Generate .tabledesc file during upgrading if missing
[ https://issues.apache.org/jira/browse/HBASE-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234382#comment-14234382 ] Yi Deng commented on HBASE-12273: - Thanks [~stack]. But this diff is not supposed to work on master, it only works for branch-1 and before. Some legacy code is necessary for generating HRegionInfo from old format while that code fragment seems been removed on master. > Generate .tabledesc file during upgrading if missing > > > Key: HBASE-12273 > URL: https://issues.apache.org/jira/browse/HBASE-12273 > Project: HBase > Issue Type: Sub-task > Components: Admin >Affects Versions: 1.0.0, 0.98.7 >Reporter: Yi Deng >Assignee: Yi Deng > Labels: upgrade > Fix For: 1.0.0, 0.98.7 > > Attachments: > 1.0-0001-INTERNAL-Add-a-tool-for-fixing-missing-TableDescript.patch > > > Generate .tabledesc file during upgrading if missing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12204: --- Attachment: 12204-1.0.txt All tests touched by the patch passes locally. Running full test suite on Linux now. > Backport HBASE-12016 'Reduce number of versions in Meta table. Make it > configurable' to branch-1 > > > Key: HBASE-12204 > URL: https://issues.apache.org/jira/browse/HBASE-12204 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Andrey Stepachev >Priority: Minor > Attachments: 12204-1.0.txt > > > The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that > HBASE-7767 was applied to master only. > This JIRA is to backport HBASE-12016 to branch-1. > This can be done when HBASE-12035 is resolved and branch-1 is brought to > parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12545) Fix backwards compatibility issue introduced with HBASE-12363
[ https://issues.apache.org/jira/browse/HBASE-12545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234400#comment-14234400 ] Lars Hofhansl commented on HBASE-12545: --- Yeah, let's close this one. > Fix backwards compatibility issue introduced with HBASE-12363 > - > > Key: HBASE-12545 > URL: https://issues.apache.org/jira/browse/HBASE-12545 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 1.0.0, 0.98.9 > > Attachments: 12545.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234396#comment-14234396 ] Ted Yu edited comment on HBASE-12204 at 12/4/14 5:50 PM: - All tests touched by the patch pass locally. Running full test suite on Linux now. was (Author: yuzhih...@gmail.com): All tests touched by the patch passes locally. Running full test suite on Linux now. > Backport HBASE-12016 'Reduce number of versions in Meta table. Make it > configurable' to branch-1 > > > Key: HBASE-12204 > URL: https://issues.apache.org/jira/browse/HBASE-12204 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Andrey Stepachev >Priority: Minor > Attachments: 12204-1.0.txt > > > The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that > HBASE-7767 was applied to master only. > This JIRA is to backport HBASE-12016 to branch-1. > This can be done when HBASE-12035 is resolved and branch-1 is brought to > parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12173) Backport: [PE] Allow random value size
[ https://issues.apache.org/jira/browse/HBASE-12173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234402#comment-14234402 ] Lars Hofhansl commented on HBASE-12173: --- I should. :) Lemme finish this today. > Backport: [PE] Allow random value size > -- > > Key: HBASE-12173 > URL: https://issues.apache.org/jira/browse/HBASE-12173 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Lars Hofhansl > Fix For: 0.94.26, 0.98.9 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234411#comment-14234411 ] Hudson commented on HBASE-12634: SUCCESS: Integrated in HBase-1.0 #539 (See [https://builds.apache.org/job/HBase-1.0/539/]) HBASE-12634 Fix the AccessController#requireGlobalPermission(ns) with NS (matteo.bertozzi: rev ffbfe01bbeb00d91f8df1ce6ddc041af9887dca7) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234418#comment-14234418 ] Lars Hofhansl commented on HBASE-12636: --- Means in the event of failure we could double replicate some entries, right? > Avoid too many write operations on zookeeper in replication > --- > > Key: HBASE-12636 > URL: https://issues.apache.org/jira/browse/HBASE-12636 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.11 >Reporter: Liu Shaohui >Assignee: Liu Shaohui > Labels: replication > Fix For: 1.0.0 > > Attachments: HBASE-12636-v1.diff > > > In our production cluster, we found there are about over 1k write operations > per second on zookeeper from hbase replication. The reason is that the > replication source will write the log position to zookeeper for every edit > shipping. If the current replicating WAL is just the WAL that regionserver is > writing to, each skipping will be very small but the frequency is very high, > which causes many write operations on zookeeper. > A simple solution is that writing log position to zookeeper when position > diff or skipped edit number is larger than a threshold, not every edit > shipping. > Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-12204: -- Assignee: Ted Yu (was: Andrey Stepachev) > Backport HBASE-12016 'Reduce number of versions in Meta table. Make it > configurable' to branch-1 > > > Key: HBASE-12204 > URL: https://issues.apache.org/jira/browse/HBASE-12204 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 12204-1.0.txt > > > The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that > HBASE-7767 was applied to master only. > This JIRA is to backport HBASE-12016 to branch-1. > This can be done when HBASE-12035 is resolved and branch-1 is brought to > parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12204: --- Status: Patch Available (was: Open) > Backport HBASE-12016 'Reduce number of versions in Meta table. Make it > configurable' to branch-1 > > > Key: HBASE-12204 > URL: https://issues.apache.org/jira/browse/HBASE-12204 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 12204-1.0.txt > > > The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that > HBASE-7767 was applied to master only. > This JIRA is to backport HBASE-12016 to branch-1. > This can be done when HBASE-12035 is resolved and branch-1 is brought to > parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12204: --- Fix Version/s: 1.0.0 > Backport HBASE-12016 'Reduce number of versions in Meta table. Make it > configurable' to branch-1 > > > Key: HBASE-12204 > URL: https://issues.apache.org/jira/browse/HBASE-12204 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 1.0.0 > > Attachments: 12204-1.0.txt > > > The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that > HBASE-7767 was applied to master only. > This JIRA is to backport HBASE-12016 to branch-1. > This can be done when HBASE-12035 is resolved and branch-1 is brought to > parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12597) Add RpcClient interface and enable changing of RpcClient implementation
[ https://issues.apache.org/jira/browse/HBASE-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12597: -- Fix Version/s: 2.0.0 1.0.0 > Add RpcClient interface and enable changing of RpcClient implementation > --- > > Key: HBASE-12597 > URL: https://issues.apache.org/jira/browse/HBASE-12597 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Jurriaan Mous > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-12597-v1.patch, HBASE-12597-v2.patch, > HBASE-12597-v3.patch, HBASE-12597-v4.patch, HBASE-12597.patch > > > Currently HConnectionImplementation works with the included RpcClient which > is a direct implementation and not defined by an interface. > It would be great to be able to swap out the default RpcClient with another > implementation which can also be controlled by the default > HConnectionImplementation. > Suggested changes: > - Create a RpcClient interface which defines all the ways > HConnectionImplementation interacts with an RPC client. Like getting a > blocking protobuf service interface or closing the client. > - Define which RpcClient implementation to construct by setting a > configuration variable which defaults to the current RpcClient. > - Possibly create an abstract RpcClient class to only load all the basic Rpc > layer configurations to be used in an implementation. > Why? It enables experimentation with RpcClients which could enable new > features or could be more performant than the included client. > I created a new RpcClient implementation based on Netty which can also be > called asynchronously. It would be great to also be able to use this > RpcClient in all the default ways and tests to see if there are any issues > with it. > https://github.com/jurmous/async-hbase-client/ > https://github.com/jurmous/async-hbase-client/blob/master/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12545) Fix backwards compatibility issue introduced with HBASE-12363
[ https://issues.apache.org/jira/browse/HBASE-12545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-12545. Resolution: Not a Problem Fix Version/s: (was: 0.98.9) (was: 1.0.0) Assignee: (was: Lars Hofhansl) Resolving as Not A Problem > Fix backwards compatibility issue introduced with HBASE-12363 > - > > Key: HBASE-12545 > URL: https://issues.apache.org/jira/browse/HBASE-12545 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Lars Hofhansl > Attachments: 12545.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12597) Add RpcClient interface and enable changing of RpcClient implementation
[ https://issues.apache.org/jira/browse/HBASE-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234426#comment-14234426 ] stack commented on HBASE-12597: --- Skimmed. Will come back with a better review (should the defines in the Interface be public so folks don't have to be in same package to make an implementation?) Marked this as for 1.0. Would be good to get it in before 1.0 goes out. Lets see what hadoopqa thinks of this patch. > Add RpcClient interface and enable changing of RpcClient implementation > --- > > Key: HBASE-12597 > URL: https://issues.apache.org/jira/browse/HBASE-12597 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Jurriaan Mous > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-12597-v1.patch, HBASE-12597-v2.patch, > HBASE-12597-v3.patch, HBASE-12597-v4.patch, HBASE-12597.patch > > > Currently HConnectionImplementation works with the included RpcClient which > is a direct implementation and not defined by an interface. > It would be great to be able to swap out the default RpcClient with another > implementation which can also be controlled by the default > HConnectionImplementation. > Suggested changes: > - Create a RpcClient interface which defines all the ways > HConnectionImplementation interacts with an RPC client. Like getting a > blocking protobuf service interface or closing the client. > - Define which RpcClient implementation to construct by setting a > configuration variable which defaults to the current RpcClient. > - Possibly create an abstract RpcClient class to only load all the basic Rpc > layer configurations to be used in an implementation. > Why? It enables experimentation with RpcClients which could enable new > features or could be more performant than the included client. > I created a new RpcClient implementation based on Netty which can also be > called asynchronously. It would be great to also be able to use this > RpcClient in all the default ways and tests to see if there are any issues > with it. > https://github.com/jurmous/async-hbase-client/ > https://github.com/jurmous/async-hbase-client/blob/master/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12597) Add RpcClient interface and enable changing of RpcClient implementation
[ https://issues.apache.org/jira/browse/HBASE-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12597: -- Status: Patch Available (was: Open) > Add RpcClient interface and enable changing of RpcClient implementation > --- > > Key: HBASE-12597 > URL: https://issues.apache.org/jira/browse/HBASE-12597 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Jurriaan Mous > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-12597-v1.patch, HBASE-12597-v2.patch, > HBASE-12597-v3.patch, HBASE-12597-v4.patch, HBASE-12597.patch > > > Currently HConnectionImplementation works with the included RpcClient which > is a direct implementation and not defined by an interface. > It would be great to be able to swap out the default RpcClient with another > implementation which can also be controlled by the default > HConnectionImplementation. > Suggested changes: > - Create a RpcClient interface which defines all the ways > HConnectionImplementation interacts with an RPC client. Like getting a > blocking protobuf service interface or closing the client. > - Define which RpcClient implementation to construct by setting a > configuration variable which defaults to the current RpcClient. > - Possibly create an abstract RpcClient class to only load all the basic Rpc > layer configurations to be used in an implementation. > Why? It enables experimentation with RpcClients which could enable new > features or could be more performant than the included client. > I created a new RpcClient implementation based on Netty which can also be > called asynchronously. It would be great to also be able to use this > RpcClient in all the default ways and tests to see if there are any issues > with it. > https://github.com/jurmous/async-hbase-client/ > https://github.com/jurmous/async-hbase-client/blob/master/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234432#comment-14234432 ] Hudson commented on HBASE-12634: SUCCESS: Integrated in HBase-0.98 #716 (See [https://builds.apache.org/job/HBase-0.98/716/]) HBASE-12634 Fix the AccessController#requireGlobalPermission(ns) with NS (matteo.bertozzi: rev 6c927313c5ac7275bce3f9cdc4d215db28264601) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234431#comment-14234431 ] Hudson commented on HBASE-12634: SUCCESS: Integrated in HBase-TRUNK #5866 (See [https://builds.apache.org/job/HBase-TRUNK/5866/]) HBASE-12634 Fix the AccessController#requireGlobalPermission(ns) with NS (matteo.bertozzi: rev 0299ab8bf69618d7e07a6ec7071ce9234d9d) * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234440#comment-14234440 ] Andrew Purtell commented on HBASE-12636: What about recording this information into a system table instead? We can start the migration of replication state out of ZooKeeper. > Avoid too many write operations on zookeeper in replication > --- > > Key: HBASE-12636 > URL: https://issues.apache.org/jira/browse/HBASE-12636 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.11 >Reporter: Liu Shaohui >Assignee: Liu Shaohui > Labels: replication > Fix For: 1.0.0 > > Attachments: HBASE-12636-v1.diff > > > In our production cluster, we found there are about over 1k write operations > per second on zookeeper from hbase replication. The reason is that the > replication source will write the log position to zookeeper for every edit > shipping. If the current replicating WAL is just the WAL that regionserver is > writing to, each skipping will be very small but the frequency is very high, > which causes many write operations on zookeeper. > A simple solution is that writing log position to zookeeper when position > diff or skipped edit number is larger than a threshold, not every edit > shipping. > Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12635: --- Fix Version/s: 0.98.9 1.0.0 > Delete acl notify znode of table after the table is deleted > --- > > Key: HBASE-12635 > URL: https://issues.apache.org/jira/browse/HBASE-12635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.11, 0.98.6 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12635-v1.diff > > > In our multi-tenant hbase cluster, we found that there are over 1M znodes > under the acl node. The reason is that users create and delete tables with > different names frequently. The acl notify znode are left there after the > tables are deleted. > Simple solution is that deleting acl notify znode of table in > AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1423#comment-1423 ] Andrew Purtell commented on HBASE-12635: +1, I'll backport and commit to all branches shortly unless objection. Sorry [~liushaohui], we should have been doing this already and I'm surprised we are not. > Delete acl notify znode of table after the table is deleted > --- > > Key: HBASE-12635 > URL: https://issues.apache.org/jira/browse/HBASE-12635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.11, 0.98.6 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12635-v1.diff > > > In our multi-tenant hbase cluster, we found that there are over 1M znodes > under the acl node. The reason is that users create and delete tables with > different names frequently. The acl notify znode are left there after the > tables are deleted. > Simple solution is that deleting acl notify znode of table in > AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12632) ThrottledInputStream/ExportSnapshot does not throttle
[ https://issues.apache.org/jira/browse/HBASE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234449#comment-14234449 ] Tobi Vollebregt commented on HBASE-12632: - Thanks Ted. Patch LGTM. I haven't tested the version with {{while}} yet though, I will try to get around to that next week. > ThrottledInputStream/ExportSnapshot does not throttle > - > > Key: HBASE-12632 > URL: https://issues.apache.org/jira/browse/HBASE-12632 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.99.2 >Reporter: Tobi Vollebregt > Attachments: 12632-v1.txt > > > I just transferred a ton of data using ExportSnapshot with bandwidth > throttling from one Hadoop cluster to another Hadoop cluster, and discovered > that ThrottledInputStream does not limit bandwidth. > The problem is that ThrottledInputStream sleeps once, for a fixed time (50 > ms), at the start of each read call, disregarding the actual amount of data > read. > ExportSnapshot defaults to a buffer size as big as the block size of the > outputFs: > {code:java} > // Use the default block size of the outputFs if bigger > int defaultBlockSize = Math.max((int) outputFs.getDefaultBlockSize(), > BUFFER_SIZE); > bufferSize = conf.getInt(CONF_BUFFER_SIZE, defaultBlockSize); > LOG.info("Using bufferSize=" + > StringUtils.humanReadableInt(bufferSize)); > {code} > In my case, this was 256MB. > Hence, the ExportSnapshot mapper will attempt to read up to 256 MB at a time, > each time sleeping only 50ms. Thus, in the worst case where each call to read > fills the 256 MB buffer in negligible time, the ThrottledInputStream cannot > reduce the bandwidth to under (256 MB) / (5 ms) = 5 GB/s. > Even in a more realistic case where read returns about 1 MB per call, it > still cannot throttle the bandwidth to under 20 MB/s. > The issue is exacerbated by the fact that you need to set a low limit because > the total bandwidth per host depends on the number of mapper slots as well. > A simple solution would change the if in throttle to a while, so that it > keeps sleeping for 50 ms until the rate is finally low enough: > {code:java} > private void throttle() throws IOException { > while (getBytesPerSec() > maxBytesPerSec) { > try { > Thread.sleep(SLEEP_DURATION_MS); > totalSleepTime += SLEEP_DURATION_MS; > } catch (InterruptedException e) { > throw new IOException("Thread aborted", e); > } > } > } > {code} > This issue affects the ThrottledInputStream in hadoop as well. > Another way to see this is that for big enough buffer sizes, > ThrottledInputStream will be throttling only the number of read calls to 20 > per second, disregarding the number of bytes read. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12358) Create ByteBuffer backed Cell
[ https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234450#comment-14234450 ] ramkrishna.s.vasudevan commented on HBASE-12358: Before we decide on BB or BR, we would like to highlight some points here so that we could decide on the behaviour of the new Cell APIs. Assume we will be working with BBs so we would introduce getXXXBuffer() APIs and also hasArray() in Cell itself directly. If we try to extend the cell or create a new Cell then everywhere we need to do instanceOf check or do type conversion and that is why adding new APIS to Cell interface itself makes sense. Plan is to use this getXXXBuffer API through out the read path. Now there are two ways to use it 1) Use getXXXBuffer along with getXXXOffset, getXXXLength like how we use now for getXXXArray APIs with the offset and length. Doing so would ensure that every where in the filters and CP one has to just replace the getXXXArray with getXXXBuffer and continue to use getXXXOffset and getXXXLength. We would do some wrapping of the byte[] with a BB incase of KeyValue type of cells so that getXXXBuffer along with offset and length holds true everywhere. Note that here if hasARray is false (for KV case) then getXXXArray would also work. 2)The other way of using this is that use only getXXXBuffer() API and ensure that the BB is always duplicated/sliced and only the portion of the total BB is returned which represents the individual component of the Cell. In this case there is no use of getXXXOffset (as it is going to be 0) and getXXXLength() is any way going to be the sliced BB's limit. But in the 2nd approach we may end up in creating lot of small objects even while doing comparison. Now the next problem that comes is what to do with the getXXXArray() APIs. If one sees hasArray() as false (a DBB backed Cell) and uses the getXXXArray() API along with offset and length - what should we do. Should we create a byte[] from the DBB and return it? Then in that case what would should the getXXXOffset() return for a getXXXBuffer or getXXXArray()? If we go with the 2nd approach then getXXXBuffer() should be clearly documented saying that it has to be used without Offset and length and use getXXXOFfset and getXXXLength only with getXXXArray(). Now if a Cell is backed by on heap BB then we could definitely return getXXXArray() also - but what to return in the getXXXOffset would be determined by what appraoch to use for getXXXBuffer. (based on (1) and (2)). We wanted to open up this topic now so that to get some feedback on what could be an option here. Since it is an user facing Interface we need to be careful with this. I would suggest that whenever a Cell is BB backed(Onheap or offheap) always hasARray would be false in that Cell impl. Every where we would use getXXXBuffer along with getXXXOffest and getXXXLength. Even in case of KV we could wrap the byte[] with BB so that we have uniformity through the read code and we don't have too many 'if' else conditions. When ever hasArray is false - using getXXXArray API would throw UnSupportedOperation Exception. AS said if we want getXXXArray to be supported as per the existing way then getXXXBuffer and getXXXOffset, getXXXLength should be clearly documented. Thoughts!!! > Create ByteBuffer backed Cell > - > > Key: HBASE-12358 > URL: https://issues.apache.org/jira/browse/HBASE-12358 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-12358.patch, HBASE-12358_1.patch, > HBASE-12358_2.patch > > > As part of HBASE-12224 and HBASE-12282 we wanted a Cell that is backed by BB. > Changing the core Cell impl would not be needed as it is used in server > only. So we will create a BB backed Cell and use it in the Server side read > path. This JIRA just creates an interface that extends Cell and adds the > needed API. > The getTimeStamp and getTypebyte() can still refer to the original Cell API > only. The getXXxOffset() and getXXXLength() can also refer to the original > Cell only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12498) Integrate HydraBase RMap in RegionManager
[ https://issues.apache.org/jira/browse/HBASE-12498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amitanand Aiyer updated HBASE-12498: Description: We need to integrate the idea of RMap (Global Region Locator) into HBase. RMap: In Hydrabase, we wanted to avoid having special tables that require to be online and available, before other servers can start processing their regions. To this end, in hydrabase, we use a notion of RMap — that tells the servers which regions to open; and also servers as a look up place for the clients to discover the set of servers handling a particular region. This RMap is stored in a well known path, as specified in the configuration. Upon initialization, RegionServers read this file and determine the set of regions that it needs to open; and the set of peers that it expects to hear from for each of those regions. Since RMap is stored in HDFS, the only way to change it is to create a new HDFS file and update the servers about it. This is intended to be a not-so-common operation; however, this may be required when a new machine joins the service, or an existing machine needs to be taken away for a significant amount of time to perform maintenance. We run a daemon, every 15 minutes, (currently in the master) to determine the set of machines going to repair and vice versa; and update the RMap accordingly. 1. In the current hydraBase model, updating the RMap is done by a single daemon; while every region server individually reads and updates itself accordingly. 2. OSS HBase seems to be using a model where a dedicated master/server manager is responsible for coordinating the regions that should be opened/served by each region server. Region servers simply follow the instructions received from the server manager. With respect to integrating both. We plan to use the servermanager to read the RMap and coordinate the region opens on various regionservers. RMap will be still stored as a HDFS file, that will be read by the ServerManager/Master to assign regions. was: We need to integrate the idea of RMap (Global Region Locator) into HBase. Amit: Can you add more details about the design on this JIRA. Thanks, Rishit > Integrate HydraBase RMap in RegionManager > - > > Key: HBASE-12498 > URL: https://issues.apache.org/jira/browse/HBASE-12498 > Project: HBase > Issue Type: Sub-task > Components: master, Region Assignment >Reporter: Rishit Shroff >Assignee: Amitanand Aiyer > > We need to integrate the idea of RMap (Global Region Locator) into HBase. > RMap: > In Hydrabase, we wanted to avoid having special tables that require to be > online and available, before other servers can start processing their > regions. To this end, in hydrabase, we use a notion of RMap — that tells the > servers which regions to open; and also servers as a look up place for the > clients to discover the set of servers handling a particular region. > This RMap is stored in a well known path, as specified in the configuration. > Upon initialization, RegionServers read this file and determine the set of > regions that it needs to open; and the set of peers that it expects to hear > from for each of those regions. > Since RMap is stored in HDFS, the only way to change it is to create a new > HDFS file and update the servers about it. This is intended to be a > not-so-common operation; however, this may be required when a new machine > joins the service, or an existing machine needs to be taken away for a > significant amount of time to perform maintenance. > We run a daemon, every 15 minutes, (currently in the master) to determine the > set of machines going to repair and vice versa; and update the RMap > accordingly. > 1. In the current hydraBase model, updating the RMap is done by a single > daemon; while every region server individually reads and updates itself > accordingly. > 2. OSS HBase seems to be using a model where a dedicated master/server > manager is responsible for coordinating the regions that should be > opened/served by each region server. Region servers simply follow the > instructions received from the server manager. > With respect to integrating both. We plan to use the servermanager to read > the RMap and coordinate the region opens on various regionservers. RMap will > be still stored as a HDFS file, that will be read by the ServerManager/Master > to assign regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12583) Allow creating reference files even the split row not lies in the storefile range if required
[ https://issues.apache.org/jira/browse/HBASE-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234455#comment-14234455 ] Andrew Purtell commented on HBASE-12583: +1 for 0.98 if earlier reviewers are ok with the particulars. Skimmed the patch, looks ok > Allow creating reference files even the split row not lies in the storefile > range if required > - > > Key: HBASE-12583 > URL: https://issues.apache.org/jira/browse/HBASE-12583 > Project: HBase > Issue Type: Improvement >Reporter: rajeshbabu >Assignee: rajeshbabu > Labels: Phoenix > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12583.patch, HBASE-12583_98.patch, > HBASE-12583_branch1.patch, HBASE-12583_v2.patch > > > Currently in HRegionFileSystem#splitStoreFile we are not creating reference > files if the split row not lies in the storefile range that means one of the > child region doesn't have any data. > {code} >// Check whether the split row lies in the range of the store file > // If it is outside the range, return directly. > if (top) { > //check if larger than last key. > KeyValue splitKey = KeyValueUtil.createFirstOnRow(splitRow); > byte[] lastKey = f.createReader().getLastKey(); > // If lastKey is null means storefile is empty. > if (lastKey == null) return null; > if (f.getReader().getComparator().compareFlatKey(splitKey.getBuffer(), > splitKey.getKeyOffset(), splitKey.getKeyLength(), lastKey, 0, > lastKey.length) > 0) { > return null; > } > } else { > //check if smaller than first key > KeyValue splitKey = KeyValueUtil.createLastOnRow(splitRow); > byte[] firstKey = f.createReader().getFirstKey(); > // If firstKey is null means storefile is empty. > if (firstKey == null) return null; > if (f.getReader().getComparator().compareFlatKey(splitKey.getBuffer(), > splitKey.getKeyOffset(), splitKey.getKeyLength(), firstKey, 0, > firstKey.length) < 0) { > return null; > } > } > {code} > In some cases when split row should be compared with part of rowkey(in > composite rowkey) mainly for secondary index tables we need to create > reference files even when split row not lies in the storefile range so that > they can be rewritten to it's child regions by some custom half store file > reader which compare the part of row key with split row. > The check of comparing split row with storefile range and returning directly > can be avoided by having special boolean attribute in table descriptor when > it set to true. Or else we can have coprocessor hooks so that in the hooks we > can create the references and bypass. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12634) Fix the AccessController#requireGlobalPermission(ns) with NS
[ https://issues.apache.org/jira/browse/HBASE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234458#comment-14234458 ] Hudson commented on HBASE-12634: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #683 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/683/]) HBASE-12634 Fix the AccessController#requireGlobalPermission(ns) with NS (matteo.bertozzi: rev 6c927313c5ac7275bce3f9cdc4d215db28264601) * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java > Fix the AccessController#requireGlobalPermission(ns) with NS > - > > Key: HBASE-12634 > URL: https://issues.apache.org/jira/browse/HBASE-12634 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12634-v2.patch, HBASE-12634-v3.patch, > HBASE-12634.patch > > > The namespace argument passed to AccessController#requireGlobalPermission API > to authorize namespace is actually not authorizing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12622) user_permission should require global admin to display global and ns permissions
[ https://issues.apache.org/jira/browse/HBASE-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234464#comment-14234464 ] Andrew Purtell commented on HBASE-12622: Ok, thanks [~ashish singhi] > user_permission should require global admin to display global and ns > permissions > > > Key: HBASE-12622 > URL: https://issues.apache.org/jira/browse/HBASE-12622 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 0.98.8, 0.99.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12622-v0.patch, HBASE-12622-v1.patch > > > user_permission check the user permission only on the table level (requiring > at least a table-level admin) > global and namespace permission listing is done without checking anything. > but only a global admins should be able to perform this operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12358) Create ByteBuffer backed Cell
[ https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234468#comment-14234468 ] Anoop Sam John commented on HBASE-12358: bq.Note that here if hasARray is false (for KV case) then getXXXArray would also work. Correction : hasArray() is true > Create ByteBuffer backed Cell > - > > Key: HBASE-12358 > URL: https://issues.apache.org/jira/browse/HBASE-12358 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-12358.patch, HBASE-12358_1.patch, > HBASE-12358_2.patch > > > As part of HBASE-12224 and HBASE-12282 we wanted a Cell that is backed by BB. > Changing the core Cell impl would not be needed as it is used in server > only. So we will create a BB backed Cell and use it in the Server side read > path. This JIRA just creates an interface that extends Cell and adds the > needed API. > The getTimeStamp and getTypebyte() can still refer to the original Cell API > only. The getXXxOffset() and getXXXLength() can also refer to the original > Cell only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12619) Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234480#comment-14234480 ] Andrew Purtell commented on HBASE-12619: I made a pass over this and it seems fine. Let me see about committing this today, it should be in place ahead of HBASE-11764 because there's a small conflict with tag type allocation. (I'll fix that in HBASE-11764). Will take a closer look then. One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be possible today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality. But that's not an issue for here, HBASE-11639 has already been reviewed and committed and this is a backport. > Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98 > --- > > Key: HBASE-12619 > URL: https://issues.apache.org/jira/browse/HBASE-12619 > Project: HBase > Issue Type: Improvement > Components: security >Affects Versions: 0.98.8 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Labels: visibility > Fix For: 0.98.9 > > Attachments: HBASE-11639_0.98_1.patch > > > Raising a back port issue for HBASE-11639. We have the patch ready just need > to see how we are handling the new CP hook in RS Observer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234485#comment-14234485 ] Andrew Purtell commented on HBASE-11639: One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be possible today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality. Sorry I couldn't come here earlier, and anyway what has been committed is fine, but we should figure out what is the best next step here, if any. One option would be introducing a replication observer interface - ReplicationObserver - and coprocessor upcall chains out of replication code. Custom replication endpoint specification could be extended to allow specification of observer stacks. /cc [~lhofhansl] [~jesse_yates] > [Visibility controller] Replicate the visibility of Cells as strings > > > Key: HBASE-11639 > URL: https://issues.apache.org/jira/browse/HBASE-11639 > Project: HBase > Issue Type: Improvement > Components: Replication, security >Affects Versions: 0.98.4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Labels: VisibilityLabels > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-11639_amend.patch, HBASE-11639_v11.patch, > HBASE-11639_v13.patch, HBASE-11639_v13.patch, HBASE-11639_v14.patch, > HBASE-11639_v15.patch, HBASE-11639_v16.patch, HBASE-11639_v17.patch, > HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch, > HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch, > HBASE-11639_v9.patch > > > This issue is aimed at persisting the visibility labels as strings in the WAL > rather than Label ordinals. This would help in replicating the label > ordinals to the replication cluster as strings directly and also that after > HBASE-11553 would help because the replication cluster could have an > implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234485#comment-14234485 ] Andrew Purtell edited comment on HBASE-11639 at 12/4/14 7:06 PM: - One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be guaranteed today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality, each implementation must take care to delegate. Sorry I couldn't come here earlier, and anyway what has been committed is fine, the VisibilityReplicationEndpoint delegates to whatever it finds in place, but we should figure out what is the best next step here, if any. One option would be introducing a replication observer interface - ReplicationObserver - and coprocessor upcall chains out of replication code. Custom replication endpoint specification could be extended to allow specification of observer stacks. /cc [~lhofhansl] [~jesse_yates] was (Author: apurtell): One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be possible today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality. Sorry I couldn't come here earlier, and anyway what has been committed is fine, but we should figure out what is the best next step here, if any. One option would be introducing a replication observer interface - ReplicationObserver - and coprocessor upcall chains out of replication code. Custom replication endpoint specification could be extended to allow specification of observer stacks. /cc [~lhofhansl] [~jesse_yates] > [Visibility controller] Replicate the visibility of Cells as strings > > > Key: HBASE-11639 > URL: https://issues.apache.org/jira/browse/HBASE-11639 > Project: HBase > Issue Type: Improvement > Components: Replication, security >Affects Versions: 0.98.4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Labels: VisibilityLabels > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-11639_amend.patch, HBASE-11639_v11.patch, > HBASE-11639_v13.patch, HBASE-11639_v13.patch, HBASE-11639_v14.patch, > HBASE-11639_v15.patch, HBASE-11639_v16.patch, HBASE-11639_v17.patch, > HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch, > HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch, > HBASE-11639_v9.patch > > > This issue is aimed at persisting the visibility labels as strings in the WAL > rather than Label ordinals. This would help in replicating the label > ordinals to the replication cluster as strings directly and also that after > HBASE-11553 would help because the replication cluster could have an > implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12619) Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234480#comment-14234480 ] Andrew Purtell edited comment on HBASE-12619 at 12/4/14 7:06 PM: - I made a pass over this and it seems fine. Let me see about committing this today, it should be in place ahead of HBASE-11764 because there's a small conflict with tag type allocation. (I'll fix that in HBASE-11764). Will take a closer look then. One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be guaranteed today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality, each implementation must take care to delegate. But that's not an issue for here, HBASE-11639 has already been reviewed and committed and this is a backport. was (Author: apurtell): I made a pass over this and it seems fine. Let me see about committing this today, it should be in place ahead of HBASE-11764 because there's a small conflict with tag type allocation. (I'll fix that in HBASE-11764). Will take a closer look then. One higher level question I have about this whole design is what happens if we want to replicate visibility tags as Strings over an existing custom ReplicationEndpoint? That won't be possible today because an endpoint is implemented by a single class, there's no opportunity for mixing in functionality. But that's not an issue for here, HBASE-11639 has already been reviewed and committed and this is a backport. > Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98 > --- > > Key: HBASE-12619 > URL: https://issues.apache.org/jira/browse/HBASE-12619 > Project: HBase > Issue Type: Improvement > Components: security >Affects Versions: 0.98.8 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Labels: visibility > Fix For: 0.98.9 > > Attachments: HBASE-11639_0.98_1.patch > > > Raising a back port issue for HBASE-11639. We have the patch ready just need > to see how we are handling the new CP hook in RS Observer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234491#comment-14234491 ] Anoop Sam John commented on HBASE-11639: I think there is no issue with custom replication endpoint. What we do is create a wrapper on top of the replication EP. This is done in VC new hook impl. So this will wrap the custom Replication EP. System calls the wrapper 1st and it convert visibility to String based and then the custom EP which can do its function. Does that sounds good enough Andy? > [Visibility controller] Replicate the visibility of Cells as strings > > > Key: HBASE-11639 > URL: https://issues.apache.org/jira/browse/HBASE-11639 > Project: HBase > Issue Type: Improvement > Components: Replication, security >Affects Versions: 0.98.4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Labels: VisibilityLabels > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-11639_amend.patch, HBASE-11639_v11.patch, > HBASE-11639_v13.patch, HBASE-11639_v13.patch, HBASE-11639_v14.patch, > HBASE-11639_v15.patch, HBASE-11639_v16.patch, HBASE-11639_v17.patch, > HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch, > HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch, > HBASE-11639_v9.patch > > > This issue is aimed at persisting the visibility labels as strings in the WAL > rather than Label ordinals. This would help in replicating the label > ordinals to the replication cluster as strings directly and also that after > HBASE-11553 would help because the replication cluster could have an > implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12619) Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234490#comment-14234490 ] Andrew Purtell commented on HBASE-12619: /cc [~lhofhansl] [~jesse_yates], private replication API changes > Backport HBASE-11639 (Replicate the visibility of Cells as strings) to 0.98 > --- > > Key: HBASE-12619 > URL: https://issues.apache.org/jira/browse/HBASE-12619 > Project: HBase > Issue Type: Improvement > Components: security >Affects Versions: 0.98.8 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Labels: visibility > Fix For: 0.98.9 > > Attachments: HBASE-11639_0.98_1.patch > > > Raising a back port issue for HBASE-11639. We have the patch ready just need > to see how we are handling the new CP hook in RS Observer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234494#comment-14234494 ] Ted Yu commented on HBASE-12204: Test suite passes on branch-1 > Backport HBASE-12016 'Reduce number of versions in Meta table. Make it > configurable' to branch-1 > > > Key: HBASE-12204 > URL: https://issues.apache.org/jira/browse/HBASE-12204 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 1.0.0 > > Attachments: 12204-1.0.txt > > > The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that > HBASE-7767 was applied to master only. > This JIRA is to backport HBASE-12016 to branch-1. > This can be done when HBASE-12035 is resolved and branch-1 is brought to > parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12565) Race condition in HRegion.batchMutate() causes partial data to be written when region closes
[ https://issues.apache.org/jira/browse/HBASE-12565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith David Winkler updated HBASE-12565: Attachment: hbase-12565.patch Applies the two part recommended solution from the description. Divided the existing batchMutate() test into two tests, one that performs batchMutate while no row locks are held, one that performs batch mutate while multiple row locks are held and concurrently another thread calls region.close(). > Race condition in HRegion.batchMutate() causes partial data to be written > when region closes > - > > Key: HBASE-12565 > URL: https://issues.apache.org/jira/browse/HBASE-12565 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 0.98.6 >Reporter: Scott Fines > Attachments: hbase-12565.patch > > > The following sequence of events is possible to occur in HRegion's > batchMutate() call: > 1. caller attempts to call HRegion.batchMutate() with a batch of N>1 records > 2. batchMutate acquires region lock in startRegionOperation, then calls > doMiniBatchMutation() > 3. doMiniBatchMutation acquires one row lock > 4. Region closes > 5. doMiniBatchMutation attempts to acquire second row lock. > When this happens, the lock acquisition will also attempt to acquire the > region lock, which fails (because the region is closing). At this stage, > doMiniBatchMutation will stop writing further, BUT it WILL write data for the > rows whose locks have already been acquired, and advance the index in > MiniBatchOperationInProgress. Then, after it terminates successfully, > batchMutate() will loop around a second time, and attempt AGAIN to acquire > the region closing lock. When that happens, a NotServingRegionException is > thrown back to the caller. > Thus, we have a race condition where partial data can be written when a > region server is closing. > The main problem stems from the location of startRegionOperation() calls in > batchMutate and doMiniBatchMutation(): > 1. batchMutate() reacquires the region lock with each iteration of the loop, > which can cause some successful writes to occur, but then fail on others > 2. getRowLock() attempts to acquire the region lock once for each row, which > allows doMiniBatchMutation to terminate early; this forces batchMutate() to > use multiple iterations and results in condition 1 being hit. > There appears to be two parts to the solution as well: > 1. open an internal path so that doMiniBatchMutation() can acquire row locks > without checking for region closure. This will have the added benefit of a > significant performance improvement during large batch mutations. > 2. move the startRegionOperation() out of the loop in batchMutate() so that > multiple iterations of doMiniBatchMutation will not cause the operation to > fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11677) Make Logger instance modifiers consistent
[ https://issues.apache.org/jira/browse/HBASE-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234501#comment-14234501 ] Andrew Purtell commented on HBASE-11677: Please make a patch for trunk [~Usha710] and then we can take these changes. > Make Logger instance modifiers consistent > - > > Key: HBASE-11677 > URL: https://issues.apache.org/jira/browse/HBASE-11677 > Project: HBase > Issue Type: Task >Reporter: Sean Busbey >Priority: Minor > Labels: beginner, sonar > Attachments: HBASE-11677-0.98.patch > > > We have some instances of Logger that are missing one of being private, > static, and final. > ex from HealthChecker.java, missing final > {code} > private static Log LOG = LogFactory.getLog(HealthChecker.class); > {code} > * Clean up where possible by making {{private static final}} > * If we can't, add a non-javadoc note about why > One way to look for problematic instances is to grep for initial assignment > for the commonly used LOG member, e.g. > * missing final: {{grep -r "LOG =" * | grep -v "final"}} > * missing static: {{grep -r "LOG =" * | grep -v "static"}} > * missing private: {{grep -r "LOG =" * | grep -v "private"}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12564) consolidate the getTableDescriptors() semantic
[ https://issues.apache.org/jira/browse/HBASE-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234504#comment-14234504 ] Andrew Purtell commented on HBASE-12564: bq. I think the behavior should be "give me all the names that I have access to" so the AccessController should filter the tables like getTableDescriptors() but checking also Read/Write rights and not only Admin/Create. Andrew Purtell? No, users with read or write access should not be granted permission to potentially sensitive information in the schema. > consolidate the getTableDescriptors() semantic > -- > > Key: HBASE-12564 > URL: https://issues.apache.org/jira/browse/HBASE-12564 > Project: HBase > Issue Type: Bug > Components: Client, master >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-12564-v0.patch, HBASE-12564-v1.patch, > HBASE-12564-v2.patch > > > Master getTableDescriptors() which is called by Admin.listTables() has a > couple of different behaviors depending on how it is called. > after HBASE-12073 with the AccessController enabled, we now get a "global > admin" required if listTables() is called without a regex otherwise we return > only the table that the user can see (we show only the tables that the user > have access to, which means or the user is a global admin or it has a > table-level create/admin). We probably should have the second behavior even > without regex, since I should able to see "my own tables". > getTableDescriptors() is returning only non system tables. Tools like > user_permission that are doing "for each listTable(): userPerm(table)" are > losing the system tables, so stuff like user_permission 'hbase:acls' will not > result any result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12565) Race condition in HRegion.batchMutate() causes partial data to be written when region closes
[ https://issues.apache.org/jira/browse/HBASE-12565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith David Winkler updated HBASE-12565: Affects Version/s: 2.0.0 Status: Patch Available (was: Open) > Race condition in HRegion.batchMutate() causes partial data to be written > when region closes > - > > Key: HBASE-12565 > URL: https://issues.apache.org/jira/browse/HBASE-12565 > Project: HBase > Issue Type: Bug > Components: Performance, regionserver >Affects Versions: 0.98.6, 2.0.0 >Reporter: Scott Fines > Attachments: hbase-12565.patch > > > The following sequence of events is possible to occur in HRegion's > batchMutate() call: > 1. caller attempts to call HRegion.batchMutate() with a batch of N>1 records > 2. batchMutate acquires region lock in startRegionOperation, then calls > doMiniBatchMutation() > 3. doMiniBatchMutation acquires one row lock > 4. Region closes > 5. doMiniBatchMutation attempts to acquire second row lock. > When this happens, the lock acquisition will also attempt to acquire the > region lock, which fails (because the region is closing). At this stage, > doMiniBatchMutation will stop writing further, BUT it WILL write data for the > rows whose locks have already been acquired, and advance the index in > MiniBatchOperationInProgress. Then, after it terminates successfully, > batchMutate() will loop around a second time, and attempt AGAIN to acquire > the region closing lock. When that happens, a NotServingRegionException is > thrown back to the caller. > Thus, we have a race condition where partial data can be written when a > region server is closing. > The main problem stems from the location of startRegionOperation() calls in > batchMutate and doMiniBatchMutation(): > 1. batchMutate() reacquires the region lock with each iteration of the loop, > which can cause some successful writes to occur, but then fail on others > 2. getRowLock() attempts to acquire the region lock once for each row, which > allows doMiniBatchMutation to terminate early; this forces batchMutate() to > use multiple iterations and results in condition 1 being hit. > There appears to be two parts to the solution as well: > 1. open an internal path so that doMiniBatchMutation() can acquire row locks > without checking for region closure. This will have the added benefit of a > significant performance improvement during large batch mutations. > 2. move the startRegionOperation() out of the loop in batchMutate() so that > multiple iterations of doMiniBatchMutation will not cause the operation to > fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12422) Use ConnectionFactory in HTable constructors
[ https://issues.apache.org/jira/browse/HBASE-12422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12422: -- Attachment: HBASE-12422.patch Agree, these should be fine if their close is going to clean up the connection. Retry hadoopqa. > Use ConnectionFactory in HTable constructors > > > Key: HBASE-12422 > URL: https://issues.apache.org/jira/browse/HBASE-12422 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Solomon Duskis >Assignee: Solomon Duskis >Priority: Minor > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-12422.patch, HBASE-12422.patch, HBASE-12422.patch > > > Use ConnectionFactory in HTable instead of ConnectionManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12564) consolidate the getTableDescriptors() semantic
[ https://issues.apache.org/jira/browse/HBASE-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234516#comment-14234516 ] Andrew Purtell commented on HBASE-12564: Anyway I looked at the v2 patch and it seems to do the right thing, we won't return descriptors unless the user has ADMIN or CREATE permissions in dominant scope. In postGetTableNames we will return table _names_ if the user has any of the Action bits in dominant scope, but that doesn't seem to be a problem, it follows that a user with any access to a table must know its name to make use of it. > consolidate the getTableDescriptors() semantic > -- > > Key: HBASE-12564 > URL: https://issues.apache.org/jira/browse/HBASE-12564 > Project: HBase > Issue Type: Bug > Components: Client, master >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-12564-v0.patch, HBASE-12564-v1.patch, > HBASE-12564-v2.patch > > > Master getTableDescriptors() which is called by Admin.listTables() has a > couple of different behaviors depending on how it is called. > after HBASE-12073 with the AccessController enabled, we now get a "global > admin" required if listTables() is called without a regex otherwise we return > only the table that the user can see (we show only the tables that the user > have access to, which means or the user is a global admin or it has a > table-level create/admin). We probably should have the second behavior even > without regex, since I should able to see "my own tables". > getTableDescriptors() is returning only non system tables. Tools like > user_permission that are doing "for each listTable(): userPerm(table)" are > losing the system tables, so stuff like user_permission 'hbase:acls' will not > result any result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12564) consolidate the getTableDescriptors() semantic
[ https://issues.apache.org/jira/browse/HBASE-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234525#comment-14234525 ] Matteo Bertozzi commented on HBASE-12564: - {quote}No, users with read or write access should not be granted permission to potentially sensitive information in the schema.{quote} yeah, sorry maybe I was not clear. I was talking about the getTableNames() which just returns the table names. In that case if the user have "access" to the table (by access I mean any RWXCA on any level of the table) the shell "list" will display the names (only names). while for the descriptor as you said only users granted with ADMIN/CREATE can see the table schema. > consolidate the getTableDescriptors() semantic > -- > > Key: HBASE-12564 > URL: https://issues.apache.org/jira/browse/HBASE-12564 > Project: HBase > Issue Type: Bug > Components: Client, master >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-12564-v0.patch, HBASE-12564-v1.patch, > HBASE-12564-v2.patch > > > Master getTableDescriptors() which is called by Admin.listTables() has a > couple of different behaviors depending on how it is called. > after HBASE-12073 with the AccessController enabled, we now get a "global > admin" required if listTables() is called without a regex otherwise we return > only the table that the user can see (we show only the tables that the user > have access to, which means or the user is a global admin or it has a > table-level create/admin). We probably should have the second behavior even > without regex, since I should able to see "my own tables". > getTableDescriptors() is returning only non system tables. Tools like > user_permission that are doing "for each listTable(): userPerm(table)" are > losing the system tables, so stuff like user_permission 'hbase:acls' will not > result any result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12622) user_permission should require global admin to display global and ns permissions
[ https://issues.apache.org/jira/browse/HBASE-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234533#comment-14234533 ] Srikanth Srungarapu commented on HBASE-12622: - +1 (non-binding). > user_permission should require global admin to display global and ns > permissions > > > Key: HBASE-12622 > URL: https://issues.apache.org/jira/browse/HBASE-12622 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 2.0.0, 0.98.8, 0.99.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12622-v0.patch, HBASE-12622-v1.patch > > > user_permission check the user permission only on the table level (requiring > at least a table-level admin) > global and namespace permission listing is done without checking anything. > but only a global admins should be able to perform this operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12583) Allow creating reference files even the split row not lies in the storefile range if required
[ https://issues.apache.org/jira/browse/HBASE-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234555#comment-14234555 ] stack commented on HBASE-12583: --- Would it make more sense and cleaner passing the splitPolicy in when you call HRegionFS#splitStoreFile? Now you have amended code base to pass a boolean. Later, splitting store files we might want to pass more info. We could pass it as extra args or just pass the policy altogether now to future proof. Otherwise, patch is fine by me. > Allow creating reference files even the split row not lies in the storefile > range if required > - > > Key: HBASE-12583 > URL: https://issues.apache.org/jira/browse/HBASE-12583 > Project: HBase > Issue Type: Improvement >Reporter: rajeshbabu >Assignee: rajeshbabu > Labels: Phoenix > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12583.patch, HBASE-12583_98.patch, > HBASE-12583_branch1.patch, HBASE-12583_v2.patch > > > Currently in HRegionFileSystem#splitStoreFile we are not creating reference > files if the split row not lies in the storefile range that means one of the > child region doesn't have any data. > {code} >// Check whether the split row lies in the range of the store file > // If it is outside the range, return directly. > if (top) { > //check if larger than last key. > KeyValue splitKey = KeyValueUtil.createFirstOnRow(splitRow); > byte[] lastKey = f.createReader().getLastKey(); > // If lastKey is null means storefile is empty. > if (lastKey == null) return null; > if (f.getReader().getComparator().compareFlatKey(splitKey.getBuffer(), > splitKey.getKeyOffset(), splitKey.getKeyLength(), lastKey, 0, > lastKey.length) > 0) { > return null; > } > } else { > //check if smaller than first key > KeyValue splitKey = KeyValueUtil.createLastOnRow(splitRow); > byte[] firstKey = f.createReader().getFirstKey(); > // If firstKey is null means storefile is empty. > if (firstKey == null) return null; > if (f.getReader().getComparator().compareFlatKey(splitKey.getBuffer(), > splitKey.getKeyOffset(), splitKey.getKeyLength(), firstKey, 0, > firstKey.length) < 0) { > return null; > } > } > {code} > In some cases when split row should be compared with part of rowkey(in > composite rowkey) mainly for secondary index tables we need to create > reference files even when split row not lies in the storefile range so that > they can be rewritten to it's child regions by some custom half store file > reader which compare the part of row key with split row. > The check of comparing split row with storefile range and returning directly > can be avoided by having special boolean attribute in table descriptor when > it set to true. Or else we can have coprocessor hooks so that in the hooks we > can create the references and bypass. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12498) Integrate HydraBase RMap in RegionManager
[ https://issues.apache.org/jira/browse/HBASE-12498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234561#comment-14234561 ] stack commented on HBASE-12498: --- I like it that master continues to tell regionservers what they should serve. Thanks [~amitanand] > Integrate HydraBase RMap in RegionManager > - > > Key: HBASE-12498 > URL: https://issues.apache.org/jira/browse/HBASE-12498 > Project: HBase > Issue Type: Sub-task > Components: master, Region Assignment >Reporter: Rishit Shroff >Assignee: Amitanand Aiyer > > We need to integrate the idea of RMap (Global Region Locator) into HBase. > RMap: > In Hydrabase, we wanted to avoid having special tables that require to be > online and available, before other servers can start processing their > regions. To this end, in hydrabase, we use a notion of RMap — that tells the > servers which regions to open; and also servers as a look up place for the > clients to discover the set of servers handling a particular region. > This RMap is stored in a well known path, as specified in the configuration. > Upon initialization, RegionServers read this file and determine the set of > regions that it needs to open; and the set of peers that it expects to hear > from for each of those regions. > Since RMap is stored in HDFS, the only way to change it is to create a new > HDFS file and update the servers about it. This is intended to be a > not-so-common operation; however, this may be required when a new machine > joins the service, or an existing machine needs to be taken away for a > significant amount of time to perform maintenance. > We run a daemon, every 15 minutes, (currently in the master) to determine the > set of machines going to repair and vice versa; and update the RMap > accordingly. > 1. In the current hydraBase model, updating the RMap is done by a single > daemon; while every region server individually reads and updates itself > accordingly. > 2. OSS HBase seems to be using a model where a dedicated master/server > manager is responsible for coordinating the regions that should be > opened/served by each region server. Region servers simply follow the > instructions received from the server manager. > With respect to integrating both. We plan to use the servermanager to read > the RMap and coordinate the region opens on various regionservers. RMap will > be still stored as a HDFS file, that will be read by the ServerManager/Master > to assign regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234569#comment-14234569 ] Enis Soztutar commented on HBASE-12204: --- I compared the patch with the 0.98 version. Seems good. +1. We should be more careful for this type of feature gaps. It seems that HBASE-12205 went into 0.98 without the branch-1 patch. > Backport HBASE-12016 'Reduce number of versions in Meta table. Make it > configurable' to branch-1 > > > Key: HBASE-12204 > URL: https://issues.apache.org/jira/browse/HBASE-12204 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 1.0.0 > > Attachments: 12204-1.0.txt > > > The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that > HBASE-7767 was applied to master only. > This JIRA is to backport HBASE-12016 to branch-1. > This can be done when HBASE-12035 is resolved and branch-1 is brought to > parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12629) Remove hbase.regionsizecalculator.enable from RegionSizeCalculator
[ https://issues.apache.org/jira/browse/HBASE-12629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234572#comment-14234572 ] Enis Soztutar commented on HBASE-12629: --- I thought that it is a safeguard if you want to disable this. Why are we removing it? > Remove hbase.regionsizecalculator.enable from RegionSizeCalculator > -- > > Key: HBASE-12629 > URL: https://issues.apache.org/jira/browse/HBASE-12629 > Project: HBase > Issue Type: Improvement >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 1.0.0, 2.0.0 > > Attachments: > 0001-HBASE-12629-Remove-hbase.regionsizecalculator.enable.patch > > > The RegionSizeCalculator has a option to disable it. It is on by default and > end-to-end use with it disabled is not tested or used anywhere except for a > simple unit test. This removes it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12204: --- Hadoop Flags: Reviewed > Backport HBASE-12016 'Reduce number of versions in Meta table. Make it > configurable' to branch-1 > > > Key: HBASE-12204 > URL: https://issues.apache.org/jira/browse/HBASE-12204 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 1.0.0 > > Attachments: 12204-1.0.txt > > > The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that > HBASE-7767 was applied to master only. > This JIRA is to backport HBASE-12016 to branch-1. > This can be done when HBASE-12035 is resolved and branch-1 is brought to > parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234580#comment-14234580 ] Enis Soztutar commented on HBASE-11639: --- While we are on the subject, the thing I did not fully understand in this patch and also in HBASE-11920 was that why do we do this way of getting a RE from config, then replacing that with a one coming from a CP. The RE is already pluggable, shouldn't the user configure RE accordingly, and then configure what RE to wrap in this new RE implementation rather. It seems very indirect. > [Visibility controller] Replicate the visibility of Cells as strings > > > Key: HBASE-11639 > URL: https://issues.apache.org/jira/browse/HBASE-11639 > Project: HBase > Issue Type: Improvement > Components: Replication, security >Affects Versions: 0.98.4 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Labels: VisibilityLabels > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-11639_amend.patch, HBASE-11639_v11.patch, > HBASE-11639_v13.patch, HBASE-11639_v13.patch, HBASE-11639_v14.patch, > HBASE-11639_v15.patch, HBASE-11639_v16.patch, HBASE-11639_v17.patch, > HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch, > HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch, > HBASE-11639_v9.patch > > > This issue is aimed at persisting the visibility labels as strings in the WAL > rather than Label ordinals. This would help in replicating the label > ordinals to the replication cluster as strings directly and also that after > HBASE-11553 would help because the replication cluster could have an > implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12493) User class should provide a way to re-use existing token
[ https://issues.apache.org/jira/browse/HBASE-12493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234584#comment-14234584 ] Yongzhi Chen commented on HBASE-12493: -- [~ghelmling], the hive HBaseStorageHandler does not have the info of the hbase connection, all your TokenUtil.addTokenForJob methods need connection, is that possible you add a method with addTokenForJob(Configuration conf, Job job, User user) Thanks > User class should provide a way to re-use existing token > > > Key: HBASE-12493 > URL: https://issues.apache.org/jira/browse/HBASE-12493 > Project: HBase > Issue Type: Task >Reporter: Brock Noland >Assignee: Gary Helmling > Fix For: 2.0.0 > > Attachments: HBASE-12493-v2.patch, HBASE-12493.patch > > > In HIVE-8874 we had to re-use HBase classes market private to re-use using > tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12485) Maintain SeqId monotonically increasing when Region Replica is on
[ https://issues.apache.org/jira/browse/HBASE-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12485: -- Fix Version/s: 2.0.0 1.0.0 > Maintain SeqId monotonically increasing when Region Replica is on > - > > Key: HBASE-12485 > URL: https://issues.apache.org/jira/browse/HBASE-12485 > Project: HBase > Issue Type: Sub-task >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-12485.patch > > > We added FLUSH, REGION CLOSE events into WAL, for each those events the > region SeqId is bumped. > The issue comes from region close operation. Because when opening a region, > we use flushed SeqId from store files while after store flush during region > close we still write COMMIT_FLUSH, REGION_CLOSE events etc which respectively > bump up SeqId. Therefore, the region opening SeqId is lower than it should > be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12485) Maintain SeqId monotonically increasing
[ https://issues.apache.org/jira/browse/HBASE-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-12485: -- Summary: Maintain SeqId monotonically increasing (was: Maintain SeqId monotonically increasing when Region Replica is on) > Maintain SeqId monotonically increasing > --- > > Key: HBASE-12485 > URL: https://issues.apache.org/jira/browse/HBASE-12485 > Project: HBase > Issue Type: Sub-task >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 1.0.0, 2.0.0 > > > We added FLUSH, REGION CLOSE events into WAL, for each those events the > region SeqId is bumped. > The issue comes from region close operation. Because when opening a region, > we use flushed SeqId from store files while after store flush during region > close we still write COMMIT_FLUSH, REGION_CLOSE events etc which respectively > bump up SeqId. Therefore, the region opening SeqId is lower than it should > be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12485) Maintain SeqId monotonically increasing
[ https://issues.apache.org/jira/browse/HBASE-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-12485: -- Attachment: (was: HBASE-12485.patch) > Maintain SeqId monotonically increasing > --- > > Key: HBASE-12485 > URL: https://issues.apache.org/jira/browse/HBASE-12485 > Project: HBase > Issue Type: Sub-task >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 1.0.0, 2.0.0 > > > We added FLUSH, REGION CLOSE events into WAL, for each those events the > region SeqId is bumped. > The issue comes from region close operation. Because when opening a region, > we use flushed SeqId from store files while after store flush during region > close we still write COMMIT_FLUSH, REGION_CLOSE events etc which respectively > bump up SeqId. Therefore, the region opening SeqId is lower than it should > be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12485) Maintain SeqId monotonically increasing
[ https://issues.apache.org/jira/browse/HBASE-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-12485: -- Attachment: HBASE-12485.patch Resubmit for QA run > Maintain SeqId monotonically increasing > --- > > Key: HBASE-12485 > URL: https://issues.apache.org/jira/browse/HBASE-12485 > Project: HBase > Issue Type: Sub-task >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-12485.patch > > > We added FLUSH, REGION CLOSE events into WAL, for each those events the > region SeqId is bumped. > The issue comes from region close operation. Because when opening a region, > we use flushed SeqId from store files while after store flush during region > close we still write COMMIT_FLUSH, REGION_CLOSE events etc which respectively > bump up SeqId. Therefore, the region opening SeqId is lower than it should > be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12493) User class should provide a way to re-use existing token
[ https://issues.apache.org/jira/browse/HBASE-12493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234613#comment-14234613 ] Gary Helmling commented on HBASE-12493: --- [~ychena] The Hive HBaseStorageHandler should create and manage the connection, doing something like: {code} Connection conn = ConnectionFactory.createConnection(job.getConfiguration()); try { UserProvider userProvider = UserProvider.instantiate(job.getConfiguration()); TokenUtil.addTokenForJob(conn, userProvider.getCurrent(), job); } finally { conn.close(); } {code} Creating the connection and managing it should really be done explicitly. It's clunky and inefficient for us to do this hidden from the client code, which leads to nasty things like hidden connection caching that we've been trying to remove from the HBase code for a long time. It's much simpler if we just make the connection handling explicit. This is why I've deprecated all methods which do _not_ take a {{Connection}} instance, and have to create one behind the scenes. Do you see any other issues? > User class should provide a way to re-use existing token > > > Key: HBASE-12493 > URL: https://issues.apache.org/jira/browse/HBASE-12493 > Project: HBase > Issue Type: Task >Reporter: Brock Noland >Assignee: Gary Helmling > Fix For: 2.0.0 > > Attachments: HBASE-12493-v2.patch, HBASE-12493.patch > > > In HIVE-8874 we had to re-use HBase classes market private to re-use using > tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12637) Compilation with Hadoop-2.4- is broken
Enis Soztutar created HBASE-12637: - Summary: Compilation with Hadoop-2.4- is broken Key: HBASE-12637 URL: https://issues.apache.org/jira/browse/HBASE-12637 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar HBASE-12554 introduced {code} public static class MockMapping extends ScriptBasedMapping { {code} which breaks compilation with earlier hadoop versions: {code} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hbase-server: Compilation failure: Compilation failure: [ERROR] /Users/enis/projects/rc-test/hbase-0.99.2/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java:[77,42] error: cannot inherit from final ScriptBasedMapping {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12637) Compilation with Hadoop-2.4- is broken
[ https://issues.apache.org/jira/browse/HBASE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12637: -- Fix Version/s: 1.0.0 > Compilation with Hadoop-2.4- is broken > --- > > Key: HBASE-12637 > URL: https://issues.apache.org/jira/browse/HBASE-12637 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 1.0.0 > > > HBASE-12554 introduced > {code} > public static class MockMapping extends ScriptBasedMapping { > {code} > which breaks compilation with earlier hadoop versions: > {code} > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile > (default-testCompile) on project hbase-server: Compilation failure: > Compilation failure: > [ERROR] > /Users/enis/projects/rc-test/hbase-0.99.2/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java:[77,42] > error: cannot inherit from final ScriptBasedMapping > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12637) Compilation with Hadoop-2.4- is broken
[ https://issues.apache.org/jira/browse/HBASE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234626#comment-14234626 ] Ted Yu commented on HBASE-12637: Thanks for catching this, Enis. > Compilation with Hadoop-2.4- is broken > --- > > Key: HBASE-12637 > URL: https://issues.apache.org/jira/browse/HBASE-12637 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 1.0.0 > > > HBASE-12554 introduced > {code} > public static class MockMapping extends ScriptBasedMapping { > {code} > which breaks compilation with earlier hadoop versions: > {code} > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile > (default-testCompile) on project hbase-server: Compilation failure: > Compilation failure: > [ERROR] > /Users/enis/projects/rc-test/hbase-0.99.2/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java:[77,42] > error: cannot inherit from final ScriptBasedMapping > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12493) User class should provide a way to re-use existing token
[ https://issues.apache.org/jira/browse/HBASE-12493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234628#comment-14234628 ] Yongzhi Chen commented on HBASE-12493: -- [~ghelmling], I have no other issues, thanks > User class should provide a way to re-use existing token > > > Key: HBASE-12493 > URL: https://issues.apache.org/jira/browse/HBASE-12493 > Project: HBase > Issue Type: Task >Reporter: Brock Noland >Assignee: Gary Helmling > Fix For: 2.0.0 > > Attachments: HBASE-12493-v2.patch, HBASE-12493.patch > > > In HIVE-8874 we had to re-use HBase classes market private to re-use using > tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12637) Compilation with Hadoop-2.4- is broken
[ https://issues.apache.org/jira/browse/HBASE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12637: -- Attachment: hbase-12637_v1.patch Simple patch. Compiled with 2.2.0, 2.3.0, 2.4.0, 2.5.0, and 2.6.0. And ran mvn test -Dtest=TestBaseLoadBalancer with 2.6.0. > Compilation with Hadoop-2.4- is broken > --- > > Key: HBASE-12637 > URL: https://issues.apache.org/jira/browse/HBASE-12637 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 1.0.0 > > Attachments: hbase-12637_v1.patch > > > HBASE-12554 introduced > {code} > public static class MockMapping extends ScriptBasedMapping { > {code} > which breaks compilation with earlier hadoop versions: > {code} > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile > (default-testCompile) on project hbase-server: Compilation failure: > Compilation failure: > [ERROR] > /Users/enis/projects/rc-test/hbase-0.99.2/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java:[77,42] > error: cannot inherit from final ScriptBasedMapping > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12637) Compilation with Hadoop-2.4- is broken
[ https://issues.apache.org/jira/browse/HBASE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234638#comment-14234638 ] Ted Yu commented on HBASE-12637: +1 > Compilation with Hadoop-2.4- is broken > --- > > Key: HBASE-12637 > URL: https://issues.apache.org/jira/browse/HBASE-12637 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 1.0.0 > > Attachments: hbase-12637_v1.patch > > > HBASE-12554 introduced > {code} > public static class MockMapping extends ScriptBasedMapping { > {code} > which breaks compilation with earlier hadoop versions: > {code} > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile > (default-testCompile) on project hbase-server: Compilation failure: > Compilation failure: > [ERROR] > /Users/enis/projects/rc-test/hbase-0.99.2/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java:[77,42] > error: cannot inherit from final ScriptBasedMapping > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12625) Deprecate certain methods in classes annotated with InterfaceAudience.Public in branch-1
[ https://issues.apache.org/jira/browse/HBASE-12625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234643#comment-14234643 ] Andrew Purtell commented on HBASE-12625: +1 for the patch on this issue In general I wouldn't expect Phoenix to compile against branch-1, see PHOENIX-1501. [~tedyu], if you have a list of compilation errors against Private audience APIs handy perhaps you'd comment on the Phoenix JIRA? > Deprecate certain methods in classes annotated with InterfaceAudience.Public > in branch-1 > > > Key: HBASE-12625 > URL: https://issues.apache.org/jira/browse/HBASE-12625 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 1.0.0 > > Attachments: 12625-v1.txt > > > Compiling Phoenix against branch-1, I got compilation error below: > {code} > [ERROR] > /Users/tyu/phoenix/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java:[345,49] > cannot find symbol > [ERROR] symbol: method estimatedSizeOf(org.apache.hadoop.hbase.Cell) > [ERROR] location: class org.apache.hadoop.hbase.CellUtil > {code} > CellUtil is annotated with InterfaceAudience.Public > estimatedSizeOf() should be deprecated in branch-1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12204) Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-12204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12204: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review, Enis. > Backport HBASE-12016 'Reduce number of versions in Meta table. Make it > configurable' to branch-1 > > > Key: HBASE-12204 > URL: https://issues.apache.org/jira/browse/HBASE-12204 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 1.0.0 > > Attachments: 12204-1.0.txt > > > The patch from HBASE-12016 doesn't apply to branch-1 due to the fact that > HBASE-7767 was applied to master only. > This JIRA is to backport HBASE-12016 to branch-1. > This can be done when HBASE-12035 is resolved and branch-1 is brought to > parity with master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12625) Deprecate certain methods in classes annotated with InterfaceAudience.Public in branch-1
[ https://issues.apache.org/jira/browse/HBASE-12625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234648#comment-14234648 ] Andrew Purtell commented on HBASE-12625: RpcScheduler, RpcSchedulerFactory, and friends is LimitedPrivate(PHOENIX) and Evolving. My interpretation of what that means is we owe a deprecation cycle, so should be deprecated in branch-1 and then we can do whatever on master. Or revert the signature change in branch-1 and master. > Deprecate certain methods in classes annotated with InterfaceAudience.Public > in branch-1 > > > Key: HBASE-12625 > URL: https://issues.apache.org/jira/browse/HBASE-12625 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 1.0.0 > > Attachments: 12625-v1.txt > > > Compiling Phoenix against branch-1, I got compilation error below: > {code} > [ERROR] > /Users/tyu/phoenix/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java:[345,49] > cannot find symbol > [ERROR] symbol: method estimatedSizeOf(org.apache.hadoop.hbase.Cell) > [ERROR] location: class org.apache.hadoop.hbase.CellUtil > {code} > CellUtil is annotated with InterfaceAudience.Public > estimatedSizeOf() should be deprecated in branch-1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12485) Maintain SeqId monotonically increasing
[ https://issues.apache.org/jira/browse/HBASE-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234649#comment-14234649 ] stack commented on HBASE-12485: --- Throw exception rather than WARN. 3547LOG.warn("Found decreasing SeqId. PreId=" + currentEditSeqId + " key=" + key 3548+ "; edit=" + val); It will never happen, right? (smile) I did not realize we were writing a file into region with the sequenceid. Better than into zk I'd say. Should the special file instead be named with a dot prefix like other special files (see head of HRegionFileSystem for list -- .regioninfo, .tmp, etc.) > Maintain SeqId monotonically increasing > --- > > Key: HBASE-12485 > URL: https://issues.apache.org/jira/browse/HBASE-12485 > Project: HBase > Issue Type: Sub-task >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-12485.patch > > > We added FLUSH, REGION CLOSE events into WAL, for each those events the > region SeqId is bumped. > The issue comes from region close operation. Because when opening a region, > we use flushed SeqId from store files while after store flush during region > close we still write COMMIT_FLUSH, REGION_CLOSE events etc which respectively > bump up SeqId. Therefore, the region opening SeqId is lower than it should > be. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234652#comment-14234652 ] stack commented on HBASE-12635: --- FYI, hadoopqa is down currently > Delete acl notify znode of table after the table is deleted > --- > > Key: HBASE-12635 > URL: https://issues.apache.org/jira/browse/HBASE-12635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.11, 0.98.6 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12635-v1.diff > > > In our multi-tenant hbase cluster, we found that there are over 1M znodes > under the acl node. The reason is that users create and delete tables with > different names frequently. The acl notify znode are left there after the > tables are deleted. > Simple solution is that deleting acl notify znode of table in > AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12625) Deprecate certain methods in classes annotated with InterfaceAudience.Public in branch-1
[ https://issues.apache.org/jira/browse/HBASE-12625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234656#comment-14234656 ] Andrew Purtell commented on HBASE-12625: bq. return type short is not compatible with int Yep, we deprecated that in 0.98 because we knew it would be an issue. The original sin was selecting a {{short}} as return type. Apologies again for that. > Deprecate certain methods in classes annotated with InterfaceAudience.Public > in branch-1 > > > Key: HBASE-12625 > URL: https://issues.apache.org/jira/browse/HBASE-12625 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 1.0.0 > > Attachments: 12625-v1.txt > > > Compiling Phoenix against branch-1, I got compilation error below: > {code} > [ERROR] > /Users/tyu/phoenix/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java:[345,49] > cannot find symbol > [ERROR] symbol: method estimatedSizeOf(org.apache.hadoop.hbase.Cell) > [ERROR] location: class org.apache.hadoop.hbase.CellUtil > {code} > CellUtil is annotated with InterfaceAudience.Public > estimatedSizeOf() should be deprecated in branch-1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12635) Delete acl notify znode of table after the table is deleted
[ https://issues.apache.org/jira/browse/HBASE-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234662#comment-14234662 ] Andrew Purtell commented on HBASE-12635: I'll be testing locally. Or would you prefer this wait until Jenkins is back up [~stack]? Let me know. > Delete acl notify znode of table after the table is deleted > --- > > Key: HBASE-12635 > URL: https://issues.apache.org/jira/browse/HBASE-12635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.11, 0.98.6 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 1.0.0, 2.0.0, 0.98.9 > > Attachments: HBASE-12635-v1.diff > > > In our multi-tenant hbase cluster, we found that there are over 1M znodes > under the acl node. The reason is that users create and delete tables with > different names frequently. The acl notify znode are left there after the > tables are deleted. > Simple solution is that deleting acl notify znode of table in > AccessController when deleting the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234663#comment-14234663 ] stack commented on HBASE-12636: --- bq. Means in the event of failure we could double replicate some entries, right? Yes. Thats ok though, right? bq. What about recording this information into a system table instead? We can start the migration of replication state out of ZooKeeper. Yes. In a separate JIRA? Patch LGTM (The background code looks to be duplicated) > Avoid too many write operations on zookeeper in replication > --- > > Key: HBASE-12636 > URL: https://issues.apache.org/jira/browse/HBASE-12636 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.11 >Reporter: Liu Shaohui >Assignee: Liu Shaohui > Labels: replication > Fix For: 1.0.0 > > Attachments: HBASE-12636-v1.diff > > > In our production cluster, we found there are about over 1k write operations > per second on zookeeper from hbase replication. The reason is that the > replication source will write the log position to zookeeper for every edit > shipping. If the current replicating WAL is just the WAL that regionserver is > writing to, each skipping will be very small but the frequency is very high, > which causes many write operations on zookeeper. > A simple solution is that writing log position to zookeeper when position > diff or skipped edit number is larger than a threshold, not every edit > shipping. > Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12273) Generate .tabledesc file during upgrading if missing
[ https://issues.apache.org/jira/browse/HBASE-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234672#comment-14234672 ] stack commented on HBASE-12273: --- [~daviddengcn] pardon me. You want this in apache hbase? > Generate .tabledesc file during upgrading if missing > > > Key: HBASE-12273 > URL: https://issues.apache.org/jira/browse/HBASE-12273 > Project: HBase > Issue Type: Sub-task > Components: Admin >Affects Versions: 1.0.0, 0.98.7 >Reporter: Yi Deng >Assignee: Yi Deng > Labels: upgrade > Fix For: 1.0.0, 0.98.7 > > Attachments: > 1.0-0001-INTERNAL-Add-a-tool-for-fixing-missing-TableDescript.patch > > > Generate .tabledesc file during upgrading if missing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234677#comment-14234677 ] Andrew Purtell commented on HBASE-12636: bq. Yes. In a separate JIRA? I'd say not ideally. Nobody's going to work on it. We keep putting it off. Now we're putting band aids on the problem. > Avoid too many write operations on zookeeper in replication > --- > > Key: HBASE-12636 > URL: https://issues.apache.org/jira/browse/HBASE-12636 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.11 >Reporter: Liu Shaohui >Assignee: Liu Shaohui > Labels: replication > Fix For: 1.0.0 > > Attachments: HBASE-12636-v1.diff > > > In our production cluster, we found there are about over 1k write operations > per second on zookeeper from hbase replication. The reason is that the > replication source will write the log position to zookeeper for every edit > shipping. If the current replicating WAL is just the WAL that regionserver is > writing to, each skipping will be very small but the frequency is very high, > which causes many write operations on zookeeper. > A simple solution is that writing log position to zookeeper when position > diff or skipped edit number is larger than a threshold, not every edit > shipping. > Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12625) Deprecate certain methods in classes annotated with InterfaceAudience.Public in branch-1
[ https://issues.apache.org/jira/browse/HBASE-12625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234680#comment-14234680 ] Ted Yu commented on HBASE-12625: w.r.t. RpcSchedulerFactory, I tried to add back the old method which accepts RegionServerServices parameter. It wasn't straight forward - AnnotationReadingPriorityFunction expects parameter of type RSRpcServices. > Deprecate certain methods in classes annotated with InterfaceAudience.Public > in branch-1 > > > Key: HBASE-12625 > URL: https://issues.apache.org/jira/browse/HBASE-12625 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 1.0.0 > > Attachments: 12625-v1.txt > > > Compiling Phoenix against branch-1, I got compilation error below: > {code} > [ERROR] > /Users/tyu/phoenix/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java:[345,49] > cannot find symbol > [ERROR] symbol: method estimatedSizeOf(org.apache.hadoop.hbase.Cell) > [ERROR] location: class org.apache.hadoop.hbase.CellUtil > {code} > CellUtil is annotated with InterfaceAudience.Public > estimatedSizeOf() should be deprecated in branch-1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12625) Deprecate certain methods in classes annotated with InterfaceAudience.Public in branch-1
[ https://issues.apache.org/jira/browse/HBASE-12625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234688#comment-14234688 ] Andrew Purtell commented on HBASE-12625: {quote} w.r.t. RpcSchedulerFactory, I tried to add back the old method which accepts RegionServerServices parameter. It wasn't straight forward - AnnotationReadingPriorityFunction expects parameter of type RSRpcServices. {quote} Ok, but it's incumbent on us to fix a LimitedPrivate interface we broke without deprecation or warning. /cc [~mbertozzi] > Deprecate certain methods in classes annotated with InterfaceAudience.Public > in branch-1 > > > Key: HBASE-12625 > URL: https://issues.apache.org/jira/browse/HBASE-12625 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 1.0.0 > > Attachments: 12625-v1.txt > > > Compiling Phoenix against branch-1, I got compilation error below: > {code} > [ERROR] > /Users/tyu/phoenix/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java:[345,49] > cannot find symbol > [ERROR] symbol: method estimatedSizeOf(org.apache.hadoop.hbase.Cell) > [ERROR] location: class org.apache.hadoop.hbase.CellUtil > {code} > CellUtil is annotated with InterfaceAudience.Public > estimatedSizeOf() should be deprecated in branch-1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)