[jira] [Commented] (HBASE-19093) Check Admin/Table to ensure all operations go via AccessControl
[ https://issues.apache.org/jira/browse/HBASE-19093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263741#comment-16263741 ] Anoop Sam John commented on HBASE-19093: bq.If we add a new method to MasterRpcServices, but don't add pre/post methods to MasterObserver. So it will still miss the ACL check? Good point. Wanted to come to this jira and check attached patch but missed in btw some thing else. I have a doubt on the general approach. The issue is when we add new client functions (say adding Quota things), there is chances that we miss the ACL checks. It is not normally seen like hook are added around the ops but missed impl in AC. Infact most of the time the AC is the prompting factor for adding hooks. We cleaned up some hooks recently which were exposing too many internal stuff to CPs (Around procedure, locks) . All those hooks were designed so as to do some AC checks. So the problem is mostly the other way around compared to what the patch is trying to do. Not sure how we can add a test for that. > Check Admin/Table to ensure all operations go via AccessControl > --- > > Key: HBASE-19093 > URL: https://issues.apache.org/jira/browse/HBASE-19093 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Balazs Meszaros >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19093.master.001.patch, > HBASE-19093.master.002.patch, RegionObserver.txt > > > A cursory review of Admin Interface has a bunch of methods as open, with out > AccessControl checks. For example, procedure executor has not check on it. > This issue is about given the Admin and Table Interfaces a once-over to see > what is missing and to fill in access control where missing. > This is a follow-on from work over in HBASE-19048 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263728#comment-16263728 ] Anoop Sam John commented on HBASE-16890: So as per ur tests the higher percentile latency values are better with async wal. But the avg latency and the throughput is on bit lower side. The test Ram did was also around throughput only. Initial time it was like async wal throughput on lower side. Later his tests did not say so. I dont know the details of these later tests. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: AsyncWAL_disruptor.patch, AsyncWAL_disruptor_1 > (2).patch, AsyncWAL_disruptor_3.patch, AsyncWAL_disruptor_3.patch, > AsyncWAL_disruptor_4.patch, AsyncWAL_disruptor_6.patch, > HBASE-16890-rc-v2.patch, HBASE-16890-rc-v3.patch, > HBASE-16890-remove-contention-v1.patch, HBASE-16890-remove-contention.patch, > Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07 > PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, Screen Shot 2016-11-04 at > 5.21.27 PM.png, Screen Shot 2016-11-04 at 5.30.18 PM.png, async.svg, > classic.svg, contention.png, contention_defaultWAL.png, > ycsb_FSHlog.vs.Async.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263258#comment-16263258 ] Anoop Sam John commented on HBASE-19301: Will give a doc patch and get it in. bq. You suggesting that ACLs updates get special treatment? Yes. The grant code is getting executed in the RS where the ACL region sits. We could directly do writes to region. The ACL grant/revoke ops may not be that perf critical. So am ok even we continue the current way. And any way that uses the short circuit connection. But this short circuit connection was created by hbase super user (starting the server).. Means the runAsLoginUser is not really working even now also. Already the connection is assigned with super user so things work. Now if changed to using connection by new API, the user is the RPC user for whom the CP was running. So any of the short circuited connection will not work with run as thing as there is no way to change the user context (As we get the user from the RPC context. Here no RPC so no rpc context) > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch, HBASE-19301_V2.patch, > HBASE-19301_V2.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19318) MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase AccessController implementation
[
https://issues.apache.org/jira/browse/HBASE-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263064#comment-16263064
]
Anoop Sam John commented on HBASE-19318:
Ya I dont see we allow plugging in Authorization. AC implemented as CP and so
Ranger or any other could do customization. What if tomorrow we remove the CP
based authorization and make the impl core to the HBase? (We discussed abt this
many a times but as no one could dedicate time for it , never happened). The
custom implementation is using the AccessControlService which is not exposed.
We can any time change this PB interface no (as per our BC guidelines). Also I
believe the custom ACL impl is dealing with acl table directly. So this is like
dealing with a system table which is not exposed. Sorry I really dont know the
impl details of Ranger. I was just trying to point to a fact of using no
exposed things which would create issues for that project later. So you can
think and propose how/whether we should be exposing these now for making
customization of AC.
I think ur patch is to use findCoprocessor(Class cls) than
findCoprocessor(String className). That looks to be harmless change any way.
Sorry am not saying Ranger is unsupported by HBase or any. Just trying to
address the bigger picture. We want our users to use exposed things only.
They using a non exposed thing now may be because we really not exposing some
thing which we must or it is wrong usage from user end. If the former , we have
to correct ourselves first. That is the whole point am trying to convey.
> MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase
> AccessController implementation
> -
>
> Key: HBASE-19318
> URL: https://issues.apache.org/jira/browse/HBASE-19318
> Project: HBase
> Issue Type: Bug
> Components: master, security
>Reporter: Sharmadha Sainath
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
>
> Sharmadha brought a failure to my attention trying to use Ranger with HBase
> 2.0 where the {{grant}} command was erroring out unexpectedly. The cluster
> had the Ranger-specific coprocessors deployed, per what was previously
> working on the HBase 1.1 line.
> After some digging, I found that the the Master is actually making a check
> explicitly for a Coprocessor that has the name
> {{org.apache.hadoop.hbase.security.access.AccessController}} (short name or
> full name), instead of looking for a deployed coprocessor which can be
> assigned to {{AccessController}} (which is what Ranger does). We have the
> CoprocessorHost methods to do the latter already implemented; it strikes me
> that we just accidentally used the wrong method in MasterRpcServices.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19318) MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase AccessController implementation
[
https://issues.apache.org/jira/browse/HBASE-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263007#comment-16263007
]
Anoop Sam John commented on HBASE-19318:
I agree that some changes broke Ranger.
bq.I'm really confused by your suggestion that authz is not meant to be
pluggable inside of HBase.
Are we allowing plugging in? Sorry I don't think so. Pls correct me if u find
else in some other place. This is the whole point I wanted to bring about.
> MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase
> AccessController implementation
> -
>
> Key: HBASE-19318
> URL: https://issues.apache.org/jira/browse/HBASE-19318
> Project: HBase
> Issue Type: Bug
> Components: master, security
>Reporter: Sharmadha Sainath
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
>
> Sharmadha brought a failure to my attention trying to use Ranger with HBase
> 2.0 where the {{grant}} command was erroring out unexpectedly. The cluster
> had the Ranger-specific coprocessors deployed, per what was previously
> working on the HBase 1.1 line.
> After some digging, I found that the the Master is actually making a check
> explicitly for a Coprocessor that has the name
> {{org.apache.hadoop.hbase.security.access.AccessController}} (short name or
> full name), instead of looking for a deployed coprocessor which can be
> assigned to {{AccessController}} (which is what Ranger does). We have the
> CoprocessorHost methods to do the latter already implemented; it strikes me
> that we just accidentally used the wrong method in MasterRpcServices.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19318) MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase AccessController implementation
[
https://issues.apache.org/jira/browse/HBASE-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262968#comment-16262968
]
Anoop Sam John commented on HBASE-19318:
Is that really advisable? ACL is a core part. Just because it is implemented
as CP, this is possible now. So the own implementation working on the system
table ACL. Am still not sure is this really correct. Neither this ACL table
nor the ACL proto things are exposed to users.
> MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase
> AccessController implementation
> -
>
> Key: HBASE-19318
> URL: https://issues.apache.org/jira/browse/HBASE-19318
> Project: HBase
> Issue Type: Bug
> Components: master, security
>Reporter: Sharmadha Sainath
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
>
> Sharmadha brought a failure to my attention trying to use Ranger with HBase
> 2.0 where the {{grant}} command was erroring out unexpectedly. The cluster
> had the Ranger-specific coprocessors deployed, per what was previously
> working on the HBase 1.1 line.
> After some digging, I found that the the Master is actually making a check
> explicitly for a Coprocessor that has the name
> {{org.apache.hadoop.hbase.security.access.AccessController}} (short name or
> full name), instead of looking for a deployed coprocessor which can be
> assigned to {{AccessController}} (which is what Ranger does). We have the
> CoprocessorHost methods to do the latter already implemented; it strikes me
> that we just accidentally used the wrong method in MasterRpcServices.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262952#comment-16262952 ] Anoop Sam John commented on HBASE-19301: Good points. The created connection having the same user (the one context which created it) is what we really need I believe. This is ok. Ya we should doc it clearly may be. The case is that as the RPC is short circuited, the run as any other user thing wont work out. No need to provide extra API for creation of non short circuited connection. CP users can use ConnectionFactory methods any way. So as part of closing this issue, some extra doc is what we need. Ok Stack? So we can not really short circuit the put to ACL region (in grant, revoke etc). If we have concerns over those writes happening via RPC, we should directly put to region. Any way the grant is an EP call and that works on a region where the row has to go in. We can get the region from CPEnv now. Stack WDYT? > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch, HBASE-19301_V2.patch, > HBASE-19301_V2.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19320) document the mysterious direct memory leak in hbase
[ https://issues.apache.org/jira/browse/HBASE-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262909#comment-16262909 ] Anoop Sam John commented on HBASE-19320: Ya 2.0 only work with DBBs. 1.x by default work with on heap BBs and which will cause NIO to allocate and keep DBBs. Ya even the replication sink time also it will use this DBBs. All reqs will go through same RPCServer route. > document the mysterious direct memory leak in hbase > > > Key: HBASE-19320 > URL: https://issues.apache.org/jira/browse/HBASE-19320 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.2.6 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: Screen Shot 2017-11-21 at 4.43.36 PM.png, Screen Shot > 2017-11-21 at 4.44.22 PM.png > > > Recently we run into a direct memory leak case, which takes some time to > trace and debug. Internally discussed with our [~saint@gmail.com], we > thought we had some findings and want to share with the community. > Basically, it is the issue described in > http://www.evanjones.ca/java-bytebuffer-leak.html and it happened to one of > our hbase clusters. > Create the jira first and will fill in more details later. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19331) Region start-key/end-key corruption in Hbase meta table
[ https://issues.apache.org/jira/browse/HBASE-19331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262903#comment-16262903 ] Anoop Sam John commented on HBASE-19331: Paste the o/p of ur test. How the keys look like (for daughter regions) when they are corrupted? U will work on a fix? > Region start-key/end-key corruption in Hbase meta table > --- > > Key: HBASE-19331 > URL: https://issues.apache.org/jira/browse/HBASE-19331 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 0.98.8 > Environment: Reproduced on HBase 0.98.8 on hadoop-2 >Reporter: Shamith kumar > > when a region split happens on a key with trailing byte equals zero, the end > key of the first resulting region and and start key of the second resulting > region in meta table gets corrupted. > Here is the link to code to reproduce this issue > https://bitbucket.org/flytxt/hbase-meta-corruption-test > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19318) MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase AccessController implementation
[
https://issues.apache.org/jira/browse/HBASE-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262895#comment-16262895
]
Anoop Sam John edited comment on HBASE-19318 at 11/22/17 4:40 PM:
--
Ranger is extending our AccessController? Is that allowed? !! I dont see AC
is exposed to CPs. It is LP but just for Conig. Means we expose the name as
such for users to config in the xml.
was (Author: anoop.hbase):
Ranger is extending our AccessController? Is that allowed. I dont see AC is
exposed to CPs. It is LP but just for Conig. Means we expose the name as such
for users to config in the xml.
> MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase
> AccessController implementation
> -
>
> Key: HBASE-19318
> URL: https://issues.apache.org/jira/browse/HBASE-19318
> Project: HBase
> Issue Type: Bug
> Components: master, security
>Reporter: Sharmadha Sainath
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
>
> Sharmadha brought a failure to my attention trying to use Ranger with HBase
> 2.0 where the {{grant}} command was erroring out unexpectedly. The cluster
> had the Ranger-specific coprocessors deployed, per what was previously
> working on the HBase 1.1 line.
> After some digging, I found that the the Master is actually making a check
> explicitly for a Coprocessor that has the name
> {{org.apache.hadoop.hbase.security.access.AccessController}} (short name or
> full name), instead of looking for a deployed coprocessor which can be
> assigned to {{AccessController}} (which is what Ranger does). We have the
> CoprocessorHost methods to do the latter already implemented; it strikes me
> that we just accidentally used the wrong method in MasterRpcServices.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19318) MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase AccessController implementation
[
https://issues.apache.org/jira/browse/HBASE-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262895#comment-16262895
]
Anoop Sam John commented on HBASE-19318:
Ranger is extending our AccessController? Is that allowed. I dont see AC is
exposed to CPs. It is LP but just for Conig. Means we expose the name as such
for users to config in the xml.
> MasterRpcServices#getSecurityCapabilities explicitly checks for the HBase
> AccessController implementation
> -
>
> Key: HBASE-19318
> URL: https://issues.apache.org/jira/browse/HBASE-19318
> Project: HBase
> Issue Type: Bug
> Components: master, security
>Reporter: Sharmadha Sainath
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-beta-1
>
>
> Sharmadha brought a failure to my attention trying to use Ranger with HBase
> 2.0 where the {{grant}} command was erroring out unexpectedly. The cluster
> had the Ranger-specific coprocessors deployed, per what was previously
> working on the HBase 1.1 line.
> After some digging, I found that the the Master is actually making a check
> explicitly for a Coprocessor that has the name
> {{org.apache.hadoop.hbase.security.access.AccessController}} (short name or
> full name), instead of looking for a deployed coprocessor which can be
> assigned to {{AccessController}} (which is what Ranger does). We have the
> CoprocessorHost methods to do the latter already implemented; it strikes me
> that we just accidentally used the wrong method in MasterRpcServices.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19320) document the mysterious direct memory leak in hbase
[ https://issues.apache.org/jira/browse/HBASE-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262006#comment-16262006 ] Anoop Sam John commented on HBASE-19320: In 2.0 we have direct BB pool to which we read reqs into. These DBBs are passed to NIO for reading reqs into. The NIO layer DBBs are not getting used then. JFYI. > document the mysterious direct memory leak in hbase > > > Key: HBASE-19320 > URL: https://issues.apache.org/jira/browse/HBASE-19320 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.2.6 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: Screen Shot 2017-11-21 at 4.43.36 PM.png, Screen Shot > 2017-11-21 at 4.44.22 PM.png > > > Recently we run into a direct memory leak case, which takes some time to > trace and debug. Internally discussed with our [~saint@gmail.com], we > thought we had some findings and want to share with the community. > Basically, it is the issue described in > http://www.evanjones.ca/java-bytebuffer-leak.html and it happened to one of > our hbase clusters. > Create the jira first and will fill in more details later. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261106#comment-16261106 ] Anoop Sam John commented on HBASE-19301: Ya.. That will be very difficult to maintain.. We can check how it will look like later. Ya may be some APIs like which passes Mutations and read APIs. > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch, HBASE-19301_V2.patch, > HBASE-19301_V2.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19301: --- Resolution: Fixed Hadoop Flags: Reviewed Release Note: Provided a way for the CP users to create a short circuitable connection with custom configs. createConnection(Configuration) is added to MasterCoprocessorEnvironment, RegionServerCoprocessorEnvironment and RegionCoprocessorEnvironment. The getConnection() version available in these Env interfaces return a single cluster connection created at the server (which server also uses) where as this new method will create new connection. The difference from connection created at client side (using ConnectionFactory APIs) is that this connection can short circuit the calls to same server avoiding the RPC paths. The connection will NOT be cached/maintained by server. That should be done the CPs. Status: Resolved (was: Patch Available) > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch, HBASE-19301_V2.patch, > HBASE-19301_V2.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-15816) Provide client with ability to set priority on Operations
[ https://issues.apache.org/jira/browse/HBASE-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15816: --- Release Note: Added setPriority(int priority) API to Put, Delete, Increment, Append, Get and Scan pojos. So for all these ops, the user can provide a custom priority level. > Provide client with ability to set priority on Operations > -- > > Key: HBASE-15816 > URL: https://issues.apache.org/jira/browse/HBASE-15816 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 3.0.0, 1.4.0 > > Attachments: HBASE-15816-v1.patch, HBASE-15816.patch, > HBASE-15816.v1.branch-1.patch, HBASE-15816.v2.patch > > > First round will just be to expose the ability to set priorities for client > operations. For more background: > http://mail-archives.apache.org/mod_mbox/hbase-dev/201604.mbox/%3CCA+RK=_BG_o=q8HMptcP2WauAinmEsL+15f3YEJuz=qbpcya...@mail.gmail.com%3E > Next step would be to remove AnnotationReadingPriorityFunction and have the > client send priorities explicitly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15816) Provide client with ability to set priority on Operations
[ https://issues.apache.org/jira/browse/HBASE-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260522#comment-16260522 ] Anoop Sam John commented on HBASE-15816: I came across this jira and was wondering why no release notes. Then I see Stack also asked this bq.churro morales Hey boss, a nice change like this needs a release note... say how to enable/use – high-level. Thanks. Am not sure whether it will be complete, but I will add the release notes abt the addition of the new API to Get/Put etc to set priority. > Provide client with ability to set priority on Operations > -- > > Key: HBASE-15816 > URL: https://issues.apache.org/jira/browse/HBASE-15816 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 3.0.0, 1.4.0 > > Attachments: HBASE-15816-v1.patch, HBASE-15816.patch, > HBASE-15816.v1.branch-1.patch, HBASE-15816.v2.patch > > > First round will just be to expose the ability to set priorities for client > operations. For more background: > http://mail-archives.apache.org/mod_mbox/hbase-dev/201604.mbox/%3CCA+RK=_BG_o=q8HMptcP2WauAinmEsL+15f3YEJuz=qbpcya...@mail.gmail.com%3E > Next step would be to remove AnnotationReadingPriorityFunction and have the > client send priorities explicitly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19309) Lower HConstants#MAX_ROW_LENGTH as guardrail in order to avoid HBASE-18987
[ https://issues.apache.org/jira/browse/HBASE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260514#comment-16260514 ] Anoop Sam John commented on HBASE-19309: So for this, do we need to reduce the max rk size globally? When the split happens and the split key is been generated, can we do truncation to make sure the length is < this new value (which is lesser than Short.MAX_VALUE).Also lesser till what value? It all depends on what are all added to it to create the Put to META right? Eg: u say table name. So what if there a crazy very long table name? I feel like this has to be handled there at split code area only. Get the split key which is less that max RK length. And then check by adding the extra things like tabName, where we reach wrt put's rk length. Accordingly truncate the split key. (I did not see any code around this area and not sure how easy or diff it will be). > Lower HConstants#MAX_ROW_LENGTH as guardrail in order to avoid HBASE-18987 > -- > > Key: HBASE-19309 > URL: https://issues.apache.org/jira/browse/HBASE-19309 > Project: HBase > Issue Type: Bug > Components: HFile, regionserver >Reporter: Esteban Gutierrez > > As discussed in HBASE-18987. A problem of having a row about the maximum size > of a row (Short.MAX_VALUE) is when a split happens, there is a possibility > that the midkey could be that row and the Put created to add the new entry in > META will exceed the maximum row size since the new row key will include the > table name and that will cause the split to abort. Since is not possible to > raise that row key size in HFileV3, a reasonable solution is to reduce the > maximum size of row key in order to avoid exceeding Short.MAX_VALUE. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19301: --- Attachment: HBASE-19301_V2.patch > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch, HBASE-19301_V2.patch, > HBASE-19301_V2.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260479#comment-16260479 ] Anoop Sam John commented on HBASE-18294: Sorry I missed to add one more thing which I wanted to. Saw some where the ask on how to handle the meta size used by CCM for meta data when the MSLAB is off heap. That is any way off heap only and not data. I would say we can just consider this as data part only. To effectively reach our data, we need this. It would be just ok. What am saying is just for CCM, we no need to add more complexity. bq.Reaching global pressure too frequently means the RS is not "healthy". Do we agree on this?? Ya reaching global barrier is not that good. Because this will make writes to be blocked for flushes. But this situation depends on many factors. It may not mean RS is not healthy. The more #regions in this RS, this chance is more. The more client pressure then also more chances. Say we have x as the global heap upper barrier and there are n regions. I consider 128 MB as flush size and 4 as blocking multiplier. Then if x>=128 * 4 * n, we would never see the barrier breach. But this is not so advisable thing any way. Just saying. Even the HDFS slowness can make the flushes to take longer and so barrier breaches. So I dont think we can say barrier breaching is always too bad and RS is unhealthy then. BTW am not getting how we can avoid/reduce that situation using these changes. The arg u made abt the selection of region when there is a global size breach make sense fully. And I feel we should correct it. So tracking the heapSize at the Region level is needed. (Remember we track it at each Segment level and can always sum all segments heapSize to know this. We do this in some flow cases). When there is a flush region selection because of global heap barrier breach, we should select region(s) which releases max heap size. For per region flush decision (128MB) IMHO we can continue to check against data size. Or may be if the BC is a concern still, we can have check like if ( dataSize >= 128 || heapSize >= 128 ) flush() > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19301: --- Attachment: HBASE-19301_V2.patch Patch with javadoc comments fixed as per review and renamed new method to createConnection(Conf) > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch, HBASE-19301_V2.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260319#comment-16260319 ] Anoop Sam John commented on HBASE-19301: Ok I will change it. I hope Stack is not really against it. :-) Stop me if am wrong boss. > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260302#comment-16260302 ] Anoop Sam John commented on HBASE-19301: bq.I left a "TODO: Use Table.put(Put) instead." in AccessControlLists when work for HBASE-18500. This can be fixed if we use the new created connection to get ACL Table. We can do in another issue? I would like to stick with what the subject says. Ya getConnection() or createConnection() both names having their good parts. I would like to go with getConnection() too because of what Stack said. You ok [~zghaobac]? > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260290#comment-16260290 ] Anoop Sam John commented on HBASE-19301: Thanks for the detailed comment Stack. Ya agree to all.. We were never doing the short circuit with avoiding the PB marshal unmarshal stuff. I believe I mentioned this in the other sub task u did for providing the OnlineRegions return from RegionCPEnv. Calling APIs on Region will be much more cheaper as we deal with POJOs then not PBs. I will remove that statement from the javadoc around the CPEnv.getConnection(). Instead will add a TODO to check it later. I dont know how we can avoid the PB overhead. For the Admin/Client stub, we need PB based args and so this is not really avoidable? Will add the doc as u said. Let me fix the javadoc issues reported by QA. > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259538#comment-16259538 ] Anoop Sam John commented on HBASE-19301: So u were suggesting that the normal way of connection creation should make the short circuited version if called within RS context? I am not sure how. Because the ShortCircuitingClusterConnection would need refs to RSRpcServices which is there with HRS only. Not sure how the normal ConnectionFactory create will get those refs. We have to make the creation within HRS code and that is why we have to add in the CPEnv which is what exposed to user. > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18298) RegionServerServices Interface cleanup for CP expose
[ https://issues.apache.org/jira/browse/HBASE-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259524#comment-16259524 ] Anoop Sam John commented on HBASE-18298: As Stack said. There is a config to decide what should be done when a CP throws an Exception. By default that config says abort the server. You should be able to make use of it. Stack asked in that issue discussion whether we should introduce some special Exception type for CPs to suggest RS to abort. Let us know whether that is needed. This config of abort on Exception is to be set false for Phoenix usage? In such case RS wont get aborted. Only that particular CP will get unloaded. In such a case, as Stack suggested we need special Exception to force RS to abort. > RegionServerServices Interface cleanup for CP expose > > > Key: HBASE-18298 > URL: https://issues.apache.org/jira/browse/HBASE-18298 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18298.patch, HBASE-18298_V2.patch, > HBASE-18298_V3.patch, HBASE-18298_V4.patch, HBASE-18298_V5.patch, > HBASE-18298_V6.patch, HBASE-18298_V7.patch, HBASE-18298_V7.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259093#comment-16259093 ] Anoop Sam John commented on HBASE-19092: Yep. That is what we want ideally. It is not so simple as we have to deal with the security tags not getting returned back to clients. As still it is not exposed to client as of now, we can not add to Cell. That is why we thought of ExtendedCell as an option. And then all the discussion as above. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258883#comment-16258883 ] Anoop Sam John commented on HBASE-19301: bq.A Connection accessing local Server will short-circuit (To be verified). Not all. Only the connection created within RS will use this way. A connection created at client side (Using ConnectionFactory#createConnection) will NOT use the short circuit path. Ya the short circuit removes the RPC things from call. It will be a direct stub call. But we have to do the PB convert. Say when mutation happens, all mutations has to be converted to PB and back to Pojo. (Some thing to see later how/whether we can avoid this) bq.But perhaps you are referring to difference between an invocation that goes via protobuf service but skips the rpc invocation vs a direct invocation, one that skips doing protobuf marshalling/unmarshaling AND rpc? This is what the short circuit connection doing right? Are there 2 short circuit stuff? I dont know. > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258858#comment-16258858 ] Anoop Sam John commented on HBASE-19092: bq. we want Tags integral to Cell, not apart as they have been, so getting Tags from Cell/ExtendedCell is where we want to be Exactly. This is what I too considered. We want Tags to be part of Cell. When we do it for say 3.0, we need getTags in Cell. Not in ExtendedCell. We have to expose tags to client side also. So if we add to ExtendedCell today, it is bound to change again tomorrow right? To be to Cell. That is what my consideration was. Then having in a static util API way is not that bad as type casting calls on ExtendedCell. In 3.0, we would deprecate that static against Cell#getTags(). Any way take a call... Am ok for any thing decided here. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258832#comment-16258832 ] Anoop Sam John commented on HBASE-19301: CPs can create own connection with custom configs as how users do that at client side. The only disadvantage will be that it will NOT be a short circuit enabled connection. Phoenix use cases introduced this short circuit based connections and so they use that in their CPs. That is why thought of a CP based getConnection. > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19301: --- Attachment: HBASE-19301.patch > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19301: --- Status: Patch Available (was: Open) > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
Anoop Sam John created HBASE-19301: -- Summary: Provide way for CPs to create short circuited connection with custom configurations Key: HBASE-19301 URL: https://issues.apache.org/jira/browse/HBASE-19301 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0-beta-1 Over in HBASE-18359 we have discussions for this. Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But this returns a pre created connection (per server). This uses the configs at hbase-site.xml at that server. Phoenix needs creating connection in CP with some custom configs. Having this custom changes in hbase-site.xml is harmful as that will affect all connections been created at that server. This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19295) The Configuration returned by CPEnv should be read-only.
[ https://issues.apache.org/jira/browse/HBASE-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258429#comment-16258429 ] Anoop Sam John commented on HBASE-19295: Moved as a sub task under HBASE-18169 (cp cleanup issue) > The Configuration returned by CPEnv should be read-only. > > > Key: HBASE-19295 > URL: https://issues.apache.org/jira/browse/HBASE-19295 > Project: HBase > Issue Type: Sub-task >Reporter: stack > > The Configuration a CP gets when it does a getConfiguration on the > environment is that of the RegionServer. The CP should not be able to modify > this config. We should throw exception if they try to write us. > Ditto w/ the Connection they can get from the env. They should not be able to > close it at min. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19295) The Configuration returned by CPEnv should be read-only.
[ https://issues.apache.org/jira/browse/HBASE-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19295: --- Issue Type: Sub-task (was: Task) Parent: HBASE-18169 > The Configuration returned by CPEnv should be read-only. > > > Key: HBASE-19295 > URL: https://issues.apache.org/jira/browse/HBASE-19295 > Project: HBase > Issue Type: Sub-task >Reporter: stack > > The Configuration a CP gets when it does a getConfiguration on the > environment is that of the RegionServer. The CP should not be able to modify > this config. We should throw exception if they try to write us. > Ditto w/ the Connection they can get from the env. They should not be able to > close it at min. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18359) CoprocessorHConnection#getConnectionForEnvironment should read config from CoprocessorEnvironment
[
https://issues.apache.org/jira/browse/HBASE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258425#comment-16258425
]
Anoop Sam John commented on HBASE-18359:
Yes Stack as Samarth Jain said.
The connection we return from CPEnv.getConnection() is a single connection
created for this RS. This will have short circuit provision. But it will use
all config options as in hbase-site.xml at RS side. Phoenix need
customizations on the configs and connection they use within CPs to have these
custom configs. The new API also should have short circuit ways. But create a
new connection using the config been passed. Will raise an issue and provide a
patch. This issue as such been created for 2.0 . Can we close this as 'Wont
fix'?
> CoprocessorHConnection#getConnectionForEnvironment should read config from
> CoprocessorEnvironment
> -
>
> Key: HBASE-18359
> URL: https://issues.apache.org/jira/browse/HBASE-18359
> Project: HBase
> Issue Type: Bug
>Reporter: Samarth Jain
> Fix For: 2.0.0
>
>
> It seems like the method getConnectionForEnvironment isn't doing the right
> thing when it is creating a CoprocessorHConnection by reading the config from
> HRegionServer and not from the env passed in.
> If coprocessors want to use a CoprocessorHConnection with some custom config
> settings, then they have no option but to configure it in the hbase-site.xml
> of the region servers. This isn't ideal as a lot of times these "global"
> level configs can have side effects. See PHOENIX-3974 as an example where
> configuring ServerRpcControllerFactory (a Phoenix implementation of
> RpcControllerFactory) could result in deadlocks. Or PHOENIX-3983 where
> presence of this global config causes our index rebuild code to incorrectly
> use handlers it shouldn't.
> If the CoprocessorHConnection created through getConnectionForEnvironment API
> used the CoprocessorEnvironment config, then it would allow co-processors to
> pass in their own config without needing to configure them in hbase-site.xml.
> The change would be simple. Basically change the below
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection((HRegionServer) services);
> }
> {code}
> to
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection(env.getConfiguration(),
> (HRegionServer) services);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17204) Make Bucket Cache off heap cache default ON
[ https://issues.apache.org/jira/browse/HBASE-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17204: --- Summary: Make Bucket Cache off heap cache default ON (was: Make L2 off heap cache default ON) > Make Bucket Cache off heap cache default ON > --- > > Key: HBASE-17204 > URL: https://issues.apache.org/jira/browse/HBASE-17204 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Blocker > Fix For: 2.0.0-beta-1 > > > L2 cache can be used for data blocks. By default it is off now. After > HBASE-11425 work, L2 off heap cache can equally perform with L1 on heap > cache. On heavy loaded workload, this can even out perform L1 cache. Pls see > recently published report by Alibaba. Also this work was backported by > Rocketfuel and similar perf improvement report from them too. > Let us turn L2 off heap cache ON. As it is off heap, we can have much larger > sized L2 BC. What should be the default size? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17204) Make Bucket Cache off heap cache default ON
[ https://issues.apache.org/jira/browse/HBASE-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258420#comment-16258420 ] Anoop Sam John commented on HBASE-17204: We fixed critical issues around BucketCache including the fragmentation one. Still the open one is we maintain at least one free bucket for every size. This gives some wastage of space when the bucket sizes are not properly tuned. That sub jira is still open ! May be we should just allow all the buckets of a size to be converted to other size as needed? No reserve of one block at least. U asked this Q some where. On the route to make off heap BC def ON , we should 1. Remove the L1, L2 concept fully. By default yes this is the way it works now also. When BC is ON, by default we will keep data blocks in BC only and index blocks in LRU cache. We have a config to make the BC a strict L2 cache (By L2 as a victim handler for L1) 2. Dont keep BC as a victim handler for LRU cache at all. Right now this is done when combined mode is true/false. Lets keep it simple. Index blocks will go to LRU and data blocks to BC. When LRU is not enough, we will see evictions there. 3. Reduce the heap % for LRU. Right now it degfaults to 40%. But when it is going to keep index blocks alone, lets make the default to be 10%? 4. Off heap mode BC is ON by def. We need a def off heap size. Remember the off heap size is not given as a % of some thing. It is exact value. May be we should give a def value of 8 GB? (or much lesser or greater?) > Make Bucket Cache off heap cache default ON > --- > > Key: HBASE-17204 > URL: https://issues.apache.org/jira/browse/HBASE-17204 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Blocker > Fix For: 2.0.0-beta-1 > > > L2 cache can be used for data blocks. By default it is off now. After > HBASE-11425 work, L2 off heap cache can equally perform with L1 on heap > cache. On heavy loaded workload, this can even out perform L1 cache. Pls see > recently published report by Alibaba. Also this work was backported by > Rocketfuel and similar perf improvement report from them too. > Let us turn L2 off heap cache ON. As it is off heap, we can have much larger > sized L2 BC. What should be the default size? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19280) Move HFileWriterImpl.compressionByName(String name) to some utility class
[
https://issues.apache.org/jira/browse/HBASE-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16257328#comment-16257328
]
Anoop Sam John commented on HBASE-19280:
How using the returned Compression.Algorithm? I can see Compression class is
marked Private and inside enum Algorithm is marked Public.
getCompressionAlgorithmByName(String compressName) is there in Compression
class also. We should decide what all the be exposed clearly from Compression
and Algorithm.
Oh ya we have setter to set compression types in HCD etc. For that may be this
Algorithm enum is exposed now. Only the type names were needed to be exposed
ideally.
> Move HFileWriterImpl.compressionByName(String name) to some utility class
> -
>
> Key: HBASE-19280
> URL: https://issues.apache.org/jira/browse/HBASE-19280
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Priority: Trivial
> Fix For: 2.0.0-beta-2
>
>
> This method can be moved to some utility (related jira PHOENIX-4368).
> {code}
> public static Compression.Algorithm compressionByName(String algoName) {
> if (algoName == null)
> return HFile.DEFAULT_COMPRESSION_ALGORITHM;
> return Compression.getCompressionAlgorithmByName(algoName);
> }
> {code}
> FYI, [~elserj]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16257298#comment-16257298 ] Anoop Sam John edited comment on HBASE-19092 at 11/17/17 5:51 PM: -- Ya for creation of Cells, they will get an ExtCellBuilder type (One which is having way to add Tags). getTags() also the best place is ExtendedCell/ RawCell.. But we can not chnage the server to work with ExtCell now. :-( Ya we can have Tags and add the static APIs like getTags(Cell), getTag(Cell, type) there? BTW we having these getTags() API in CellUtil, so we cannot really remove them now. We will deprecate for 2.0 right? was (Author: anoop.hbase): Ya for creation of Cells, they will get an ExtCellBuilder type (One which is having way to add Tags). getTags() also the best place is ExtendedCell/ RawCell.. But we can not chnage the server to work with ExtCell now. :-( Ya we can have Tags and add the static APIs like getTags(Cell), getTag(Cell, type) there? > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16257298#comment-16257298 ] Anoop Sam John commented on HBASE-19092: Ya for creation of Cells, they will get an ExtCellBuilder type (One which is having way to add Tags). getTags() also the best place is ExtendedCell/ RawCell.. But we can not chnage the server to work with ExtCell now. :-( Ya we can have Tags and add the static APIs like getTags(Cell), getTag(Cell, type) there? > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16257283#comment-16257283 ] Anoop Sam John commented on HBASE-19092: All the CPs get the type as Cell not ExtendedCell. Now if we add APIs in ExtendedCell alone, CP users has to know that even if type is Cell all objects are of type ExtendedCell and then call with type casting. This will look as a bad design IMHO. Till now they could get tags using CellUtil API. This will be far better (I mean using any static method).. I agree add to ExtendedCell/RawCell is the best way. But then we have to make sure it is ExtendedCell/ RawCell type flowing through this server side CPs. We cannot do that at least as of now. That is why thinking ExtendedCell is not the right place. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18359) CoprocessorHConnection#getConnectionForEnvironment should read config from CoprocessorEnvironment
[
https://issues.apache.org/jira/browse/HBASE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256726#comment-16256726
]
Anoop Sam John commented on HBASE-18359:
This issue (As stated in subject) is no longer valid for latest HBase. On
older versions also it is not as the
CoprocessorHConnection#getConnectionForEnvironment was taking a
CoprocessorEnvironment instance not config. So to change the configs, u will
end up changing the actual region level config. Ya to solve the issue (said in
this jira) having a workaround of CoprocessorHConnectionTableFactory. Ya what
that doing is correct. From 2.0 u can not have this
CoprocessorHConnectionTableFactory. What we need in HBase is an API
CoprocessorEnvironment#getConnection(Config). This call will always make new
connection (which is short circuit enabled). So caching of this and reuse the
callee has to take care. Is that ok? Then we can provide a quick patch and
make it avail in beta-1 release. I will close down this jira and open a new
one with correct title and desc.
> CoprocessorHConnection#getConnectionForEnvironment should read config from
> CoprocessorEnvironment
> -
>
> Key: HBASE-18359
> URL: https://issues.apache.org/jira/browse/HBASE-18359
> Project: HBase
> Issue Type: Bug
>Reporter: Samarth Jain
> Fix For: 2.0.0
>
>
> It seems like the method getConnectionForEnvironment isn't doing the right
> thing when it is creating a CoprocessorHConnection by reading the config from
> HRegionServer and not from the env passed in.
> If coprocessors want to use a CoprocessorHConnection with some custom config
> settings, then they have no option but to configure it in the hbase-site.xml
> of the region servers. This isn't ideal as a lot of times these "global"
> level configs can have side effects. See PHOENIX-3974 as an example where
> configuring ServerRpcControllerFactory (a Phoenix implementation of
> RpcControllerFactory) could result in deadlocks. Or PHOENIX-3983 where
> presence of this global config causes our index rebuild code to incorrectly
> use handlers it shouldn't.
> If the CoprocessorHConnection created through getConnectionForEnvironment API
> used the CoprocessorEnvironment config, then it would allow co-processors to
> pass in their own config without needing to configure them in hbase-site.xml.
> The change would be simple. Basically change the below
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection((HRegionServer) services);
> }
> {code}
> to
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection(env.getConfiguration(),
> (HRegionServer) services);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18359) CoprocessorHConnection#getConnectionForEnvironment should read config from CoprocessorEnvironment
[
https://issues.apache.org/jira/browse/HBASE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256688#comment-16256688
]
Anoop Sam John commented on HBASE-18359:
Ya that make sense James. So what u say is u can not make this config changes
globally in xml in the RS side . Correct? With current 2.0 you can not do
this. Ya u can create connection how u make at client side. But will not get
the short circuit benefit then. So if u need this, we will have to add new way
then. Let us know.
> CoprocessorHConnection#getConnectionForEnvironment should read config from
> CoprocessorEnvironment
> -
>
> Key: HBASE-18359
> URL: https://issues.apache.org/jira/browse/HBASE-18359
> Project: HBase
> Issue Type: Bug
>Reporter: Samarth Jain
> Fix For: 2.0.0
>
>
> It seems like the method getConnectionForEnvironment isn't doing the right
> thing when it is creating a CoprocessorHConnection by reading the config from
> HRegionServer and not from the env passed in.
> If coprocessors want to use a CoprocessorHConnection with some custom config
> settings, then they have no option but to configure it in the hbase-site.xml
> of the region servers. This isn't ideal as a lot of times these "global"
> level configs can have side effects. See PHOENIX-3974 as an example where
> configuring ServerRpcControllerFactory (a Phoenix implementation of
> RpcControllerFactory) could result in deadlocks. Or PHOENIX-3983 where
> presence of this global config causes our index rebuild code to incorrectly
> use handlers it shouldn't.
> If the CoprocessorHConnection created through getConnectionForEnvironment API
> used the CoprocessorEnvironment config, then it would allow co-processors to
> pass in their own config without needing to configure them in hbase-site.xml.
> The change would be simple. Basically change the below
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection((HRegionServer) services);
> }
> {code}
> to
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection(env.getConfiguration(),
> (HRegionServer) services);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19112) Suspect methods on Cell to be deprecated
[
https://issues.apache.org/jira/browse/HBASE-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256603#comment-16256603
]
Anoop Sam John commented on HBASE-19112:
So the RawCell is CP exposed. We have some APIs like getTags() there which CP
users might want to use. The issue is it is Cell type that flaws throughout
the server code base. Even to CPs. So CP users has to know that it is RawCell
types actually coming in in all places and have to type cast and call these
APIs. This is ugly.
Otherwise we have to change the flows in server side so as to pass the RawCell
types instead. This also we can not do fully. Because some hooks like the
postGet passes a Result to CP. The cells what we get from Result is Cell type
and we can not change the return type there as Result is public exposed. Like
that there may be conflicts in areas like Filter etc. After all this will be
huge change.. The idea I agree mostly. But not at all sure we can do for 2.0.
In the other issue of expose tags to CPs, me and Ram were discussing this same
topic.
> Suspect methods on Cell to be deprecated
>
>
> Key: HBASE-19112
> URL: https://issues.apache.org/jira/browse/HBASE-19112
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Josh Elser
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> [~chia7712] suggested on the [mailing
> list|https://lists.apache.org/thread.html/e6de9af26d9b888a358ba48bf74655ccd893573087c032c0fcf01585@%3Cdev.hbase.apache.org%3E]
> that we have some methods on Cell which should be deprecated for removal:
> * {{#getType()}}
> * {{#getTimestamp()}}
> * {{#getTag()}}
> * {{#getSequenceId()}}
> Let's make a pass over these (and maybe the rest) to make sure that there
> aren't others which are either implementation details or methods returning
> now-private-marked classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18359) CoprocessorHConnection#getConnectionForEnvironment should read config from CoprocessorEnvironment
[
https://issues.apache.org/jira/browse/HBASE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255686#comment-16255686
]
Anoop Sam John commented on HBASE-18359:
Phoenix want a connection which allows the short circuiting when possible.
There is no way to create new fresh connection (Using custom configs) from
CPs. (Am missing some way?) This is what we need to give.
CoprocessorHConnection , they were using before. Now that is gone.
> CoprocessorHConnection#getConnectionForEnvironment should read config from
> CoprocessorEnvironment
> -
>
> Key: HBASE-18359
> URL: https://issues.apache.org/jira/browse/HBASE-18359
> Project: HBase
> Issue Type: Bug
>Reporter: Samarth Jain
> Fix For: 2.0.0-beta-1
>
>
> It seems like the method getConnectionForEnvironment isn't doing the right
> thing when it is creating a CoprocessorHConnection by reading the config from
> HRegionServer and not from the env passed in.
> If coprocessors want to use a CoprocessorHConnection with some custom config
> settings, then they have no option but to configure it in the hbase-site.xml
> of the region servers. This isn't ideal as a lot of times these "global"
> level configs can have side effects. See PHOENIX-3974 as an example where
> configuring ServerRpcControllerFactory (a Phoenix implementation of
> RpcControllerFactory) could result in deadlocks. Or PHOENIX-3983 where
> presence of this global config causes our index rebuild code to incorrectly
> use handlers it shouldn't.
> If the CoprocessorHConnection created through getConnectionForEnvironment API
> used the CoprocessorEnvironment config, then it would allow co-processors to
> pass in their own config without needing to configure them in hbase-site.xml.
> The change would be simple. Basically change the below
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection((HRegionServer) services);
> }
> {code}
> to
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection(env.getConfiguration(),
> (HRegionServer) services);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18359) CoprocessorHConnection#getConnectionForEnvironment should read config from CoprocessorEnvironment
[
https://issues.apache.org/jira/browse/HBASE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255405#comment-16255405
]
Anoop Sam John commented on HBASE-18359:
bq.We should make the Configuration returned out of the Env be read-only?
Seems yes. This will avoid the misuse.
For this particular issue (and I see how Phoenix having a workaround for this),
there is no way to achieve in HBase 2.0. Because there only one cluster
connection already created in HRS and Env.getConnection() returns that. Here
seems we should give a getConnection() API with param Conf and that may have to
recreate the connection (Short circuited). May be phoenix should take care of
caching this connection and reuse?
> CoprocessorHConnection#getConnectionForEnvironment should read config from
> CoprocessorEnvironment
> -
>
> Key: HBASE-18359
> URL: https://issues.apache.org/jira/browse/HBASE-18359
> Project: HBase
> Issue Type: Bug
>Reporter: Samarth Jain
>
> It seems like the method getConnectionForEnvironment isn't doing the right
> thing when it is creating a CoprocessorHConnection by reading the config from
> HRegionServer and not from the env passed in.
> If coprocessors want to use a CoprocessorHConnection with some custom config
> settings, then they have no option but to configure it in the hbase-site.xml
> of the region servers. This isn't ideal as a lot of times these "global"
> level configs can have side effects. See PHOENIX-3974 as an example where
> configuring ServerRpcControllerFactory (a Phoenix implementation of
> RpcControllerFactory) could result in deadlocks. Or PHOENIX-3983 where
> presence of this global config causes our index rebuild code to incorrectly
> use handlers it shouldn't.
> If the CoprocessorHConnection created through getConnectionForEnvironment API
> used the CoprocessorEnvironment config, then it would allow co-processors to
> pass in their own config without needing to configure them in hbase-site.xml.
> The change would be simple. Basically change the below
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection((HRegionServer) services);
> }
> {code}
> to
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection(env.getConfiguration(),
> (HRegionServer) services);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18359) CoprocessorHConnection#getConnectionForEnvironment should read config from CoprocessorEnvironment
[
https://issues.apache.org/jira/browse/HBASE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254736#comment-16254736
]
Anoop Sam John commented on HBASE-18359:
[~samarth.j...@gmail.com], [~sergey.soldatov]. Coming to this jira again.
In trunk, things are changed a lot and there is no CoprocessorHConnection any
more. We have CoprocessorEnvironment#getConnection(). This will give short
circuited connection. Means any calls, via this connection , on regions in same
RS will go a short circuited path.
On the old code u mentioned, we were passing CoprocessorEnvironment. This is
the object u r getting from HBase core and you can just GET config from that.
This is the same config that Region uses. So am not sure how u can create
custom conf. You can not really set any config in CoprocessorEnvironment
object. I guess what u were doing is just changing /setting some config
attributes on the config getting from CoprocessorEnvironment. That will be
wrong. Because the conf object is a shared one for every thing in this Region.
All CPs over this Region and Region, Store and level down will use this changed
config. What ideally should be done is clone the config and set the new
attributes. If we were having an API which takes Config, it would have been
possible. Also CoprocessorHConnection was a Private class.
So this is not the ideal path for what you want to do.
Checking the new trunk code also, seems you can not do what u really want.
Because all calls CoprocessorEnvironment#getConnection() will return a same
Cluster connection object which is created in HRS. So if u need to have
custom configs to be used for these connection, only way would be to have a
CoprocessorEnvironment#getConnection(Config) API and create new connection
(Short circuited) in the impl. Not good to create the connection on every
call. We have to decide who will do the caching of the connection and reuse.
cc [~Stack]
> CoprocessorHConnection#getConnectionForEnvironment should read config from
> CoprocessorEnvironment
> -
>
> Key: HBASE-18359
> URL: https://issues.apache.org/jira/browse/HBASE-18359
> Project: HBase
> Issue Type: Bug
>Reporter: Samarth Jain
>
> It seems like the method getConnectionForEnvironment isn't doing the right
> thing when it is creating a CoprocessorHConnection by reading the config from
> HRegionServer and not from the env passed in.
> If coprocessors want to use a CoprocessorHConnection with some custom config
> settings, then they have no option but to configure it in the hbase-site.xml
> of the region servers. This isn't ideal as a lot of times these "global"
> level configs can have side effects. See PHOENIX-3974 as an example where
> configuring ServerRpcControllerFactory (a Phoenix implementation of
> RpcControllerFactory) could result in deadlocks. Or PHOENIX-3983 where
> presence of this global config causes our index rebuild code to incorrectly
> use handlers it shouldn't.
> If the CoprocessorHConnection created through getConnectionForEnvironment API
> used the CoprocessorEnvironment config, then it would allow co-processors to
> pass in their own config without needing to configure them in hbase-site.xml.
> The change would be simple. Basically change the below
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection((HRegionServer) services);
> }
> {code}
> to
> {code}
> if (services instanceof HRegionServer) {
> return new CoprocessorHConnection(env.getConfiguration(),
> (HRegionServer) services);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16253511#comment-16253511 ] Anoop Sam John commented on HBASE-19092: Latest patch is there in RB? If not pls put. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[ https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19235: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the reviews. > CoprocessorEnvironment should be exposed to CPs > --- > > Key: HBASE-19235 > URL: https://issues.apache.org/jira/browse/HBASE-19235 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0-alpha-4 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19235.patch, HBASE-19235_V2.patch, > HBASE-19235_V3.patch, HBASE-19235_V3.patch > > > Its sub interfaces are exposed with > LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions > in this are. Better we mark CoprocessorEnvironment also as CP exposed to > avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[ https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19235: --- Attachment: HBASE-19235_V3.patch > CoprocessorEnvironment should be exposed to CPs > --- > > Key: HBASE-19235 > URL: https://issues.apache.org/jira/browse/HBASE-19235 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0-alpha-4 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19235.patch, HBASE-19235_V2.patch, > HBASE-19235_V3.patch, HBASE-19235_V3.patch > > > Its sub interfaces are exposed with > LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions > in this are. Better we mark CoprocessorEnvironment also as CP exposed to > avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251393#comment-16251393 ] Anoop Sam John commented on HBASE-19092: TagUtil (no 's') if for each of the Tag. It act on one Tag. Let the Cell related APIs be there in CellUtil or ExtendedCell. This includes API to retrieve tag(s) from a Cell. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092_001-branch-2.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[ https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19235: --- Attachment: HBASE-19235_V3.patch > CoprocessorEnvironment should be exposed to CPs > --- > > Key: HBASE-19235 > URL: https://issues.apache.org/jira/browse/HBASE-19235 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0-alpha-4 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19235.patch, HBASE-19235_V2.patch, > HBASE-19235_V3.patch > > > Its sub interfaces are exposed with > LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions > in this are. Better we mark CoprocessorEnvironment also as CP exposed to > avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[
https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251222#comment-16251222
]
Anoop Sam John commented on HBASE-19092:
bq.{@link PrivateCellUtil#MAX_TAGS_LENGTH} define, in PCU and not in Tag?
This is the length of all tags. So Tag may be not the apt place. If we have
moved methods for extracting tag(s) from a Cell within ExtendedCell, ya we can
see whether we can keep it in ExtendedCell.
bq.Can TagType be a subclass of Tag?
I dont think TagType is-a tag. This is just a constants class for keeping the
system tags types.
> Make Tag IA.LimitedPrivate and expose for CPs
> -
>
> Key: HBASE-19092
> URL: https://issues.apache.org/jira/browse/HBASE-19092
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19092-branch-2.patch,
> HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch,
> HBASE-19092_001-branch-2.patch
>
>
> We need to make tags as LimitedPrivate as some use cases are trying to use
> tags like timeline server. The same topic was discussed in dev@ and also in
> HBASE-18995.
> Shall we target this for beta1 - cc [~saint@gmail.com].
> So once we do this all related Util methods and APIs should also move to
> LimitedPrivate Util classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19254) Write up how InterfaceAudience works over class hierarchy
[ https://issues.apache.org/jira/browse/HBASE-19254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251211#comment-16251211 ] Anoop Sam John commented on HBASE-19254: +1 on [~chia7712] suggestion. > Write up how InterfaceAudience works over class hierarchy > - > > Key: HBASE-19254 > URL: https://issues.apache.org/jira/browse/HBASE-19254 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack > > Over on the tail of HBASE-19235, there is a discussion around marking classes > with InterfaceAudicence in a hierarchy. One camp suggests that it could be > possible that a super class might be IA.Private but subclasses (and their > methods) could then be IA.LP. General opinion held the opposite. [~chia7712] > said it best: > bq. It makes sense to me that sub-class can always narrow the declaration > (ex. IA.Public -> IA.Private/IA.LP). Widening the declaration is like a issue > we need to fix. > This issue is about adding a note to the IA section in refguide (should > probably pass it by dev list before it goes into refguide). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[ https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251200#comment-16251200 ] Anoop Sam John commented on HBASE-19235: If we have to avoid the instance of check while calling startup or shutdown, we have to change public abstract class CoprocessorHost> Note BaseEnvironment instead of CoprocessorEnvironment. Subsequently we will have to make changes in all CPHost impls. Now we have RegionServerCoprocessorHost extends CoprocessorHost RegionCoprocessorHost extends CoprocessorHost Like these. I feel all these changes will be much more ugly than having an instance of check. Infact we need all our internal impls of the CPEnv to be extending BaseEnv. So we can even have just an assert and type casting way.. In future any one make a new impl of CPEnv not extending BaseEnv, this will break any way. > CoprocessorEnvironment should be exposed to CPs > --- > > Key: HBASE-19235 > URL: https://issues.apache.org/jira/browse/HBASE-19235 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0-alpha-4 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19235.patch, HBASE-19235_V2.patch > > > Its sub interfaces are exposed with > LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions > in this are. Better we mark CoprocessorEnvironment also as CP exposed to > avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19253) Dynamic change MemStoreFlusher FlushHandler workers
[ https://issues.apache.org/jira/browse/HBASE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19253: --- Issue Type: Improvement (was: Bug) > Dynamic change MemStoreFlusher FlushHandler workers > --- > > Key: HBASE-19253 > URL: https://issues.apache.org/jira/browse/HBASE-19253 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin > > Use update_config to dynamic change MemStoreFlusher FlushHandler workers -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[ https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250785#comment-16250785 ] Anoop Sam John commented on HBASE-19235: Appy. IMHO #2 is not at all good. Am not sure any other place we have like super interface is private and sub type as LP or public. I did not check. If so, we have to fix. I raised this issue after seeing a patch in Phoenix. The sub interfaces are LP exposed. So any functions available to it, can be called by user. Else the CP user has to really check in which interface it is declared and see its declaration. It will be very difficult task and confusing for sure. Ya we were having CPEnv interface as private till now. But we can cleanup now. Better late than never. I understood Chia comment to avoid the instance of check. Let me see. I thought we can make it simple. Like in old way. Let me see. > CoprocessorEnvironment should be exposed to CPs > --- > > Key: HBASE-19235 > URL: https://issues.apache.org/jira/browse/HBASE-19235 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0-alpha-4 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19235.patch, HBASE-19235_V2.patch > > > Its sub interfaces are exposed with > LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions > in this are. Better we mark CoprocessorEnvironment also as CP exposed to > avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[ https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19235: --- Attachment: HBASE-19235_V2.patch With #2. This is how we were doing before. > CoprocessorEnvironment should be exposed to CPs > --- > > Key: HBASE-19235 > URL: https://issues.apache.org/jira/browse/HBASE-19235 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0-alpha-4 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19235.patch, HBASE-19235_V2.patch > > > Its sub interfaces are exposed with > LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions > in this are. Better we mark CoprocessorEnvironment also as CP exposed to > avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19112) Suspect methods on Cell to be deprecated
[
https://issues.apache.org/jira/browse/HBASE-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249140#comment-16249140
]
Anoop Sam John commented on HBASE-19112:
bq.Chia-Ping Tsai Unrelated, I can't remember where, but in an issue you raised
the notion of a 'RawCell'. I keep meaning to say that I like this notion. We
currently have 'EnhancedCell'. I think RawCell a much better name (Anoop Sam
John & ramkrishna.s.vasudevan – what you think lads?).
So Chia_Ping says we can have Cell, RawCell and ExtendedCell. Correct?
> Suspect methods on Cell to be deprecated
>
>
> Key: HBASE-19112
> URL: https://issues.apache.org/jira/browse/HBASE-19112
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Josh Elser
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
>
> [~chia7712] suggested on the [mailing
> list|https://lists.apache.org/thread.html/e6de9af26d9b888a358ba48bf74655ccd893573087c032c0fcf01585@%3Cdev.hbase.apache.org%3E]
> that we have some methods on Cell which should be deprecated for removal:
> * {{#getType()}}
> * {{#getTimestamp()}}
> * {{#getTag()}}
> * {{#getSequenceId()}}
> Let's make a pass over these (and maybe the rest) to make sure that there
> aren't others which are either implementation details or methods returning
> now-private-marked classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[ https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249132#comment-16249132 ] Anoop Sam John commented on HBASE-19235: Ya I too feel #2 is fine. > CoprocessorEnvironment should be exposed to CPs > --- > > Key: HBASE-19235 > URL: https://issues.apache.org/jira/browse/HBASE-19235 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0-alpha-4 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19235.patch > > > Its sub interfaces are exposed with > LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions > in this are. Better we mark CoprocessorEnvironment also as CP exposed to > avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19187) Remove option to create on heap bucket cache
[ https://issues.apache.org/jira/browse/HBASE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19187: --- Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) Fix Version/s: (was: 2.1.0) 2.0.0-beta-1 Status: Resolved (was: Patch Available) Thanks all for the reviews. > Remove option to create on heap bucket cache > > > Key: HBASE-19187 > URL: https://issues.apache.org/jira/browse/HBASE-19187 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0-alpha-3 >Reporter: Biju Nair >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19187.patch, HBASE-19187_V2.patch > > > Since we are moving read and write paths to use direct memory, option to > configure on heap bucket cache is irrelevant. We should remove that option. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[
https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247278#comment-16247278
]
Anoop Sam John commented on HBASE-19235:
I see now.. The startup and shutdown were there in Environment only(Now
BaseEnv) . All 4 types of CPEnv impls are sub types of BaseEnv. Previously
we were doing type casting and call these 2 methods.
{code}
if (env instanceof Environment) {
((Environment)env).startup();
}
if (e instanceof Environment) {
if (LOG.isDebugEnabled()) {
LOG.debug("Stop coprocessor " + e.getInstance().getClass().getName());
}
((Environment)e).shutdown();
}
{code}
I think we can continue this way. Its ok and so avoid these 2 APIs being
exposed to CPs
> CoprocessorEnvironment should be exposed to CPs
> ---
>
> Key: HBASE-19235
> URL: https://issues.apache.org/jira/browse/HBASE-19235
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Affects Versions: 2.0.0-alpha-4
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Minor
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19235.patch
>
>
> Its sub interfaces are exposed with
> LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions
> in this are. Better we mark CoprocessorEnvironment also as CP exposed to
> avoid confusion.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[ https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247230#comment-16247230 ] Anoop Sam John commented on HBASE-19235: >From ObserverContext the CP user get RegionCPEnv. This is marked LP for CP and >any functions available there can be used . That is what we say. So all >functions from super type is already exposed right? Checking other CP >exposed interfaces, I see we have their super type also as LP not private. We >should some way clean this. Discussed with Stack offline. We both felt its ok >to expose.. We have some getters which might not be really needed for CPs >but harmless to expose. Start and Shutdown are some thing which should not be >used by CP users. Start() even if called it is a noop in impl. Shutdown ya... Checking some old code base, we did not have these start and shutdown over in CoprocessorEnvironment. Seems some recent jira added this.. Why? We can even avoid. Ya I see this startup was there in CPHost . > CoprocessorEnvironment should be exposed to CPs > --- > > Key: HBASE-19235 > URL: https://issues.apache.org/jira/browse/HBASE-19235 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0-alpha-4 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19235.patch > > > Its sub interfaces are exposed with > LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions > in this are. Better we mark CoprocessorEnvironment also as CP exposed to > avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[ https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19235: --- Attachment: HBASE-19235.patch Simple patch. > CoprocessorEnvironment should be exposed to CPs > --- > > Key: HBASE-19235 > URL: https://issues.apache.org/jira/browse/HBASE-19235 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0-alpha-4 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19235.patch > > > Its sub interfaces are exposed with > LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions > in this are. Better we mark CoprocessorEnvironment also as CP exposed to > avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
[ https://issues.apache.org/jira/browse/HBASE-19235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19235: --- Status: Patch Available (was: Open) > CoprocessorEnvironment should be exposed to CPs > --- > > Key: HBASE-19235 > URL: https://issues.apache.org/jira/browse/HBASE-19235 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0-alpha-4 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19235.patch > > > Its sub interfaces are exposed with > LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions > in this are. Better we mark CoprocessorEnvironment also as CP exposed to > avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19235) CoprocessorEnvironment should be exposed to CPs
Anoop Sam John created HBASE-19235: -- Summary: CoprocessorEnvironment should be exposed to CPs Key: HBASE-19235 URL: https://issues.apache.org/jira/browse/HBASE-19235 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0-alpha-4 Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Minor Fix For: 2.0.0-beta-1 Its sub interfaces are exposed with LimitedPrivate(HBaseInterfaceAudience.COPROC). So ideally all the functions in this are. Better we mark CoprocessorEnvironment also as CP exposed to avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19187) Remove option to create on heap bucket cache
[ https://issues.apache.org/jira/browse/HBASE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19187: --- Release Note: Removing the on heap Bucket cache feature. The config "hbase.bucketcache.ioengine" no longer support the 'heap' value. Its supported values now are 'offheap', 'file:', 'files:' and 'mmap:' > Remove option to create on heap bucket cache > > > Key: HBASE-19187 > URL: https://issues.apache.org/jira/browse/HBASE-19187 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0-alpha-3 >Reporter: Biju Nair >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-19187.patch, HBASE-19187_V2.patch > > > Since we are moving read and write paths to use direct memory, option to > configure on heap bucket cache is irrelevant. We should remove that option. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19187) Remove option to create on heap bucket cache
[ https://issues.apache.org/jira/browse/HBASE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19187: --- Attachment: HBASE-19187_V2.patch Correction of test failures. Removed the on heap BC tests. Fixed one javadoc warn. > Remove option to create on heap bucket cache > > > Key: HBASE-19187 > URL: https://issues.apache.org/jira/browse/HBASE-19187 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0-alpha-3 >Reporter: Biju Nair >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-19187.patch, HBASE-19187_V2.patch > > > Since we are moving read and write paths to use direct memory, option to > configure on heap bucket cache is irrelevant. We should remove that option. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19187) Remove option to create on heap bucket cache
[ https://issues.apache.org/jira/browse/HBASE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19187: --- Attachment: HBASE-19187.patch > Remove option to create on heap bucket cache > > > Key: HBASE-19187 > URL: https://issues.apache.org/jira/browse/HBASE-19187 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0-alpha-3 >Reporter: Biju Nair >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-19187.patch > > > Since we are moving read and write paths to use direct memory, option to > configure on heap bucket cache is irrelevant. We should remove that option. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19187) Remove option to create on heap bucket cache
[ https://issues.apache.org/jira/browse/HBASE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19187: --- Hadoop Flags: Incompatible change Status: Patch Available (was: Open) > Remove option to create on heap bucket cache > > > Key: HBASE-19187 > URL: https://issues.apache.org/jira/browse/HBASE-19187 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0-alpha-3 >Reporter: Biju Nair >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-19187.patch > > > Since we are moving read and write paths to use direct memory, option to > configure on heap bucket cache is irrelevant. We should remove that option. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18624) Added support for clearing BlockCache based on table name
[ https://issues.apache.org/jira/browse/HBASE-18624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245235#comment-16245235 ] Anoop Sam John commented on HBASE-18624: [~zyork] Release Notes pls? > Added support for clearing BlockCache based on table name > - > > Key: HBASE-18624 > URL: https://issues.apache.org/jira/browse/HBASE-18624 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.3.0 >Reporter: Ajay Jadhav >Assignee: Zach York > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18624.branch-1.001.patch, > HBASE-18624.master.001.patch, HBASE-18624.master.002.patch, > HBASE-18624.master.003.patch, HBASE-18624.master.004.patch, > HBASE-18624.master.005.patch, HBASE-18624.master.006.patch, > HBASE-18624.master.007.patch, HBASE-18624.master.008.patch, > HBASE-18624.master.009.patch, HBASE-18624.master.010.patch, > HBASE-18624.master.011.patch, HBASE-18624.master.012.patch, > HBASE-18624.master.013.patch > > > Bulk loading the primary HBase cluster triggers a lot of compactions > resulting in archival/ creation > of multiple HFiles. This process will cause a lot of items to become stale in > replica’s BlockCache. > This patch will help users to clear the block cache for a given table by > either using shell or API. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18294) Flush is based on data size instead of heap size
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244286#comment-16244286 ] Anoop Sam John commented on HBASE-18294: bq.to a tuple of 3 - data size, heap size, and off-heap size Not sure why 3 items to be tracked. data size and heap size is enough IMO. Data size can be on heap or off heap based on the MSLAB in use. The main thing will be that at Region level have to track not just the data size but the heap size also. We do this at segment level any way. > Flush is based on data size instead of heap size > > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-19187) Remove option to create on heap bucket cache
[ https://issues.apache.org/jira/browse/HBASE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John reassigned HBASE-19187: -- Assignee: Anoop Sam John > Remove option to create on heap bucket cache > > > Key: HBASE-19187 > URL: https://issues.apache.org/jira/browse/HBASE-19187 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0-alpha-3 >Reporter: Biju Nair >Assignee: Anoop Sam John >Priority: Minor > Fix For: 2.1.0 > > > Since we are moving read and write paths to use direct memory, option to > configure on heap bucket cache is irrelevant. We should remove that option. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19187) Remove option to create on heap bucket cache
[ https://issues.apache.org/jira/browse/HBASE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243270#comment-16243270 ] Anoop Sam John commented on HBASE-19187: Better we can remove in 2.0 itself. We are trying to remove some other stuff like DLR, Prefix tree. (Reasons are diff there though).. Will give patch soon. > Remove option to create on heap bucket cache > > > Key: HBASE-19187 > URL: https://issues.apache.org/jira/browse/HBASE-19187 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0-alpha-3 >Reporter: Biju Nair >Priority: Minor > Fix For: 2.1.0 > > > Since we are moving read and write paths to use direct memory, option to > configure on heap bucket cache is irrelevant. We should remove that option. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19116) Currently the tail of hfiles with CellComparator* classname makes it so hbase1 can't open hbase2 written hfiles; fix
[ https://issues.apache.org/jira/browse/HBASE-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240555#comment-16240555 ] Anoop Sam John commented on HBASE-19116: The cluster is a mix of some servers based on 2.0 while others 1.x. One region was there in 2.0 server and wrote some HFiles. Now this RS went down and region got assigned to 1.x based server and now that 2.0 based file is opened by 1.x server. > Currently the tail of hfiles with CellComparator* classname makes it so > hbase1 can't open hbase2 written hfiles; fix > > > Key: HBASE-19116 > URL: https://issues.apache.org/jira/browse/HBASE-19116 > Project: HBase > Issue Type: Sub-task > Components: HFile, migration >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > > See tail of HBASE-19052 for discussion which concludes we should try and make > it so operators do not have to go to latest hbase version before they > upgrade, at least if we can avoid it. > The necessary change of our default comparator from KV to Cell naming has > hfiles with tails that have the classname CellComparator in them in place of > KeyValueComparator. If an hbase1 tries to open them, it will fail not having > a CellComparator in its classpath (We have name of comparator in tail because > different files require different comparators... perhaps we write an alias > instead of a class one day... TODO). HBASE-16189 and HBASE-19052 are about > trying to carry knowledge of hbase2 back to hbase1, a brittle approach making > it so operators will have to upgrade to the latest branch-1 before they can > go to hbase2. > This issue is about undoing our writing of an incompatible (to hbase1) tail, > not unless we really have to (and it sounds like we could do without writing > an incompatible tail) to see if we can avoid requiring operators go to > lastest branch-1 (we may end up needing this but lets a have a really good > reason for it if we do). > Oh, let this filing be an answer to our [~anoop.hbase]'s old high-level > question over in HBASE-16189: > bq. ...means when rolling upgrade done to 2.0, first users have to upgrade to > some 1.x versions which is having this fix and then to 2.0.. What do you guys > think Whether we should avoid this kind of indirection? cc Enis Soztutar, > Stack, Ted Yu, Matteo Bertozzi > Yeah, lets try to avoid this if we can... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19187) Remove option to create on heap bucket cache
[ https://issues.apache.org/jira/browse/HBASE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240236#comment-16240236 ] Anoop Sam John commented on HBASE-19187: [~gsbiju] U will work on this issue or want me to provide a patch? Tks > Remove option to create on heap bucket cache > > > Key: HBASE-19187 > URL: https://issues.apache.org/jira/browse/HBASE-19187 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0-alpha-3 >Reporter: Biju Nair >Priority: Minor > Fix For: 2.0.0-beta-1 > > > Since we are moving read and write paths to use direct memory, option to > configure on heap bucket cache is irrelevant. We should remove that option. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19187) Remove option to create on heap bucket cache
[ https://issues.apache.org/jira/browse/HBASE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19187: --- Fix Version/s: 2.0.0-beta-1 > Remove option to create on heap bucket cache > > > Key: HBASE-19187 > URL: https://issues.apache.org/jira/browse/HBASE-19187 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0-alpha-3 >Reporter: Biju Nair >Priority: Minor > Fix For: 2.0.0-beta-1 > > > Since we are moving read and write paths to use direct memory, option to > configure on heap bucket cache is irrelevant. We should remove that option. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18708) Configure on-heap bucketCache size using percentage of heap size
[
https://issues.apache.org/jira/browse/HBASE-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John resolved HBASE-18708.
Resolution: Won't Fix
Assignee: (was: Biju Nair)
HBASE-19187 will remove the on heap version of BC. So resolving this as 'Wont
fix'
> Configure on-heap bucketCache size using percentage of heap size
>
>
> Key: HBASE-18708
> URL: https://issues.apache.org/jira/browse/HBASE-18708
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Biju Nair
>Priority: Trivial
> Attachments: HBASE-18708-BRANCH-1.PATCH
>
>
> Currently heap allocations for RS memory structures like {{memstore}} and
> {{lruCache}} are configured as percentage of total RS heap size. Since
> on-heap bucketCache uses RS heap, configuring it as a percentage of heap size
> will improve usability. Currently this can be configured either as a
> percentage of heap or a memory size in MiB and we can remove the latter
> option which is applicable to external or off-heap bucketCache.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19165) TODO Handle stuck in transition: rit=OPENING, location=ve0538....
[ https://issues.apache.org/jira/browse/HBASE-19165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239924#comment-16239924 ] Anoop Sam John commented on HBASE-19165: Can the state cell be there with empty value? You also say this is not possible ideally and if that means some issue? Correct me if am wrong. May be specific handling can be there. Ya agree on return as null and handle in using places. > TODO Handle stuck in transition: rit=OPENING, location=ve0538 > - > > Key: HBASE-19165 > URL: https://issues.apache.org/jira/browse/HBASE-19165 > Project: HBase > Issue Type: Bug > Components: migration >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19165.master.001.patch > > > A few of us trying migration from hbase1 to hbase2 have run into various > assignment issues. In my case, I see that Master coming online, reading > hbase:meta, finding an issue in OPENING state and then just not assigning. My > thought is that its expecting a callback to come in. Eventually log is fills > with this sort of stuff: > 2017-11-02 15:02:54,237 WARN [ProcExecTimeout] assignment.AssignmentManager: > TODO Handle stuck in transition: rit=OPENING, > location=ve0528.halxg.cloudera.com,16020,1509657692581, > table=IntegrationTestBigLinkedList, region=919cc6636ffbed17f628f335d3a58726 > Let me get a cleaner run. This current log is polluted by a few restarts. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19077) Have Region*CoprocessorEnvironment provide an ImmutableOnlineRegions
[ https://issues.apache.org/jira/browse/HBASE-19077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239918#comment-16239918 ] Anoop Sam John commented on HBASE-19077: We have a problem here. OnlineRegions extending Server. So we are exposing Server interface also. It will be dangerous to expose all the APIs in server to CPs. Even the RS can be aborted. It exposes ChoreService, CoordinatedStateManager , ZooKeeperWatcher etc which should not be ideally. At Master CP side , we are not exposing the Server. We can change OnlineRegions not to extend Server. Let the RegionServerServices only extend Server. We can do it as an addendum here or need a new issue? [~stack] > Have Region*CoprocessorEnvironment provide an ImmutableOnlineRegions > > > Key: HBASE-19077 > URL: https://issues.apache.org/jira/browse/HBASE-19077 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-19077.master.001.patch, > HBASE-19077.master.002.patch > > > This is about restoring some of the hbase1 parity... You used to be able to > get list of Region on the local server and talk to them directly. > Suggested by [~anoop.hbase] up in review. Makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Issue Comment Deleted] (HBASE-19077) Have Region*CoprocessorEnvironment provide an ImmutableOnlineRegions
[ https://issues.apache.org/jira/browse/HBASE-19077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19077: --- Comment: was deleted (was: We have a problem here. OnlineRegions extending Server. So we are exposing Server interface also. It will be dangerous to expose all the APIs in server to CPs. Even the RS can be aborted. It exposes ChoreService, CoordinatedStateManager , ZooKeeperWatcher etc which should not be ideally. At Master CP side , we are not exposing the Server. We can change OnlineRegions not to extend Server. Let the RegionServerServices only extend Server. We can do it as an addendum here or need a new issue? [~stack]) > Have Region*CoprocessorEnvironment provide an ImmutableOnlineRegions > > > Key: HBASE-19077 > URL: https://issues.apache.org/jira/browse/HBASE-19077 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-19077.master.001.patch, > HBASE-19077.master.002.patch > > > This is about restoring some of the hbase1 parity... You used to be able to > get list of Region on the local server and talk to them directly. > Suggested by [~anoop.hbase] up in review. Makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19160) Re-expose CellComparator
[
https://issues.apache.org/jira/browse/HBASE-19160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239886#comment-16239886
]
Anoop Sam John commented on HBASE-19160:
+1
> Re-expose CellComparator
>
>
> Key: HBASE-19160
> URL: https://issues.apache.org/jira/browse/HBASE-19160
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19160.patch, HBASE-19160.v2.patch,
> HBASE-19160.v3.patch, HBASE-19160.v4.patch
>
>
> On HBASE-18995 we moved a bunch of public methods to Private places. This
> inadvertently breaks donwstream consumers. Let's see if we can ease up on
> some of the lockdown and make life easier for them.
> Copying [~ram_krish]'s previous analysis:
> {quote}
> I read the Crunch projec't hbase-support related code.
> -> It uses both CellUtil (Public exposed) and KeyValueUtil (@Private) classes
> for helper methods.
> -> All methods in CellUtil that are getting used are even now exposed in
> branch-2's CellUtil and they are very common helper methods. So we are safe
> here.
> -> Wrt KeyValueUtil the API is createFirstOnRow(). It is used in test cases
> and in some core code. In most of the places they are trying to create the
> splitKeys from the region's start keys and that is also getting persisted. I
> think here they can safely create a cell out of the given byte[] of the row.
> But there is one place where they are trying to do some scanning on a
> HFileScanner directly (@Private) scanner. So this should be changed because
> it is an internal interface for us. And on this scanner they have copied our
> seekTo() code into their source files for some scanning purpose. In this code
> they are actually using the KvUtil.createFirstOnRow() to seek to that first
> cell of that row.
> More over I think in branch-2 we are restricting even CPs from accessing some
> of our internal scanners and they can only use InternalScanner interface. So
> this code in crunch needs heavy refactoring to work with branch-2 in case
> they want to fit into the Public/Private exposed semantics that HBase
> presents to the downstreamers.
> -> If still they want some APIs like this we can expose
> CellUtil#createFirstOnRow, createLastOnRow, createFirstOnCol and
> createLastOnCol at the maximum. I think others are not useful and are more
> internal stuffs.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19165) TODO Handle stuck in transition: rit=OPENING, location=ve0538....
[ https://issues.apache.org/jira/browse/HBASE-19165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239838#comment-16239838 ] Anoop Sam John commented on HBASE-19165: bq. if (cell == null || cell.getValueLength() == 0) return null; Does it make sense to return OFFLINE state itself from here rather than null and handle the null in other place? Any other impact? Can we have this state Cell in META entry row but with no value ( cell.getValueLength() == 0) case? Seems no as per my check. > TODO Handle stuck in transition: rit=OPENING, location=ve0538 > - > > Key: HBASE-19165 > URL: https://issues.apache.org/jira/browse/HBASE-19165 > Project: HBase > Issue Type: Bug > Components: migration >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19165.master.001.patch > > > A few of us trying migration from hbase1 to hbase2 have run into various > assignment issues. In my case, I see that Master coming online, reading > hbase:meta, finding an issue in OPENING state and then just not assigning. My > thought is that its expecting a callback to come in. Eventually log is fills > with this sort of stuff: > 2017-11-02 15:02:54,237 WARN [ProcExecTimeout] assignment.AssignmentManager: > TODO Handle stuck in transition: rit=OPENING, > location=ve0528.halxg.cloudera.com,16020,1509657692581, > table=IntegrationTestBigLinkedList, region=919cc6636ffbed17f628f335d3a58726 > Let me get a cleaner run. This current log is polluted by a few restarts. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19035) Miss metrics when coprocessor use region scanner to read data
[ https://issues.apache.org/jira/browse/HBASE-19035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239818#comment-16239818 ] Anoop Sam John commented on HBASE-19035: I see. So those were already in place. COuld not see in this patch and got a doubt. I did not check the existing flow for these.. If so fine then. +1. Get a final go from Stack also. Tks > Miss metrics when coprocessor use region scanner to read data > - > > Key: HBASE-19035 > URL: https://issues.apache.org/jira/browse/HBASE-19035 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19035.master.001.patch, > HBASE-19035.master.002.patch > > > Region interface is exposed to coprocessor. So coprocessor use getScanner to > get a region scanner to read data. But the scan metrics was only updated in > region server level. So we will miss some scan metrics for the read from > coprocessor. > || Region Operation || When to update requests metric || > | get | update read metric in nextRaw() | > | put | update write metric in batchMutate() | > | delete | update write metric in batchMutate() | > | increment | update read metric by get() and update write metric in > doDelta() | > | append | update read metric by get() and update write metric in doDelta() | > | mutateRow | update write metric in processRowsWithLocks() | > | mutateRowsWithLocks | update write metric in processRowsWithLocks() | > | batchMutate | update write metric in batchMutate() | > | checkAndMutate | update read metric by get() and update write metric by > mutateRow() | > | checkAndRowMutate | update read metric by get() and update write metric by > doBatchMutate() | > | processRowsWithLocks | update write metric in processRowsWithLocks() | > Meanwhile, remove requestRowActionCount in RSRpcServices. This metric can be > computed by region's readRequestsCount and writeRequestsCount. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19035) Miss metrics when coprocessor use region scanner to read data
[ https://issues.apache.org/jira/browse/HBASE-19035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239468#comment-16239468 ] Anoop Sam John commented on HBASE-19035: What updation of writeRequestsCount when HRegion#batchMutate()? What abt HRegion append/increment/checkAndXXX ops? > Miss metrics when coprocessor use region scanner to read data > - > > Key: HBASE-19035 > URL: https://issues.apache.org/jira/browse/HBASE-19035 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19035.master.001.patch, > HBASE-19035.master.002.patch > > > Region interface is exposed to coprocessor. So coprocessor use getScanner to > get a region scanner to read data. But the scan metrics was only updated in > region server level. So we will miss some scan metrics for the read from > coprocessor. > || Region Operation || When to update requests metric || > | get | update read metric in nextRaw() | > | put | update write metric in batchMutate() | > | delete | update write metric in batchMutate() | > | increment | update read metric by get() and update write metric in > doDelta() | > | append | update read metric by get() and update write metric in doDelta() | > | mutateRow | update write metric in processRowsWithLocks() | > | mutateRowsWithLocks | update write metric in processRowsWithLocks() | > | batchMutate | update write metric in batchMutate() | > | checkAndMutate | update read metric by get() and update write metric by > mutateRow() | > | checkAndRowMutate | update read metric by get() and update write metric by > doBatchMutate() | > | processRowsWithLocks | update write metric in processRowsWithLocks() | > Meanwhile, remove requestRowActionCount in RSRpcServices. This metric can be > computed by region's readRequestsCount and writeRequestsCount. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17819) Reduce the heap overhead for BucketCache
[ https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16238776#comment-16238776 ] Anoop Sam John commented on HBASE-17819: Typo. eviction of blocks on HFile close is default off only (We have a config to turn this ON). bq.How did you do the size compare? Tested with Java Instrumentation > Reduce the heap overhead for BucketCache > > > Key: HBASE-17819 > URL: https://issues.apache.org/jira/browse/HBASE-17819 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch, > HBASE-17819_V3.patch > > > We keep Bucket entry map in BucketCache. Below is the math for heapSize for > the key , value into this map. > BlockCacheKey > --- > String hfileName - Ref - 4 > long offset - 8 > BlockType blockType - Ref - 4 > boolean isPrimaryReplicaBlock - 1 > Total = 12 (Object) + 17 = 29 > BucketEntry > > int offsetBase - 4 > int length - 4 > byte offset1 - 1 > byte deserialiserIndex - 1 > long accessCounter - 8 > BlockPriority priority - Ref - 4 > volatile boolean markedForEvict - 1 > AtomicInteger refCount - 16 + 4 > long cachedTime - 8 > Total = 12 (Object) + 51 = 63 > ConcurrentHashMap Map.Entry - 40 > blocksByHFile ConcurrentSkipListSet Entry - 40 > Total = 29 + 63 + 80 = 172 > For 10 million blocks we will end up having 1.6GB of heap size. > This jira aims to reduce this as much as possible -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18708) Configure on-heap bucketCache size using percentage of heap size
[
https://issues.apache.org/jira/browse/HBASE-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16237583#comment-16237583
]
Anoop Sam John commented on HBASE-18708:
Can we close this and raise another issue to remove the BC on-heap mode?
> Configure on-heap bucketCache size using percentage of heap size
>
>
> Key: HBASE-18708
> URL: https://issues.apache.org/jira/browse/HBASE-18708
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Biju Nair
>Assignee: Biju Nair
> Attachments: HBASE-18708-BRANCH-1.PATCH
>
>
> Currently heap allocations for RS memory structures like {{memstore}} and
> {{lruCache}} are configured as percentage of total RS heap size. Since
> on-heap bucketCache uses RS heap, configuring it as a percentage of heap size
> will improve usability. Currently this can be configured either as a
> percentage of heap or a memory size in MiB and we can remove the latter
> option which is applicable to external or off-heap bucketCache.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-17819) Reduce the heap overhead for BucketCache
[
https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16237495#comment-16237495
]
Anoop Sam John edited comment on HBASE-17819 at 11/3/17 11:52 AM:
--
To let know the approach. This is bit diff from V2 patch. Major changes are
1. BucketEntry is extended to make the SharedMemory BucketEntry. For file
mode, there is no need to keep the ref count as that is not shared memory type.
So I removed those new states added for 11425 from BucketEntry. For off heap
mode BucketEntry, we have an extension now where we have the new states.
2. Removed the CSLM for keeping the HFilename based blocks info. The
evictBlocksByHfileName will have a perf impact as it has to iterate through all
the entries to know each of the block entry belong to this file or not. For
that changed the evictBlocksByHfileName to be an async op way. A dedicated
eviction thread will do this work. ANy way even if we dont remove these blocks
or have delay in removal, eventually these block will get removed as we have
LRU algo for the eviction. So when there are no space left for the new blocks
addition, eviction would happen, removing unused blocks. More over, eviction
of blocks on HFile close is default off only (We have a config to turn this
off). When it is compaction , for the compacted files, we have evictByHFiles
happening now. There will be bit more delay for the actual removal of the
blocks.
But we save lot of heap memory per entry now as per this approach. The math is
there in above comment
{quote}
Now - 32 + 64 + 40 + 40 = 176
After patch - 32 + 48 + 40 = 120
Tested with Java Instrumentation
{quote}
was (Author: anoop.hbase):
To let know the approach. This is bit diff from V2 patch. Major changes are
1. BucketEntry is extended to make the SharedMemory BucketEntry. For file
mode, there is no need to keep the ref count as that is not shared memory type.
So I removed those new states added for 11425 from BucketEntry. For off heap
mode BucketEntry, we have an extension now where we have the new states.
2. Removed the CSLM for keeping the HFilename based blocks info. The
evictBlocksByHfileName will have a perf impact as it has to iterate through all
the entries to know each of the block entry belong to this file or not. For
that changed the evictBlocksByHfileName to be an async op way. A dedicated
eviction thread will do this work. ANy way even if we dont remove these blocks
or have delay in removal, eventually these block will get removed as we have
LRU algo for the eviction. So when there are no space left for the new blocks
addition, eviction would happen, removing unused blocks. More over, eviction
of blocks on HFile close is default off only (We have a config to turn this
off). When it is compaction , for the compacted files, we have evictByHFiles
happening now. There will be bit more delay for the actual removal of the
blocks.
But we save lot of heap memory per entry now as per this approach. The math is
there in above comment
> Reduce the heap overhead for BucketCache
>
>
> Key: HBASE-17819
> URL: https://issues.apache.org/jira/browse/HBASE-17819
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch,
> HBASE-17819_V3.patch
>
>
> We keep Bucket entry map in BucketCache. Below is the math for heapSize for
> the key , value into this map.
> BlockCacheKey
> ---
> String hfileName - Ref - 4
> long offset - 8
> BlockType blockType - Ref - 4
> boolean isPrimaryReplicaBlock - 1
> Total = 12 (Object) + 17 = 29
> BucketEntry
>
> int offsetBase - 4
> int length - 4
> byte offset1 - 1
> byte deserialiserIndex - 1
> long accessCounter - 8
> BlockPriority priority - Ref - 4
> volatile boolean markedForEvict - 1
> AtomicInteger refCount - 16 + 4
> long cachedTime - 8
> Total = 12 (Object) + 51 = 63
> ConcurrentHashMap Map.Entry - 40
> blocksByHFile ConcurrentSkipListSet Entry - 40
> Total = 29 + 63 + 80 = 172
> For 10 million blocks we will end up having 1.6GB of heap size.
> This jira aims to reduce this as much as possible
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17819) Reduce the heap overhead for BucketCache
[ https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17819: --- Attachment: HBASE-17819_V3.patch To let know the approach. This is bit diff from V2 patch. Major changes are 1. BucketEntry is extended to make the SharedMemory BucketEntry. For file mode, there is no need to keep the ref count as that is not shared memory type. So I removed those new states added for 11425 from BucketEntry. For off heap mode BucketEntry, we have an extension now where we have the new states. 2. Removed the CSLM for keeping the HFilename based blocks info. The evictBlocksByHfileName will have a perf impact as it has to iterate through all the entries to know each of the block entry belong to this file or not. For that changed the evictBlocksByHfileName to be an async op way. A dedicated eviction thread will do this work. ANy way even if we dont remove these blocks or have delay in removal, eventually these block will get removed as we have LRU algo for the eviction. So when there are no space left for the new blocks addition, eviction would happen, removing unused blocks. More over, eviction of blocks on HFile close is default off only (We have a config to turn this off). When it is compaction , for the compacted files, we have evictByHFiles happening now. There will be bit more delay for the actual removal of the blocks. But we save lot of heap memory per entry now as per this approach. The math is there in above comment > Reduce the heap overhead for BucketCache > > > Key: HBASE-17819 > URL: https://issues.apache.org/jira/browse/HBASE-17819 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch, > HBASE-17819_V3.patch > > > We keep Bucket entry map in BucketCache. Below is the math for heapSize for > the key , value into this map. > BlockCacheKey > --- > String hfileName - Ref - 4 > long offset - 8 > BlockType blockType - Ref - 4 > boolean isPrimaryReplicaBlock - 1 > Total = 12 (Object) + 17 = 29 > BucketEntry > > int offsetBase - 4 > int length - 4 > byte offset1 - 1 > byte deserialiserIndex - 1 > long accessCounter - 8 > BlockPriority priority - Ref - 4 > volatile boolean markedForEvict - 1 > AtomicInteger refCount - 16 + 4 > long cachedTime - 8 > Total = 12 (Object) + 51 = 63 > ConcurrentHashMap Map.Entry - 40 > blocksByHFile ConcurrentSkipListSet Entry - 40 > Total = 29 + 63 + 80 = 172 > For 10 million blocks we will end up having 1.6GB of heap size. > This jira aims to reduce this as much as possible -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19153) LruBlockCache cache too big blocks logic error
[
https://issues.apache.org/jira/browse/HBASE-19153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235688#comment-16235688
]
Anoop Sam John commented on HBASE-19153:
What is the issue? When the block's size > maxBlockSize, we want to return
with out caching to the LRU cache. The extra check is to limit the WARN logs.
Not logging for every occasion but only at every 50th (means 2% of total)
if (stats.failInsert() % 50 != 0)
{ return; }
This means not actually cache at 2% times.. That is not what we want
> LruBlockCache cache too big blocks logic error
> --
>
> Key: HBASE-19153
> URL: https://issues.apache.org/jira/browse/HBASE-19153
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
>Affects Versions: 2.0.0-alpha-3
>Reporter: Zhang Quanjin
>
> The latest version of LruBolckCache, I found the code logic of cache too big
> bolcks is inconsistent with annotation.
> If follow the notes, the code should look like this:
> if (buf.heapSize() > maxBlockSize) {
> // If there are a lot of blocks that are too
> // big this can make the logs way too noisy.
> // So we log 2%
> if (stats.failInsert() % 50 != 0) {
> return;
> }
> LOG.warn("Trying to cache too large a block "
> + cacheKey.getHfileName() + " @ "
> + cacheKey.getOffset()
> + " is " + buf.heapSize()
> + " which is larger than " + maxBlockSize);
>
> }
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore
[ https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16234404#comment-16234404 ] Anoop Sam John commented on HBASE-16421: Use the new config 'hbase.regionserver.offheap.global.memstore.size' which says the global off heap memstore size. (Size in MBs) We still need 'hbase.regionserver.global.memstore.size' also which says the heap % for max global size. Also to make sure that the system using MSLAB and pool. hbase.hregion.memstore.mslab.enabled -> True hbase.hregion.memstore.chunkpool.maxsize -> 1 These configs def value only Also pls make sure that HBASE_OFFHEAPSIZE is having appropriate value (hbase-env.sh).. Need to add some more than the global off heap memstore size (Like 2 GB or so). The ByteBufferPool at RPC layer uses off heap memory. Also the NIO layer need some off heap memory as well. > Introducing the CellChunkMap as a new additional index variant in the MemStore > -- > > Key: HBASE-16421 > URL: https://issues.apache.org/jira/browse/HBASE-16421 > Project: HBase > Issue Type: Umbrella >Reporter: Anastasia Braginsky >Priority: Major > Attachments: CellChunkMapRevived.pdf, ChunkCell_creation.png, > IntroductiontoNewFlatandCompactMemStore.pdf > > > Follow up for HBASE-14921. This is going to be the umbrella JIRA to include > all the parts of integration of the CellChunkMap to the MemStore. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233661#comment-16233661 ] Anoop Sam John commented on HBASE-16890: Lets work on this after Alpha-4. This is a very imp jira for 2.0 > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: AsyncWAL_disruptor.patch, AsyncWAL_disruptor_1 > (2).patch, AsyncWAL_disruptor_3.patch, AsyncWAL_disruptor_3.patch, > AsyncWAL_disruptor_4.patch, AsyncWAL_disruptor_6.patch, > HBASE-16890-rc-v2.patch, HBASE-16890-rc-v3.patch, > HBASE-16890-remove-contention-v1.patch, HBASE-16890-remove-contention.patch, > Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07 > PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, Screen Shot 2016-11-04 at > 5.21.27 PM.png, Screen Shot 2016-11-04 at 5.30.18 PM.png, async.svg, > classic.svg, contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17204) Make L2 off heap cache default ON
[ https://issues.apache.org/jira/browse/HBASE-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233658#comment-16233658 ] Anoop Sam John commented on HBASE-17204: Working on this.. Doing some more perf tests based on Alpha-3 release and on smaller data size. Will do long run test also soon. > Make L2 off heap cache default ON > - > > Key: HBASE-17204 > URL: https://issues.apache.org/jira/browse/HBASE-17204 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Blocker > Fix For: 2.0.0-beta-1 > > > L2 cache can be used for data blocks. By default it is off now. After > HBASE-11425 work, L2 off heap cache can equally perform with L1 on heap > cache. On heavy loaded workload, this can even out perform L1 cache. Pls see > recently published report by Alibaba. Also this work was backported by > Rocketfuel and similar perf improvement report from them too. > Let us turn L2 off heap cache ON. As it is off heap, we can have much larger > sized L2 BC. What should be the default size? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-18375: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: (was: 2.0.0-beta-1) 2.0.0-alpha-3 Status: Resolved (was: Patch Available) > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02-branch-2.patch, > HBASE-18375-V02.patch, HBASE-18375-V03.patch, HBASE-18375-V04.patch, > HBASE-18375-V05.patch, HBASE-18375-V06.patch, HBASE-18375-V07.patch, > HBASE-18375-V08.patch, HBASE-18375-V09.patch, HBASE-18375-V10.patch, > HBASE-18375-V11.patch, HBASE-18375-branch-2.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19122) preCompact and preFlush can bypass by returning null scanner; shut it down
[
https://issues.apache.org/jira/browse/HBASE-19122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227117#comment-16227117
]
Anoop Sam John commented on HBASE-19122:
We can better move this issue as subtask under that CP cleanup jira?
> preCompact and preFlush can bypass by returning null scanner; shut it down
> --
>
> Key: HBASE-19122
> URL: https://issues.apache.org/jira/browse/HBASE-19122
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, Scanners
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
>
> Noticed by [~anoop.hbase] during review of HBASE-18770, preCompact and
> preFlush can bypass normal processing by returning null. They are not
> bypasable by ordained route. We should shut down this avenue.
> The preCompact at least may be new coming in with:
> {code}
> tree dbf13093842f85a713f023d7219caccf8f4eb05f
> parent a4dcf51415616772e462091ce93622f070ea8810
> author zhangduo Sat Apr 9 16:18:08 2016 +0800
> committer zhangduo Sun Apr 10 09:26:28 2016 +0800
> HBASE-15527 Refactor Compactor related classes
> {code}
> Would have to dig in more to figure for sure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18770) Remove bypass method in ObserverContext and implement the 'bypass' logic case by case
[ https://issues.apache.org/jira/browse/HBASE-18770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227107#comment-16227107 ] Anoop Sam John commented on HBASE-18770: Lets commit this.. Any more things to cleanup, we can open up new issue. +1. This great work > Remove bypass method in ObserverContext and implement the 'bypass' logic case > by case > - > > Key: HBASE-18770 > URL: https://issues.apache.org/jira/browse/HBASE-18770 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18770.master.001.patch, > HBASE-18770.master.002.patch, HBASE-18770.master.003.patch, > HBASE-18770.master.004.patch, HBASE-18770.master.005.patch, > HBASE-18770.master.006.patch, HBASE-18770.master.007.patch, > HBASE-18770.master.008.patch, HBASE-18770.master.009.patch, > HBASE-18770.master.010.patch, HBASE-18770.master.011.patch, > HBASE-18770.master.012.patch > > > http://search-hadoop.com/m/HBase/YGbbXd0RDCIHSC1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226516#comment-16226516 ] Anoop Sam John commented on HBASE-19092: What all CellUtil APIs we would need in LP exposed util? Can u list down pls.. Ya I dont think we need a getter in Env for CellUtil stuff. Public exposed CellUtil is any way a Util class and LP users also might need them. Why to have a diff way for the extra exposing for CPs?But ya we would need an extends way for the Util classes as noted by some comments in this jira (Or the other one?) ie. PrivateCellUtil extends LPCellUtil LPCellUtil extends CellUtil. For this making the constructor to be at least package private . I think that is ok. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092_001-branch-2.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226491#comment-16226491 ] Anoop Sam John commented on HBASE-19092: bq.I think instead of LPTagUtil we could add all the static methods to Tag interface itself. Not sure how many such methods will come to Tag then. But seems ok bq.One way is create TagsUtil (notice the 's'). Have an static INSTANCE and getInstance() in that. We have a public CellUtil and a private one now right. We need a LP CellUtil for sure. This is to add things like matchingXXX methods for Cells. Or the LP CellUtil is just for Tags? If not just for Tags, I dont see any issue in adding tag's' related static methods to CellUtil.. This is about getting Tags from a Cell.. Like the matching methods, these APIs also act on a Cell. If we are going to have a LP CellUtil version, I would say, put the Tag's' related methods there.. Per tag related , ya we have it in TagUtil now. If all APIs there can become CP exposed, dont think any issue in exposing that. Adding to Tag interface also seems ok provided it wont pollute the interface. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092_001-branch-2.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19100) Missing break in catch block of InterruptedException in HRegion#waitForFlushesAndCompactions
[ https://issues.apache.org/jira/browse/HBASE-19100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226331#comment-16226331 ] Anoop Sam John commented on HBASE-19100: +1 > Missing break in catch block of InterruptedException in > HRegion#waitForFlushesAndCompactions > > > Key: HBASE-19100 > URL: https://issues.apache.org/jira/browse/HBASE-19100 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 19100.v1.txt, 19100.v1.txt > > > Over in HBASE-19072, Anoop reminded me that there was a missing break in > HRegion#waitForFlushesAndCompactions as well. > This issue would fix the defect. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226293#comment-16226293 ] Anoop Sam John commented on HBASE-18375: bq.FAILURE: Integrated in Jenkins build HBase-2.0 #467 (See https://builds.apache.org/job/HBase-2.0/467/) Ya seems this is already there in branch-2. Around 7th Sep. So which version? Alpha-3? [~stack]? > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02-branch-2.patch, > HBASE-18375-V02.patch, HBASE-18375-V03.patch, HBASE-18375-V04.patch, > HBASE-18375-V05.patch, HBASE-18375-V06.patch, HBASE-18375-V07.patch, > HBASE-18375-V08.patch, HBASE-18375-V09.patch, HBASE-18375-V10.patch, > HBASE-18375-V11.patch, HBASE-18375-branch-2.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
