Ok… So… I see you followed up to your earlier post.
Lets walk through this to make sure we're on the same page. You put() row 1 in table 1. The post_put() wants to insert a value in table 2. Now suppose I have an htable pool of connections in my Region Observer. (Is this overkill? I'm treating the RO as if its a client connecting to HBase.) RO performs a put() into the second table. The RPC handlers are a Region server resource, yes? So I can always increase them from the default (10) … but the point is that I can have 10 clients updating table A and then have a couple of different regions on the RS for the table making RO requests. Is that the case? On Oct 10, 2013, at 2:23 PM, Vladimir Rodionov <[email protected]> wrote: > Nope. It is not so obvious, but definitely the anti-pattern is still there. > > Each RPC call is served by a thread from RPC-handlers pool (which is 10? by > default). > > Making RPC call from within handler's therad is: > > A. expensive > B. may result in some deadlock -type of situations when no more incoming RPC > calls can be accepted because > everyone is connected to everyone in a spaghetti way and waiting on RPC calls > to complete > > Let us say we have 2 region servers for simplicity: > RS1 and RS2. Both have pool of handler threads = 10 > > What happen when all 10 handlers in RS1 are trying to RPC RS2 and all 10 > handlers are trying to RPC RS2? > > The same deadlock. Its all probabilistic . > > Best regards, > Vladimir Rodionov > Principal Platform Engineer > Carrier IQ, www.carrieriq.com > e-mail: [email protected] > > ________________________________________ > From: Vladimir Rodionov > Sent: Thursday, October 10, 2013 12:09 PM > To: [email protected] > Subject: RE: Coprocessor Increments > > Classical deadlock : > > CP-Region1 updates counter in CP-Region2 (waits on RPC) > CP-Region2 updates counter in CP-Region1 (waits on RPC) > > I think its an anti-pattern. Do not do cross region calls in region CP code. > > Best regards, > Vladimir Rodionov > Principal Platform Engineer > Carrier IQ, www.carrieriq.com > e-mail: [email protected] > > ________________________________________ > From: Michael Segel [[email protected]] > Sent: Thursday, October 10, 2013 9:55 AM > To: [email protected] > Cc: Ted Yu > Subject: Re: Coprocessor Increments > > I think Andrew has a handle on it… my take is that you end up running out of > resources to handle an RPC connection while within your coprocessor and > you're waiting for a resource to be free and it can't because another > coprocessor has an RPC resource and is also waiting for a free resource. > > Maybe its an over simplification, but if that's the case… you could always > try thing to limit the RPC call, which would delay updating the counter. > (Which may not be a problem) or redesign the coprocessors so that the > coprocessors don't share the same RPC resources. > > But the key is to really understanding and confirming what's causing the > Deadlock in detail. > > On Oct 10, 2013, at 11:15 AM, John Weatherford > <[email protected]> wrote: > >> Michael, >> I would also really like to know how this issue is caused also. I can't even >> give a solid way to reproduce our deadlock. It *seems* to happen more under >> load, but nothing can be proved yet. While google-ing and looking for an >> answer I came across that old message post >> (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E). >> This seemed to line up with what we are doing, so we _hope_ this will be a >> fix for us, but we aren't entirely sure. >> >> >> >> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote: >>> Can we just take a quick pause… >>> >>> John you wrote the following: >>> "We have been running into an RPC deadlock issue on HBase and from >>> investigation, we believe the root of the issue is in us doing cross >>> region increments from a coprocessor. After some further searching and >>> reading over this >>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E> >>> we think that we can solve this by doing the increments locally on the >>> region. " >>> >>> Which goes back to some thing Andrew wrote concerning an RPC deadlock. >>> >>> Can either you or Andrew explain in detail what is meant by the RPC >>> deadlock? >>> >>> This goes back to rethink how to implement coprocessors. >>> >>> >>> On Oct 9, 2013, at 11:03 PM, John Weatherford >>> <[email protected] <mailto:[email protected]>> >>> wrote: >>> >>>> Thank you ted. I might have to rethink my key design to accomodate >>>> this. With this design and using the totals in keys as you suggested, >>>> I would have to scan the entire "California" group to find the >>>> totals. Thank you very much for your help. >>>> >>>> -John >>>> >>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote: >>>>> John: >>>>> Suppose 'California-**12346' is the start key of region 1 and >>>>> 'California-** >>>>> 95424' is the start key of region 2. You can choose >>>>> 'California-**12346#total' >>>>> to be the row key where increment is done for region 1 and >>>>> 'California-**95424#total' >>>>> to be the row key where increment is done for region 2. >>>>> >>>>> w.r.t. verification of whether given row key is indeed inside the >>>>> hosting >>>>> region, see the following: >>>>> >>>>> void checkRow(final byte [] row, String op) throws IOException { >>>>> >>>>> if (!rowIsInRange(getRegionInfo(), row)) { >>>>> >>>>> throw new WrongRegionException("Requested row out of range for " + >>>>> >>>>> op + " on HRegion " + this + ", startKey='" + >>>>> >>>>> Cheers >>>>> >>>>> >>>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford < >>>>> [email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>>> First, Thank you everyone for the quick response. >>>>>> >>>>>> Ted: >>>>>> The custom split policy could be an interesting solution. >>>>>> Regarding the >>>>>> other option you sent, if I pull the end row key, I could construct >>>>>> an end >>>>>> key, but if I suffix the row key with the end key of the region, >>>>>> would that >>>>>> actually solve my problem? In the contrived case, wouldn't the >>>>>> suffixed key >>>>>> now be "California-total-California-**12346" and the other region's >>>>>> counter would be "California-total-California-**95424" and both of >>>>>> these >>>>>> keys would actually end up on the second region since the sorting >>>>>> of the >>>>>> keys would impose that "California-t*" comes after "California-[1-9]". >>>>>> >>>>>> Part of the question is that I don't understand if calling >>>>>> incrementColumnValue() on the region will always execute on the region >>>>>> called, regardles of rowkey. If so, what happens when those regions are >>>>>> merged? >>>>>> >>>>>> Thanks again for the help! >>>>>> >>>>>> >>>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote: >>>>>> >>>>>>> bq. 'California-total' row in each region >>>>>>> >>>>>>> Of course such row key needs to be suffixed with either start or >>>>>>> end key >>>>>>> of >>>>>>> the corresponding region. >>>>>>> >>>>>>> >>>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> John: >>>>>>>> Looks like you need a 'California-total' row in each region. >>>>>>>> >>>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in: >>>>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541> >>>>>>>> >>>>>>>> Cheers >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov < >>>>>>>> [email protected] <mailto:[email protected]> >>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>> >>>>>>>> Contrived Example >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call >>>>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all on >>>>>>>>>>> >>>>>>>>>> Region 1. >>>>>>>>> >>>>>>>>> This would likely be on an insert on the same region. But as >>>>>>>>> the table >>>>>>>>>>> grows, this secondary insert could end up on another region. >>>>>>>>>>> If it is >>>>>>>>>>> confined, then suppose we later insert "California-95424" >>>>>>>>>>> which still >>>>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of >>>>>>>>>>> "California-total" all on Region 2. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> Are we now left with two rowkeys of "California-total"? One on each >>>>>>>>>>> region server? If so, what happens if these two regions are >>>>>>>>>>> compacted >>>>>>>>>>> into one? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> Nope, Your "California-total" will migrate to Region 2 after region >>>>>>>>> split >>>>>>>>> is complete. >>>>>>>>> >>>>>>>>> Vladimir Rodionov >>>>>>>>> Principal Platform Engineer >>>>>>>>> Carrier IQ, www.carrieriq.com >>>>>>>>> e-mail: [email protected] >>>>>>>>> >>>>>>>>> ______________________________**__________ >>>>>>>>> >>>>>>>>> Confidentiality Notice: The information contained in this message, >>>>>>>>> including any attachments hereto, may be confidential and is >>>>>>>>> intended >>>>>>>>> to be >>>>>>>>> read only by the individual or entity to whom this message is >>>>>>>>> addressed. If >>>>>>>>> the reader of this message is not the intended recipient or an >>>>>>>>> agent or >>>>>>>>> designee of the intended recipient, please note that any review, >>>>>>>>> use, >>>>>>>>> disclosure or distribution of this message or its attachments, >>>>>>>>> in any >>>>>>>>> form, >>>>>>>>> is strictly prohibited. If you have received this message in error, >>>>>>>>> please >>>>>>>>> immediately notify the sender and/or [email protected] and >>>>>>>>> delete or destroy any copy of this message and its attachments. >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>> >> > > > Confidentiality Notice: The information contained in this message, including > any attachments hereto, may be confidential and is intended to be read only > by the individual or entity to whom this message is addressed. If the reader > of this message is not the intended recipient or an agent or designee of the > intended recipient, please note that any review, use, disclosure or > distribution of this message or its attachments, in any form, is strictly > prohibited. If you have received this message in error, please immediately > notify the sender and/or [email protected] and delete or destroy > any copy of this message and its attachments. >
smime.p7s
Description: S/MIME cryptographic signature
