Re: HConnection thread waiting on blocking queue indefinitely

Any possible solution for this? As the number of clusters are increasing we are hitting thread limit frequently. Any suggrestions? Regards, Mukund Murrali On Sat, Jul 25, 2015 at 4:39 PM, mukund murrali wrote: > Sorry for the delay. No it was not a client scanner

Re: HConnection thread waiting on blocking queue indefinitely

Sorry for the delay. No it was not a client scanner call. It happens during first region split with user thread hbase.hconnection.threads.max. It is a 6 node cluster with 4 RS, 1 Master and 1 backup master. All RS machines have 1TB HDD, 32GB RAM. And master machine have 500GB HDD, 16GB RAM.

Re: HConnection thread waiting on blocking queue indefinitely

Just curious to know, your client is getting stuck while using ClientSmallReversedScanner? Can you try using normal scanner and see how it goes? what's your cluster setup? # of nodes? hardware? On Sun, Jul 5, 2015 at 10:21 PM, mukund murrali mukundmurra...@gmail.com wrote: Anyone has update on

Re: HConnection thread waiting on blocking queue indefinitely

Anyone has update on this? We really want to restrict the number of threads (hbase.hconnection.threads.max) to a minimum value. But, this issue seems to come up in that case. Also no response for JIRA too :( Restricting the threads not advisably? If so such a client stall can still happen if

Re: HConnection thread waiting on blocking queue indefinitely

Hi, We are still facing this issue in production. Any help is appreciated. Thanks, Praneesh On Mon, Jun 22, 2015 at 11:19 AM, mukund murrali mukundmurra...@gmail.com wrote: Hi All I have moved this as jira. https://issues.apache.org/jira/browse/HBASE-13942 Please post all your

Re: HConnection thread waiting on blocking queue indefinitely

I was out of the country this past week where access to gmail was difficult. Looking at client stack trace, it seems the hang corresponded to the following: at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:145) at

Re: HConnection thread waiting on blocking queue indefinitely

Hi All I have moved this as jira. https://issues.apache.org/jira/browse/HBASE-13942 Please post all your opinions there. Thanks On Mon, Jun 22, 2015 at 10:53 AM, Ted Yu yuzhih...@gmail.com wrote: I was out of the country this past week where access to gmail was difficult. Looking at

Fwd: HConnection thread waiting on blocking queue indefinitely

Even with 1.1.0 the issue persists. Client side blocking wait still happens during first region split. Tried in distributed set up with 1.0.0 as suggested by you and had the same results. Client jstack - http://pastebin.com/Ptw0JhdG RS Hosting Table Log - http://pastebin.com/ZSD4YUE5 One point

HConnection thread waiting on blocking queue indefinitely

Hi Ted, Even in hbase 1.1.0, the client connection stalls when region split occurs. Thread dump of master http://pastebin.com/rTFyuAqC Thread dump of RS http://pastebin.com/Vf52Z1ni Client waits in same BoundedCompletionService.take call Regards, Praneesh On Thursday 11 June 2015, Ted Yu

Re: HConnection thread waiting on blocking queue indefinitely

Looking at the revision history for ClientSmallReversedScanner.java which appeared in the stack trace, there have been several bug fixes on top of the hbase release you're using. Can you try hbase 1.1.0 to see if the problem can be reproduced (in cluster deployment) ? Thanks On Tue, Jun 9, 2015

Re: HConnection thread waiting on blocking queue indefinitely

Can you pastebin the complete stack trace for the region server ? Thanks On Jun 9, 2015, at 10:52 PM, mukund murrali mukundmurra...@gmail.com wrote: We are using HBase-1.0.0. Just before the client stalled, in RS there were few handler threads that were blocked for MVCC(thread stack

Re: HConnection thread waiting on blocking queue indefinitely

Kindly look into this for full trace of RS. http://pastebin.com/VS17vVd8 Thanks On Wed, Jun 10, 2015 at 11:35 AM, Ted Yu yuzhih...@gmail.com wrote: Can you pastebin the complete stack trace for the region server ? Thanks On Jun 9, 2015, at 10:52 PM, mukund murrali

Re: HConnection thread waiting on blocking queue indefinitely

Hi I wrote a sample program with default client configurations and created a single connection. I spawn client threads hbase.hconnection.threads.max from my client application and each thread insert data to hbase cluster. Once a region split happens, all the hconnection threads(core pool and max

Re: HConnection thread waiting on blocking queue indefinitely

We are using HBase-1.0.0. Just before the client stalled, in RS there were few handler threads that were blocked for MVCC(thread stack below) check. Not sure if it could cause a problem. I don't see anything unusual in RS threads. Also the same client can connect to regionserver after restart. At

Re: HConnection thread waiting on blocking queue indefinitely

Can you see at this time, what the threads at RS doing? Handlers mainly.. which version oh hbase? On Tuesday, June 9, 2015, mukund murrali mukundmurra...@gmail.com wrote: Hi I wrote a sample program with default client configurations and created a single connection. I spawn client threads

Re: HConnection thread waiting on blocking queue indefinitely

Sorry for misleading by specifying it as meta split. It was meta update during a user region split. This had caused the stallation probably. We have right now reverting client configs. Till now we didn't face the issue again. Those changes causing some kindof exceptions or timeout was what we

Re: HConnection thread waiting on blocking queue indefinitely

Not very sure on what could be the problem when the meta update happened. I would think that when the region split happened, there was some issue on the meta update (as you said in the later mail). The splitted regions would not have been updated properly in the META. So any client updates/reads

Re: HConnection thread waiting on blocking queue indefinitely

When you started your client server was the META table assigned. May be some thing happened around that time and the client app was just waiting on the meta table to be assigned. It would have retried - Can you check the logs.? So the best part here is the stand alone client was able to be

Re: HConnection thread waiting on blocking queue indefinitely

Hi In our case there at that instance when the client thread stalled, there was a hbase:meta region split happening. So what went wrong? If there is a split why should hconnection thread stall? Since we changed the client configuration caused this? I am once again specifying our client related

Re: HConnection thread waiting on blocking queue indefinitely

Hbase:meta getting split? It may b some user region, can u check that? If ur meta was splitting then there is something wrong. Can u attach the log snippets. Sent from phone. Excuse typos. On Jun 5, 2015 6:00 PM, mukund murrali mukundmurra...@gmail.com wrote: Hi In our case there at that

Re: HConnection thread waiting on blocking queue indefinitely

I would suggest reverting client config changes back to defaults. At least we will know if the issue is somehow related to client config changes. On Jun 5, 2015 6:15 AM, ramkrishna vasudevan ramkrishna.s.vasude...@gmail.com wrote: Hbase:meta getting split? It may b some user region, can u check

Re: HConnection thread waiting on blocking queue indefinitely

Hi Ram, The cluster ran without any problem for about 2 to 3 days with low load, once we enabled it for high load we immediately faced this issue. Regards, Praneesh. On Thursday 4 June 2015, ramkrishna vasudevan ramkrishna.s.vasude...@gmail.com wrote: Is your cluster in working condition.

Re: HConnection thread waiting on blocking queue indefinitely

The client connection was in stalled state. But there was only one hconnection thread found in our thread dump, which was waiting indefinitely in BoundedCompletionService.take call. Meanwhile we ran a standalone test program which was successful. Once we restarted the client server, the problem

Re: HConnection thread waiting on blocking queue indefinitely

Can you tell us more. Is your client not working at all and it is stalled ? Are you seeing some results but you find it slow than you expected? What type of workload are you running? All the tables are healthy? Are you able to read or write to them individually using the hbase shell? On Fri,

Re: HConnection thread waiting on blocking queue indefinitely

Is your cluster in working condition. Can you see if the META has been assigned properly? If the META table is not initialized and opened then your client thread will hang. Regards Ram On Thu, Jun 4, 2015 at 9:05 PM, PRANEESH KUMAR praneesh.san...@gmail.com wrote: Hi, We are using

HConnection thread waiting on blocking queue indefinitely

Hi, We are using Hbase-1.0.0. We also facing the same issue that client connection thread is waiting at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1200). Any help is appreciated. Regards, Praneesh

Re: HConnection thread waiting on blocking queue indefinitely

Hi We still facing this issue. Any one can help us out of this. Thanks Regards Mukund Murrali

Re: HConnection thread waiting on blocking queue indefinitely

Hi Ted We are using hbase-1.0.0. The issue seems to suddenly arise. It does not happen during startup of client process. The servers hosting hbase:meta seems normal during the issue. We have modified some of the hbase client properties apart from max and core threads like

Re: HConnection thread waiting on blocking queue indefinitely

Which release of hbase are you using ? Is the server hosting hbase:meta operating normally ? Thanks On Tue, Jun 2, 2015 at 4:29 AM, Meeran mohamed.mohid...@zohocorp.com wrote: Hi All, HBase client waits indefinitely for performing any operation. On taking, jstack we saw all the threads

HConnection thread waiting on blocking queue indefinitely

Hi All, HBase client waits indefinitely for performing any operation. On taking, jstack we saw all the threads waiting on BoundedCompletionService.take() call. The hconnection pool's stack is hconnection-0x58ca857d-shared--pool1-t96 daemon prio=10 tid=0x7f29d4510800 nid=0x84d0

HConnection thread waiting on blocking queue indefinitely

Hi All, HBase client waits indefinitely for performing any operation. On taking, jstack we saw all the threads waiting on BoundedCompletionService.take() call. Regards, Meeran.​