Any possible solution for this? As the number of clusters are increasing we
are hitting thread limit frequently. Any suggrestions?
Regards,
Mukund Murrali
On Sat, Jul 25, 2015 at 4:39 PM, mukund murrali
wrote:
> Sorry for the delay. No it was not a client scanner call. It happens
> during firs
Sorry for the delay. No it was not a client scanner call. It happens during
first region split with user thread > hbase.hconnection.threads.max.
It is a 6 node cluster with 4 RS, 1 Master and 1 backup master.
All RS machines have 1TB HDD, 32GB RAM. And master machine have 500GB HDD,
16GB RAM.
Al
Just curious to know, your client is getting stuck while using
ClientSmallReversedScanner?
Can you try using normal scanner and see how it goes?
what's your cluster setup? # of nodes? hardware?
On Sun, Jul 5, 2015 at 10:21 PM, mukund murrali
wrote:
> Anyone has update on this? We really want to
Anyone has update on this? We really want to restrict the number of threads
(hbase.hconnection.threads.max) to a minimum value. But, this issue seems
to come up in that case. Also no response for JIRA too :( Restricting the
threads not advisably? If so such a client stall can still happen if number
Hi,
We are still facing this issue in production.
Any help is appreciated.
Thanks,
Praneesh
On Mon, Jun 22, 2015 at 11:19 AM, mukund murrali
wrote:
> Hi All
>
> I have moved this as jira.
>
> https://issues.apache.org/jira/browse/HBASE-13942
>
> Please post all your opinions there.
>
> Thank
Hi All
I have moved this as jira.
https://issues.apache.org/jira/browse/HBASE-13942
Please post all your opinions there.
Thanks
On Mon, Jun 22, 2015 at 10:53 AM, Ted Yu wrote:
> I was out of the country this past week where access to gmail was
> difficult.
>
> Looking at client stack trace,
I was out of the country this past week where access to gmail was difficult.
Looking at client stack trace, it seems the hang corresponded to the
following:
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:145)
at
org.apache.hadoop.h
Looking at the revision history for ClientSmallReversedScanner.java which
appeared in the stack trace, there have been several bug fixes on top of
the hbase release you're using.
Can you try hbase 1.1.0 to see if the problem can be reproduced (in cluster
deployment) ?
Thanks
On Tue, Jun 9, 2015
Kindly look into this for full trace of RS.
http://pastebin.com/VS17vVd8
Thanks
On Wed, Jun 10, 2015 at 11:35 AM, Ted Yu wrote:
> Can you pastebin the complete stack trace for the region server ?
>
> Thanks
>
>
>
> > On Jun 9, 2015, at 10:52 PM, mukund murrali
> wrote:
> >
> > We are using HBa
Can you pastebin the complete stack trace for the region server ?
Thanks
> On Jun 9, 2015, at 10:52 PM, mukund murrali wrote:
>
> We are using HBase-1.0.0. Just before the client stalled, in RS there were
> few handler threads that were blocked for MVCC(thread stack below) check.
> Not sure
We are using HBase-1.0.0. Just before the client stalled, in RS there were
few handler threads that were blocked for MVCC(thread stack below) check.
Not sure if it could cause a problem. I don't see anything unusual in RS
threads. Also the same client can connect to regionserver after restart. At
Can you see at this time, what the threads at RS doing? Handlers mainly..
which version oh hbase?
On Tuesday, June 9, 2015, mukund murrali wrote:
> Hi
>
> I wrote a sample program with default client configurations and created a
> single connection. I spawn client threads > hbase.hconnection.thre
Hi
I wrote a sample program with default client configurations and created a
single connection. I spawn client threads > hbase.hconnection.threads.max
from my client application and each thread insert data to hbase cluster.
Once a region split happens, all the hconnection threads(core pool and max
Not very sure on what could be the problem when the meta update happened.
I would think that when the region split happened, there was some issue on
the meta update (as you said in the later mail). The splitted regions would
not have been updated properly in the META. So any client updates/reads
h
Sorry for misleading by specifying it as meta split. It was meta update
during a user region split. This had caused the stallation probably. We
have right now reverting client configs. Till now we didn't face the issue
again. Those changes causing some kindof exceptions or timeout was what we
expec
I would suggest reverting client config changes back to defaults. At least
we will know if the issue is somehow related to client config changes.
On Jun 5, 2015 6:15 AM, "ramkrishna vasudevan" <
ramkrishna.s.vasude...@gmail.com> wrote:
> Hbase:meta getting split? It may b some user region, can u c
Hbase:meta getting split? It may b some user region, can u check that? If
ur meta was splitting then there is something wrong.
Can u attach the log snippets.
Sent from phone. Excuse typos.
On Jun 5, 2015 6:00 PM, "mukund murrali" wrote:
> Hi
>
> In our case there at that instance when the client
Hi
In our case there at that instance when the client thread stalled, there
was a hbase:meta region split happening. So what went wrong? If there is a
split why should hconnection thread stall? Since we changed the client
configuration caused this? I am once again specifying our client related
cha
When you started your client server was the META table assigned. May be
some thing happened around that time and the client app was just waiting on
the meta table to be assigned. It would have retried - Can you check the
logs.?
So the best part here is the stand alone client was able to be succ
The client connection was in stalled state. But there was only one
hconnection thread found in our thread dump, which was waiting indefinitely
in BoundedCompletionService.take call. Meanwhile we ran a standalone test
program which was successful.
Once we restarted the client server, the problem go
Can you tell us more. Is your client not working at all and it is stalled ?
Are you seeing some results but you find it slow than you expected?
What type of workload are you running? All the tables are healthy? Are
you able to read or write to them individually using the hbase shell?
On Fri, Ju
Hi Ram,
The cluster ran without any problem for about 2 to 3 days with low load,
once we enabled it for high load we immediately faced this issue.
Regards,
Praneesh.
On Thursday 4 June 2015, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:
> Is your cluster in working condition.
Is your cluster in working condition. Can you see if the META has been
assigned properly? If the META table is not initialized and opened then
your client thread will hang.
Regards
Ram
On Thu, Jun 4, 2015 at 9:05 PM, PRANEESH KUMAR
wrote:
> Hi,
>
> We are using Hbase-1.0.0. We also facing the
Hi
We still facing this issue. Any one can help us out of this.
Thanks
Regards
Mukund Murrali
Hi Ted
We are using hbase-1.0.0.
The issue seems to suddenly arise.
It does not happen during startup of
client process.
The servers hosting hbase:meta seems normal during the issue.
We have modified some of the hbase client properties
apart from max and core threads
like
hbase.client.retries.n
Which release of hbase are you using ?
Is the server hosting hbase:meta operating normally ?
Thanks
On Tue, Jun 2, 2015 at 4:29 AM, Meeran
wrote:
> Hi All,
>
>
>
> HBase client waits indefinitely for performing any operation. On taking,
> jstack we saw all the threads waiting on BoundedComple
26 matches
Mail list logo
