How many Thrift servers did you start ? The log snippet was INFO logs which didn't give us more clue.
Can you pastebin more of the logs (from Thrift servers) ? Thanks On Wed, Mar 8, 2017 at 11:20 AM, Motty Cruz <motty.c...@gmail.com> wrote: > Hello, in the past three weeks I have seen the following errors on HBase, > the Thrift servers stop talking to each other. Anybody seen this? > > > > Mar 8, 5:31:49.660 AM INFO org.apache.hadoop.hbase. > client.AsyncProcess > > #7147032, waiting for some tasks to finish. Expected max=0, > tasksInProgress=25 > > Mar 8, 5:31:49.662 AM INFO org.apache.hadoop.hbase. > client.AsyncProcess > > #7143730, table=LogSum, attempt=32/35 succeeded on h24-node.fqdn.com > ,60020,1474305457742, tracking started Wed Mar 08 03:52:39 PST 2017 > > Mar 8, 5:31:49.662 AM INFO org.apache.hadoop.hbase. > client.AsyncProcess > > #7143730, waiting for some tasks to finish. Expected max=0, > tasksInProgress=21 > > > > Any ideas? > > > > Thanks, > Motty > > > > > > From: Ravi Prakash [mailto:ravihad...@gmail.com] > Sent: Tuesday, March 07, 2017 11:32 AM > To: Motty Cruz > Cc: user > Subject: Re: last exception: java.io.IOException: Call to > e26-node.fqdn.com/10.12.1.209:60020 failed on local exception > > > > You should probably email Hbase mailing lists rather than Hadoop : > https://hbase.apache.org/mail-lists.html > > > > On Thu, Mar 2, 2017 at 10:02 AM, Motty Cruz <motty.c...@gmail.com> wrote: > > Hello, in the past two weeks, I see the following error on HBase Thrift > servers, we have total of about 10 Thrift servers and randomly get the > following errors: > > > > 2017-02-28 10:45:56,541 INFO org.apache.hadoop.hbase.client.AsyncProcess: > #4087940, table=MBData, attempt=11/35 failed=2ops, last exception: ja > > va.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on > local exception: org.apache.hadoop.hbase.ipc.CallTimeoutExcepti > > on: Call id=36009061, waitTime=180001, operationTimeout=180000 expired. on > e26-node.fqdn.com,60020,1487800633128, tracking started null, > > retrying after=10060ms, replay=2ops > > 2017-02-28 10:45:57,207 INFO org.apache.hadoop.hbase.client.AsyncProcess: > #4084675, waiting for some tasks to finish. Expected max=0, tasksInPro > > gress=22 > > > > Restarting the Thrift server resolves the issue. Any ideas where should I > be looking for? > > > > Thanks, > Motty > > > >
