Go to CDH3 if you can. CDH2 is also old. St.Ack
On Wed, Jun 29, 2011 at 7:15 AM, Srikanth P. Shreenivas <[email protected]> wrote: > Thanks St. Ack for the inputs. > > Will upgrading to CDH3 help or is there a version within CDH2 that you > recommend we should upgrade to? > > Regards, > Srikanth > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of Stack > Sent: Wednesday, June 29, 2011 11:16 AM > To: [email protected] > Subject: Re: HBase Read and Write Issues in Mutlithreaded Environments > > Can you upgrade? That release is > 18 months old. A bunch has > happened in the meantime. > > For retries exhausted, check whats going on on the remote regionserver > that you are trying to write too. Its probably struggling and thats > why requests are not going through -- or the client missed the fact > that region moved (all stuff that should be working better in latest > hbase). > > St.Ack > > On Tue, Jun 28, 2011 at 9:51 PM, Srikanth P. Shreenivas > <[email protected]> wrote: >> Hi, >> >> We are using HBase 0.20.3 (hbase-0.20-0.20.3-1.cloudera.noarch.rpm) cluster >> in distributed mode with Hadoop 0.20.2 (hadoop-0.20-0.20.2+320-1.noarch). >> We are using pretty much default configuration, and only thing we have >> customized is that we have allocated 4GB RAM in >> /etc/hbase-0.20/conf/hbase-env.sh >> >> In our setup, we have a web application that reads a record from HBase and >> writes a record as part of each web request. The application is hosted in >> Apache Tomcat 7 and is a stateless web application providing a REST-like web >> service API. >> >> We are observing that our reads and writes times out once in a while. This >> happens more for writes. >> We see below exception in our application logs: >> >> >> Exception Type 1 - During Get: >> --------------------------------------- >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact >> region server 10.1.68.36:60020 for region >> employeedata,be8784ac8b57c45625a03d52be981b88097c2fdc,1308657957879, row >> 'd51b74eb05e07f96cee0ec556f5d8d161e3281f3', but failed after 10 attempts. >> Exceptions: >> java.io.IOException: Call to /10.1.68.36:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> java.nio.channels.ClosedByInterruptException >> java.nio.channels.ClosedByInterruptException >> java.nio.channels.ClosedByInterruptException >> java.nio.channels.ClosedByInterruptException >> java.nio.channels.ClosedByInterruptException >> java.nio.channels.ClosedByInterruptException >> java.nio.channels.ClosedByInterruptException >> java.nio.channels.ClosedByInterruptException >> java.nio.channels.ClosedByInterruptException >> >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048) >> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:417) >> <snip> >> >> Exception Type 2 - During Put: >> --------------------------------------------- >> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying >> to contact region server 10.1.68.34:60020 for region >> audittable,,1309183872019, row '2a012017120f80a801b28f5f66a83dc2a8882d1b', >> but failed after 10 attempts. >> Exceptions: >> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> java.io.IOException: Call to /10.1.68.34:60020 failed on local exception: >> java.nio.channels.ClosedByInterruptException >> >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1048) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3.doCall(HConnectionManager.java:1239) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1161) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1247) >> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:609) >> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:474) >> <snip> >> >> Any inputs on why this is happening, or how to rectify it will be of immense >> help. >> >> Thanks, >> Srikanth >> >> >> >> Srikanth P Shreenivas|Principal Consultant | MindTree Ltd.|Global Village, >> RVCE Post, Mysore Road, Bangalore-560 059, INDIA|Voice +91 80 26264000 / Fax >> +91 80 2626 4100| Mob: 9880141059|email: >> [email protected]<mailto:[email protected]> >> |www.mindtree.com<http://www.mindtree.com/> | >> >> >> ________________________________ >> >> http://www.mindtree.com/email/disclaimer.html >> >
