Hi Lucas, This is good to hear. Thanks for writing back to let us know.
My understanding is Clint will look into issues regarding THBase and ITHBase and log splitting soon. - Andy ________________________________ From: Lucas Nazário dos Santos <nazario.lu...@gmail.com> To: hbase-user@hadoop.apache.org Sent: Friday, September 18, 2009 6:28:16 AM Subject: Re: HBase cracks on EC2 Just to let you know, I followed Andrew's suggestion of using a cluster with three EC2 instances and HBase has been running smoothly for more than one day with five map reduce jobs continuously adding and retrieving data. In previous tests HBase would go down in a couple of minutes. Anyway, I'm still a bit scared about the error java.io.IOException: wrong key class: org.apache.hadoop.hbase.regionserver.HLogKey is not class org.apache.hadoop.hbase.regionserver.transactional.THLogKey Lucas On Fri, Sep 18, 2009 at 7:23 AM, Lucas Nazário dos Santos < nazario.lu...@gmail.com> wrote: > Thanks, Clint. > > Lucas > > > > > On Thu, Sep 17, 2009 at 7:28 PM, Clint Morgan <clint.mor...@troove.net>wrote: > >> Looks like a case that is not handled by the ITHbase: if the regionserver >> dies, master tries to split his logs but its not configured to use the >> THLogKeys... >> >> Should be easy enough to fix, I'll try to look into it next week. But yeah >> I >> guess this is not the *root* problem because its caused by regionserver >> failure, but a case that ITHBase should handle.. >> >> -clint >> >> On Thu, Sep 17, 2009 at 4:25 AM, Lucas Nazário dos Santos < >> nazario.lu...@gmail.com> wrote: >> >> > Thanks, Clint. In fact, I've enabled the indexing (and not transactions) >> > with the following properties inside the hbase-site.xml file as >> described >> > in >> > >> > >> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/client/tableindexed/package-summary.html >> > . >> > >> > <property> >> > <name>hbase.regionserver.class</name> >> > <value>org.apache.hadoop.hbase.ipc.IndexedRegionInterface</value> >> > </property> >> > <property> >> > <name>hbase.regionserver.impl</name> >> > >> > >> > >> <value>org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer</value> >> > </property> >> > >> > Any clue on what should I do to solve the "wrong key class" exception? >> > >> > Lucas >> > >> > >> > >> > On Wed, Sep 16, 2009 at 8:25 PM, Andrew Purtell <apurt...@apache.org> >> > wrote: >> > >> > > Good catch! It looks like there is more than one problem here. >> > > >> > > - Andy >> > > >> > > >> > > >> > > >> > > ________________________________ >> > > From: Clint Morgan <clint.mor...@troove.net> >> > > To: hbase-user@hadoop.apache.org >> > > Sent: Wednesday, September 16, 2009 2:43:22 PM >> > > Subject: Re: HBase cracks on EC2 >> > > >> > > This looks like the problem to me: >> > > >> > > java.io.IOException: wrong key class: >> > > org.apache.hadoop.hbase.regionserver.HLogKey is not class >> > > org.apache.hadoop.hbase.regionserver.transactional.THLogKey >> > > at >> > > >> > > Have you enable transactional hbase? Looks like you are using regular >> > HBase >> > > to recover a THBase log... >> > > >> > > >> > > On Wed, Sep 16, 2009 at 11:32 AM, Lucas Nazário dos Santos < >> > > nazario.lu...@gmail.com> wrote: >> > > >> > > > Thanks, Andy. I'll give it a shot. >> > > > >> > > > >> > > > On Wed, Sep 16, 2009 at 3:18 PM, Andrew Purtell < >> apurt...@apache.org> >> > > > wrote: >> > > > >> > > > > Hi Lucas, >> > > > > >> > > > > The log fragment you sent is actually after the problem occurred, >> and >> > > it >> > > > > does not look like 1777 but there's really insufficient >> information >> > to >> > > > say >> > > > > for sure. >> > > > > >> > > > > I find that the hypervisor steals 20-50% of CPU time on m1.large >> or >> > > > > m1.xlarge instance types as reported by top. I don't recommend any >> > > > instance >> > > > > type other than c1.xlarge (0% stolen as reported by top). Also I >> > would >> > > > > recommend a cluster of 3 c1.xlarge instances to start. Also please >> > have >> > > a >> > > > > look at the troubleshooting wiki and be sure to follow the advice >> for >> > > > items >> > > > > #5, #6, and #7. >> > > > > >> > > > > Best regards, >> > > > > >> > > > > - Andy >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > ________________________________ >> > > > > From: Lucas Nazário dos Santos <nazario.lu...@gmail.com> >> > > > > To: hbase-user@hadoop.apache.org >> > > > > Sent: Wednesday, September 16, 2009 10:48:53 AM >> > > > > Subject: Re: HBase cracks on EC2 >> > > > > >> > > > > Thanks, Andy. >> > > > > >> > > > > It's only a test with one large instance (7.5Gb of memory and 4 >> EC2 >> > > > compute >> > > > > units). I've already tried a cluster with 3 small instances but >> the >> > > same >> > > > > error occurred (I know small instances are not recommended and >> that >> > is >> > > > why >> > > > > I'm trying large instances now). Can it be the number of instances >> > I'm >> > > > > using? >> > > > > >> > > > > Hadoop logs are clear. >> > > > > >> > > > > The line bellow caught my attention. I'm wondering if the problem >> can >> > > be >> > > > > related to the issue >> http://issues.apache.org/jira/browse/HBASE-1777 >> > . >> > > > > >> > > > > 2009-09-16 09:02:59,846 FATAL >> > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: >> > > > > Replay of hlog required. Forcing server shutdown >> > > > > java.io.IOException: Server not running, aborting >> > > > > at org.apache.hadoop.hbase. >> > > > > regionserver.HRegionServer.checkOpen(HRegionServer.java:2299) >> > > > > at >> > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.regionserver.HRegionServer.exists(HRegionServer.java:1775) >> > > > > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown >> > Source) >> > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >> > > Source) >> > > > > at java.lang.reflect.Method.invoke(Unknown Source) >> > > > > at >> > > > > >> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) >> > > > > at >> > > > > >> > > >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) >> > > > > 2009-09-16 09:02:59,863 INFO >> > > > > org.apache.hadoop.hbase.regionserver.CompactSplitThread: >> > > > > regionserver/10.209.207.80:60020.compactor exiting >> > > > > >> > > > > Lucas >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > On Wed, Sep 16, 2009 at 1:37 PM, Andrew Purtell < >> apurt...@apache.org >> > > >> > > > > wrote: >> > > > > >> > > > > > Lucas, >> > > > > > >> > > > > > > [...] ServerManager: 0 region servers, 1 dead, average load >> > > > > > >> > > > > > Is this a pseudo distributed test? I.e. everything running on >> one >> > EC2 >> > > > > > instance? >> > > > > > >> > > > > > What instance type are you using? >> > > > > > >> > > > > > The reported exceptions indicate a possible problem with DFS. >> Are >> > > there >> > > > > > exceptions in the DataNode logs? Or the NameNode log? >> > > > > > >> > > > > > Best regards, >> > > > > > >> > > > > > - Andy >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > ________________________________ >> > > > > > From: Lucas Nazário dos Santos <nazario.lu...@gmail.com> >> > > > > > To: hbase-user@hadoop.apache.org >> > > > > > Sent: Wednesday, September 16, 2009 6:21:27 AM >> > > > > > Subject: HBase cracks on EC2 >> > > > > > >> > > > > > Hi, >> > > > > > >> > > > > > I'm trying to migrate my solution to Amazon EC2 but the >> following >> > > > errors >> > > > > > crack HBase. I'm using HBase and Hadoop 0.20.0. Does anybody >> know >> > > what >> > > > > may >> > > > > > be the cause of it? >> > > > > > >> > > > > > Lucas >> > > > > > >> > > > > > >> > > > > > >> > > > > > HBASE MASTER >> > > > > > 2009-09-16 09:02:41,271 INFO >> > > > > org.apache.hadoop.hbase.master.ServerManager: >> > > > > > 1 >> > > > > > region servers, 0 dead, average load 6.0 >> > > > > > 2009-09-16 09:02:41,453 INFO >> > > > org.apache.hadoop.hbase.master.BaseScanner: >> > > > > > RegionManager.rootScanner scanning meta region {server: >> > > > > > 10.209.207.80:60020, >> > > > > > regionname: -ROOT-,,0, startKey: <>} >> > > > > > 2009-09-16 09:02:41,472 INFO >> > > > org.apache.hadoop.hbase.master.BaseScanner: >> > > > > > RegionManager.rootScanner scan of 1 row(s) of meta region >> {server: >> > > > > > 10.209.207.80:60020, regionname: -ROOT-,,0, startKey: <>} >> complete >> > > > > > 2009-09-16 09:02:41,553 INFO >> > > > org.apache.hadoop.hbase.master.BaseScanner: >> > > > > > RegionManager.metaScanner scanning meta region {server: >> > > > > > 10.209.207.80:60020, >> > > > > > regionname: .META.,,1, startKey: <>} >> > > > > > 2009-09-16 09:02:41,604 INFO >> > > > org.apache.hadoop.hbase.master.BaseScanner: >> > > > > > RegionManager.metaScanner scan of 4 row(s) of meta region >> {server: >> > > > > > 10.209.207.80:60020, regionname: .META.,,1, startKey: <>} >> complete >> > > > > > 2009-09-16 09:02:41,605 INFO >> > > > org.apache.hadoop.hbase.master.BaseScanner: >> > > > > > All >> > > > > > 1 .META. region(s) scanned >> > > > > > 2009-09-16 09:03:01,905 INFO >> > > > > org.apache.hadoop.hbase.master.ServerManager: >> > > > > > domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743 >> znode >> > > > > expired >> > > > > > 2009-09-16 09:03:01,921 INFO >> > > > > org.apache.hadoop.hbase.master.RegionManager: >> > > > > > -ROOT- region unset (but not set to be reassigned) >> > > > > > 2009-09-16 09:03:01,921 INFO >> > > > > org.apache.hadoop.hbase.master.RegionManager: >> > > > > > META region removed from onlineMetaRegions >> > > > > > 2009-09-16 09:03:01,923 INFO >> > > > > > org.apache.hadoop.hbase.master.RegionServerOperation: process >> > > shutdown >> > > > of >> > > > > > server >> > domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743: >> > > > > > logSplit: false, rootRescanned: false, numberOfMetaRegions: 1, >> > > > > > onlineMetaRegions.size(): 0 >> > > > > > 2009-09-16 09:03:01,935 INFO >> > > org.apache.hadoop.hbase.regionserver.HLog: >> > > > > > Splitting 2 hlog(s) in >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> hdfs://domU-12-31-39-07-CC-A2.compute-1.internal:9000/hbase/.logs/domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743 >> > > > > > 2009-09-16 09:03:01,943 WARN >> > > org.apache.hadoop.hbase.regionserver.HLog: >> > > > > > Exception processing >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> hdfs://domU-12-31-39-07-CC-A2.compute-1.internal:9000/hbase/.logs/domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743/hlog.dat.1253103102168 >> > > > > > -- continuing. Possible DATA LOSS! >> > > > > > java.io.IOException: wrong key class: >> > > > > > org.apache.hadoop.hbase.regionserver.HLogKey is not class >> > > > > > org.apache.hadoop.hbase.regionserver.transactional.THLogKey >> > > > > > at >> > > > > > >> > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1824) >> > > > > > at >> > > > > > >> > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876) >> > > > > > at >> > > > > org.apache.hadoop.hbase.regionserver.HLog.splitLog(HLog.java:880) >> > > > > > at >> > > > > org.apache.hadoop.hbase.regionserver.HLog.splitLog(HLog.java:802) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.master.ProcessServerShutdown.process(ProcessServerShutdown.java:274) >> > > > > > at >> > > > > > >> > > > >> > >> org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:492) >> > > > > > at >> > > org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:426) >> > > > > > 2009-09-16 09:03:01,951 WARN >> > > org.apache.hadoop.hbase.regionserver.HLog: >> > > > > > Exception processing >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> hdfs://domU-12-31-39-07-CC-A2.compute-1.internal:9000/hbase/.logs/domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743/hlog.dat.1253105692784 >> > > > > > -- continuing. Possible DATA LOSS! >> > > > > > java.io.IOException: wrong key class: >> > > > > > org.apache.hadoop.hbase.regionserver.HLogKey is not class >> > > > > > org.apache.hadoop.hbase.regionserver.transactional.THLogKey >> > > > > > at >> > > > > > >> > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1824) >> > > > > > at >> > > > > > >> > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876) >> > > > > > at >> > > > > org.apache.hadoop.hbase.regionserver.HLog.splitLog(HLog.java:880) >> > > > > > at >> > > > > org.apache.hadoop.hbase.regionserver.HLog.splitLog(HLog.java:802) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.master.ProcessServerShutdown.process(ProcessServerShutdown.java:274) >> > > > > > at >> > > > > > >> > > > >> > >> org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:492) >> > > > > > at >> > > org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:426) >> > > > > > 2009-09-16 09:03:01,956 INFO >> > > org.apache.hadoop.hbase.regionserver.HLog: >> > > > > > hlog >> > > > > > file splitting completed in 29 millis for >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> hdfs://domU-12-31-39-07-CC-A2.compute-1.internal:9000/hbase/.logs/domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743 >> > > > > > 2009-09-16 09:03:01,956 INFO >> > > > > > org.apache.hadoop.hbase.master.RegionServerOperation: Log split >> > > > complete, >> > > > > > meta reassignment and scanning: >> > > > > > 2009-09-16 09:03:01,956 INFO >> > > > > > org.apache.hadoop.hbase.master.RegionServerOperation: >> > > > > ProcessServerShutdown >> > > > > > reassigning ROOT region >> > > > > > 2009-09-16 09:03:01,956 INFO >> > > > > org.apache.hadoop.hbase.master.RegionManager: >> > > > > > -ROOT- region unset (but not set to be reassigned) >> > > > > > 2009-09-16 09:03:01,956 INFO >> > > > > org.apache.hadoop.hbase.master.RegionManager: >> > > > > > ROOT inserted into regionsInTransition >> > > > > > 2009-09-16 09:03:01,956 INFO >> > > > > > org.apache.hadoop.hbase.master.RegionServerOperation: >> > > > > ProcessServerShutdown >> > > > > > setting to unassigned: {server: 10.209.207.80:60020, >> regionname: >> > > > > > .META.,,1, >> > > > > > startKey: <>} >> > > > > > 2009-09-16 09:03:41,273 INFO >> > > > > org.apache.hadoop.hbase.master.ServerManager: >> > > > > > 0 >> > > > > > region servers, 1 dead, average load >> > > > > > >> NaN[domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743] >> > > > > > >> > > > > > >> > > > > > >> > > > > > HBASE REGIONSERVER >> > > > > > 2009-09-16 09:02:56,571 WARN org.apache.hadoop.hdfs.DFSClient: >> > > > > > NotReplicatedYetException sleeping >> > > > > > /hbase/news/698339833/charset/3497317581208054515 retries left 1 >> > > > > > 2009-09-16 09:02:59,844 WARN org.apache.hadoop.hdfs.DFSClient: >> > > > > DataStreamer >> > > > > > Exception: org.apache.hadoop.ipc.RemoteException: >> > > java.io.IOException: >> > > > > File >> > > > > > /hbase/news/698339833/charset/3497317581208054515 could only be >> > > > > replicated >> > > > > > to 0 nodes, instead of 1 >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1256) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422) >> > > > > > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown >> > > Source) >> > > > > > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >> > > > Source) >> > > > > > at java.lang.reflect.Method.invoke(Unknown Source) >> > > > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) >> > > > > > at >> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) >> > > > > > at >> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) >> > > > > > at java.security.AccessController.doPrivileged(Native >> > Method) >> > > > > > at javax.security.auth.Subject.doAs(Unknown Source) >> > > > > > at >> org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) >> > > > > > >> > > > > > at org.apache.hadoop.ipc.Client.call(Client.java:739) >> > > > > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) >> > > > > > at $Proxy1.addBlock(Unknown Source) >> > > > > > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown >> > Source) >> > > > > > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >> > > > Source) >> > > > > > at java.lang.reflect.Method.invoke(Unknown Source) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >> > > > > > at $Proxy1.addBlock(Unknown Source) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234) >> > > > > > >> > > > > > 2009-09-16 09:02:59,844 WARN org.apache.hadoop.hdfs.DFSClient: >> > Error >> > > > > > Recovery for block null bad datanode[0] nodes == null >> > > > > > 2009-09-16 09:02:59,844 WARN org.apache.hadoop.hdfs.DFSClient: >> > Could >> > > > not >> > > > > > get >> > > > > > block locations. Source file >> > > > > > "/hbase/news/698339833/charset/3497317581208054515" - >> Aborting... >> > > > > > 17581208054515" - Aborting... >> > > > > > 2009-09-16 09:02:59,846 FATAL >> > > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of >> > hlog >> > > > > > required. Forcing server shutdown >> > > > > > org.apache.hadoop.hbase.DroppedSnapshotException: region: >> > > > > > news,,1253103842893 >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:950) >> > > > > > at >> > > > > > >> > > > >> > >> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:843) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149) >> > > > > > Caused by: org.apache.hadoop.ipc.RemoteException: >> > > java.io.IOException: >> > > > > File >> > > > > > /hbase/news/698339833/charset/3497317581208054515 could only be >> > > > > replicated >> > > > > > to 0 nodes, instead of 1 >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1256) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422) >> > > > > > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown >> > > Source) >> > > > > > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >> > > > Source) >> > > > > > at java.lang.reflect.Method.invoke(Unknown Source) >> > > > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) >> > > > > > at >> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) >> > > > > > at >> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) >> > > > > > at java.security.AccessController.doPrivileged(Native >> > Method) >> > > > > > at javax.security.auth.Subject.doAs(Unknown Source) >> > > > > > at >> org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) >> > > > > > >> > > > > > at org.apache.hadoop.ipc.Client.call(Client.java:739) >> > > > > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) >> > > > > > at $Proxy1.addBlock(Unknown Source) >> > > > > > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown >> > Source) >> > > > > > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >> > > > Source) >> > > > > > at java.lang.reflect.Method.invoke(Unknown Source) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >> > > > > > at $Proxy1.addBlock(Unknown Source) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234) >> > > > > > 2009-09-16 09:02:59,849 INFO >> > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of >> > metrics: >> > > > > > request=0.0, regions=6, stores=27, storefiles=36, >> > > storefileIndexSize=0, >> > > > > > memstoreSize=0, usedHeap=195, maxHeap=996, >> blockCacheSize=82678352, >> > > > > > blockCacheFree=126368384, blockCacheCount=1168, >> > blockCacheHitRatio=99 >> > > > > > 2009-09-16 09:02:59,849 INFO >> > > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: >> > > > > > regionserver/10.209.207.80:60020.cacheFlusher exiting >> > > > > > 2009-09-16 09:02:59,849 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 1086 on 60020, call exists([...@466057, row= >> > > > > > http://www.bloomberg.com/news/commentary/marinis.html#, >> > > maxVersions=1, >> > > > > > timeRange=[0,9223372036854775807), families=ALL) from >> > > > > 10.209.207.80:47866: >> > > > > > error: java.io.IOException: Server not running, aborting >> > > > > > java.io.IOException: Server not running, aborting >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299) >> > > > > > at >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.regionserver.HRegionServer.exists(HRegionServer.java:1775) >> > > > > > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown >> > > Source) >> > > > > > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >> > > > Source) >> > > > > > at java.lang.reflect.Method.invoke(Unknown Source) >> > > > > > at >> > > > > > >> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) >> > > > > > at >> > > > > > >> > > > >> > >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) >> > > > > > 2009-09-16 09:02:59,863 INFO >> > > > > > org.apache.hadoop.hbase.regionserver.CompactSplitThread: >> > > > > > regionserver/10.209.207.80:60020.compactor exiting >> > > > > > 2009-09-16 09:03:00,027 INFO org.apache.hadoop.ipc.HBaseServer: >> > > > Stopping >> > > > > > server on 60020 >> > > > > > 2009-09-16 09:03:00,028 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 1 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,039 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 1092 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,042 INFO org.apache.hadoop.ipc.HBaseServer: >> > > > Stopping >> > > > > > IPC >> > > > > > Server listener on 60020 >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 3 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 4 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 7 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 8 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 9 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 10 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 18 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 19 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 20 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 22 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 23 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 24 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,044 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 25 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,044 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 26 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,044 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 27 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,044 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 29 on 60020: exiting >> > > > > > 2009-09-16 09:03:00,044 INFO org.apache.hadoop.ipc.HBaseServer: >> IPC >> > > > > Server >> > > > > > handler 30 on 60020: exiting >> > > > > > 2009-09-16 09:03:01,898 INFO >> > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: worker >> thread >> > > > exiting >> > > > > > 2009-09-16 09:03:01,898 INFO org.apache.zookeeper.ZooKeeper: >> > Closing >> > > > > > session: 0x123c2c3d8bb0001 >> > > > > > 2009-09-16 09:03:01,899 INFO org.apache.zookeeper.ClientCnxn: >> > Closing >> > > > > > ClientCnxn for session: 0x123c2c3d8bb0001 >> > > > > > 2009-09-16 09:03:01,905 INFO org.apache.zookeeper.ClientCnxn: >> > > Exception >> > > > > > while closing send thread for session 0x123c2c3d8bb0001 : Read >> > error >> > > rc >> > > > = >> > > > > > -1 >> > > > > > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] >> > > > > > 2009-09-16 09:03:02,008 INFO org.apache.zookeeper.ClientCnxn: >> > > > > Disconnecting >> > > > > > ClientCnxn for session: 0x123c2c3d8bb0001 >> > > > > > 2009-09-16 09:03:02,008 INFO org.apache.zookeeper.ZooKeeper: >> > Session: >> > > > > > 0x123c2c3d8bb0001 closed >> > > > > > 2009-09-16 09:03:02,008 INFO org.apache.zookeeper.ClientCnxn: >> > > > EventThread >> > > > > > shut down >> > > > > > 2009-09-16 09:03:02,110 INFO >> > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: >> regionserver/ >> > > > > > 10.209.207.80:60020 exiting >> > > > > > 2009-09-16 09:03:03,603 INFO org.apache.hadoop.hbase.Leases: >> > > > > > regionserver/10.209.207.80:60020.leaseChecker closing leases >> > > > > > 2009-09-16 09:03:03,603 INFO org.apache.hadoop.hbase.Leases: >> > > > > > regionserver/10.209.207.80:60020.leaseChecker closed leases >> > > > > > 2009-09-16 09:03:03,604 INFO >> > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Starting >> > shutdown >> > > > > > thread. >> > > > > > 2009-09-16 09:03:03,604 INFO >> > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown >> thread >> > > > > > complete >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > > >> > > >> > >> > >
