Side note: 0.94.3 is not certified for hadoop 2.0 Cheers
On Wed, Jan 23, 2013 at 10:01 AM, ramkrishna vasudevan < ramkrishna.s.vasude...@gmail.com> wrote: > This is something to do with HDFS also. Regarding data loss, the client > should have known that the put has failed. So there should not be silent > data loss as far as i can think of. > Any way further investigation can make things clearer. > > Regards > Ram > > On Wed, Jan 23, 2013 at 11:29 PM, Kevin O'dell <kevin.od...@cloudera.com > >wrote: > > > Varun, > > > > I suspect there might be a FATAL or ERROR above that message? That > > message typically means the HLog can't write to HDFS, you may want to > look > > at the DN logs around that time. > > > > On Wed, Jan 23, 2013 at 12:57 PM, Varun Sharma <va...@pinterest.com> > > wrote: > > > > > Hi, > > > > > > I had 2 region servers crash with this (running hbase-0.94.3 with > > > hadoop-2.0.0). There is also stuff saying that Memstore was rolled > back. > > Is > > > this some kind of data loss happening ? > > > > > > 2013-01-23 09:14:23,119 FATAL > > > org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. > Requesting > > > close of hlog > > > java.io.IOException: Reflection > > > at > > > > > > > > > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304) > > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1328) > > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1437) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5205) > > > at > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2245) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1973) > > > at > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3493) > > > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) > > > at > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > > at java.lang.reflect.Method.invoke(Method.java:597) > > > at > > > > > > > > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > > > at > > > > > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1438) > > > Caused by: java.lang.reflect.InvocationTargetException > > > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > > > at > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > > at java.lang.reflect.Method.invoke(Method.java:597) > > > at > > > > > > > > > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302) > > > ... 11 more > > > Caused by: > > > > > > > > > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): > > > No lease on > > > > > > > > > /hbase/.logs/ip-10-10-162-152.ec2.internal,60020,1358833785984/ip-10-10-162-152.ec2.internal%2C60020%2C1358833785984.1358932305725 > > > File does not exist. Holder DFSClient_NONMAPREDUCE_-1821696410_41 does > > not > > > have any open files. > > > at > > > > > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2308) > > > at > > > > > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2299) > > > at > > > > > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2231) > > > at > > > > > > > > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:501) > > > at > > > > > > > > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:316) > > > at > > > > > > > > > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44082) > > > at > > > > > > > > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > > > > > > 4163,64-71 1% > > > > > > Thanks > > > Varun > > > > > > > > > > > -- > > Kevin O'Dell > > Customer Operations Engineer, Cloudera > > >
