Hi all
Recently I have upgraded my cluster from Hbase 0.90.1 to 0.90.4 (using
cloudera from cdh3u0 to cdh3u2)
Everything was ok till I ran pig extract on the new cluster, from the old
cluster everything worked well.
Now each time i run the extract in conjunction to other work performed on
the cluster I get the following exception during job execution, and the job
is finally killed.
It appears only on heavy extract (For small extract < minute execution
everything is ok).
Any suggestion?
Below a stacktrace from the task.
2012-02-06 01:18:26,561 INFO
org.apache.pig.backend.hadoop.hbase.HBaseTableInputFormat:
setScan with ranges: 374419525347267552315334599026232694541505353300281 -
374419525347267552315334599026232694541509698599225 ( 4345298944)
2012-02-06 01:19:32,836 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2012-02-06 01:19:32,845 WARN org.apache.hadoop.mapred.Child: Error running
child org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'-7220618182832784549'
does not exist
at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java: 230)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:
1862)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java: 570)
at
org.apache.hadoop.hbase.ipc.HBaseServer $Handler.run(HBaseServer.java:1039)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:
39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:
27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:
96)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:
83)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:
38)
at
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:
1019)
at
org.apache.hadoop.hbase.client.HTable $ClientScanner.next(HTable.java:1151)
at
org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:
133)
at
org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:
142)
at
org.apache.pig.backend.hadoop.hbase.HBaseTableInputFormat
$HBaseTableRecordReader.nextKeyValue(HBaseTableInputFormat.java:162)
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.getNext(HBaseStorage.java:
321)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:
187)
at
org.apache.hadoop.mapred.MapTask
$NewTrackingRecordReader.nextKeyValue(MapTask.java:456)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java: 67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264) 2012-02-06
01:19:32,978 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the
task
Thanks
Mikael.S
