I made more test. I think my guess may be the cause. In a latest test, there are 11 killed map tasks, and there are also 11 expired leases.
I think may be the MapReduce code should be improved. Schubert On Wed, Mar 11, 2009 at 4:38 PM, schubert zhang <[email protected]> wrote: > Hi all, > I am running a MapReduce job to read regions(row ranges) in Map task. The > job can work completely and correctly. > > But, usually, I find following INFO in hbase regionservre's log file. > > 2009-03-11 15:57:06,428 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > -6710483098243538624 lease expired > > To debug this issue, I add some new log information in: > src/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > (1) in addScanner(), I add: > LOG.info("----Schubert Add Scanner " + scannerName + " lease create!"); > (2) in close(), I add: > LOG.info("----Schubert close " + scannerName + " lease cancel!"); > > Then I redo my mapreduce job. And I found some leases/scanners are not > closed. > > I can make sure that my mapreduce RecordReader implements the close() > method and calls the close() method of the scanner. > > following log info is copy from one region server of my HBase clasuer, and > to make clear, I move the Add and Close together. > > 2009-03-11 15:52:41,406 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > 5785283094618320498 lease create! > 2009-03-11 15:54:08,423 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close > 5785283094618320498 lease cancel! > > 2009-03-11 15:52:46,493 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > -3726092592914221244 lease create! > 2009-03-11 15:54:11,676 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close > -3726092592914221244 lease cancel! > > 2009-03-11 15:52:51,984 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > 313013793996434532 lease create! > 2009-03-11 15:54:18,428 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close > 313013793996434532 lease cancel! > > 2009-03-11 15:54:10,213 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > -5389407006476765661 lease create! > 2009-03-11 15:55:48,337 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close > -5389407006476765661 lease cancel! > > 2009-03-11 15:54:13,505 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > 8463558669492796494 lease create! > 2009-03-11 15:55:59,152 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close > 8463558669492796494 lease cancel! > > 2009-03-11 15:54:20,118 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > 8149590914348186380 lease create! > 2009-03-11 15:56:04,350 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close > 8149590914348186380 lease cancel! > > 2009-03-11 15:54:29,466 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > 4137760362936278647 lease create! > 2009-03-11 15:56:10,765 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close > 4137760362936278647 lease cancel! > > 2009-03-11 15:55:34,786 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > -5919619533272076177 lease create! > 2009-03-11 15:56:50,472 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > -5919619533272076177 lease expired > > 2009-03-11 15:55:36,961 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > -7919803182075534481 lease create! > 2009-03-11 15:57:04,200 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > -7919803182075534481 lease expired > > 2009-03-11 15:55:45,992 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > 2812856283312715383 lease create! > 2009-03-11 15:57:14,442 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > 2812856283312715383 lease expired > > 2009-03-11 15:56:03,033 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner > -6710483098243538624 lease create! > 2009-03-11 15:57:06,428 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner > -6710483098243538624 lease expired > > Wen can see, the > Scanners -5919619533272076177, -7919803182075534481, 2812856283312715383 > and -6710483098243538624 are not closed. > > I guess, it maybe caused by MapReduce kill task (speculative). > > Schubert >
