.META. went offline during second flow attempt. The time out I mentioned happened for 1st and 3rd attempts. HBase was restarted before the 1st and 3rd attempts.
Here is jstack: http://pastebin.com/EHMSvsRt On Thu, Feb 10, 2011 at 3:04 PM, Stack <[email protected]> wrote: > So, .META. is not online? What happens if you use shell at this time. > > Your attachement did not come across Ted. Mind postbin'ing it? > > St.Ack > > On Thu, Feb 10, 2011 at 2:41 PM, Ted Yu <[email protected]> wrote: > > I replaced hbase jar with hbase-0.90.1.jar > > I also upgraded client side jar to hbase-0.90.1.jar > > > > Our map tasks were running faster than before for about 50 minutes. > However, > > map tasks then timed out calling flushCommits(). This happened even after > > fresh restart of hbase. > > > > I don't see any exception in region server logs. > > > > In master log, I found: > > > > 2011-02-10 18:24:15,286 DEBUG > > org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region > > -ROOT-,,0.70236052 on sjc1-hadoop6.X.com,60020,1297362251595 > > 2011-02-10 18:24:15,349 INFO > org.apache.hadoop.hbase.catalog.CatalogTracker: > > Failed verification of .META.,,1 at address=null; > > org.apache.hadoop.hbase.NotServingRegionException: > > org.apache.hadoop.hbase.NotServingRegionException: Region is not online: > > .META.,,1 > > 2011-02-10 18:24:15,350 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: > > master:60000-0x12e10d0e31e0000 Creating (or updating) unassigned node for > > 1028785192 with OFFLINE state > > > > I am attaching region server (which didn't respond to stop-hbase.sh) > jstack. > > > > FYI > > > > On Thu, Feb 10, 2011 at 10:10 AM, Stack <[email protected]> wrote: > >> > >> Thats probably enough Ted. The 0.90.1 hbase-default.xml has an extra > >> config. to enable the experimental HBASE-3455 feature but you can copy > >> that over if you want to try playing with it (it defaults off so you'd > >> copy over the config. if you wanted to set it to true). > >> > >> St.Ack > > > > >
