Re: NotServingRegionException

I use org.apache.hadoop.hbase.mapreduce.Import to import which is launched on the same VM. On Wed, Mar 3, 2010 at 11:37 AM, Jean-Daniel Cryans wrote: > Yes that's one thing, also make sure your client has connectivity... > doesn't seem so. > > J-D > > On Wed, Mar 3, 2010 at 11:32 AM, Ted Yu wrot

Re: NotServingRegionException

Yes that's one thing, also make sure your client has connectivity... doesn't seem so. J-D On Wed, Mar 3, 2010 at 11:32 AM, Ted Yu wrote: > But querying zookeeper shows: >  lsr /hbase > > hbase >    safe-mode >    rs >   1267640372165 >    root-region-server >    master >    shutdown > > > On

Re: NotServingRegionException

Looks like a connectivity issue, it says: > 10/03/03 10:45:55 WARN zookeeper.ZooKeeperWrapper: Failed to create /hbase > -- check quorum servers, currently=tyu-linux:2181 Do what it says to do ;) Also make sure that that client can reach that address. In my experience using a VM can be troublesom

Re: NotServingRegionException

Hi, J-D: I restarted hbase and am not seeing NotServingRegionException now. I tried to import a table that I exported from hbase 0.20.1 into this 0.20.3 instance. After sometime I got: 10/03/03 10:45:55 WARN zookeeper.ZooKeeperWrapper: Failed to create /hbase -- check quorum servers, currently=tyu

Re: NotServingRegionException

Ted, With such a small snippet it's hard to tell ;) Looks like that region server wasn't assigned with .META. but -ROOT- contains that address for that region. Look at the logs for when 1) the master assigns the region and 2) when the region server opens the region. In between I expect you should

Re: NotServingRegionException

Previous attempt wasn't delivered. On Wed, Mar 3, 2010 at 9:30 AM, Ted Yu wrote: > Hi, > I started hbase 0.20.3 successfully on my Linux VM. Master and regionserver > are on the same VM. > There're two empty tables. > > Soon I saw the following in regionserver.log: > 2010-03-03 09:18:31,643 INFO

Re: NotServingRegionException

IIya, This is an INFO level log so it's normal. Basically it just responds to the client saying that a certain region isn't there anymore (rebalanced, splitting, etc). J-D On Thu, Oct 15, 2009 at 12:24 PM, Ilya Sukhar wrote: > I'm seeing the following exceptions: > http://pastebin.com/m28d7bad4

Re: NotServingRegionException on .META. table?!

Okay. Thanks for your kind answer.I will give here another mail with the DEBUG level log when it happens again. Sincerely, Woosuk Suh. 2009/10/9 Amandeep Khurana > Can you change the logging level to DEBUG and post the logs again.. That'll > give a better idea on whats happening. > > You dont n

Re: NotServingRegionException on .META. table?!

Can you change the logging level to DEBUG and post the logs again.. That'll give a better idea on whats happening. You dont need to attach the logs. Use pastebin or pastie.. Those are convenient. -ak Amandeep Khurana Computer Science Graduate Student University of California, Santa Cruz On We

Re: NotServingRegionException as indication of cluster overloading

very large tables. Best regards, - Andy > From: Michael Dagaev > Subject: Re: NotServingRegionException as indication of cluster overloading > To: hbase-user@hadoop.apache.org, apurt...@apache.org > Date: Thursday, March 26, 2009, 11:17 AM > > Andrew, thank you for the detailed

Re: NotServingRegionException as indication of cluster overloading

formative. But to be credible, any alternative to > HDFS would have to handle appends. See HADOOP-1700 and > HADOOP-4379 for more information there. > > Best regards, > >   - Andy > >> From: Michael Dagaev >> Subject: Re: NotServingRegionException as indication of c

Re: NotServingRegionException as indication of cluster overloading

See below. On Thu, Mar 26, 2009 at 4:18 PM, stack wrote: > So, does that region fail to deploy?  Does it ever come on line? Do you mean a region or region server ? > Can you download that file successfully using hadoop command-line: > > ./bin/hadoop fs -get /hbase/ENTITIES/267503732/oldlogfile.

Re: NotServingRegionException as indication of cluster overloading

would be most informative. But to be credible, any alternative to HDFS would have to handle appends. See HADOOP-1700 and HADOOP-4379 for more information there. Best regards, - Andy > From: Michael Dagaev > Subject: Re: NotServingRegionException as indication of cluster overloadin

Re: NotServingRegionException as indication of cluster overloading

So, does that region fail to deploy? Does it ever come on line? Can you download that file successfully using hadoop command-line: ./bin/hadoop fs -get /hbase/ENTITIES/267503732/oldlogfile.log . Do you have the xceivers' bumped up on your cluster and the dfsclient timeout set to zero? St.Ack

Re: NotServingRegionException as indication of cluster overloading

Stack Currenlty, one region server of 7 throws a lot of NotServingRegionException. No map reduce jobs and no CPU starvation on this host. There also a lot of IO exceptions like that: java.io.IOException: Could not obtain block: blk_-3762232304446475286_1462869 file=/hbase/ENTITIES/26750373

Re: NotServingRegionException as indication of cluster overloading

Michael: What is happening on your cluster? Are you doing any mapreduce jobs or is there any other kind of heavy access afoot? Look in your logs to see if you can figure whats going on. What else is going on on these machines? Are the other processes starving the hbase regionservers? St.Ack

Re: NotServingRegionException as indication of cluster overloading

Thanks, Schubert. I wonder why it is starting to occur very frequently and make also clients fail. On Thu, Mar 26, 2009 at 2:05 PM, schubert zhang wrote: > This NotServingRegionException would happen when assignment.When > region-splitting, master should assign the new regions to servers, the > p

Re: NotServingRegionException as indication of cluster overloading

This NotServingRegionException would happen when assignment.When region-splitting, master should assign the new regions to servers, the processing need time. In this duration, the regions are not accessable. On Thu, Mar 26, 2009 at 7:20 PM, Michael Dagaev wrote: > Hi, all > >It looks like No

Re: NotServingRegionException - Map/Reduce process fails

St.Ack and J-D, Thanks for your help. Upgrading to the latest 0.19.0 and changing the region size back to 256MB along with the Premature EOF settings from Jean Adrien fixed the issues I was seeing. Dru On Oct 23, 2008, at 4:04 PM, stack wrote: Dru Jensen wrote: Stack, Sorry for the co

Re: NotServingRegionException - Map/Reduce process fails

Dru Jensen wrote: Stack, Sorry for the confusion, I am not using the old implementation of TableReduce. The new 0.19.0 changed this to an interface. The reduce process is performing calculations. It's not just writing to the table and requires the sort. Or try running with even more reduc

Re: NotServingRegionException - Map/Reduce process fails

Stack, Sorry for the confusion, I am not using the old implementation of TableReduce. The new 0.19.0 changed this to an interface. The reduce process is performing calculations. It's not just writing to the table and requires the sort. I will change the region size back and see if that

Re: NotServingRegionException - Map/Reduce process fails

Any reason you need to use TableReduce? If you delay the insert into hbase till reduce-time, it means 1.), the MR framework has spent a bunch of resources shuffling and sorting your data, a sort that is going to happen on hbase insert anyways, and 2). your inserts are going into hbase in order

Re: NotServingRegionException - Map/Reduce process fails

I do not see any swapping. I have a 3 node cluster with 8GB memory and 4 cpu's each and 2TB HDFS. Node 1 is acting as master. I am reducing 32M map results into about 2M rows, several column families with 10's of columns each. I am writing them to a table using TableReduce class. grepi

Re: NotServingRegionException - Map/Reduce process fails

Dru: If compactions are taking 4minutes, then your instance is being overrun; its unable to keep up with your rate of upload. Whats your upload rate like? How are you doing it? Or is it that your servers are buckled carrying the load? Are they swapping? Usually compaction runs fast. It'l

Re: NotServingRegionException - Map/Reduce process fails

Dru. To make sure it's not 921, check if the region that NSRE is currently reachable (provided that you did not reboot). It should be assigned in META but missing from the region server. J-D On Thu, Oct 23, 2008 at 3:07 PM, Dru Jensen <[EMAIL PROTECTED]> wrote: > Stack and J-D, Thanks for your

Re: NotServingRegionException - Map/Reduce process fails

Stack and J-D, Thanks for your responses. It looks like the RetriesExhaustedException occurred during: 2008-10-23 11:08:55,180 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region ... 1224785065371 in 4mins, 25sec It doesn't look like I am having the HBASE-921

Re: NotServingRegionException - Map/Reduce process fails

Dru. See also if it's a case of HBASE-921because it would make sense if not using hbase 0.18.1 and under a heavy load. J-D On Thu, Oct 23, 2008 at 2:30 PM, stack <[EMAIL PROTECTED]> wrote: > Find the MR task that failed. Click through the UI to

Re: NotServingRegionException - Map/Reduce process fails

Find the MR task that failed. Click through the UI to look at its logs. It may have interesting info. Its probably complaining about a region not being available (NSRE). Figure which region it is. Use the region historian or grep in the master logs -- 'grep -v metaScanner REGIONNAME' so yo

RE: NotServingRegionException revisited

Hi Again After going through the logs a bit more carefully I found a FNFE while trying to do a compaction on that particular region. The relevant log follows attached. After the failed compaction because of the FNFE the region is still online in .META. but no longer among the onli