Matthew,
What did you set your max region size to be for this table?
14K files totalling 650GB means you have a lot of small files...
On average ~45MB (rough calc).
How many regions? Do you have mslabs set up?
(GC tuning?)
Sorry for jumping in on the end of this conversation.
-Mike
> From: matt...@tovbin.com
> Date: Mon, 31 Oct 2011 12:07:53 +0200
> Subject: Re: Region has been OPENING for too long
> To: user@hbase.apache.org
>
> Ted, thanks for such a rapid response.
>
> You're right, we use hbase 0.90.3 from cdh3u1.
>
> So, I suppose I need to make bulk loading in smaller bulks then. Any other
> suggestions?
>
>
> Best regards,
> Matthew Tovbin =)
>
> >
> >
> >I assume you're using HBase 0.90.x where HBASE-4015 isn't available.
> >
> >>> 5. And so on, till some of Slaves fail with "java.net.SocketException:
> >Too many open files".
> >Do you have some monitoring setup so that you can know the number of open
> >file handles ?
> >
> >Cheers
> >
> >On Sun, Oct 30, 2011 at 7:21 AM, Matthew Tovbin <[EMAIL PROTECTED]> wrote:
> >
> >> Hi guys,
> >>
> >> I've bulkloaded a solid amount of data (650GB, ~14000 files) into Hbase
> >> (1master + 3regions) and now enabling the table results the
> >> following behavior on the cluster:
> >>
> >> 1. Master says that opening started -
> >> "org.apache.hadoop.hbase.master.AssignmentManager: Handling
> >> transition=RS_ZK_REGION_OPENING, server=slave..."
> >> 2. Slaves report about opening files in progress -
> >> "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
> >> 3. Then after ~10 mins the following error occurs on hmaster -
> >> "org.apache.hadoop.hbase.master.AssignmentManager: Regions in
> transition
> >> timed out / Region has been OPENING for too long, reassigning
> region=..."
> >> 4. More slaves report about opening files in progress -
> >> "org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://...."
> >> 5. And so on, till some of Slaves fail with "java.net.SocketException:
> >> Too many open files".
> >>
> >>
> >> What I've done already to solve the issue (which DID NOT help though):
> >>
> >> 1. Set 'ulimit -n 65536' for hbase user
> >> 2. Set hbase.hbasemaster.maxregionopen=3600000 (1 hour) in
> hbase-site.xml
> >>
> >>
> >> What else can I try?!
> >>
> >>
> >> Best regards,
> >> Matthew Tovbin =)
> >>
