Hi,
I have a HBase/Hadoop CDH3 cluster with 4 machines (1 Master + 3 Slaves),
running on Amazon EC2. The master is a High-Memory Extra Large Instance
(m2.xlarge) with NameNode, JobTracker, HMaster and Zookeeper. The slaves
are Extra Large Instances (m1.xlarge) running Datanode, TaskTracker,
Region
(set it to 0) and your JVM params.
Sandy
> -Original Message-
> From: Leonardo Gamas [mailto:leoga...@jusbrasil.com.br]
> Sent: Thursday, December 29, 2011 07:44
> To: user@hbase.apache.org
> Subject: RegionServer dying every two or three days
>
> Hi,
>
> I
; Sent: Thursday, December 29, 2011 07:44
> > To: user@hbase.apache.org
> > Subject: RegionServer dying every two or three days
> >
> > Hi,
> >
> > I have a HBase/Hadoop CDH3 cluster with 4 machines (1 Master + 3 Slaves),
> > running on Amazon EC2. The m
gt;> to the crash.
>>
>> If it turns out to be GC, you might want to look at your kernel
>> swappiness setting (set it to 0) and your JVM params.
>>
>> Sandy
>>
>>
>> > -Original Message-
>> > From: Leonardo Gamas [mailto:leoga..
statements leading up
>>> to the crash.
>>>
>>> If it turns out to be GC, you might want to look at your kernel
>>> swappiness setting (set it to 0) and your JVM params.
>>>
>>> Sandy
>>>
>>>
>>> > -Original Messa
omewhat.
Sandy
> -Original Message-
> From: Leonardo Gamas [mailto:leoga...@jusbrasil.com.br]
> Sent: Wednesday, January 04, 2012 12:17
> To: user@hbase.apache.org
> Subject: Re: RegionServer dying every two or three days
>
> Sandy,
>
> It happened again:
>
>
timestamps of the log statements leading
up
>>> to the crash.
>>>
>>> If it turns out to be GC, you might want to look at your kernel
>>> swappiness setting (set it to 0) and your JVM params.
>>>
>>> Sandy
>>>
>>>
>>> >
and it was due to GC config. Pls check
> your GC.
>
> Regards
> Ram
>
> -Original Message-
> From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack
> Sent: Thursday, January 05, 2012 2:50 AM
> To: user@hbase.apache.org
> Subject: Re: Re
tenured ratio might be 1:3. Were the new gen GCs
taking too long? This is probably orthogonal to your immediate issue, though.
-Original Message-
From: Leonardo Gamas [mailto:leoga...@jusbrasil.com.br]
Sent: Thursday, January 05, 2012 5:33 AM
To: user@hbase.apache.org
Subject: Re: R
jects in the generation, and CMS does indeed cause
> STW pauses). A typical new to tenured ratio might be 1:3. Were the new
> gen GCs taking too long? This is probably orthogonal to your immediate
> issue, though.
>
>
>
> -Original Message-
> From: Leonardo Gam
on.
> -Original Message-
> From: Leonardo Gamas [mailto:leoga...@jusbrasil.com.br]
> Sent: Thursday, January 05, 2012 13:15
> To: user@hbase.apache.org
> Subject: Re: RegionServer dying every two or three days
>
> I checked the CPU Utilization graphics provided by Amaz
ence with
> Amazon so I don't know if that sort of thing is common.
>
> > -Original Message-
> > From: Leonardo Gamas [mailto:leoga...@jusbrasil.com.br]
> > Sent: Thursday, January 05, 2012 13:15
> > To: user@hbase.apache.org
> > Subject: Re: Regio
ng GC, leaving us to
>> think the GC ran long when it didn't? I don't have a lot of experience
>> with Amazon so I don't know if that sort of thing is common.
>>
>> > -Original Message-
>> > From: Leonardo Gamas [mailto:leoga...@jusbrasil.c
ppens to be during GC, leaving us to
> >> think the GC ran long when it didn't? I don't have a lot of experience
> >> with Amazon so I don't know if that sort of thing is common.
> >>
> >> > -Original Message-
> >> > From: Leonardo
5 Sandy Pratt
> > >
> > >> You think it's an Amazon problem maybe? Like they paused or migrated
> > >> your virtual machine, and it just happens to be during GC, leaving us
> to
> > >> think the GC ran long when it didn't? I don't have a
> > > > I/O Performance: High
> > > > API name: m1.xlarge
> > > > I was not expecting to suffer from this problems, or at least not
> much.
> > > >
> > > >
> > > > 2012/1/5 Sandy Pratt
> > > >
> &g
> > > >
> > > > > 15 GB memory
> > > > > 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
> > > > > 1,690 GB instance storage
> > > > > 64-bit platform
> > > > > I/O Performance: High
> > > > > A
l
> > > > > > machine.
> > > > > >
> > > > > > But the intance type i'm using:
> > > > > >
> > > > > > *Extra Large Instance*
> > > > > >
> > > > > > 15 GB memory
>
er
> > > > > > > continue to work normally.
> > > > > > > But it can be related to the shared environment nature of
> Amazon,
> > > > maybe
> > > > > > > some spike in I/O caused by another virtualized server in the
> >
iveOpts in
> my
> > to
> > > > > > tests.
> > > > > > > >> >
> > > > > > > >> > I don't have payed much attention to the value of the new
> > size
> > > > > &
's an Amazon/Virtualized Environment
> > inherent
> > > > > > issue.
> > > > > > > >
> > > > > > > > Thanks for the great help provided so far.
> > > > > > > >
> > > > > > > > 2012/1
ems related to I/O. It starts to timeout
> > > many
> > > > > > > > > connections, new connections to/from the machine timeout
> too,
> > > and
> > > > > > > finally
> > > > > > > > > the RegionServer dies because of YouA
22 matches
Mail list logo
