Setting cache blocks to false while running an MR Jon serves 2 purposes - it
helps real time requests by not out caches for MR jobs which don't really
need/use them, it helps prevent churn/fragmentation in block cache which helps
GC
There are properties you need to set in hbase-site.xml for tha
.
> > >
> > > My suggestion is you could try to decrease the MR task numbers to
> around
> > > 12 or lesser, and see if the frequency of crash decrease?
> > >
> > > Best regards,
> > > Henry Hung
> > >
> > > -Original Message-
&
> 12 or lesser, and see if the frequency of crash decrease?
> >
> > Best regards,
> > Henry Hung
> >
> > -Original Message-
> > From: Dhaval Shah [mailto:prince_mithi...@yahoo.co.in]
> > Sent: Tuesday, May 27, 2014 8:03 AM
> > To: user@
nd
>> 12 or lesser, and see if the frequency of crash decrease?
>>
>> Best regards,
>> Henry Hung
>>
>> -Original Message-
>> From: Dhaval Shah [mailto:prince_mithi...@yahoo.co.in]
>> Sent: Tuesday, May 27, 2014 8:03 AM
>> To: user@hbase.
ers to around
> 12 or lesser, and see if the frequency of crash decrease?
>
> Best regards,
> Henry Hung
>
> -Original Message-
> From: Dhaval Shah [mailto:prince_mithi...@yahoo.co.in]
> Sent: Tuesday, May 27, 2014 8:03 AM
> To: user@hbase.apache.org
> Subject: Re: HBas
decrease?
Best regards,
Henry Hung
-Original Message-
From: Dhaval Shah [mailto:prince_mithi...@yahoo.co.in]
Sent: Tuesday, May 27, 2014 8:03 AM
To: user@hbase.apache.org
Subject: Re: HBase cluster design
A few things pop out to me on cursory glance:
- You are using CMSIncrementalMode which
aults here
On top of these, when you kick off your MR job to scan HBase you should
setCacheBlocks to false
Regards,
Dhaval
From: Flavio Pompermaier
To: user@hbase.apache.org; Dhaval Shah
Sent: Friday, 23 May 2014 3:16 AM
Subject: Re: HBase cluster d
gt;
>
> Regards,
> Dhaval
>
>
>
> From: Flavio Pompermaier
> To: user@hbase.apache.org
> Sent: Saturday, 17 May 2014 2:49 AM
> Subject: Re: HBase cluster design
>
>
> Could you tell me please in detail the parameters you'd l
Can you share your hbase-env.sh and hbase-site.xml? And hardware specs of your
cluster?
Regards,
Dhaval
From: Flavio Pompermaier
To: user@hbase.apache.org
Sent: Saturday, 17 May 2014 2:49 AM
Subject: Re: HBase cluster design
Could you tell me please
Could you tell me please in detail the parameters you'd like to see so i
can look for them and learn the important ones?i'm using cloudera, cdh4 in
one cluster and cdh5 in the other.
Best,
Flavio
On May 17, 2014 2:48 AM, "prince_mithi...@yahoo.co.in" <
prince_mithi...@yahoo.co.in> wrote:
> Can yo
Can you describe your setup in more detail? Specifically the amount of heap
hbase region servers have and your GC settings. Is your server swapping when
your MR obs are running? Also do your regions go down or your region servers?
We run many MR jobs simultaneously on our hbase tables (size is i
Thanks for the response. Actually I'm still trying to understand why some
of the regions of my Hbase goes down from time to time during my mapred job
if table updates occur because in the logs there's nothing interesting..The
updates usually happens in bursts of 10/100 sequential puts per seconds.
On Tue, May 13, 2014 at 3:14 AM, Flavio Pompermaier wrote:
> So just to summarize the result of this discussion..
> do you confirm me that the last version of HBase should (in theory) support
> mapreduce jobs on tables that in the meantime could be updated by external
> processes (i.e. not by the
Of course I did..but that's not very helpful, is too broad!
I need something more specific to the tuning of HBase for the support of
the run of multiple mapred jobs while updating the tables..
Best,
Flavio
On Tue, May 13, 2014 at 3:20 PM, Ted Yu wrote:
> Have you looked at http://hbase.apache.
Have you looked at http://hbase.apache.org/book/performance.html ?
Cheers
On May 13, 2014, at 3:14 AM, Flavio Pompermaier wrote:
> So just to summarize the result of this discussion..
> do you confirm me that the last version of HBase should (in theory) support
> mapreduce jobs on tables that i
So just to summarize the result of this discussion..
do you confirm me that the last version of HBase should (in theory) support
mapreduce jobs on tables that in the meantime could be updated by external
processes (i.e. not by the mapred job)?
One of the answer about this was saying: "Poorly tuned
Today I was able to catch an error during a mapreduce job that actually
mimes the rowCount more or less.
The error I saw is:
ould not sync. Requesting close of hlog
java.io.IOException: Reflection
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter
You have one other thing to consider.
Did you oversubscribe on the m/r tuning side of things.
Many people want to segment their HBase to a portion of the cluster.
This should be the exception to the design not the primary cluster design.
If you over subscribe your cluster, you will run out o
The 'Connection refused' message was logged at WARN level.
If you can pastebin more of the region server log before its crash, I would
be take a deeper look.
BTW I assume your zookeeper quorum was healthy during that period of time.
On Fri, Apr 4, 2014 at 7:29 AM, Flavio Pompermaier wrote:
> Y
Yes I know I should update HBase, this is something I'm going to do really
soon. Bad me..
I just wanted to know if the fact of adding/updating rows in HBase while
running a mapred job could be problematic or not..
>From what you told me it's not, so the problem could be caused by the old
version of
Was the updating performed by one of the mapreduce jobs ?
HBase should be able to serve multiple mapreduce jobs in the same cluster.
Can you provide more detail on the crash ?
BTW, there are 3 major releases after 0.92
Please consider upgrading your cluster to newer release.
Cheers
On Apr 4,
Hi to everybody,
I have a probably stupid question: is it a problem to run many mapreduce
jobs on the same HBase table at the same time? And multiple jobs on
different tables on the same cluster?
Should I use Hoya to have a better cluster usage..?
In my current cluster I noticed that the region s
22 matches
Mail list logo
