Re: Tune MapReduce over HBase to insert data

gt; replicated 3 times. > > > > >> > > > 6GB/80s = 75MB/s, twice that if you do not disable the WAL, > > > which > > > > a > > > > >> > > > reasonable machine should be able to absorb. > > > > >> > > > The fact that deferre

Re: Tune MapReduce over HBase to insert data

hat if you do not disable the WAL, >> > which >> > > a >> > > >> > > > reasonable machine should be able to absorb. >> > > >> > > > The fact that deferred log flush does not help you seems to >> > > indicate >> > > >> >

Re: Tune MapReduce over HBase to insert data

>> > > > > > > >> > > > What's your memstore flush size? Potentially the data is > written > > > many > > > >> > > > times during compactions. > > > >> > > > > > > >> > > >

Re: Tune MapReduce over HBase to insert data

written > > many > > >> > > > times during compactions. > > >> > > > > > >> > > > > > >> > > > In your case you dial down the HDFS replication, since you only > > have > > >> > two > >

Re: Tune MapReduce over HBase to insert data

at's your memstore flush size? Potentially the data is written > many > >> > > > times during compactions. > >> > > > > >> > > > > >> > > > In your case you dial down the HDFS replication, since you only > have > >> > two > >> > > >

Re: Tune MapReduce over HBase to insert data

gt;> > > > >> > > > >> > > > In your case you dial down the HDFS replication, since you only have >> > two >> > > > physical machines anyway. >> > > > (Set it to 2. If you do not specify any failure zones, you might as >> > well >> > &

Re: Tune MapReduce over HBase to insert data

set it to 1... You will lose data if one of your server machines dies > > > > anyway). > > > > > > > > It does not really make that much sense to deploy HBase and HDFS on > > > > virtual nodes like this. > > > > -- Lars > > >

Re: Tune MapReduce over HBase to insert data

e data if one of your server machines dies > > > anyway). > > > > > > It does not really make that much sense to deploy HBase and HDFS on > > > virtual nodes like this. > > > -- Lars > > > > > > > > > > > > ___

Re: Tune MapReduce over HBase to insert data

your server machines dies > > anyway). > > > > It does not really make that much sense to deploy HBase and HDFS on > > virtual nodes like this. > > -- Lars > > > > > > > > > > From: Farrokh Shahriari > > &g

Re: Tune MapReduce over HBase to insert data

Note: if you have a huge amount of data using bulk inserts is much faster than using Puts. Regards, Gerrit On Tue, Jan 8, 2013 at 7:04 AM, Farrokh Shahriari < mohandes.zebeleh...@gmail.com> wrote: > Tnx Ted, > How can I tune it ? can you tell me !! > I have not yet decided for upgrading,does it

Re: Tune MapReduce over HBase to insert data

odes like this. > -- Lars > > > > > From: Farrokh Shahriari > To: user@hbase.apache.org > Sent: Monday, January 7, 2013 9:38 PM > Subject: Re: Tune MapReduce over HBase to insert data > > Hi again, > I'm using HBase 0.92.1-cdh4.0.0.

Re: Tune MapReduce over HBase to insert data

y, January 7, 2013 9:38 PM Subject: Re: Tune MapReduce over HBase to insert data Hi again, I'm using HBase 0.92.1-cdh4.0.0. I have two server machine with 48Gb RAM,12 physical core & 24 logical core that contain 12 nodes(6 nodes on each server). Each node has 8Gb RAM & 2 VCPU. I

Re: Tune MapReduce over HBase to insert data

Please take a look at http://hbase.apache.org/book.html#jvm Section 12.2.3, “JVM Garbage Collection Logs”should be read as well. There is more recent effort to reduce GC activity. Namely HBASE-7404 Bucket Cache:A solution about CMS,Heap Fragment a

Re: Tune MapReduce over HBase to insert data

Have you tuned the JVM parameter of hbase ? If you have Ganglia, did you observe high variation in network latency on the 6 nodes ? HBase 0.92.2 has been released. Do you plan to upgrade to 0.92.2 or 0.94.3 ? Cheers On Mon, Jan 7, 2013 at 9:38 PM, Farrokh Shahriari < mohandes.zebeleh...@gmail.co

Re: Tune MapReduce over HBase to insert data

Hi again, I'm using HBase 0.92.1-cdh4.0.0. I have two server machine with 48Gb RAM,12 physical core & 24 logical core that contain 12 nodes(6 nodes on each server). Each node has 8Gb RAM & 2 VCPU. I've set some parameter that get better result like set WAL=off on put,but some parameters like Heap-s

Re: Tune MapReduce over HBase to insert data

Have you read through http://hbase.apache.org/book.html#performance ? What version of HBase are you using ? Cheers On Mon, Jan 7, 2013 at 9:05 PM, Farrokh Shahriari < mohandes.zebeleh...@gmail.com> wrote: > Hi there > I have a cluster with 12 nodes that each of them has 2 core of CPU. Now,I > w