Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-04-02 Thread anil gupta
at the JT would prevent speculative > execution no matter what the client says, but I could be mistaken. > > Sandy > > > -Original Message- > > From: anil gupta [mailto:anilgupt...@gmail.com] > > Sent: Friday, March 30, 2012 14:53 > > To: user@hbase.a

RE: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-04-02 Thread Sandy Pratt
com] > Sent: Friday, March 30, 2012 14:53 > To: user@hbase.apache.org > Subject: HBase bulk loader doing speculative execution when it set to false in > mapred-site.xml > > Hi All, > > I am using cdh3u2. I ran HBase bulk loading with property > "mapred.reduce.tasks.spec

Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-04-02 Thread anil gupta
+common-u...@hadoop.apache.org Hi Harsh, Thanks for the information. Is there any way to differentiate between a client side property and server-side property?or a Document which enlists whether a property is server or client-side? Many times i have to speculate over this and try out test runs.

Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-04-02 Thread Mikael Sitruk
Hash I think you have a good point here - It is a good practice that the utilities given from HBase also adapt HBase own recommendations. For example the RowCounter (org.apache.hadoop.hbase.mapreduce.RowCounter.java) utility is neither setting the speculative execution to 'false', nor the scan cach

Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-03-30 Thread Harsh J
Anil, You can also disable speculative execution on a per-job basis. See http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setMapSpeculativeExecution(boolean) (Which is why it is called a client-sided property - it applies per-job). If HBase strongly recommends

Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-03-30 Thread anil gupta
Hi Doug, Yes, that's why i had set that property as false in my mapred-site.xml. But, to my surprise i didnt know that setting that property would be useless for Hadoop jobs unless the mapred-site.xml is in classpath. The idea of client side property is a little confusing to me at present since th

Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-03-30 Thread Doug Meil
Speculative execution is on by default in Hadoop. One of the Performance recommendations in the Hbase RefGuide is to turn it off. On 3/30/12 6:12 PM, "Jean-Daniel Cryans" wrote: >Well that's not an HBase configuration, that's Hadoop. I'm not sure if >this is listed anywhere, maybe in the b

Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-03-30 Thread Jean-Daniel Cryans
Well that's not an HBase configuration, that's Hadoop. I'm not sure if this is listed anywhere, maybe in the book. BTW usually HBase has a "client" somewhere in the same to indicate it's client side. J-D On Fri, Mar 30, 2012 at 3:08 PM, anil gupta wrote: > Thanks for the quick reply, Jean. Is t

Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-03-30 Thread anil gupta
Thanks for the quick reply, Jean. Is there any link where i can find the name of all client-side configuration for HBase? ~Anil On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans wrote: > This is a client-side configuration so if your mapred-site.xml is > _not_ on your classpath when you start

Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-03-30 Thread Jean-Daniel Cryans
This is a client-side configuration so if your mapred-site.xml is _not_ on your classpath when you start the bulk load, it's not going to pick it up. So either have that file on your classpath, or put it in whatever other configuration file you have. J-D On Fri, Mar 30, 2012 at 2:52 PM, anil gupt

HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

2012-03-30 Thread anil gupta
Hi All, I am using cdh3u2. I ran HBase bulk loading with property "mapred.reduce.tasks.speculative.execution" set to "false" in mapred-site.xml. Still, i can see 6 killed task in Bulk Loading job and after short analysis i realized that these jobs are killed because another worker node completed t