Here are some of the answers: >> How many concurrent reducers run on each node? Default two? I was assuming 2 on each node would be the default. If not, this could be a problem. Please let me know.
>>'d suggest you spend a bit of time figuring where your MR jobs are spending their time? I agree. Will do some more research :) >>How much of this overall time is spent in reduce phase? Mostly time is spent in the Reduce phases, because that's where most of the critical code is. >>Are inserts to a new table? Yes, all inserts will always be in a new table. In fact, I disable/drop HTables during this process. Not using any special indexes, should I be? >>I'm a little surprised that all worked on the small instances, that your jobs completed. But, really, shouldn't Amazon guarantee predictability :) After all I am paying for these instances.. albeit a small amount! >>Are you opening a new table inside each task or once up in the config? I open HTable in the 'setup' method for each mapper/reducer, and close table in the 'cleanup' method. >>You have to temper the above general rule with the fact that... I will try a few combinations. >>How big is your dataset? This one in particular is not big, but the real production ones will be. Here's approximately how many rows get processed: Phase 1: 300 rows Phase 2 thru 8: 100 rows. (Note: Each phase does complex calculations on the row.) Thanks for the help. On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans <[email protected]>wrote: > How big is your dataset? > > J-D > > On Tue, Jan 26, 2010 at 8:47 AM, Something Something > <[email protected]> wrote: > > I have noticed some strange performance numbers on EC2. If someone can > give > > me some hints to improve performance that would be greatly appreciated. > > Here are the details: > > > > I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase > > 0.20.2 I ran the *exact* same process with following configurations: > > > > 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper > (*c1.medium*) > > with *8 Reducers *for every Reduce task. The process completed in *849* > > seconds. > > > > 2) 1 Master, 4 Workers & 1 Zookeeper *ALL m1.small* instances with *8 > > Reducers *for every Reduce task. The process completed in *906* seconds. > > > > 3) 1 Master, *11* Workers & *3* Zookeepers *ALL m1.small* instances with > *20 > > Reducers *for every Reduce task. The process completed in *984* seconds! > > > > > > Two main questions: > > > > 1) It's totally surprising that when I have 11 workers with 20 Reducers > it > > runs slower than when I have exactly same type of fewer machines with > fewer > > reducers.. > > 2) As expected it runs faster on c1.xlarge, but the performance > improvement > > doesn't justify the high cost difference. I must not be utilizing the > > machine power, but I don't know how to do that. > > > > Here are some of the performance improvements tricks that I have learnt > from > > this mailing list in the past that I am using: > > > > 1) conf.set("hbase.client.scanner.caching", "30"); I have this for all > > jobs. > > > > 2) Using the following code every time I open a HTable: > > this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ"); > > table.setAutoFlush(false); > > table.setWriteBufferSize(1024 * 1024 * 12); > > > > 3) For every Put I do this: > > Put put = new Put(Bytes.toBytes(out)); > > put.setWriteToWAL(false); > > > > 4) Change the No. of Reducers as per the No. of Workers. I believe the > > formula is: # of workers * 1.75. > > > > Any other hints? As always, greatly appreciate the help. Thanks. > > >
