how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?

hi,maillist: i recent get a problem,when i run MR job, it happened OOM in shuffle process,the options about MR is default,not changed,which option should i tuning? thanks

Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?

version is rewally important here.. - If 1.x, then Where (NN , JT , TT ?) - if 2.x, then where? (AM, NM, ... ?) <-- probably less likely here, since the resources are ephemeral. I know that some older 1x versions had an issue with the jobtracker having an ever-expanding hashmap or something like

Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?

Hi Jay, Can you point out a detail about 1.x that job tracker having an ever-expanding hash map. I want fix it. On Tue, Dec 3, 2013 at 3:08 AM, Jay Vyas wrote: > version is rewally important here.. > > - If 1.x, then Where (NN , JT , TT ?) > - if 2.x, then where? (AM, NM, ... ?) <-- probably le