Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?

2013-12-03 Thread Azuryy Yu 
Hi Jay,
Can you point out a detail about 1.x that job tracker having an
ever-expanding hash map. I want fix it.


On Tue, Dec 3, 2013 at 3:08 AM, Jay Vyas jayunit...@gmail.com wrote:

 version is rewally important here..

 - If 1.x, then Where (NN , JT , TT ?)
 - if 2.x, then where? (AM, NM, ... ?) -- probably less likely here, since
 the resources are ephemeral.

 I know that some older 1x versions had an issue with the jobtracker having
 an ever-expanding hashmap or something like that, so that if you ran 100s
 of jobs, you could get OOM erros on the JobTracker.

Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?

2013-12-02 Thread Jay Vyas 
version is rewally important here..

- If 1.x, then Where (NN , JT , TT ?)
- if 2.x, then where? (AM, NM, ... ?) -- probably less likely here, since
the resources are ephemeral.

I know that some older 1x versions had an issue with the jobtracker having
an ever-expanding hashmap or something like that, so that if you ran 100s
of jobs, you could get OOM erros on the JobTracker.