io.sort.mb = 256 MB On Monday, March 25, 2013, Harsh J wrote:
> The MapTask may consume some memory of its own as well. What is your > io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to? > > On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi > <nagarjuna.kanamarlap...@gmail.com <javascript:;>> wrote: > > Hi, > > > > I configured my child jvm heap to 2 GB. So, I thought I could really > read > > 1.5GB of data and store it in memory (mapper/reducer). > > > > I wanted to confirm the same and wrote the following piece of code in the > > configure method of mapper. > > > > @Override > > > > public void configure(JobConf job) { > > > > System.out.println("FREE MEMORY -- " > > > > + Runtime.getRuntime().freeMemory()); > > > > System.out.println("MAX MEMORY ---" + Runtime.getRuntime().maxMemory()); > > > > } > > > > > > Surprisingly the output was > > > > > > FREE MEMORY -- 341854864 = 320 MB > > MAX MEMORY ---1908932608 = 1.9 GB > > > > > > I am just wondering what processes are taking up that extra 1.6GB of heap > > which I configured for the child jvm heap. > > > > > > Appreciate in helping me understand the scenario. > > > > > > > > Regards > > > > Nagarjuna K > > > > > > > > > > -- > Harsh J > -- Sent from iPhone