I have a lookup file which I need in the mapper. So I am trying to read the whole file and load it into list in the mapper.
For each and every record Iook in this file which I got from distributed cache. — Sent from iPhone On Mon, Mar 25, 2013 at 6:39 PM, Hemanth Yamijala <yhema...@thoughtworks.com> wrote: > Hmm. How are you loading the file into memory ? Is it some sort of memory > mapping etc ? Are they being read as records ? Some details of the app will > help > On Mon, Mar 25, 2013 at 2:14 PM, nagarjuna kanamarlapudi < > nagarjuna.kanamarlap...@gmail.com> wrote: >> Hi Hemanth, >> >> I tried out your suggestion loading 420 MB file into memory. It threw java >> heap space error. >> >> I am not sure where this 1.6 GB of configured heap went to ? >> >> >> On Mon, Mar 25, 2013 at 12:01 PM, Hemanth Yamijala < >> yhema...@thoughtworks.com> wrote: >> >>> Hi, >>> >>> The free memory might be low, just because GC hasn't reclaimed what it >>> can. Can you just try reading in the data you want to read and see if that >>> works ? >>> >>> Thanks >>> Hemanth >>> >>> >>> On Mon, Mar 25, 2013 at 10:32 AM, nagarjuna kanamarlapudi < >>> nagarjuna.kanamarlap...@gmail.com> wrote: >>> >>>> io.sort.mb = 256 MB >>>> >>>> >>>> On Monday, March 25, 2013, Harsh J wrote: >>>> >>>>> The MapTask may consume some memory of its own as well. What is your >>>>> io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to? >>>>> >>>>> On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi >>>>> <nagarjuna.kanamarlap...@gmail.com> wrote: >>>>> > Hi, >>>>> > >>>>> > I configured my child jvm heap to 2 GB. So, I thought I could really >>>>> read >>>>> > 1.5GB of data and store it in memory (mapper/reducer). >>>>> > >>>>> > I wanted to confirm the same and wrote the following piece of code in >>>>> the >>>>> > configure method of mapper. >>>>> > >>>>> > @Override >>>>> > >>>>> > public void configure(JobConf job) { >>>>> > >>>>> > System.out.println("FREE MEMORY -- " >>>>> > >>>>> > + Runtime.getRuntime().freeMemory()); >>>>> > >>>>> > System.out.println("MAX MEMORY ---" + >>>>> Runtime.getRuntime().maxMemory()); >>>>> > >>>>> > } >>>>> > >>>>> > >>>>> > Surprisingly the output was >>>>> > >>>>> > >>>>> > FREE MEMORY -- 341854864 = 320 MB >>>>> > MAX MEMORY ---1908932608 = 1.9 GB >>>>> > >>>>> > >>>>> > I am just wondering what processes are taking up that extra 1.6GB of >>>>> heap >>>>> > which I configured for the child jvm heap. >>>>> > >>>>> > >>>>> > Appreciate in helping me understand the scenario. >>>>> > >>>>> > >>>>> > >>>>> > Regards >>>>> > >>>>> > Nagarjuna K >>>>> > >>>>> > >>>>> > >>>>> >>>>> >>>>> >>>>> -- >>>>> Harsh J >>>>> >>>> >>>> >>>> -- >>>> Sent from iPhone >>>> >>> >>> >>