Hmm. How are you loading the file into memory ? Is it some sort of memory
mapping etc ? Are they being read as records ? Some details of the app will
help
On Mon, Mar 25, 2013 at 2:14 PM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlap...@gmail.com> wrote:
> Hi Hemanth,
>
> I tried out your suggestion loading 420 MB file into memory. It threw java
> heap space error.
>
> I am not sure where this 1.6 GB of configured heap went to ?
>
>
> On Mon, Mar 25, 2013 at 12:01 PM, Hemanth Yamijala <
> yhema...@thoughtworks.com> wrote:
>
>> Hi,
>>
>> The free memory might be low, just because GC hasn't reclaimed what it
>> can. Can you just try reading in the data you want to read and see if that
>> works ?
>>
>> Thanks
>> Hemanth
>>
>>
>> On Mon, Mar 25, 2013 at 10:32 AM, nagarjuna kanamarlapudi <
>> nagarjuna.kanamarlap...@gmail.com> wrote:
>>
>>> io.sort.mb = 256 MB
>>>
>>>
>>> On Monday, March 25, 2013, Harsh J wrote:
>>>
>>>> The MapTask may consume some memory of its own as well. What is your
>>>> io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to?
>>>>
>>>> On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi
>>>> <nagarjuna.kanamarlap...@gmail.com> wrote:
>>>> > Hi,
>>>> >
>>>> > I configured my child jvm heap to 2 GB. So, I thought I could really
>>>> read
>>>> > 1.5GB of data and store it in memory (mapper/reducer).
>>>> >
>>>> > I wanted to confirm the same and wrote the following piece of code in
>>>> the
>>>> > configure method of mapper.
>>>> >
>>>> > @Override
>>>> >
>>>> > public void configure(JobConf job) {
>>>> >
>>>> > System.out.println("FREE MEMORY -- "
>>>> >
>>>> > + Runtime.getRuntime().freeMemory());
>>>> >
>>>> > System.out.println("MAX MEMORY ---" +
>>>> Runtime.getRuntime().maxMemory());
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> > Surprisingly the output was
>>>> >
>>>> >
>>>> > FREE MEMORY -- 341854864 = 320 MB
>>>> > MAX MEMORY ---1908932608 = 1.9 GB
>>>> >
>>>> >
>>>> > I am just wondering what processes are taking up that extra 1.6GB of
>>>> heap
>>>> > which I configured for the child jvm heap.
>>>> >
>>>> >
>>>> > Appreciate in helping me understand the scenario.
>>>> >
>>>> >
>>>> >
>>>> > Regards
>>>> >
>>>> > Nagarjuna K
>>>> >
>>>> >
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>> --
>>> Sent from iPhone
>>>
>>
>>
>
