Hi Hemanth/Koji,
Seems the above script doesn't work for me. Can u look into the following
and suggest what more can I do
hadoop fs -cat /user/ims-b/dump.sh
#!/bin/sh
hadoop dfs -put myheapdump.hprof /tmp/myheapdump_ims/${PWD//\//_}.hprof
hadoop jar LL.jar
Couple of things to check:
Does your class com.hadoop.publicationMrPOC.Launcher implement the Tool
interface ? You can look at an example at (
http://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.html#Source+Code-N110D0).
That's what accepts the -D params on command line. Alternatively, you can
Hi Hemanth,
Nice to see this. I didnot know about this till now.
But few one more issue.. the dump file did not get created.. The
following are the logs
ttempt_201302211510_81218_m_00_0:
Hi,
Dumping heap to ./heapdump.hprof
File myheapdump.hprof does not exist.
The file names don't match - can you check your script / command line args.
Thanks
hemanth
On Wed, Mar 27, 2013 at 3:21 PM, nagarjuna kanamarlapudi
nagarjuna.kanamarlap...@gmail.com wrote:
Hi Hemanth,
Nice to
Awesome,
Working good .. need to start analysing why only 300MB is free out of
configured 1.9GB heap for mappers and reducers.
On Wed, Mar 27, 2013 at 3:25 PM, Hemanth Yamijala yhema...@thoughtworks.com
wrote:
Hi,
Dumping heap to ./heapdump.hprof
File myheapdump.hprof does not exist.
If your task is running out of memory, you could add the option *
-XX:+HeapDumpOnOutOfMemoryError *
*to *mapred.child.java.opts (along with the heap memory). However, I am not
sure where it stores the dump.. You might need to experiment a little on
it.. Will try and send out the info if I get
Hi,
I tried to use the -XX:+HeapDumpOnOutOfMemoryError. Unfortunately, like I
suspected, the dump goes to the current work directory of the task attempt
as it executes on the cluster. This directory is cleaned up once the task
is done. There are options to keep failed task files or task files
Create a dump.sh on hdfs.
$ hadoop dfs -cat /user/knoguchi/dump.sh
#!/bin/sh
hadoop dfs -put myheapdump.hprof /tmp/myheapdump_knoguchi/${PWD//\//_}.hprof
Run your job with
-Dmapred.create.symlink=yes
-Dmapred.cache.files=hdfs:///user/knoguchi/dump.sh#dump.sh
Koji,
Works beautifully. Thanks a lot. I learnt at least 3 different things with
your script today !
Hemanth
On Tue, Mar 26, 2013 at 9:41 PM, Koji Noguchi knogu...@yahoo-inc.comwrote:
Create a dump.sh on hdfs.
$ hadoop dfs -cat /user/knoguchi/dump.sh
#!/bin/sh
hadoop dfs -put
Hi,
The free memory might be low, just because GC hasn't reclaimed what it can.
Can you just try reading in the data you want to read and see if that works
?
Thanks
Hemanth
On Mon, Mar 25, 2013 at 10:32 AM, nagarjuna kanamarlapudi
nagarjuna.kanamarlap...@gmail.com wrote:
io.sort.mb = 256 MB
Hi Hemanth,
I tried out your suggestion loading 420 MB file into memory. It threw java
heap space error.
I am not sure where this 1.6 GB of configured heap went to ?
On Mon, Mar 25, 2013 at 12:01 PM, Hemanth Yamijala
yhema...@thoughtworks.com wrote:
Hi,
The free memory might be low, just
Hmm. How are you loading the file into memory ? Is it some sort of memory
mapping etc ? Are they being read as records ? Some details of the app will
help
On Mon, Mar 25, 2013 at 2:14 PM, nagarjuna kanamarlapudi
nagarjuna.kanamarlap...@gmail.com wrote:
Hi Hemanth,
I tried out your
I have a lookup file which I need in the mapper. So I am trying to read the
whole file and load it into list in the mapper.
For each and every record Iook in this file which I got from distributed cache.
—
Sent from iPhone
On Mon, Mar 25, 2013 at 6:39 PM, Hemanth Yamijala
Hi,
One option to find what could be taking the memory is to use jmap on the
running task. The steps I followed are:
- I ran a sleep job (which comes in the examples jar of the distribution -
effectively does nothing in the mapper / reducer).
- From the JobTracker UI looked at a map task attempt
Hi hemanth,
This sounds interesting, will out try out that on the pseudo cluster. But
the real problem for me is, the cluster is being maintained by third party.
I only have have a edge node through which I can submit the jobs.
Is there any other way of getting the dump instead of physically
Hi,
I configured my child jvm heap to 2 GB. So, I thought I could really read
1.5GB of data and store it in memory (mapper/reducer).
I wanted to confirm the same and wrote the following piece of code in the
configure method of mapper.
@Override
public void configure(JobConf job) {
did you set the min heap size == your max head size? if you didn't,
free memory only shows you the difference between used and commit, not
used and max.
On 3/24/13, nagarjuna kanamarlapudi nagarjuna.kanamarlap...@gmail.com wrote:
Hi,
I configured my child jvm heap to 2 GB. So, I thought I
Hi Ted,
As far as i can recollect, I onl configured these parameters
property
namemapred.child.java.opts/name
value-Xmx2048m/value
descriptionthis number is the number of megabytes of memory that
each mapper and each reducers will have available to use. If jobs start
running out
The MapTask may consume some memory of its own as well. What is your
io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to?
On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi
nagarjuna.kanamarlap...@gmail.com wrote:
Hi,
I configured my child jvm heap to 2 GB. So, I thought I could
io.sort.mb = 256 MB
On Monday, March 25, 2013, Harsh J wrote:
The MapTask may consume some memory of its own as well. What is your
io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to?
On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi
nagarjuna.kanamarlap...@gmail.com
20 matches
Mail list logo
