Hi Guys, Thanks very much for getting back to me.
Thanks Chris - the idea of slitting the data is a great suggestion. Yes Wangda, I was restarting after changing the configs I’ve been checking the relationship between what I thought was in my config files and what hadoop thought were in them. With: // Print out Config file settings for testing. for (Entry<String, String> entry: conf){ System.out.printf("%s=%s\n", entry.getKey(), entry.getValue()); } There were anomalies ;0( Now that my hadoop reflects the values that are in my config files - I just get the message “Killed” without any explanation. Unfortunately, where I was applying changes incrementally and testing I’ve applied all the changes all at once. I’m now backing out the changes I made slowly to see where it starts to reflect what I expect. Regards, Chris MacKenzie telephone: 0131 332 6967 email: stu...@chrismackenziephotography.co.uk corporate: www.chrismackenziephotography.co.uk <http://www.chrismackenziephotography.co.uk/> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts> <http://www.linkedin.com/in/chrismackenziephotography/> From: Chris Mawata <chris.maw...@gmail.com> Reply-To: <user@hadoop.apache.org> Date: Thursday, 17 July 2014 16:15 To: <user@hadoop.apache.org> Subject: Re: Configuration set up questions - Container killed on request. Exit code is 143 Another thing to try is smaller input splits if your data can be broken up into smaller files that can be independently processed. That way s you get more but smaller map tasks. You could also use more but smaller reducers. The many files will tax your NameNode more but you might get to use all you cores. On Jul 17, 2014 9:07 AM, "Chris MacKenzie" <stu...@chrismackenziephotography.co.uk> wrote: Hi Chris, Thanks for getting back to me. I will set that value to 10 I have just tried this. https://support.gopivotal.com/hc/en-us/articles/201462036-Mapreduce-YARN-Me mory-Parameters Setting both to mapreduce.map.memory.mb mapreduce.reduce.memory.mb. Though after setting it I didn’t get the expected change. As the output was still 2.1 GB of 2.1 GB virtual memory used. Killing container Regards, Chris MacKenzie telephone: 0131 332 6967 email: stu...@chrismackenziephotography.co.uk corporate: www.chrismackenziephotography.co.uk <http://www.chrismackenziephotography.co.uk> <http://www.chrismackenziephotography.co.uk/> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts> <http://www.linkedin.com/in/chrismackenziephotography/> From: Chris Mawata <chris.maw...@gmail.com> Reply-To: <user@hadoop.apache.org> Date: Thursday, 17 July 2014 13:36 To: Chris MacKenzie <stu...@chrismackenziephotography.co.uk> Cc: <user@hadoop.apache.org> Subject: Re: Configuration set up questions - Container killed on request. Exit code is 143 Hi Chris MacKenzie, I have a feeling (I am not familiar with the kind of work you are doing) that your application is memory intensive. 8 cores per node and only 12GB is tight. Try bumping up the yarn.nodemanager.vmem-pmem-ratio Chris Mawata On Wed, Jul 16, 2014 at 11:37 PM, Chris MacKenzie <stu...@chrismackenziephotography.co.uk> wrote: Hi, Thanks Chris Mawata I’m working through this myself, but wondered if anyone could point me in the right direction. I have attached my configs. I’m using hadoop 2.41 My system is: 32 Clusters 8 processors per machine 12 gb ram Available disk space per node 890 gb This is my current error: mapreduce.Job (Job.java:printTaskEvents(1441)) - Task Id : attempt_1405538067846_0006_r_000000_1, Status : FAILED Container [pid=25848,containerID=container_1405538067846_0006_01_000004] is running beyond virtual memory limits. Current usage: 439.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1405538067846_0006_01_000004 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 25853 25848 25848 25848 (java) 2262 193 2268090368 112050 /usr/java/latest//bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx768m -Djava.io.tmpdir=/tmp/hadoop-cm469/nm-local-dir/usercache/cm469/appcache/ap plication_1405538067846_0006/container_1405538067846_0006_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/scratch/extra/cm469/hadoop-2.4.1/logs/userlog s/application_1405538067846_0006/container_1405538067846_0006_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 137.195.143.103 59056 attempt_1405538067846_0006_r_000000_1 4 |- 25848 25423 25848 25848 (bash) 0 0 108613632 333 /bin/bash -c /usr/java/latest//bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx768m -Djava.io.tmpdir=/tmp/hadoop-cm469/nm-local-dir/usercache/cm469/appcache/ap plication_1405538067846_0006/container_1405538067846_0006_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/scratch/extra/cm469/hadoop-2.4.1/logs/userlog s/application_1405538067846_0006/container_1405538067846_0006_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 137.195.143.103 59056 attempt_1405538067846_0006_r_000000_1 4 1>/scratch/extra/cm469/hadoop-2.4.1/logs/userlogs/application_1405538067846 _0006/container_1405538067846_0006_01_000004/stdout 2>/scratch/extra/cm469/hadoop-2.4.1/logs/userlogs/application_1405538067846 _0006/container_1405538067846_0006_01_000004/stderr Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 Regards, Chris MacKenzie telephone: 0131 332 6967 email: stu...@chrismackenziephotography.co.uk corporate: www.chrismackenziephotography.co.uk <http://www.chrismackenziephotography.co.uk> <http://www.chrismackenziephotography.co.uk> <http://www.chrismackenziephotography.co.uk/> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts> <http://www.linkedin.com/in/chrismackenziephotography/> From: Chris Mawata <chris.maw...@gmail.com> Reply-To: <user@hadoop.apache.org> Date: Thursday, 17 July 2014 02:10 To: <user@hadoop.apache.org> Subject: Re: Can someone shed some light on this ? - java.io.IOException: Spill failed I would post the configuration files -- easier for someone to spot something wrong than to imagine what configuration would get you to that stacktrace. The part Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1405523201400_0006_m_000000_0_spill_8.out would suggest you might not have hadoop.tmp.dir set (?) On Wed, Jul 16, 2014 at 1:02 PM, Chris MacKenzie <stu...@chrismackenziephotography.co.uk> wrote: Hi, Is this a coding or a setup issue ? I¹m using Hadoop 2.41 My program is doing a concordance on 500,000 sequences of 400 chars. My cluster set is 32 data nodes and two masters. The exact error is: Error: java.io.IOException: Spill failed at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas k.java:1535) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput OutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp er.java:112) at par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java: 96) at par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java: 1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j ava:1556) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1405523201400_0006_m_000000_0_spill_8.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW rite(LocalDirAllocator.java:402) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat or.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat or.java:131) at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil es.java:159) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java: 1566) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85 3) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja va:1505) Regards, Chris