RE: Task process exit with nonzero status of 134
Alex I don't know why but now I got other error code : On Jobtracker : java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) On taskTracker : 2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.DefaultTaskController: Output from DefaultTaskController's launchTask follows: 2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.TaskController: 2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM Not killed jvm_201112271514_0001_m_-1820714542 but just removed 2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201112271514_0001_m_-1820714542 exited with exit code 1. Number of tasks it ran: 0 2011-12-27 15:14:50,666 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201112271514_0001_m_02_0 : Child Error java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) 2011-12-27 15:14:53,676 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2 From: sokar6...@hotmail.com To: wget.n...@googlemail.com; common-user@hadoop.apache.org Subject: RE: Task process exit with nonzero status of 134 Date: Tue, 27 Dec 2011 13:54:22 + Alex -XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions) Anyway, I tested it but same result. Is it wise to test hadoop with the new jdk7_2 ? Anthony Date: Tue, 27 Dec 2011 13:47:03 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.n...@googlemail.com To: sokar6...@hotmail.com Anthony, 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest update, could be a bug inside. Can you test SE 7u2? Todd mentioned in a older post to use -XX:+UseCompressedOops (hadoop-env.sh). Another option could be to take a closer look at garbage collection with compressed option. - Alex On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier sokar6...@hotmail.com wrote: Alex, Memory available on namenode / Jobtracker : Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 15360M total,11609M used, 3750M free, 311M buffers Swap: 2047M total,1M used, 2046M free, 8833M cached On datanode / Tasktracker : top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26 Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 72373M total, 4321M used,68051M free, 348M buffers Swap: 2047M total,0M used, 2047M free, 2771M cached src/mapred/mapred-default.xml : property namemapred.child.java.opts/name value-Xmx200m/value /property So there should be enough memory Anthony Date: Tue, 27 Dec 2011 11:58:46 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.n...@googlemail.com To: sokar6...@hotmail.com Anthony, How much memory you have available? Did the system going into swap? - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx). - what says top -Hc? - Alex On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier sokar6...@hotmail.com wrote: Hi, I got Nothing in the dmesg I've checked the Tasktracker and this is what I got : / STARTUP_MSG: Starting TaskTracker STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.203.0 STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 / 2011-12-23 15:11:02,275 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2011-12-23 15:11:02,330 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2011-12-23 15:11:02,331 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2011-12-23 15:11:02,331 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started 2011-12-23 15:11:02,597 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-12-23 15:11:02,803 INFO
RE: Task process exit with nonzero status of 134
Alex java -version -Xmx1024m is running fine I was also able to create directory in /users/hadoop-0.20.203.0/log/userlogs But anyway I've doe a fresh install with the 0.20.205.0 version and the mapreduce job completed successfully ! Thank you for your help. Anthony Date: Tue, 27 Dec 2011 17:06:28 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.n...@googlemail.com To: sokar6...@hotmail.com Anthoney, That could be when you try to allocate some dedicated space and it is not available or present on your machine. Did -Xmx has an dedicated memory value that you can provide to JVM on your machine (e.g. -Xmx1024m)? Second - filesystem full (mostly hadoop/userlogs). Ext3 has an limit of 32k links per node. Try to create a directory in (mkdir hadoop/userlogs/123), if you have too much directories an error will occur. - Alex On Tue, Dec 27, 2011 at 3:18 PM, anthony garnier sokar6...@hotmail.com wrote: Alex I don't know why but now I got other error code : On Jobtracker : java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) On taskTracker : 2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.DefaultTaskController: Output from DefaultTaskController's launchTask follows: 2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.TaskController: 2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM Not killed jvm_201112271514_0001_m_-1820714542 but just removed 2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201112271514_0001_m_-1820714542 exited with exit code 1. Number of tasks it ran: 0 2011-12-27 15:14:50,666 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201112271514_0001_m_02_0 : Child Error java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) 2011-12-27 15:14:53,676 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2 From: sokar6...@hotmail.com To: wget.n...@googlemail.com; common-user@hadoop.apache.org Subject: RE: Task process exit with nonzero status of 134 Date: Tue, 27 Dec 2011 13:54:22 + Alex -XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions) Anyway, I tested it but same result. Is it wise to test hadoop with the new jdk7_2 ? Anthony Date: Tue, 27 Dec 2011 13:47:03 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.n...@googlemail.com To: sokar6...@hotmail.com Anthony, 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest update, could be a bug inside. Can you test SE 7u2? Todd mentioned in a older post to use -XX:+UseCompressedOops (hadoop-env.sh). Another option could be to take a closer look at garbage collection with compressed option. - Alex On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier sokar6...@hotmail.com wrote: Alex, Memory available on namenode / Jobtracker : Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 15360M total,11609M used, 3750M free, 311M buffers Swap: 2047M total,1M used, 2046M free, 8833M cached On datanode / Tasktracker : top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26 Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 72373M total, 4321M used,68051M free, 348M buffers Swap: 2047M total,0M used, 2047M free, 2771M cached src/mapred/mapred-default.xml : property namemapred.child.java.opts/name value-Xmx200m/value /property So there should be enough memory Anthony Date: Tue, 27 Dec 2011 11:58:46 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.n...@googlemail.com To: sokar6...@hotmail.com Anthony, How much memory you have available? Did the system going into swap? - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx). - what says top -Hc? - Alex On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier sokar6...@hotmail.com wrote: Hi, I got Nothing in the dmesg I've checked the Tasktracker and this is what I got : / STARTUP_MSG: Starting TaskTracker STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86
Re: Task process exit with nonzero status of 134
Anthony, personally I haven't tested yet, jdk7 have bugs already. Was only a hint to see if the error occurs. I would focus on memory issues, the installed RAM are okay? No errors? My next step would be to downgrade into one JDK earlier to check for a bug. Did you update the OS before? - Alex On Tue, Dec 27, 2011 at 2:54 PM, anthony garnier sokar6...@hotmail.com wrote: Alex -XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions) Anyway, I tested it but same result. Is it wise to test hadoop with the new jdk7_2 ? Anthony Date: Tue, 27 Dec 2011 13:47:03 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.n...@googlemail.com To: sokar6...@hotmail.com Anthony, 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest update, could be a bug inside. Can you test SE 7u2? Todd mentioned in a older post to use -XX:+UseCompressedOops (hadoop-env.sh). Another option could be to take a closer look at garbage collection with compressed option. - Alex On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier sokar6...@hotmail.com wrote: Alex, Memory available on namenode / Jobtracker : Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 15360M total, 11609M used, 3750M free, 311M buffers Swap: 2047M total, 1M used, 2046M free, 8833M cached On datanode / Tasktracker : top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26 Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 72373M total, 4321M used, 68051M free, 348M buffers Swap: 2047M total, 0M used, 2047M free, 2771M cached src/mapred/mapred-default.xml : property namemapred.child.java.opts/name value-Xmx200m/value /property So there should be enough memory Anthony Date: Tue, 27 Dec 2011 11:58:46 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.n...@googlemail.com To: sokar6...@hotmail.com Anthony, How much memory you have available? Did the system going into swap? - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx). - what says top -Hc? - Alex On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier sokar6...@hotmail.com wrote: Hi, I got Nothing in the dmesg I've checked the Tasktracker and this is what I got : / STARTUP_MSG: Starting TaskTracker STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.203.0 STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 / 2011-12-23 15:11:02,275 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2011-12-23 15:11:02,330 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2011-12-23 15:11:02,331 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2011-12-23 15:11:02,331 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started 2011-12-23 15:11:02,597 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2011-12-23 15:11:02,827 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as root 2011-12-23 15:11:02,870 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered. 2011-12-23 15:11:02,871 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered. 2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting SocketReader 2011-12-23 15:11:02,900 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort58709 registered. 2011-12-23 15:11:02,900 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
RE: Task process exit with nonzero status of 134
Alex -XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions) Anyway, I tested it but same result. Is it wise to test hadoop with the new jdk7_2 ? Anthony Date: Tue, 27 Dec 2011 13:47:03 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.n...@googlemail.com To: sokar6...@hotmail.com Anthony, 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest update, could be a bug inside. Can you test SE 7u2? Todd mentioned in a older post to use -XX:+UseCompressedOops (hadoop-env.sh). Another option could be to take a closer look at garbage collection with compressed option. - Alex On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier sokar6...@hotmail.com wrote: Alex, Memory available on namenode / Jobtracker : Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 15360M total,11609M used, 3750M free, 311M buffers Swap: 2047M total,1M used, 2046M free, 8833M cached On datanode / Tasktracker : top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26 Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 72373M total, 4321M used,68051M free, 348M buffers Swap: 2047M total,0M used, 2047M free, 2771M cached src/mapred/mapred-default.xml : property namemapred.child.java.opts/name value-Xmx200m/value /property So there should be enough memory Anthony Date: Tue, 27 Dec 2011 11:58:46 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.n...@googlemail.com To: sokar6...@hotmail.com Anthony, How much memory you have available? Did the system going into swap? - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx). - what says top -Hc? - Alex On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier sokar6...@hotmail.com wrote: Hi, I got Nothing in the dmesg I've checked the Tasktracker and this is what I got : / STARTUP_MSG: Starting TaskTracker STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.203.0 STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 / 2011-12-23 15:11:02,275 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2011-12-23 15:11:02,330 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2011-12-23 15:11:02,331 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2011-12-23 15:11:02,331 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started 2011-12-23 15:11:02,597 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2011-12-23 15:11:02,827 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as root 2011-12-23 15:11:02,870 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered. 2011-12-23 15:11:02,871 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered. 2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting SocketReader 2011-12-23 15:11:02,900 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort58709 registered. 2011-12-23 15:11:02,900 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort58709 registered. 2011-12-23 15:11:02,903 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 58709: starting 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 58709: starting 2011-12-23 15:11:02,904 INFO
Re: Task process exit with nonzero status of 134
with retainTimeStamp:1324735862967 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231312_0002 for user-log deletion with retainTimeStamp:1324735862967 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231142_0002 for user-log deletion with retainTimeStamp:1324735862967 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112230933_0002 for user-log deletion with retainTimeStamp:1324735862967 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231357_0001 for user-log deletion with retainTimeStamp:1324735862967 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231052_0002 for user-log deletion with retainTimeStamp:1324735862967 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231426_0001 for user-log deletion with retainTimeStamp:1324735862967 2011-12-23 15:11:51,501 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201112231511_0001_m_02_0 task's state:UNASSIGNED 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201112231511_0001_m_02_0 which needs 1 slots 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201112231511_0001_m_02_0 which needs 1 slots 2011-12-23 15:11:51,633 WARN org.apache.hadoop.conf.Configuration: /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a attempt to override final parameter: hadoop.tmp.dir; Ignoring. 2011-12-23 15:11:51,635 WARN org.apache.hadoop.conf.Configuration: /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a attempt to override final parameter: dfs.data.dir; Ignoring. 2011-12-23 15:11:51,636 WARN org.apache.hadoop.conf.Configuration: /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a attempt to override final parameter: dfs.name.dir; Ignoring. 2011-12-23 15:11:51,685 INFO org.apache.hadoop.mapred.JobLocalizer: Initializing user root on this TT. 2011-12-23 15:11:51,720 WARN org.apache.hadoop.conf.Configuration: /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a attempt to override final parameter: hadoop.tmp.dir; Ignoring. 2011-12-23 15:11:51,722 WARN org.apache.hadoop.conf.Configuration: /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a attempt to override final parameter: dfs.data.dir; Ignoring. 2011-12-23 15:11:51,723 WARN org.apache.hadoop.conf.Configuration: /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a attempt to override final parameter: dfs.name.dir; Ignoring. 2011-12-23 15:11:51,845 WARN org.apache.hadoop.conf.Configuration: /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a attempt to override final parameter: hadoop.tmp.dir; Ignoring. 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration: /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a attempt to override final parameter: dfs.data.dir; Ignoring. 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration: /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a attempt to override final parameter: dfs.name.dir; Ignoring. 2011-12-23 15:11:51,858 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201112231511_0001_m_-1653678851 2011-12-23 15:11:51,859 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201112231511_0001_m_-1653678851 spawned. 2011-12-23 15:11:51,861 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_02_0/taskjvm.sh 2011-12-23 15:11:52,427 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201112231511_0001_m_-1653678851 given task: attempt_201112231511_0001_m_02_0 2011-12-23 15:11:52,675 WARN org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 134 2011-12-23 15:11:52,675 INFO org.apache.hadoop.mapred.DefaultTaskController: Output from DefaultTaskController's launchTask follows: 2011-12-23 15:11:52,675 INFO org.apache.hadoop.mapred.TaskController: 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of tasks it ran: 0 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201112231511_0001_m_02_0 : Child Error java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) I also checked the logs of the task : logs
RE: Task process exit with nonzero status of 134
Alex, Memory available on namenode / Jobtracker : Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 15360M total,11609M used, 3750M free, 311M buffers Swap: 2047M total,1M used, 2046M free, 8833M cached On datanode / Tasktracker : top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26 Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 72373M total, 4321M used,68051M free, 348M buffers Swap: 2047M total,0M used, 2047M free, 2771M cached src/mapred/mapred-default.xml : property namemapred.child.java.opts/name value-Xmx200m/value /property So there should be enough memory Anthony Date: Tue, 27 Dec 2011 11:58:46 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.n...@googlemail.com To: sokar6...@hotmail.com Anthony, How much memory you have available? Did the system going into swap? - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx). - what says top -Hc? - Alex On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier sokar6...@hotmail.com wrote: Hi, I got Nothing in the dmesg I've checked the Tasktracker and this is what I got : / STARTUP_MSG: Starting TaskTracker STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.203.0 STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 / 2011-12-23 15:11:02,275 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2011-12-23 15:11:02,330 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2011-12-23 15:11:02,331 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2011-12-23 15:11:02,331 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started 2011-12-23 15:11:02,597 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2011-12-23 15:11:02,827 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as root 2011-12-23 15:11:02,870 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered. 2011-12-23 15:11:02,871 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered. 2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting SocketReader 2011-12-23 15:11:02,900 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort58709 registered. 2011-12-23 15:11:02,900 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort58709 registered. 2011-12-23 15:11:02,903 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 58709: starting 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 58709: starting 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 58709: starting 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 58709: starting 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:58709 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 2011-12-23 15:11:02,908 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 58709: starting 2011-12-23 15:11:02,958 INFO org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@7054f9f1 2011-12-23 15:11:02,960 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 2011-12-23 15:11:02,964 INFO
Task process exit with nonzero status of 134
Hi folks, I've just done a fresh install of Hadoop, Namenode and datanode are up, Task/job Tracker also up, but when I run the Map reduce worcount exemple I got this error on Task tracker: 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of tasks it ran: 0 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201112231511_0001_m_02_0 : Child Error java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) And job tracker stuck : # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/ 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to process : 1 11/12/23 15:11:49 INFO mapred.JobClient: Running job: job_201112231511_0001 11/12/23 15:11:50 INFO mapred.JobClient: map 0% reduce 0% I'running hadoop 0.20.203.0, java 1.6.0 rev 25 I've done some googling, apparently the JVM crash hard (maybe Out of memory), does someone have any hint ? Regards, Anthony Garnier /DSIN/ASTI/ETSO IT Center PSA Peugeot Citroen Bessoncourt 90160
Re: Task process exit with nonzero status of 134
Hi, take a look into the logs for the failed attempt at your Tasktracker. Also check the system logs with dmesg or /var/log/kern*. Could be a syskill (segfault). - Alex On Fri, Dec 23, 2011 at 3:32 PM, anthony garnier sokar6...@hotmail.com wrote: Hi folks, I've just done a fresh install of Hadoop, Namenode and datanode are up, Task/job Tracker also up, but when I run the Map reduce worcount exemple I got this error on Task tracker: 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of tasks it ran: 0 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201112231511_0001_m_02_0 : Child Error java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) And job tracker stuck : # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/ 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to process : 1 11/12/23 15:11:49 INFO mapred.JobClient: Running job: job_201112231511_0001 11/12/23 15:11:50 INFO mapred.JobClient: map 0% reduce 0% I'running hadoop 0.20.203.0, java 1.6.0 rev 25 I've done some googling, apparently the JVM crash hard (maybe Out of memory), does someone have any hint ? Regards, Anthony Garnier /DSIN/ASTI/ETSO IT Center PSA Peugeot Citroen Bessoncourt 90160 -- Alexander Lorenz http://mapredit.blogspot.com P Think of the environment: please don't print this email unless you really need to.
Re: java.io.IOException: Task process exit with nonzero status of 134
Here is some update of the problem. I tried a very simple example (word count) and tried to compress the reducer output using default Codec, or GzipCodec. I didn't try LZO to avoid further troubles. I didn't use the combiner class, and set the number of reducer to 1. I am trying it on a 64-bit Debian. My java version is java version 1.6.0 Java(TM) SE Runtime Environment (build 1.6.0-b105) Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-b105, mixed mode) I use SequenceFileOutputFormat, and the output value class is a Vector. At first, I didn't specify the compressOutput and the there was no problem. The sequenceFile was generated correctly. However, when I compressed the output adding the following 3-line-command: SequenceFileOutputFormat.setCompressOutput(conf,true); SequenceFileOutputFormat.setOutputCompressorClass(conf, DefaultCodec.class); SequenceFileOutputFormat.setOutputCompressionType(conf, SequenceFile.CompressionType.BLOCK); The reducer kept on generating error and the task finally crashed. 11/03/09 12:28:05 INFO mapred.JobClient: map 100% reduce 33% 11/03/09 12:28:09 INFO mapred.JobClient: map 100% reduce 0% 11/03/09 12:28:09 INFO mapred.JobClient: Task Id : attempt_201103081457_0024_r_00_0, Status : FAILED java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418) 11/03/09 12:28:19 INFO mapred.JobClient: map 100% reduce 29% 11/03/09 12:28:21 INFO mapred.JobClient: map 100% reduce 0% 11/03/09 12:28:22 INFO mapred.JobClient: Task Id : attempt_201103081457_0024_r_00_1, Status : FAILED java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418) I checked the logs of the reducer (only 1), there is no error in syslog. But in the stdout file, there is an error information: An unexpected error has been detected by Java Runtime Environment: # # SIGFPE (0x8) at pc=0x2b22eecc7b83, pid=13306, tid=1076017504 # # Java VM: Java HotSpot(TM) 64-Bit Server VM (1.6.0-b105 mixed mode) # Problematic frame: # C [ld-linux-x86-64.so.2+0x7b83] # # An error report file with more information is saved as hs_err_pid13306.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp So I would like to know how should I solve this problem. Should I upgrade anything? I guess this problem is not new. Thanks for the information. Shi On 3/8/2011 4:04 PM, Shi Yu wrote: What is the true reason of causing this? I realized there are many reports on web, but couldn't find the exact solution? I have this problem when using compressed sequence file output. SequenceFileOutputFormat.setCompressOutput(conf, true); SequenceFileOutputFormat.setOutputCompressorClass(conf, GzipCodec.class); SequenceFileOutputFormat.setOutputCompressionType(conf, CompressionType.BLOCK); If I remove that 3 lines, everything is fine. I am using hadoop 0.19.2, is there any way to avoid the problem without upgrading hadoop? Thanks! Shi
Re: java.io.IOException: Task process exit with nonzero status of 134
Shi Yu wrote: An unexpected error has been detected by Java Runtime Environment: # # SIGFPE (0x8) at pc=0x2b22eecc7b83, pid=13306, tid=1076017504 # # Java VM: Java HotSpot(TM) 64-Bit Server VM (1.6.0-b105 mixed mode) # Problematic frame: # C [ld-linux-x86-64.so.2+0x7b83] # # An error report file with more information is saved as hs_err_pid13306.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp So it's a 64-bit JVM, and it's invoking a 64-bit ld.so, presumably to load a native library. Is your zlib also 64-bit? I vaguely recall seeing a SIGFPE a while back, but I don't recall what it turned out to be. (Seems unlikely that a 32/64-bit mismatch might cause one, but that's all that comes to mind offhand.) Btw, the zlib codec exists in both native (JNI) and Java forms (well, with the native part buried inside the JDK), so you could also try disabling native support (set io.native.lib.available to false). If it still fails, it's not related to shared libraries. You could also use the bzip2 codec rather than the default one; bzip2 support exists only in Java form for some inexplicable reason. (Perhaps users were blinded by its blazing speed.) Greg
java.io.IOException: Task process exit with nonzero status of 134
What is the true reason of causing this? I realized there are many reports on web, but couldn't find the exact solution? I have this problem when using compressed sequence file output. SequenceFileOutputFormat.setCompressOutput(conf, true); SequenceFileOutputFormat.setOutputCompressorClass(conf, GzipCodec.class); SequenceFileOutputFormat.setOutputCompressionType(conf, CompressionType.BLOCK); If I remove that 3 lines, everything is fine. I am using hadoop 0.19.2, is there any way to avoid the problem without upgrading hadoop? Thanks! Shi
