Hello,
I see in the archives that last month somebody had a question along these
lines, but there didn't seem to be a real resolution. I am unable to run
any processes due to some issue with the task tracker that I don't
understand.
Command line interactions:
$ bin/hadoop dfs -mkdir input
$ bin/hadoop dfs -copyFromLocal ~/inputfiles.* input
$ bin/hadoop jar hadoop-0.19.0-examples.jar wordcount input output
09/10/01 13:02:56 INFO mapred.FileInputFormat: Total input paths to
process : 10
09/10/01 13:02:57 INFO mapred.JobClient: Running job: job_200910011254_0003
09/10/01 13:02:58 INFO mapred.JobClient: map 0% reduce 0%
09/10/01 13:03:07 INFO mapred.JobClient: Task Id :
attempt_200910011254_0003_m_000045_0, Status : FAILED
java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:424)
09/10/01 13:03:07 WARN mapred.JobClient: Error reading task
outputhttp://tile.cse.ohio-state.edu:50060/tasklog?plaintext=true&taskid=attempt_200910011254_0003_m_000045_0&filter=stdout
09/10/01 13:03:07 WARN mapred.JobClient: Error reading task
outputhttp://tile.cse.ohio-state.edu:50060/tasklog?plaintext=true&taskid=attempt_200910011254_0003_m_000045_0&filter=stderr
09/10/01 13:03:10 INFO mapred.JobClient: Task Id :
With subsequent attempts that give the same error.
From the log file,
hadoop-bromberg-tasktracker-roof.cse.ohio-state.edu.log:
2009-10-01 12:37:48,242 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = roof.cse.ohio-state.edu/164.107.116.105
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.19.1-dev
STARTUP_MSG: build = -r ; compiled by 'cbrew' on Tue Jan 20 15:22:44 EST 2009
************************************************************/
2009-10-01 12:37:49,236 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2009-10-01 12:37:49,261 INFO org.mortbay.util.Credential: Checking
Resource aliases
2009-10-01 12:37:50,437 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.WebAppli
cationhand...@149d886
2009-10-01 12:37:50,567 INFO org.mortbay.util.Container: Started
WebApplicationContext[/static,/sta
tic]
2009-10-01 12:37:50,858 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.WebAppli
cationhand...@19f3736
2009-10-01 12:37:50,859 INFO org.mortbay.util.Container: Started
WebApplicationContext[/logs,/logs]
2009-10-01 12:37:51,127 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.WebAppli
cationhand...@c8376b
2009-10-01 12:37:51,138 INFO org.mortbay.util.Container: Started
WebApplicationContext[/,/]
2009-10-01 12:37:51,148 INFO org.mortbay.http.SocketListener: Started
SocketListener on 0.0.0.0:500
60
2009-10-01 12:37:51,148 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.ser...@dc6a77
2009-10-01 12:37:51,163 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics wit
h processName=TaskTracker, sessionId=2009-10-01 12:37:51,215 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=TaskTracker, port=43163
2009-10-01 12:37:51,347 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2009-10-01 12:37:51,350 INFO org.apache.hadoop.ipc.Server: IPC Server listener
on 43163: starting
2009-10-01 12:37:51,352 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 43163: starting
2009-10-01 12:37:51,354 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1
on 43163: starting
2009-10-01 12:37:51,354 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2
on 43163: starting
2009-10-01 12:37:51,362 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker
up at: localhost.loc
aldomain/127.0.0.1:43163
2009-10-01 12:37:51,362 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tracker tracker_roof.cs
e.ohio-state.edu:localhost.localdomain/127.0.0.1:43163
2009-10-01 12:37:51,363 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 43163: starting
2009-10-01 12:37:51,473 INFO org.apache.hadoop.mapred.TaskTracker: Starting
thread: Map-events fetc
her for all reduce tasks on
tracker_roof.cse.ohio-state.edu:localhost.localdomain/127.0.0.1:43163
2009-10-01 12:37:51,476 INFO org.apache.hadoop.mapred.IndexCache: IndexCache
created with max memor
y = 10485760
2009-10-01 12:37:51,483 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker's tasksMaxMem is not set. TaskMemoryManager is disabled.
2009-10-01 12:39:31,965 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask):
attempt_200910011237_0001_m_000045_0
2009-10-01 12:39:31,974 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_20091
0011237_0001_m_000045_0
2009-10-01 12:39:31,974 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free sl
ots : 2 and trying to launch attempt_200910011237_0001_m_000045_0
2009-10-01 12:39:32,750 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner
constructed JVM ID:
jvm_200910011237_0001_m_-834551651
2009-10-01 12:39:32,751 INFO org.apache.hadoop.mapred.JvmManager: JVM
Runner jvm_200910011237_0001_m_-834551651 spawned.
2009-10-01 12:39:32,773 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_200910011237_0001_m_-83
4551651 exited. Number of tasks it ran: 0
2009-10-01 12:39:32,776 WARN org.apache.hadoop.mapred.TaskRunner:
attempt_200910011237_0001_m_00004
5_0 Child Error
java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:424)
2009-10-01 12:39:35,783 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_200910011237_0001_m_000045_0 done; removing files.
I believe that's where it starts to repeat. Similar log files for the
other nodes in the cloud.
I have looked in the conf/ files for a place to set the tasksMaxMem but
can only find in hadoop-default.xml:
<property>
<name>mapred.tasktracker.taskmemorymanager.monitoring-interval</name>
<value>5000</value>
</property>
I am using hadoop-0.19.0. I had a working system a month ago, but after a
complete system reboot, I've had some issues with hadoop. Everything
seems to go okay now with copying data in/out of the hdfs, but I can't run
any processes. When I do jps on the namenode (roof), I get:
5417 DataNode
5287 NameNode
5779 TaskTracker
5653 JobTracker
5578 SecondaryNameNode
6566 Jps
So it seems like everything loads up fine.
Any pointers on how to fix the tasktracker problem? Or is there a hidden
problem here?
Thanks,
Ilana
Ilana Heintz
Department of Linguistics
Ohio State University
http://www.ling.ohio-state.edu/~bromberg
