Thanks Rekha, I was missing the new library
(hadoop-0.20.1-hdfs-core.jar) in my client.
It seems to run a little further but I'm now getting a
ClassCastException returned by the mapper. Note, this worked with the
0.19 load, so I'm assuming there's something additional in the
configuration that I'm missing. Can anyone help?
java.lang.ClassCastException: org.apache.hadoop.mapred.MultiFileSplit
cannot be cast to org.apache.hadoop.mapred.FileSplit
at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat
.java:54)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Cheers Arv
-----Original Message-----
From: Rekha Joshi [mailto:rekha...@yahoo-inc.com]
Sent: November 26, 2009 11:45 PM
To: common-user@hadoop.apache.org
Subject: Re: Hadoop 0.20 map/reduce Failing for old API
The exit status of 1 usually indicates configuration issues, incorrect
command invocation in hadoop 0.20 (incorrect params), if not JVM crash.
In your logs there is no indication of crash, but some paths/command can
be the cause. Can you check if your lib paths/data paths are correct?
If it is a memory intensive task, you may also try values on
mapred.child.java.opts /mapred.job.map.memory.mb.Thanks!
On 11/27/09 1:28 AM, "Arv Mistry" <a...@kindsight.net> wrote:
Hi,
We've recently upgraded to hadoop 0.20. Writing to HDFS seems to be
working fine, but the map/reduce jobs are failing with the following
exception. Note, we have not moved to the new map/reduce API yet. In the
client that launches the job, the only change I have made is to now load
the three files; core-site.xml, hdfs-site.xml and mapred-site.xml rather
than the hadoop-site.xml. Any ideas?
INFO | jvm 1 | 2009/11/26 13:47:26 | 2009-11-26 13:47:26,328 INFO
[FileInputFormat] Total input paths to process : 711
INFO | jvm 1 | 2009/11/26 13:47:28 | 2009-11-26 13:47:28,033 INFO
[JobClient] Running job: job_200911241319_0003
INFO | jvm 1 | 2009/11/26 13:47:29 | 2009-11-26 13:47:29,036 INFO
[JobClient] map 0% reduce 0%
INFO | jvm 1 | 2009/11/26 13:47:36 | 2009-11-26 13:47:36,068 INFO
[JobClient] Task Id : attempt_200911241319_0003_m_000003_0, Status :
FAILED
INFO | jvm 1 | 2009/11/26 13:47:36 | java.io.IOException: Task
process exit with nonzero status of 1.
INFO | jvm 1 | 2009/11/26 13:47:36 | at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
INFO | jvm 1 | 2009/11/26 13:47:36 |
INFO | jvm 1 | 2009/11/26 13:47:36 | 2009-11-26 13:47:36,094 WARN
[JobClient] Error reading task
outputhttp://dev-cs1.ca.kindsight.net:50060/tasklog?plaintext=true&taski
d=attempt_200911241319_0003_m_000003_0&filter=stdout
INFO | jvm 1 | 2009/11/26 13:47:36 | 2009-11-26 13:47:36,096 WARN
[JobClient] Error reading task
outputhttp://dev-cs1.ca.kindsight.net:50060/tasklog?plaintext=true&taski
d=attempt_200911241319_0003_m_000003_0&filter=stderr
INFO | jvm 1 | 2009/11/26 13:47:51 | 2009-11-26 13:47:51,162 INFO
[JobClient] Task Id : attempt_200911241319_0003_m_000000_0, Status :
FAILED
INFO | jvm 1 | 2009/11/26 13:47:51 | java.io.IOException: Task
process exit with nonzero status of 1.
INFO | jvm 1 | 2009/11/26 13:47:51 | at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
INFO | jvm 1 | 2009/11/26 13:47:51 |
INFO | jvm 1 | 2009/11/26 13:47:51 | 2009-11-26 13:47:51,166 WARN
[JobClient] Error reading task
outputhttp://dev-cs1.ca.kindsight.net:50060/tasklog?plaintext=true&taski
d=attempt_200911241319_0003_m_000000_0&filter=stdout
INFO | jvm 1 | 2009/11/26 13:47:51 | 2009-11-26 13:47:51,167 WARN
[JobClient] Error reading task
outputhttp://dev-cs1.ca.kindsight.net:50060/tasklog?plaintext=true&taski
d=attempt_200911241319_0003_m_000000_0&filter=stderr
INFO | jvm 1 | 2009/11/26 13:47:52 | 2009-11-26 13:47:52,173 INFO
[JobClient] map 50% reduce 0%
INFO | jvm 1 | 2009/11/26 13:48:03 | 2009-11-26 13:48:03,219 INFO
[JobClient] Task Id : attempt_200911241319_0003_m_000001_0, Status :
FAILED
INFO | jvm 1 | 2009/11/26 13:48:03 | Map output lost, rescheduling:
getMapOutput(attempt_200911241319_0003_m_000001_0,0) failed :
INFO | jvm 1 | 2009/11/26 13:48:03 |
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_200911241319_0003/attempt_200911241319_0003_m_0
00001_0/output/file.out.index in any of the configured local directories
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathT
oRead(LocalDirAllocator.java:389)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAlloca
tor.java:138)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.
java:2886)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:2
16)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandler
Collection.java:230)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.Server.handle(Server.java:324)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConne
ction.java:864)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:
409)
INFO | jvm 1 | 2009/11/26 13:48:03 | at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
:522)
INFO | jvm 1 | 2009/11/26 13:48:03 |
INFO | jvm 1 | 2009/11/26 13:48:06 | 2009-11-26 13:48:06,235 INFO
[JobClient] Task Id : attempt_200911241319_0003_m_000000_1, Status :
FAILED
INFO | jvm 1 | 2009/11/26 13:48:06 | java.io.IOException: Task
process exit with nonzero status of 1.
INFO | jvm 1 | 2009/11/26 13:48:06 | at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
INFO | jvm 1 | 2009/11/26 13:48:06 |
INFO | jvm 1 | 2009/11/26 13:48:06 | java.io.IOException: Task
process exit with nonzero status of 1.
INFO | jvm 1 | 2009/11/26 13:48:06 | at
org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
INFO | jvm 1 | 2009/11/26 13:48:06 |
INFO | jvm 1 | 2009/11/26 13:48:06 | 2009-11-26 13:48:06,239 WARN
[JobClient] Error reading task
outputhttp://dev-cs1.ca.kindsight.net:50060/tasklog?plaintext=true&taski
d=attempt_200911241319_0003_m_000000_1&filter=stdout
INFO | jvm 1 | 2009/11/26 13:48:06 | 2009-11-26 13:48:06,245 WARN
[JobClient] Error reading task
outputhttp://dev-cs1.ca.kindsight.net:50060/tasklog?plaintext=true&taski
d=attempt_200911241319_0003_m_000000_1&filter=stderr
INFO | jvm 1 | 2009/11/26 13:48:13 | 2009-11-26 13:48:13,302 INFO
[JobClient] map 0% reduce 0%
INFO | jvm 1 | 2009/11/26 13:48:16 | 2009-11-26 13:48:16,315 INFO
[JobClient] map 50% reduce 0%
INFO | jvm 1 | 2009/11/26 13:48:18 | 2009-11-26 13:48:18,324 INFO
[JobClient] Task Id : attempt_200911241319_0003_m_000000_2, Status :
FAILED
INFO | jvm 1 | 2009/11/26 13:48:18 | java.io.IOException: Task
process exit with nonzero status of 1.
