[
https://issues.apache.org/jira/browse/MAPREDUCE-5169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636558#comment-13636558
]
Owen O'Malley edited comment on MAPREDUCE-5169 at 4/19/13 9:07 PM:
-------------------------------------------------------------------
Client Logs
{code}
/usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop/hadoop-examples-1.X.Y.jar
wordcount "-Dmapreduce.reduce.input.limit=-1"
/user/hrt_qa/test_mapred_ha/medium_wordcount_input
/user/hrt_qa/test_mapred_ha/jobtracker-near-submit_wc_output
13/04/17 22:04:10 INFO input.FileInputFormat: Total input paths to process : 20
13/04/17 22:04:10 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
13/04/17 22:04:10 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library [hadoop-lzo rev cf4e7cbf8ed0f0622504d008101c2729dc0c9ff3]
13/04/17 22:04:10 WARN snappy.LoadSnappy: Snappy native library is available
13/04/17 22:04:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/04/17 22:04:10 INFO snappy.LoadSnappy: Snappy native library loaded
13/04/17 22:04:11 INFO mapred.JobClient: Running job: job_201304172049_0008
13/04/17 22:04:12 INFO mapred.JobClient: map 0% reduce 0%
13/04/17 22:04:32 INFO ipc.Client: Retrying connect to server: host:50300.
Already tried 0 time(s); retry policy is MultipleLinearRandomRetry[6x10000ms,
10x60000ms]
java.io.IOException: The job appears to have been removed.
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.updateStatus(JobClient.java:241)
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.isComplete(JobClient.java:321)
at org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1382)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:583)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
{code}
Jobtracker logs
{code}
2013-04-17 22:04:11,305 INFO org.apache.hadoop.mapred.JobInProgress:
job_201304172049_0008: nMaps=180 nReduces=1 max=-1
2013-04-17 22:04:11,357 INFO org.apache.hadoop.mapred.JobQueuesManager: Job
job_201304172049_0008 submitted to queue default
2013-04-17 22:04:11,357 INFO org.apache.hadoop.mapred.JobTracker: Job
job_201304172049_0008 added successfully for user 'hrt_qa' to queue 'default'
2013-04-17 22:04:14,268 INFO org.apache.hadoop.mapred.JobInitializationPoller:
Passing to Initializer Job Id :job_201304172049_0008 User: user Queue : default
2013-04-17 22:04:15,181 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at
************************************************************/
2013-04-17 22:04:15,204 INFO org.apache.hadoop.mapred.JobInitializationPoller:
Initializing job : job_201304172049_0008 in Queue default For user : user
2013-04-17 22:04:15,204 INFO org.apache.hadoop.mapred.JobTracker: Initializing
job_201304172049_0008
2013-04-17 22:04:15,204 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201304172049_0008
2013-04-17 22:04:30,900 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host =
STARTUP_MSG: args = []
STARTUP_MSG: version =
STARTUP_MSG: build =
STARTUP_MSG: java = 1.6.0_31
************************************************************/
...
...
...
2013-04-17 22:04:31,862 WARN org.apache.hadoop.mapred.JobTracker: Job
job_201304172049_0008 does not have valid info/token file so ignoring for
recovery
{code}
was (Author: arpitgupta):
Client Logs
{code}
/usr/lib/hadoop/bin/hadoop jar
/usr/lib/hadoop/hadoop-examples-1.3.0.1.3.0.0-15.jar wordcount
"-Dmapreduce.reduce.input.limit=-1"
/user/hrt_qa/test_mapred_ha/medium_wordcount_input
/user/hrt_qa/test_mapred_ha/jobtracker-near-submit_wc_output
13/04/17 22:04:10 INFO input.FileInputFormat: Total input paths to process : 20
13/04/17 22:04:10 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
13/04/17 22:04:10 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library [hadoop-lzo rev cf4e7cbf8ed0f0622504d008101c2729dc0c9ff3]
13/04/17 22:04:10 WARN snappy.LoadSnappy: Snappy native library is available
13/04/17 22:04:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/04/17 22:04:10 INFO snappy.LoadSnappy: Snappy native library loaded
13/04/17 22:04:11 INFO mapred.JobClient: Running job: job_201304172049_0008
13/04/17 22:04:12 INFO mapred.JobClient: map 0% reduce 0%
13/04/17 22:04:32 INFO ipc.Client: Retrying connect to server:
hor1n02.gq1.ygridcore.net/68.142.244.21:50300. Already tried 0 time(s); retry
policy is MultipleLinearRandomRetry[6x10000ms, 10x60000ms]
java.io.IOException: The job appears to have been removed.
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.updateStatus(JobClient.java:241)
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.isComplete(JobClient.java:321)
at org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1382)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:583)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
{code}
Jobtracker logs
{code}
2013-04-17 22:04:11,305 INFO org.apache.hadoop.mapred.JobInProgress:
job_201304172049_0008: nMaps=180 nReduces=1 max=-1
2013-04-17 22:04:11,357 INFO org.apache.hadoop.mapred.JobQueuesManager: Job
job_201304172049_0008 submitted to queue default
2013-04-17 22:04:11,357 INFO org.apache.hadoop.mapred.JobTracker: Job
job_201304172049_0008 added successfully for user 'hrt_qa' to queue 'default'
2013-04-17 22:04:14,268 INFO org.apache.hadoop.mapred.JobInitializationPoller:
Passing to Initializer Job Id :job_201304172049_0008 User: user Queue : default
2013-04-17 22:04:15,181 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at
************************************************************/
2013-04-17 22:04:15,204 INFO org.apache.hadoop.mapred.JobInitializationPoller:
Initializing job : job_201304172049_0008 in Queue default For user : user
2013-04-17 22:04:15,204 INFO org.apache.hadoop.mapred.JobTracker: Initializing
job_201304172049_0008
2013-04-17 22:04:15,204 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201304172049_0008
2013-04-17 22:04:30,900 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host =
STARTUP_MSG: args = []
STARTUP_MSG: version =
STARTUP_MSG: build =
STARTUP_MSG: java = 1.6.0_31
************************************************************/
...
...
...
2013-04-17 22:04:31,862 WARN org.apache.hadoop.mapred.JobTracker: Job
job_201304172049_0008 does not have valid info/token file so ignoring for
recovery
{code}
> Job recovery fails if job tracker is restarted after the job is submitted but
> before its initialized
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5169
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 1.2.0
> Reporter: Arpit Gupta
>
> This was noticed when within 5 seconds of submitting a word count job, the
> job tracker was restarted. Upon restart the job failed to recover
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
