[jira] [Created] (MAPREDUCE-7114) Make FrameworkUploader symlink ignore improvement

2018-06-22 Thread Gergo Repas (JIRA) 
Gergo Repas created MAPREDUCE-7114:
--

 Summary: Make FrameworkUploader symlink ignore improvement
 Key: MAPREDUCE-7114
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7114
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 3.1.0
Reporter: Gergo Repas
Assignee: Gergo Repas


/dir1/dir2/./hadoop-common.jar style symlinks are not ignored, even if they 
link to a file within the same directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7028) Concurrent task progress updates causing NPE in Application Master

2017-12-20 Thread Gergo Repas (JIRA) 
Gergo Repas created MAPREDUCE-7028:
--

 Summary: Concurrent task progress updates causing NPE in 
Application Master
 Key: MAPREDUCE-7028
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7028
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
Reporter: Gergo Repas
Assignee: Gergo Repas


Concurrent task progress updates can cause a NullPointerException in the 
Application Master (stack trace is with code at current trunk):

{quote}
2017-12-20 06:49:42,369 INFO [IPC Server handler 9 on 39501] 
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
attempt_1513780867907_0001_m_02_0 is : 0.02677883
2017-12-20 06:49:42,369 INFO [IPC Server handler 13 on 39501] 
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
attempt_1513780867907_0001_m_02_0 is : 0.02677883
2017-12-20 06:49:42,383 FATAL [AsyncDispatcher event handler] 
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.NullPointerException
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:2450)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:2433)
at 
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
at 
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at 
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
at 
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1362)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:154)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1543)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1535)
at 
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at 
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
at java.lang.Thread.run(Thread.java:748)
2017-12-20 06:49:42,385 INFO [IPC Server handler 13 on 39501] 
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
attempt_1513780867907_0001_m_02_0 is : 0.02677883
2017-12-20 06:49:42,386 INFO [AsyncDispatcher ShutDown handler] 
org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
{quote}

This happened naturally in several big wordcount runs, and I could reproduce 
this reliably by artificially making task updates more frequent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6984) MR AM to clean up temporary files from previous attempt

2017-10-17 Thread Gergo Repas (JIRA) 
Gergo Repas created MAPREDUCE-6984:
--

 Summary: MR AM to clean up temporary files from previous attempt
 Key: MAPREDUCE-6984
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6984
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: applicationmaster
Affects Versions: 3.0.0-beta1
Reporter: Gergo Repas


When the MR AM restarts, the {outputDir}/_temporary/{appAttemptNumber} 
directory remains on HDFS, even though this directory is not used during the 
next attempt if the restart has been done without recovery. So if recovery is 
not used for the AM restart, then the deletion of this directory can be done 
earlier (at the start of the next attempt). The benefit is that more free HDFS 
space is available for the next attempt.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org