[jira] [Commented] (MAPREDUCE-6895) Job end notification not send due to YarnRuntimeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048711#comment-16048711
]
yunjiong zhao commented on MAPREDUCE-6895:
--
Thanks for your time review this patch.
> Job end notification not send due to YarnRuntimeException
> -
>
> Key: MAPREDUCE-6895
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6895
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.4.1, 2.8.0, 2.7.3
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: MAPREDUCE-6895.001.patch, MAPREDUCE-6895.002.patch
>
>
> MRAppMaster.this.stop() throw out YarnRuntimeException as below log shows, it
> caused job end notification not send.
> {quote}
> 2017-05-24 12:14:02,165 WARN [Thread-693]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:531)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:360)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1476)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1090)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:605)
> Caused by: java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1528)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
> org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1754)
> at
> org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1088)
> at org.apache.avro.io.JsonEncoder.flush(JsonEncoder.java:67)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventWriter.write(EventWriter.java:67)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$MetaInfo.writeEvent(JobHistoryEventHandler.java:886)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:520)
> ... 11 more
> 2017-05-24 12:14:02,165 INFO [Thread-693]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Exiting MR AppMaster..GoodBye!
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6895) Job end notification not send due to YarnRuntimeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yunjiong zhao updated MAPREDUCE-6895:
-
Attachment: MAPREDUCE-6895.002.patch
Thanks [~leftnoteasy] and [~djp] for review.
Upload one patch to remove below lines.
{code}
// Do we need job-end notification?
if (userUrl == null) {
Log.getLog().info("Job end notification URL not set, skipping.");
return;
}
{code}
Personally I think it's better to keep it in case in future someone called
notifier.notify(report) without MRJobConfig.MR_JOB_END_NOTIFICATION_URL set.
> Job end notification not send due to YarnRuntimeException
> -
>
> Key: MAPREDUCE-6895
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6895
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.4.1, 2.8.0, 2.7.3
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: MAPREDUCE-6895.001.patch, MAPREDUCE-6895.002.patch
>
>
> MRAppMaster.this.stop() throw out YarnRuntimeException as below log shows, it
> caused job end notification not send.
> {quote}
> 2017-05-24 12:14:02,165 WARN [Thread-693]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:531)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:360)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1476)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1090)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:605)
> Caused by: java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1528)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
> org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1754)
> at
> org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1088)
> at org.apache.avro.io.JsonEncoder.flush(JsonEncoder.java:67)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventWriter.write(EventWriter.java:67)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$MetaInfo.writeEvent(JobHistoryEventHandler.java:886)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:520)
> ... 11 more
> 2017-05-24 12:14:02,165 INFO [Thread-693]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Exiting MR AppMaster..GoodBye!
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6895) Job end notification not send due to YarnRuntimeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yunjiong zhao updated MAPREDUCE-6895:
-
Affects Version/s: 2.8.0
2.7.3
Status: Patch Available (was: Open)
> Job end notification not send due to YarnRuntimeException
> -
>
> Key: MAPREDUCE-6895
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6895
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.7.3, 2.8.0, 2.4.1
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: MAPREDUCE-6895.001.patch
>
>
> MRAppMaster.this.stop() throw out YarnRuntimeException as below log shows, it
> caused job end notification not send.
> {quote}
> 2017-05-24 12:14:02,165 WARN [Thread-693]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:531)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:360)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1476)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1090)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:605)
> Caused by: java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1528)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
> org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1754)
> at
> org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1088)
> at org.apache.avro.io.JsonEncoder.flush(JsonEncoder.java:67)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventWriter.write(EventWriter.java:67)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$MetaInfo.writeEvent(JobHistoryEventHandler.java:886)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:520)
> ... 11 more
> 2017-05-24 12:14:02,165 INFO [Thread-693]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Exiting MR AppMaster..GoodBye!
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6895) Job end notification not send due to YarnRuntimeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yunjiong zhao updated MAPREDUCE-6895:
-
Attachment: MAPREDUCE-6895.001.patch
This patch will send notification in finally block if it didn't send due to
runtime exception.
> Job end notification not send due to YarnRuntimeException
> -
>
> Key: MAPREDUCE-6895
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6895
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.4.1
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: MAPREDUCE-6895.001.patch
>
>
> MRAppMaster.this.stop() throw out YarnRuntimeException as below log shows, it
> caused job end notification not send.
> {quote}
> 2017-05-24 12:14:02,165 WARN [Thread-693]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:531)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:360)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1476)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1090)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:605)
> Caused by: java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1528)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
> org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1754)
> at
> org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1088)
> at org.apache.avro.io.JsonEncoder.flush(JsonEncoder.java:67)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventWriter.write(EventWriter.java:67)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$MetaInfo.writeEvent(JobHistoryEventHandler.java:886)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:520)
> ... 11 more
> 2017-05-24 12:14:02,165 INFO [Thread-693]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Exiting MR AppMaster..GoodBye!
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6895) Job end notification not send due to YarnRuntimeException
yunjiong zhao created MAPREDUCE-6895:
Summary: Job end notification not send due to YarnRuntimeException
Key: MAPREDUCE-6895
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6895
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster
Affects Versions: 2.4.1
Reporter: yunjiong zhao
Assignee: yunjiong zhao
MRAppMaster.this.stop() throw out YarnRuntimeException as below log shows, it
caused job end notification not send.
{quote}
2017-05-24 12:14:02,165 WARN [Thread-693]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.nio.channels.ClosedChannelException
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:531)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:360)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1476)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1090)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:605)
Caused by: java.nio.channels.ClosedChannelException
at
org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1528)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1754)
at
org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1088)
at org.apache.avro.io.JsonEncoder.flush(JsonEncoder.java:67)
at
org.apache.hadoop.mapreduce.jobhistory.EventWriter.write(EventWriter.java:67)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$MetaInfo.writeEvent(JobHistoryEventHandler.java:886)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:520)
... 11 more
2017-05-24 12:14:02,165 INFO [Thread-693]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Exiting MR AppMaster..GoodBye!
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-5541) Improved algorithm for whether need speculative task
[ https://issues.apache.org/jira/browse/MAPREDUCE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yunjiong zhao resolved MAPREDUCE-5541. -- Resolution: Won't Fix > Improved algorithm for whether need speculative task > > > Key: MAPREDUCE-5541 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5541 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 1.2.1 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: MAPREDUCE-5541-branch-1.2.patch, > MAPREDUCE-5541-branch-1.2.patch > > > Most of time, tasks won't start running at same time. > In this case hasSpeculativeTask in TaskInProgress not working very well. > Some times, some tasks just start running, and scheduler already decide it > need speculative task to run. > And this waste a lot of resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
