[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606221#comment-17606221 ] Daniel Ma commented on MAPREDUCE-7351: -- [~prabhujoseph] Thanks for your clarification. it seems Sqoop use abortJob() in a different way. I'll fix the inconsistency in Sqoop side. > CleanupJob during handle of SIGTERM signal > -- > > Key: MAPREDUCE-7351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Shubham Gupta >Priority: Minor > Fix For: 3.3.0 > > Attachments: MAPREDUCE-7351-001.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Currently MR CleanupJob happens when the job is either successful or fail. > But during kill, it is not handled. This leaves all the temporary folders > under the output path. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606137#comment-17606137 ] Prabhu Joseph commented on MAPREDUCE-7351: -- This patch removes the _temporary directory under output path and not the output path. And in case of job succeed or failed, even without this patch the _temporary directory under output path will be removed. > CleanupJob during handle of SIGTERM signal > -- > > Key: MAPREDUCE-7351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Shubham Gupta >Priority: Minor > Fix For: 3.3.0 > > Attachments: MAPREDUCE-7351-001.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Currently MR CleanupJob happens when the job is either successful or fail. > But during kill, it is not handled. This leaves all the temporary folders > under the output path. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605648#comment-17605648 ] Daniel Ma commented on MAPREDUCE-7351: -- This MR will clear all the output data of a MR job including normal scenario. > CleanupJob during handle of SIGTERM signal > -- > > Key: MAPREDUCE-7351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Shubham Gupta >Priority: Minor > Fix For: 3.3.0 > > Attachments: MAPREDUCE-7351-001.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Currently MR CleanupJob happens when the job is either successful or fail. > But during kill, it is not handled. This leaves all the temporary folders > under the output path. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17376197#comment-17376197 ] Prabhu Joseph commented on MAPREDUCE-7351: -- Thanks [~shubhamod] for the patch. Have committed it to trunk. > CleanupJob during handle of SIGTERM signal > -- > > Key: MAPREDUCE-7351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Shubham Gupta >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.0 > > Attachments: MAPREDUCE-7351-001.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Currently MR CleanupJob happens when the job is either successful or fail. > But during kill, it is not handled. This leaves all the temporary folders > under the output path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372962#comment-17372962 ] Shubham Gupta commented on MAPREDUCE-7351: -- Hi, I tried putting it as log.error, but the log is not coming in the log file, mostly logger is also getting closed before this > CleanupJob during handle of SIGTERM signal > -- > > Key: MAPREDUCE-7351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Shubham Gupta >Priority: Minor > Fix For: 3.3.0 > > Attachments: MAPREDUCE-7351-001.patch > > > Currently MR CleanupJob happens when the job is either successful or fail. > But during kill, it is not handled. This leaves all the temporary folders > under the output path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372847#comment-17372847 ] Steve Loughran commented on MAPREDUCE-7351: --- 1. Submit it as a github PR 2. we use SLF4J for logging' thanks > CleanupJob during handle of SIGTERM signal > -- > > Key: MAPREDUCE-7351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Shubham Gupta >Priority: Minor > Fix For: 3.3.0 > > Attachments: MAPREDUCE-7351-001.patch > > > Currently MR CleanupJob happens when the job is either successful or fail. > But during kill, it is not handled. This leaves all the temporary folders > under the output path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372429#comment-17372429 ] Surendra Singh Lilhore commented on MAPREDUCE-7351: --- [~shubhamod], Assigned this Jira to you. > CleanupJob during handle of SIGTERM signal > -- > > Key: MAPREDUCE-7351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Shubham Gupta >Priority: Minor > Attachments: mrappcleanuptempfiles.patch > > > Currently MR CleanupJob happens when the job is either successful or fail. > But during kill, it is not handled. This leaves all the temporary folders > under the output path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371848#comment-17371848 ] Shubham Gupta commented on MAPREDUCE-7351: -- Hey , I have uploaded the patch, please have a look! We will call the "committer.abortJob(jobContext, State.KILLED);" in MRAppMasterShutdownHook. I have tested it by killing the jobs and the _temporary files got cleared up. > CleanupJob during handle of SIGTERM signal > -- > > Key: MAPREDUCE-7351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > > Currently MR CleanupJob happens when the job is either successful or fail. > But during kill, it is not handled. This leaves all the temporary folders > under the output path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362473#comment-17362473 ] Prabhu Joseph commented on MAPREDUCE-7351: -- Yes right [~ste...@apache.org]. > CleanupJob during handle of SIGTERM signal > -- > > Key: MAPREDUCE-7351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > > Currently MR CleanupJob happens when the job is either successful or fail. > But during kill, it is not handled. This leaves all the temporary folders > under the output path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[
https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362352#comment-17362352
]
Steve Loughran commented on MAPREDUCE-7351:
---
do you mean calling {{OutputCommitter.cleanupJob()}}? Makes sense
> CleanupJob during handle of SIGTERM signal
> --
>
> Key: MAPREDUCE-7351
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
>
> Currently MR CleanupJob happens when the job is either successful or fail.
> But during kill, it is not handled. This leaves all the temporary folders
> under the output path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
