[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577241#comment-13577241
]
Hadoop QA commented on MAPREDUCE-4643:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569119/MAPREDUCE-4643-branch-1-addendum.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3330//console
This message is automatically generated.
> Make job-history cleanup-period configurable
>
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 1.0.3
>Reporter: Karthik Kambatla
>Assignee: Sandy Ryza
> Fix For: 1.2.0
>
> Attachments: MAPREDUCE-4643-branch1-2.patch,
> MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch,
> MAPREDUCE-4643-branch-1-5.patch, MAPREDUCE-4643-branch-1-6.patch,
> MAPREDUCE-4643-branch-1-addendum.patch, MAPREDUCE-4643-branch1.patch,
> MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1
> month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
> to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577238#comment-13577238 ] Sandy Ryza commented on MAPREDUCE-4643: --- The test appears to be failing occasionally due to time zone issues. Uploading an addendum patch that fixes the failure. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Fix For: 1.2.0 > > Attachments: MAPREDUCE-4643-branch1-2.patch, > MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch, > MAPREDUCE-4643-branch-1-5.patch, MAPREDUCE-4643-branch-1-6.patch, > MAPREDUCE-4643-branch-1-addendum.patch, MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574902#comment-13574902 ] Alejandro Abdelnur commented on MAPREDUCE-4643: --- +1. did a clean build and run relevant test. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1-2.patch, > MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch, > MAPREDUCE-4643-branch-1-5.patch, MAPREDUCE-4643-branch-1-6.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574896#comment-13574896
]
Hadoop QA commented on MAPREDUCE-4643:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568649/MAPREDUCE-4643-branch-1-6.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3320//console
This message is automatically generated.
> Make job-history cleanup-period configurable
>
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 1.0.3
>Reporter: Karthik Kambatla
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1-2.patch,
> MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch,
> MAPREDUCE-4643-branch-1-5.patch, MAPREDUCE-4643-branch-1-6.patch,
> MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch,
> MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1
> month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
> to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574888#comment-13574888 ] Sandy Ryza commented on MAPREDUCE-4643: --- Latest patch renames run and tests the case where only some of the files within a directory need to be deleted. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1-2.patch, > MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch, > MAPREDUCE-4643-branch-1-5.patch, MAPREDUCE-4643-branch-1-6.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574167#comment-13574167
]
Karthik Kambatla commented on MAPREDUCE-4643:
-
Cool test, Sandy. The latest patch looks mostly good; minor comments:
# We should rename {{run(long now)}} to {{runCleaner(long now)}} to avoid
confusion.
# The test doesn't seem to test the case where only some of the files within a
directory need to be deleted. To add this case to the test, we might have to
calculate the paths etc. instead of hardcoding them.
> Make job-history cleanup-period configurable
>
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 1.0.3
>Reporter: Karthik Kambatla
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1-2.patch,
> MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch,
> MAPREDUCE-4643-branch-1-5.patch, MAPREDUCE-4643-branch1.patch,
> MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1
> month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
> to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574164#comment-13574164
]
Hadoop QA commented on MAPREDUCE-4643:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568512/MAPREDUCE-4643-branch-1-5.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3319//console
This message is automatically generated.
> Make job-history cleanup-period configurable
>
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 1.0.3
>Reporter: Karthik Kambatla
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1-2.patch,
> MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch,
> MAPREDUCE-4643-branch-1-5.patch, MAPREDUCE-4643-branch1.patch,
> MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1
> month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
> to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574156#comment-13574156 ] Sandy Ryza commented on MAPREDUCE-4643: --- Latest patch adds test that runs in less than a second. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1-2.patch, > MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch, > MAPREDUCE-4643-branch-1-5.patch, MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13573096#comment-13573096
]
Hadoop QA commented on MAPREDUCE-4643:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568349/MAPREDUCE-4643-branch-1-4.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3315//console
This message is automatically generated.
> Make job-history cleanup-period configurable
>
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 1.0.3
>Reporter: Karthik Kambatla
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1-2.patch,
> MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch,
> MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch,
> MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1
> month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
> to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13573091#comment-13573091 ] Sandy Ryza commented on MAPREDUCE-4643: --- Latest patch adds the default values into mapred-default.xml and takes out the test, which will be covered in MAPREDUCE-4676. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1-2.patch, > MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13571899#comment-13571899 ] Alejandro Abdelnur commented on MAPREDUCE-4643: --- patch looks good, 2 things though: * mapred-default.xml should have the default values as values (instead of empty) * The TestJobHistory test takes 333secs, this seems a bit too much. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1-2.patch, > MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13571835#comment-13571835 ] Karthik Kambatla commented on MAPREDUCE-4643: - +1. Thanks Sandy - the latest patch looks good to me. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1-2.patch, > MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13571828#comment-13571828
]
Hadoop QA commented on MAPREDUCE-4643:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568109/MAPREDUCE-4643-branch-1-3.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3307//console
This message is automatically generated.
> Make job-history cleanup-period configurable
>
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 1.0.3
>Reporter: Karthik Kambatla
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1-2.patch,
> MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch1.patch,
> MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1
> month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
> to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13571820#comment-13571820 ] Sandy Ryza commented on MAPREDUCE-4643: --- uploaded refresh patch that includes test > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1-2.patch, > MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471968#comment-13471968
]
Hadoop QA commented on MAPREDUCE-4643:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548332/MAPREDUCE-4643-branch1-2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2918//console
This message is automatically generated.
> Make job-history cleanup-period configurable
>
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 1.0.3
>Reporter: Karthik Kambatla
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1-2.patch,
> MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch,
> MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1
> month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
> to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471966#comment-13471966 ] Sandy Ryza commented on MAPREDUCE-4643: --- I uploaded a patch that makes that revision. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1-2.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471934#comment-13471934
]
Alejandro Abdelnur commented on MAPREDUCE-4643:
---
oh well. looks good, one small NIT though
{code}
+return result.getTimeInMillis() -
+result.getTimeInMillis() % HistoryCleaner.ONE_DAY_IN_MS;
{code}
Assign getTimeInMillis() to a local var and then use that for the expression
above.
> Make job-history cleanup-period configurable
>
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 1.0.3
>Reporter: Karthik Kambatla
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch,
> MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1
> month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
> to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471926#comment-13471926 ] Sandy Ryza commented on MAPREDUCE-4643: --- trunk already uses milliseconds. Should branch-1 be different? > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471913#comment-13471913 ] Alejandro Abdelnur commented on MAPREDUCE-4643: --- using ms as unit for a value that will normally be in days seems a bit off. I understand that for testing purposes we want something short enough, can we use seconds instead? (saving 3 digits). Ideally we should be able to specify units on the time interval (ie '2ms', '2d'), but this is completely out of scope of this JIRA. Thx > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463023#comment-13463023
]
Hadoop QA commented on MAPREDUCE-4643:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12546557/MAPREDUCE-4643-branch1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2877//console
This message is automatically generated.
> Make job-history cleanup-period configurable
>
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 1.0.3
>Reporter: Karthik Kambatla
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch,
> MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1
> month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
> to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463022#comment-13463022 ] Sandy Ryza commented on MAPREDUCE-4643: --- Uploaded a patch that fixes the comments and includes the test > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1.patch, > MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462408#comment-13462408 ] Hadoop QA commented on MAPREDUCE-4643: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546425/MAPREDUCE-4643-branch1.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2875//console This message is automatically generated. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462346#comment-13462346
]
Karthik Kambatla commented on MAPREDUCE-4643:
-
Thanks for the patch, Sandy. It looks like it should work.
Minor comments:
# In {{JobHistory.directoryTime()}}, the initial comments do not apply anymore
and can be removed
# Do you want to add a test to verify the timely deletion of history files?
(Now, I see that this test would fix MAPREDUCE-4676 as well). Do you think you
can add a test here and mark the other one duplicate?
> Make job-history cleanup-period configurable
>
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 1.0.3
>Reporter: Karthik Kambatla
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1
> month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it
> to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462285#comment-13462285 ] Sandy Ryza commented on MAPREDUCE-4643: --- I spoke with Karthik and we resolved that the patch does work for smaller ages, but that for directories older than a day, it could work more efficiently by deleting the entire directory. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462241#comment-13462241 ] Sandy Ryza commented on MAPREDUCE-4643: --- It should work for significantly smaller ages as well. It uses the directory names to decide which to look in, but checks the modification times on the files inside them. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462220#comment-13462220 ] Karthik Kambatla commented on MAPREDUCE-4643: - Thanks for your patch, Sandy. The patch looks like it should work for (max-age > 1 day). How do we handle significantly smaller ages? We should probably address it the same it is addressed in trunk - use the modification time? Thoughts? > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462215#comment-13462215 ] Hadoop QA commented on MAPREDUCE-4643: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546397/MAPREDUCE-4643.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2874//console This message is automatically generated. > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4643.patch > > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461585#comment-13461585 ] Sandy Ryza commented on MAPREDUCE-4643: --- Ah, didn't see this JIRA until now. The patch I'm working on for MAPREDUCE-4676 leaves the directory structure the same and deletes based on the modification time (only checking files in old enough directories). For the testJobHistoryCleaner test, it would be ideal for it to be configurable to milliseconds. What do you think? > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450412#comment-13450412 ] Karthik Kambatla commented on MAPREDUCE-4643: - The two aspects of making the job-history cleanup period configurable seem to be: # unit of time to configure it on (minutes vs days) # directory naming to reflect the creation time For the directory naming, we should be able to leave it as /mm/dd format. If the cleanup period is in hours/minutes, we can read the file modification time and use that to do the comparison. Trunk currently uses the modification time even for higher values of cleanup period (days) and has a TODO note that we should use the directory structure instead to reduce the load on HDFS. With regards to the unit of time for configuration, minutes seems to be the better option. Thoughts? > Make job-history cleanup-period configurable > > > Key: MAPREDUCE-4643 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla > > Job history cleanup should be made configurable. Currently, it is set to 1 > month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it > to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
