[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000235#comment-13000235 ] Hadoop QA commented on MAPREDUCE-1730: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449081/MAPREDUCE-1730.patch against trunk revision 1075216. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/83//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/83//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/83//console This message is automatically generated. > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: MAPREDUCE-1730.patch, MAPREDUCE-1730.patch, > MAPREDUCE-1730.patch, TestJobRetired.patch, TestJobRetired.patch, > TestRetiredJobs-ydist-security-patch.txt, > TestRetiredJobs-ydist-security-patch.txt, TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890177#action_12890177 ] Iyappan Srinivasan commented on MAPREDUCE-1730: --- Sorry. I have put in the dependency. Please check in after you check in the MAPREDUCE-1713. > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: MAPREDUCE-1730.patch, MAPREDUCE-1730.patch, > MAPREDUCE-1730.patch, TestJobRetired.patch, TestJobRetired.patch, > TestRetiredJobs-ydist-security-patch.txt, > TestRetiredJobs-ydist-security-patch.txt, TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890100#action_12890100
]
Konstantin Boudnik commented on MAPREDUCE-1730:
---
I have tried to compile MR with the patch in place and have gotten
{noformat}
[javac]
/Users/cos/work/H0.22/mapreduce/src/test/system/test/org/apache/hadoop/mapred/TestRetiredJobs.java:116:
cannot find symbol
[javac] symbol : method isJobStarted(org.apache.hadoop.mapred.JobID)
[javac] location: class org.apache.hadoop.mapreduce.test.system.JTClient
[javac] boolean jobStarted =
cluster.getJTClient().isJobStarted(jobID);
{noformat}
Apparently, the JIRA needs some other patch to be applied and that dependency
isn't listed here. Until the blocking JIRA is committed I can't commit this
patch.
> Automate test scenario for successful/killed jobs' memory is properly removed
> from jobtracker after these jobs retire.
> --
>
> Key: MAPREDUCE-1730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 0.21.0
>Reporter: Iyappan Srinivasan
>Assignee: Iyappan Srinivasan
> Attachments: MAPREDUCE-1730.patch, MAPREDUCE-1730.patch,
> MAPREDUCE-1730.patch, TestJobRetired.patch, TestJobRetired.patch,
> TestRetiredJobs-ydist-security-patch.txt,
> TestRetiredJobs-ydist-security-patch.txt, TestRetiredJobs.patch
>
>
> Automate using herriot framework, test scenario for successful/killed jobs'
> memory is properly removed from jobtracker after these jobs retire.
> This should test when successful and failed jobs are retired, their
> jobInProgress object are removed properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888706#action_12888706 ] Iyappan Srinivasan commented on MAPREDUCE-1730: --- The two errors are unrelated to the patch. > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: MAPREDUCE-1730.patch, MAPREDUCE-1730.patch, > MAPREDUCE-1730.patch, TestJobRetired.patch, TestJobRetired.patch, > TestRetiredJobs-ydist-security-patch.txt, > TestRetiredJobs-ydist-security-patch.txt, TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888623#action_12888623 ] Hadoop QA commented on MAPREDUCE-1730: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449081/MAPREDUCE-1730.patch against trunk revision 963986. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/300/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/300/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/300/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/300/console This message is automatically generated. > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: MAPREDUCE-1730.patch, MAPREDUCE-1730.patch, > MAPREDUCE-1730.patch, TestJobRetired.patch, TestJobRetired.patch, > TestRetiredJobs-ydist-security-patch.txt, > TestRetiredJobs-ydist-security-patch.txt, TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887482#action_12887482 ] Konstantin Boudnik commented on MAPREDUCE-1730: --- +1 looks good to me. > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1730.patch, MAPREDUCE-1730.patch, > MAPREDUCE-1730.patch, TestJobRetired.patch, TestJobRetired.patch, > TestRetiredJobs-ydist-security-patch.txt, > TestRetiredJobs-ydist-security-patch.txt, TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885784#action_12885784 ] Konstantin Boudnik commented on MAPREDUCE-1730: --- I think we have discussed this on a number of occasions: please do not use Thread.sleep(5); directly. I believe there's a utility method for this. > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1730.patch, TestJobRetired.patch, > TestJobRetired.patch, TestRetiredJobs-ydist-security-patch.txt, > TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880067#action_12880067 ] Vinay Kumar Thota commented on MAPREDUCE-1730: -- Code looks good. +1 for the patch. > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan > Fix For: 0.22.0 > > Attachments: TestJobRetired.patch, TestJobRetired.patch, > TestRetiredJobs-ydist-security-patch.txt, TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863286#action_12863286
]
Vinay Kumar Thota commented on MAPREDUCE-1730:
--
Code looks good. However, there are two more comments.
{noformat}
static final Log LOG = LogFactory.
getLog(TestRetiredJobs.class);
Indentation issue in the above statement. the continuation line should have
four spaces before starting.
{noformat}
{noformat}
int count = 0;
while (jInfo != null && !jInfo.getStatus().isJobComplete()) {
Thread.sleep(1);
jInfo = remoteJTClient.getJobInfo(rJob.getID());
//If the count goes more than 100 seconds, then fail; This is to
//avoid infinite loop
if (count > 10) {
Assert.fail("Since the job has not started even after" +
"100 seconds, failing at this point");
}
}
* Don't see the count increment in the above block.
* Indentation issue in the Assert statement.
{noformat}
> Automate test scenario for successful/killed jobs' memory is properly removed
> from jobtracker after these jobs retire.
> --
>
> Key: MAPREDUCE-1730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 0.22.0
>Reporter: Iyappan Srinivasan
> Fix For: 0.22.0
>
> Attachments: TestJobRetired.patch, TestRetiredJobs.patch
>
>
> Automate using herriot framework, test scenario for successful/killed jobs'
> memory is properly removed from jobtracker after these jobs retire.
> This should test when successful and failed jobs are retired, their
> jobInProgress object are removed properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863247#action_12863247 ] Iyappan Srinivasan commented on MAPREDUCE-1730: --- I don't see where the test is really verifying that job in progress object is really removed. I don't see any assert conditions specific to retired job in the code. - getJobInfo returns JobinProgress Object. So, the presence or absence of Jobinprogress object is used to verify job retirement. Assert conditions are present for the presence and absence of this object. cluster.isJobStarted(jobID); -> this method is available in the MRCluster. - This method is used now in the testcase. Why are your checking the jobInfo second time? - Double checking is removed. There is only one checking now. If jobinfo available after completion of job, it should fail and shows the message saying that 'Job information is still available after completion of job'. - This message is added, with a slight change. "Job information is still available after retirement of job" > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan > Fix For: 0.22.0 > > Attachments: TestJobRetired.patch, TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862534#action_12862534 ] Vinay Kumar Thota commented on MAPREDUCE-1730: -- Also I don't see any assert conditions specific to retired job in the code. > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan > Fix For: 0.22.0 > > Attachments: TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862533#action_12862533
]
Vinay Kumar Thota commented on MAPREDUCE-1730:
--
{noformat}
int count = 0;
while (jInfo.getStatus().getRunState() != JobStatus.RUNNING) {
UtilsForTests.waitFor(1);
count++;
jInfo = wovenClient.getJobInfo(jobID);
//If the count goes more than 100 seconds, then fail; This is to
//avoid infinite loop
if (count > 10) {
Assert.fail("Since the job has not started even after" +
" 100 seconds, failing at this point");
}
}
For above functionality can use the below statement.
clsuster.isJobStarted(jobID); -> this method is available in the
MRCluster.
{noformat}
{noformat}
JobInfo jobInfo = cluster.getJTClient().getProxy()
.getJobInfo(jobID);
Assert.assertNotNull("The Job information is not present ", jobInfo);
Why are your checking the jobInfo second time? is there any reason behind it?
Already checking the jobinfo in the top whether its null or not.In that case,
why do you need the second time checking?
{noformat}
{noformat}
Assert.assertNull("The Job information is not present ", jobInfo);
The message in Assert statement might be wrong, because after completion of job
the information should be null right.So you are checking the assertNull
condition and its correct.But the message is not correct.
If jobinfo available after completion of job, it should fail and shows the
message saying that 'Job information is still available after completion of
job'.
{noformat}
> Automate test scenario for successful/killed jobs' memory is properly removed
> from jobtracker after these jobs retire.
> --
>
> Key: MAPREDUCE-1730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 0.22.0
>Reporter: Iyappan Srinivasan
> Fix For: 0.22.0
>
> Attachments: TestRetiredJobs.patch
>
>
> Automate using herriot framework, test scenario for successful/killed jobs'
> memory is properly removed from jobtracker after these jobs retire.
> This should test when successful and failed jobs are retired, their
> jobInProgress object are removed properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862228#action_12862228 ] Balaji Rajagopalan commented on MAPREDUCE-1730: --- I don't see where the test is really verifying that job in progress object is really removed. > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.22.0 >Reporter: Iyappan Srinivasan > Fix For: 0.22.0 > > Attachments: TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
