[jira] Commented: (MAPREDUCE-1741) Automate the test scenario of job related files are moved from history directory to done directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1396#comment-1396 ] Hadoop QA commented on MAPREDUCE-1741: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451420/1741-ydist-security-patch.txt against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/81//console This message is automatically generated. Automate the test scenario of job related files are moved from history directory to done directory --- Key: MAPREDUCE-1741 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1741 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 0.21.0 Reporter: Iyappan Srinivasan Assignee: Iyappan Srinivasan Attachments: 1741-ydist-security-patch.txt, MAPREDUCE-1741.patch, MAPREDUCE-1741.patch, MAPREDUCE-1741.patch, TestJobHistoryLocation-ydist-security-patch.txt, TestJobHistoryLocation-ydist-security-patch.txt, TestJobHistoryLocation-ydist-security-patch.txt, TestJobHistoryLocation.patch, TestJobHistoryLocation.patch, TestJobHistoryLocation.patch Job related files are moved from history directory to done directory, when 1) Job succeeds 2) Job is killed 3) When 100 files are put in the done directory 4) When multiple jobs are completed at the same time, some successful, some failed. Also, two files, conf.xml and job files should be present in the done directory. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1741) Automate the test scenario of job related files are moved from history directory to done directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12887484#action_12887484 ] Konstantin Boudnik commented on MAPREDUCE-1741: --- Patch looks good. +1 Automate the test scenario of job related files are moved from history directory to done directory --- Key: MAPREDUCE-1741 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1741 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 0.21.0 Reporter: Iyappan Srinivasan Attachments: MAPREDUCE-1741.patch, MAPREDUCE-1741.patch, MAPREDUCE-1741.patch, TestJobHistoryLocation-ydist-security-patch.txt, TestJobHistoryLocation-ydist-security-patch.txt, TestJobHistoryLocation-ydist-security-patch.txt, TestJobHistoryLocation.patch, TestJobHistoryLocation.patch, TestJobHistoryLocation.patch Job related files are moved from history directory to done directory, when 1) Job succeeds 2) Job is killed 3) When 100 files are put in the done directory 4) When multiple jobs are completed at the same time, some successful, some failed. Also, two files, conf.xml and job files should be present in the done directory. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1741) Automate the test scenario of job related files are moved from history directory to done directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885426#action_12885426
]
Vinay Kumar Thota commented on MAPREDUCE-1741:
--
{noformat}
+Assert.assertTrue(jobFileFound is false. Job History +
+File is not found in the done directory,
+ jobFileFound);
+Assert.assertEquals(Both the job related files are not found,
+fileCount, 2);
{noformat}
Remove the assert statements in private method and check them in the test
method block.
Automate the test scenario of job related files are moved from history
directory to done directory
---
Key: MAPREDUCE-1741
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1741
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 0.22.0
Reporter: Iyappan Srinivasan
Fix For: 0.22.0
Attachments: MAPREDUCE-1741.patch,
TestJobHistoryLocation-ydist-security-patch.txt,
TestJobHistoryLocation-ydist-security-patch.txt,
TestJobHistoryLocation.patch, TestJobHistoryLocation.patch,
TestJobHistoryLocation.patch
Job related files are moved from history directory to done directory, when
1) Job succeeds
2) Job is killed
3) When 100 files are put in the done directory
4) When multiple jobs are completed at the same time, some successful, some
failed.
Also, two files, conf.xml and job files should be present in the done
directory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1741) Automate the test scenario of job related files are moved from history directory to done directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12880068#action_12880068
]
Vinay Kumar Thota commented on MAPREDUCE-1741:
--
{noformat}
+ int fileCount = 0;
+ jobFileFound = false;
+ for (FileStatus jobHistoryDoneFileStatus : jobHistoryDoneFileStatuses) {
+LOG.info(jobHistoryDoneFileStatus path is : +
+jobHistoryDoneFileStatus.getPath().toString());
+LOG.info(jobIDString path is : + jobIDString);
+StringBuffer jobHistoryDoneFileBuffer = new
+StringBuffer(jobHistoryDoneFileStatus.getPath().toString());
+
+if ( jobHistoryDoneFileBuffer.indexOf(jobIDString) != -1 ) {
+ jobFileFound = true;
+ fileCount++;
+ //Both the conf file and the job file has to be present
+ if (fileCount == 2) {
+break;
+ }
+}
{noformat}
{noformat}
+ //Wait for the job to start running.
+ int count = 0;
+ while (jInfo != null jInfo.getStatus().getRunState()
+ != JobStatus.RUNNING) {
+UtilsForTests.waitFor(1);
+count++;
+jInfo = remoteJTClient.getJobInfo(jobID);
+//If the count goes beyond 100 seconds, then break; This is to avoid
+//infinite loop.
+if (count 10) {
+ Assert.fail(job has not reached running state for more than +
+ 100 seconds. Failing at this point);
+}
+ }
{noformat}
The above code blocks are repeating couple of times in the test. please make it
common functions and utilize it.
Automate the test scenario of job related files are moved from history
directory to done directory
---
Key: MAPREDUCE-1741
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1741
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 0.22.0
Reporter: Iyappan Srinivasan
Fix For: 0.22.0
Attachments: TestJobHistoryLocation-ydist-security-patch.txt,
TestJobHistoryLocation.patch, TestJobHistoryLocation.patch,
TestJobHistoryLocation.patch
Job related files are moved from history directory to done directory, when
1) Job succeeds
2) Job is killed
3) When 100 files are put in the done directory
4) When multiple jobs are completed at the same time, some successful, some
failed.
Also, two files, conf.xml and job files should be present in the done
directory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1741) Automate the test scenario of job related files are moved from history directory to done directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12862553#action_12862553
]
Vinay Kumar Thota commented on MAPREDUCE-1741:
--
{noformat}
Test names are bit confusing for me. Can you please change to some appropriate
names?
Here is my suggestion.
testRetiredJobsCompletedLocation() - testRetiredJobsHistoryLocation()
testRetiredMultipleJobsCompletedLocation() -
testRetiredMultipleJobsHistoryLocation()
{noformat}
{noformat}
String hadoopLogDirString = jconf.get(hadoop.log.dir);
jobHistoryDonePathString =
jconf.get(mapred.job.tracker.history.completed.location);
String jobHistoryPathString = jconf.get(hadoop.job.history.location);
Can please check the above three statements with assert condition. Otherwise if
any of these attributes are not available in the configuration
then test fails w/ NPE w/out any message. So please check the above three
values before proceeding to further steps
{noformat}
{noformat}
Assert.assertTrue(jobFileFound is false, jobFileFound);
You are checking the above condition while running the job right, in this case
the above statement would be wrong because
the jobFileFound should be false and it should be like below.
Assert.assertFalse(Job history files available in history location for running
job itself.,jobFileFound)
{noformat}
How do you confirm whether its moved to retired job state or not after
completion of job ?
I don't see any specific condition in the code for retire job. Is there any
default interval available in the configuration for moving jobs to retire?
{noformat}
Put the assert statement like below after completion of job.So that user will
have clear information whenever test fails.
Assert.assertTrue(Job history files are not available in history location
after job retired., jobFileFound);
{noformat}
Even I have same question that Balaji has asked regarding creating 1000 files
in done location. Can you please
elaborate more on this in your comments.
Can you please put the annotation @Test after java doc comments instead of top.
Automate the test scenario of job related files are moved from history
directory to done directory
---
Key: MAPREDUCE-1741
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1741
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 0.22.0
Reporter: Iyappan Srinivasan
Fix For: 0.22.0
Attachments: TestJobHistoryLocation.patch
Job related files are moved from history directory to done directory, when
1) Job succeeds
2) Job is killed
3) When 100 files are put in the done directory
4) When multiple jobs are completed at the same time, some successful, some
failed.
Also, two files, conf.xml and job files should be present in the done
directory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1741) Automate the test scenario of job related files are moved from history directory to done directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12862623#action_12862623 ] Iyappan Srinivasan commented on MAPREDUCE-1741: --- What is the reason we create manually 100 files, please provide the documentation, looking at the code it is not evident to me. 100 files are created for the test scenario in which the movemnt of history files to done directory should still happen, even iftehre are 100 dummy files in that directory. In the multiple job scenario are the jobs launches simultaneously or sequentially one after the other ? What is the reason we create manually 100 files, please provide the documentation, looking at the code it is not evident to me. 2. In the multiple job scenario are the jobs launches simultaneously or sequentially one after the other ? - It is sequential, with a difference of few seconds. The test scenario is when a bunch of testcases finish almost at the same time, still ,the history files have to be moved to done directory Test names are bit confusing for me. Can you please change to some appropriate names? Here is my suggestion. testRetiredJobsCompletedLocation() - testRetiredJobsHistoryLocation() testRetiredMultipleJobsCompletedLocation() - testRetiredMultipleJobsHistoryLocation() - Done String hadoopLogDirString = jconf.get(hadoop.log.dir); jobHistoryDonePathString = jconf.get(mapred.job.tracker.history.completed.location); String jobHistoryPathString = jconf.get(hadoop.job.history.location); Can please check the above three statements with assert condition. Otherwise if any of these attributes are not available in the configuration then test fails w/ NPE w/out any message. So please check the above three values before proceeding to further steps - hadoop.log.dir is not required . Removed it. - Added a Assert statement for mapred.job.tracker.history.completed.location - hadoop.job.history.location is a optional parm , which may or may not be present. And it si checked for null, before usage. So, no need for assert. How do you confirm whether its moved to retired job state or not after completion of job ? I don't see any specific condition in the code for retire job. Is there any default interval available in the configuration for moving jobs to retire? - The retired jbo movement is done as a inline method and will happen soon after job is completed. So, the code waits for a maximum of 30 seconds to make absolutely sure that it is moved to retired. Put the assert statement like below after completion of job.So that user will have clear information whenever test fails. Assert.assertTrue(Job history files are not available in history location after job retired., jobFileFound); - It is checking in the done directory. Job history file should be found here. If it si not found, then it asserts. I have elaborated the message as Job History File is not found in the done directory Can you please put the annotation @Test after java doc comments instead of top. [ Show ยป ] Vinay Kumar Thota added a comment - 30/Apr/10 02:06 AM Test names are bit confusing for me. Can you please change to some appropriate names? Here is my suggestion. testRetiredJobsCompletedLocation() - testRetiredJobsHistoryLocation() testRetiredMultipleJobsCompletedLocation() - testRetiredMultipleJobsHistoryLocation() String hadoopLogDirString = jconf.get(hadoop.log.dir); jobHistoryDonePathString = jconf.get(mapred.job.tracker.history.completed.location); String jobHistoryPathString = jconf.get(hadoop.job.history.location); Can please check the above three statements with assert condition. Otherwise if any of these attributes are not available in the configuration then test fails w/ NPE w/out any message. So please check the above three values before proceeding to further steps Assert.assertTrue(jobFileFound is false, jobFileFound); You are checking the above condition while running the job right, in this case the above statement would be wrong because the jobFileFound should be false and it should be like below. Assert.assertFalse(Job history files available in history location for running job itself.,jobFileFound) How do you confirm whether its moved to retired job state or not after completion of job ? I don't see any specific condition in the code for retire job. Is there any default interval available in the configuration for moving jobs to retire? Put the assert statement like below after completion of job.So that user will have clear information whenever test fails. Assert.assertTrue(Job history files are not available in history location after job retired., jobFileFound); Even I have same question that Balaji has asked regarding creating 1000 files in done location. Can you please elaborate more on this in your comments. Can you please put the annotation @Test after java doc comments instead of
[jira] Commented: (MAPREDUCE-1741) Automate the test scenario of job related files are moved from history directory to done directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12862234#action_12862234 ] Balaji Rajagopalan commented on MAPREDUCE-1741: --- 1. What is the reason we create manually 100 files, please provide the documentation, looking at the code it is not evident to me. 2. In the multiple job scenario are the jobs launches simultaneously or sequentially one after the other ? Automate the test scenario of job related files are moved from history directory to done directory --- Key: MAPREDUCE-1741 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1741 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 0.22.0 Reporter: Iyappan Srinivasan Fix For: 0.22.0 Attachments: TestJobHistoryLocation.patch Job related files are moved from history directory to done directory, when 1) Job succeeds 2) Job is killed 3) When 100 files are put in the done directory 4) When multiple jobs are completed at the same time, some successful, some failed. Also, two files, conf.xml and job files should be present in the done directory. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
