[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles updated MAPREDUCE-7262:
--
Fix Version/s: 2.10.1
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1
>
> Attachments: MAPREDUCE-7262-branch-2.10.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEven
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7262-branch-2.10.002.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: MAPREDUCE-7262-branch-2.10.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEve
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles updated MAPREDUCE-7262:
--
Fix Version/s: 3.2.2
3.1.4
3.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
+1. Thanks, [~ahussein]. I committed this to trunk, branch-3.2, and branch-3.1.
This patch doesn't apply to branch-2.10 due to the Logger vs Log api. With a
small change, this patch could apply to the branch-2.10.
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf,
> MAPREDUCE-7262.001.patch, MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usag
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: (was: MAPREDUCE-7261.002.patch)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf,
> MAPREDUCE-7262.001.patch, MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Status: Open (was: Patch Available)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAtt
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7262.002.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAtt
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Status: Patch Available (was: Open)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAtt
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7261.002.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
> TaskAttem
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7262.001.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> org
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7262-elapsedTimes.pdf
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> org.apache.hadoop.mapr
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Status: Patch Available (was: Open)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> org.apache.hadoop.mapreduce.v2.app (67 usages found)
> TestAMInfos (1 usage
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Summary: MRApp helpers block for long intervals (500ms) (was: MRApp
helpers blocks for long intervals)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> org.apache.hadoop.
