[jira] Updated: (MAPREDUCE-825) JobClient completion poll interval of 5s causes slow tests in local mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-825: Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Aaron! > JobClient completion poll interval of 5s causes slow tests in local mode > > > Key: MAPREDUCE-825 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-825 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball >Priority: Minor > Fix For: 0.21.0 > > Attachments: completion-poll-interval.patch, MAPREDUCE-825.2.patch > > > The JobClient.NetworkedJob.waitForCompletion() method polls for job > completion every 5 seconds. When running a set of short tests in > pseudo-distributed mode, this is unnecessarily slow and causes lots of wasted > time. When bandwidth is not scarce, setting the poll interval to 100 ms > results in a 4x speedup in some tests. This interval should be parametrized > to allow users to control the interval for testing purposes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-825) JobClient completion poll interval of 5s causes slow tests in local mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-825: Status: Patch Available (was: Open) > JobClient completion poll interval of 5s causes slow tests in local mode > > > Key: MAPREDUCE-825 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-825 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball >Priority: Minor > Attachments: completion-poll-interval.patch, MAPREDUCE-825.2.patch > > > The JobClient.NetworkedJob.waitForCompletion() method polls for job > completion every 5 seconds. When running a set of short tests in > pseudo-distributed mode, this is unnecessarily slow and causes lots of wasted > time. When bandwidth is not scarce, setting the poll interval to 100 ms > results in a 4x speedup in some tests. This interval should be parametrized > to allow users to control the interval for testing purposes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-825) JobClient completion poll interval of 5s causes slow tests in local mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-825: Attachment: MAPREDUCE-825.2.patch Attaching a new patch that also includes a new "jobclient.progress.monitor.poll.interval" setting; default is 1000 ms. Modified TestTaskFail to set the completion poll interval to 50 ms. With default (5000) ms timeout, test runtime was 3 minutes 15 seconds. Setting the timeout to 50 ms reduced test runtime to 3 minutes 8 seconds. If we expect an average of 2500 milliseconds wasted per job in the default case, then this is 2500*3 = 7500 ms expected to be wasted, so the observed speedup seems correct. To be sure, I also set the timeout to 2 ms; test runtime went up to 3 minutes 52 seconds. So there's definitely a correlation. > JobClient completion poll interval of 5s causes slow tests in local mode > > > Key: MAPREDUCE-825 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-825 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball >Priority: Minor > Attachments: completion-poll-interval.patch, MAPREDUCE-825.2.patch > > > The JobClient.NetworkedJob.waitForCompletion() method polls for job > completion every 5 seconds. When running a set of short tests in > pseudo-distributed mode, this is unnecessarily slow and causes lots of wasted > time. When bandwidth is not scarce, setting the poll interval to 100 ms > results in a 4x speedup in some tests. This interval should be parametrized > to allow users to control the interval for testing purposes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-825) JobClient completion poll interval of 5s causes slow tests in local mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-825: Status: Open (was: Patch Available) > JobClient completion poll interval of 5s causes slow tests in local mode > > > Key: MAPREDUCE-825 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-825 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball >Priority: Minor > Attachments: completion-poll-interval.patch > > > The JobClient.NetworkedJob.waitForCompletion() method polls for job > completion every 5 seconds. When running a set of short tests in > pseudo-distributed mode, this is unnecessarily slow and causes lots of wasted > time. When bandwidth is not scarce, setting the poll interval to 100 ms > results in a 4x speedup in some tests. This interval should be parametrized > to allow users to control the interval for testing purposes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-825) JobClient completion poll interval of 5s causes slow tests in local mode
[
https://issues.apache.org/jira/browse/MAPREDUCE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Kimball updated MAPREDUCE-825:
Attachment: completion-poll-interval.patch
This patch adds the {{jobclient.completion.poll.interval}} mapreduce parameter,
which defaults to the existing value of 5000 ms. No new tests because this is a
very small change; it's not clear what new (or any) functionality needs
additional verification.
> JobClient completion poll interval of 5s causes slow tests in local mode
>
>
> Key: MAPREDUCE-825
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-825
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Aaron Kimball
>Assignee: Aaron Kimball
>Priority: Minor
> Attachments: completion-poll-interval.patch
>
>
> The JobClient.NetworkedJob.waitForCompletion() method polls for job
> completion every 5 seconds. When running a set of short tests in
> pseudo-distributed mode, this is unnecessarily slow and causes lots of wasted
> time. When bandwidth is not scarce, setting the poll interval to 100 ms
> results in a 4x speedup in some tests. This interval should be parametrized
> to allow users to control the interval for testing purposes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-825) JobClient completion poll interval of 5s causes slow tests in local mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-825: Status: Patch Available (was: Open) > JobClient completion poll interval of 5s causes slow tests in local mode > > > Key: MAPREDUCE-825 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-825 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Aaron Kimball >Assignee: Aaron Kimball >Priority: Minor > Attachments: completion-poll-interval.patch > > > The JobClient.NetworkedJob.waitForCompletion() method polls for job > completion every 5 seconds. When running a set of short tests in > pseudo-distributed mode, this is unnecessarily slow and causes lots of wasted > time. When bandwidth is not scarce, setting the poll interval to 100 ms > results in a 4x speedup in some tests. This interval should be parametrized > to allow users to control the interval for testing purposes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
