[
https://issues.apache.org/jira/browse/CRUNCH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434477#comment-13434477
]
Gabriel Reid commented on CRUNCH-42:
------------------------------------
It seems that there is still a hardcoded sleep and interval in Hadoop 1.0.3
that isn't present in 2.0 or in CDH3, so the performance improvements will just
remain much larger when running with either 2.0 or CDH3. I've pushed this to
the main repo as it seems that there isn't anything that can be done for 1.0.3
(and it still provides a verify small improvement).
Running with 2.0 or CDH3 allows running the crunch integration tests in
something under 7 minutes on my laptop.
> Improve speed of running integration tests
> ------------------------------------------
>
> Key: CRUNCH-42
> URL: https://issues.apache.org/jira/browse/CRUNCH-42
> Project: Crunch
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Priority: Minor
> Fix For: 0.3.0
>
> Attachments: 0001-Use-short-job-poll-interval-when-in-local-mode.patch
>
>
> The CrunchJobControl currently uses a hard-coded poll interval of 5 seconds
> to check on running jobs. This is valid when running jobs on large datasets
> on a cluster, but results in unnecessary waiting when running in local mode.
> By using a smaller poll interval when running in local mode, the performance
> of the integration tests can be greatly improved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
