[
https://issues.apache.org/jira/browse/CRUNCH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434370#comment-13434370
]
Gabriel Reid commented on CRUNCH-42:
------------------------------------
Yep, it seems to be due to a change in the dependencies -- running with hadoop
1.0.3 still seems to be pretty slow, and running with CDH3 (the version that
was in the pom when I made the original patch) was fast. Using hadoop 2.0 also
seems to run quickly with this change. I was actually starting to think that I
had imagined running "mvn verify" in under 10 minutes, and it was starting to
worry me :-)
In any case, I believe that there's a config setting that can be set somewhere
in hadoop 1.x to change the poll interval as well -- I'm currently trying to
hunt that down and find the best place to set it, and then the performance
improvement should work everywhere.
> Improve speed of running integration tests
> ------------------------------------------
>
> Key: CRUNCH-42
> URL: https://issues.apache.org/jira/browse/CRUNCH-42
> Project: Crunch
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Priority: Minor
> Attachments: 0001-Use-short-job-poll-interval-when-in-local-mode.patch
>
>
> The CrunchJobControl currently uses a hard-coded poll interval of 5 seconds
> to check on running jobs. This is valid when running jobs on large datasets
> on a cluster, but results in unnecessary waiting when running in local mode.
> By using a smaller poll interval when running in local mode, the performance
> of the integration tests can be greatly improved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
