[
https://issues.apache.org/jira/browse/PIG-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485246#comment-13485246
]
Rohini Palaniswamy commented on PIG-2898:
-----------------------------------------
+1. Thanks Ivan. Tested the new patch. It is now faster even for running two or
three tests. It would have been nicer if you had named the new methods
something more meaningful like onGroupRunSetup()/onGroupRunCleanup() instead of
globalSetup2() or cleanupSetup2(). But I guess it is ok since it is only the
e2e test script.
> Parallel execution of e2e tests
> -------------------------------
>
> Key: PIG-2898
> URL: https://issues.apache.org/jira/browse/PIG-2898
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Affects Versions: 0.10.0
> Reporter: Andrey Klochkov
> Assignee: Ivan A. Veselovsky
> Labels: test
> Attachments: PIG-2898-branch-0.10-6-final.patch,
> PIG-2898-branch-0.10-7.patch, PIG-2898-trunk-3.patch,
> PIG-2898-trunk-6-final.patch, PIG-2898-trunk-7.patch
>
>
> Today it takes ~19 hours to run the full set of e2e tests in mapred mode. The
> bottleneck here is the client side, and per our observations it can help a
> lot if the e2e harness would be able to run tests in parallel threads.
> We prototyped changes in e2e harness allowing to run tests in a configurable
> number of threads. Preliminary results show more than 6x reduction in
> execution time when using a small 3-nodes M/R cluster with modest
> configuration. Going to share a patch shortly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
