[
https://issues.apache.org/jira/browse/PIG-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan A. Veselovsky updated PIG-2898:
------------------------------------
Attachment: PIG-2898-against-trunk.patch
Thanks, Rohini,
I fixed some issues in the code and prepared the patch against "trunk". Later I
will attach the patch for 0.10 also.
Some more comments to this change:
1) --startat (-st) test_harness.pl option is not supported in the group fork
factor is greater than 1. This is because of the fact that several test groups
are executed simultaneously.
2) In a forked mode "Results so far, ..." lines on the console may show results
that are less than the actual results achieved so far. This is because each
subprocess prints only its own data and does not know about the progress of
other subprocesses.
> Multithreaded execution of e2e tests
> ------------------------------------
>
> Key: PIG-2898
> URL: https://issues.apache.org/jira/browse/PIG-2898
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Reporter: Andrey Klochkov
> Assignee: Ivan A. Veselovsky
> Attachments: PIG-2898-against-trunk.patch,
> pig-2898-for-svn-branch-0.9.patch
>
>
> Today it takes ~19 hours to run the full set of e2e tests in mapred mode. The
> bottleneck here is the client side, and per our observations it can help a
> lot if the e2e harness would be able to run tests in parallel threads.
> We prototyped changes in e2e harness allowing to run tests in a configurable
> number of threads. Preliminary results show more than 6x reduction in
> execution time when using a small 3-nodes M/R cluster with modest
> configuration. Going to share a patch shortly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
