[
https://issues.apache.org/jira/browse/PIG-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463537#comment-13463537
]
Rohini Palaniswamy commented on PIG-2898:
-----------------------------------------
Ilya,
Thanks it worked for local mode. But there were some additional test
failures than usual in H23. H20 tests are still running and I will update
status on them tomorrow.
Few comments:
1) Can you change yarn.nodemanager.local-dirs to mapreduce.cluster.local.dir.
yarn.nodemanager.local-dirs was a wrong suggestion from me. Tested with
mapreduce.cluster.local.dir and it works.
2) Can you change name of hadoop.mapred.dir to hadoop.mapred.local.dir as it is
slightly confusing and make it configurable through commandline. In many cases
/tmp gets full and would be good to have the ability to point to some other dir.
3) I had some comments in the reviewboard. Can you incorporate them too and
post an updated patch in reviewboard.
> Parallel execution of e2e tests
> -------------------------------
>
> Key: PIG-2898
> URL: https://issues.apache.org/jira/browse/PIG-2898
> Project: Pig
> Issue Type: Improvement
> Components: e2e harness
> Affects Versions: 0.10.0
> Reporter: Andrey Klochkov
> Assignee: Ivan A. Veselovsky
> Labels: test
> Attachments: PIG-2898-against-trunk-2.patch,
> PIG-2898-fix-sub-prototypes.patch, PIG-2898-trunk-3.patch
>
>
> Today it takes ~19 hours to run the full set of e2e tests in mapred mode. The
> bottleneck here is the client side, and per our observations it can help a
> lot if the e2e harness would be able to run tests in parallel threads.
> We prototyped changes in e2e harness allowing to run tests in a configurable
> number of threads. Preliminary results show more than 6x reduction in
> execution time when using a small 3-nodes M/R cluster with modest
> configuration. Going to share a patch shortly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
