----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7053/#review11749 -----------------------------------------------------------
General comments: 1) It would be good to rename file.fork.factor to fork.factor.conf.file and fork.factor to fork.factor.group for easy understanding of the intention of the parameters. 2) [exec] Results so far, PASSED: 1 FAILED: 0 SKIPPED: 0 ABORTED: 0 FAILED DEPENDENCY: 0 - It would be good to have another context[] before Results, which puts the forkname like [nightly.conf-CoGroup] or [nightly.conf] if it is a parallel execution. Without it reading results is difficult. 3) dbg and dumpHash There are lot of commented dbg and dumpHash calls which need to be uncommented manually for debug purposes. Could we have these printed based on a -debug option or -De2e.debug=true system property instead of having commented code and rename dumpHash to dbgDumpHash. Issues: - Running test in local mode hangs and does not work . Needs to be fixed. - When running only two tests the time taken was twice sequential mode because of the mkdir/rmr. That could be improved. More details in the code section. Performance: - Performance improvement is really good when I tested it. Test for mapred mode was down to 2.5 hrs from 9 hrs with benchmark results already cached in our setup. http://svn.apache.org/repos/asf/pig/trunk/test/e2e/harness/TestDriver.pm <https://reviews.apache.org/r/7053/#comment25316> Context of the fork here http://svn.apache.org/repos/asf/pig/trunk/test/e2e/harness/TestDriver.pm <https://reviews.apache.org/r/7053/#comment25315> Context of the fork here. http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/drivers/TestDriverPig.pm <https://reviews.apache.org/r/7053/#comment25314> This was taking 40 secs for each group in parallel mode as opposed to 6 secs in sequential mode and I am suspecting it is due to lot of forking. So running two tests with tests.to.run option took double the time of sequential (5 min to 12 min). That cost is immaterial if we are running the whole suite. But reducing it helps as most of the time you will be running a subset for testing. Would be good to optimize by combining the two hdfs mkdirs into one command and use perl mkdir instead of IPC::Run (which internally does fork and 1 sec sleep). - Rohini Palaniswamy On Sept. 12, 2012, 8:31 a.m., Ivan Veselovsky wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7053/ > ----------------------------------------------------------- > > (Updated Sept. 12, 2012, 8:31 a.m.) > > > Review request for pig and Rohini Palaniswamy. > > > Description > ------- > > Please see https://issues.apache.org/jira/browse/PIG-2898 for details. > > > This addresses bug https://issues.apache.org/jira/browse/PIG-2898. > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/PIG-2898 > > > Diffs > ----- > > http://svn.apache.org/repos/asf/pig/trunk/test/e2e/harness/TestDriver.pm > 1383357 > http://svn.apache.org/repos/asf/pig/trunk/test/e2e/harness/test_harness.pl > 1383357 > http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/build.xml 1383357 > > http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/deployers/ExistingClusterDeployer.pm > 1383357 > > http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/drivers/TestDriverPig.pm > 1383357 > > Diff: https://reviews.apache.org/r/7053/diff/ > > > Testing > ------- > > Tested e2e tests execution in both sequential (default) and parallel modes. > The test run duration measurement data (in dependency on the fork parameters) > will be available soon. > > > Thanks, > > Ivan Veselovsky > >
