[
https://issues.apache.org/jira/browse/HIVE-19514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-19514:
---------------------------------------
Description:
Ptest tries to copy ~20G worth of logs from the worker nodes to the server. The
total aggregate time spent by all the Ptest server in copying these files is
close to 15 min (although much of that is in parallel threads). Most of the
times we don't check for the logs. We seldom check for logs of successful
tests. We should ignore copying successful test logs by default to improve
runtime.
{noformat}
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:235 Executed 13568 tests
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ExecutionPhase
took 71 minutes
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF:
ExecutionPhase.TotalRsyncElapsedTime took 15 minutes
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: PrepPhase took
5 minutes
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ReportingPhase
took 0 minutes
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: TestCheckPhase
took 0 minutes
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: YetusPhase
took 0 minutes
{noformat}
was:
Ptest tries to copy ~20G worth of logs from the worker nodes to the server. The
total aggregate time spent by all the hosts in copying these files is close to
15 min in copying files. Most of the times we don't check for the logs. We
seldom check for logs of successful tests. We should ignore copying successful
test logs by default to improve runtime.
{noformat}
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:235 Executed 13568 tests
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ExecutionPhase
took 71 minutes
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF:
ExecutionPhase.TotalRsyncElapsedTime took 15 minutes
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: PrepPhase took
5 minutes
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: ReportingPhase
took 0 minutes
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: TestCheckPhase
took 0 minutes
2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: YetusPhase
took 0 minutes
{noformat}
> Successful test logs should be copied
> -------------------------------------
>
> Key: HIVE-19514
> URL: https://issues.apache.org/jira/browse/HIVE-19514
> Project: Hive
> Issue Type: Sub-task
> Reporter: Vihang Karajgaonkar
> Assignee: Vihang Karajgaonkar
> Priority: Major
>
> Ptest tries to copy ~20G worth of logs from the worker nodes to the server.
> The total aggregate time spent by all the Ptest server in copying these files
> is close to 15 min (although much of that is in parallel threads). Most of
> the times we don't check for the logs. We seldom check for logs of successful
> tests. We should ignore copying successful test logs by default to improve
> runtime.
> {noformat}
> 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:235 Executed 13568
> tests
> 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF:
> ExecutionPhase took 71 minutes
> 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF:
> ExecutionPhase.TotalRsyncElapsedTime took 15 minutes
> 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: PrepPhase
> took 5 minutes
> 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF:
> ReportingPhase took 0 minutes
> 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF:
> TestCheckPhase took 0 minutes
> 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: YetusPhase
> took 0 minutes
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
