[ https://issues.apache.org/jira/browse/HIVE-19514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473752#comment-16473752 ]
Vihang Karajgaonkar commented on HIVE-19514: -------------------------------------------- Ptest server now only copies successful logs. This inprovement shaves off 10 min from the total execution time. {noformat} 2018-05-14 01:29:45,702 INFO [TestExecutor] PTest.run:237 PERF: ExecutionPhase took 64 minutes 2018-05-14 01:29:45,702 INFO [TestExecutor] PTest.run:237 PERF: ExecutionPhase.TotalRsyncElapsedTime took 8 minutes 2018-05-14 01:29:45,702 INFO [TestExecutor] PTest.run:237 PERF: PrepPhase took 5 minutes 2018-05-14 01:29:45,702 INFO [TestExecutor] PTest.run:237 PERF: ReportingPhase took 0 minutes 2018-05-14 01:29:45,702 INFO [TestExecutor] PTest.run:237 PERF: TestCheckPhase took 0 minutes 2018-05-14 01:29:45,702 INFO [TestExecutor] PTest.run:237 PERF: YetusPhase took 0 minutes {noformat} > Successful test logs should be copied > ------------------------------------- > > Key: HIVE-19514 > URL: https://issues.apache.org/jira/browse/HIVE-19514 > Project: Hive > Issue Type: Sub-task > Reporter: Vihang Karajgaonkar > Assignee: Vihang Karajgaonkar > Priority: Major > > Ptest tries to copy ~20G worth of logs from the worker nodes to the server. > The total aggregate time spent by all the Ptest server in copying these files > is close to 15 min (although much of that is in parallel threads). Most of > the times we don't check for the logs. We seldom check for logs of successful > tests. We should ignore copying successful test logs by default to improve > runtime. > {noformat} > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:235 Executed 13568 > tests > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: > ExecutionPhase took 71 minutes > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: > ExecutionPhase.TotalRsyncElapsedTime took 15 minutes > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: PrepPhase > took 5 minutes > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: > ReportingPhase took 0 minutes > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: > TestCheckPhase took 0 minutes > 2018-05-13 16:10:15,243 INFO [TestExecutor] PTest.run:237 PERF: YetusPhase > took 0 minutes > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)