[
https://issues.apache.org/jira/browse/HIVE-26807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643313#comment-17643313
]
Stamatis Zampetakis commented on HIVE-26807:
--------------------------------------------
First of all, I extracted the test results in CSV files with the following
structure (testname@classname@time).
{noformat}
zgrep -a "<testcase" test-results-1514.tgz | sed 's/.*name="\(.*\)"
classname="\(.*\)" time="\([0-9\.]\+\)".*/\1@\2@\3/' | sort -k2 -k1 >
/tmp/master-1514.csv
zgrep -a "<testcase" test-results-1495.tgz | sed 's/.*name="\(.*\)"
classname="\(.*\)" time="\([0-9\.]\+\)".*/\1@\2@\3/' | sort -k2 -k1 >
/tmp/master-1495.csv
{noformat}
To facilitate the analysis, I imported the CSV files into Postgres tables.
{code:sql}
CREATE TABLE master_1514 (testname VARCHAR, classname VARCHAR, time DECIMAL);
CREATE TABLE master_1495 (testname VARCHAR, classname VARCHAR, time DECIMAL);
COPY master_1514 FROM '/tmp/master-1514.csv' WITH DELIMITER '@';
COPY master_1495 FROM '/tmp/master-1495.csv' WITH DELIMITER '@';
{code}
The combination of testname, classname is not unique due to parameterized tests
so we need an way to distinguish duplicate tests if we want to perform joins.
The trick is to use the ROW_NUMBER window function and assign incrementing
integers to seemingly duplicate tests; it is not 100% precise but satisfactory
for our needs.
{code:sql}
SELECT testname, classname, time, ROW_NUMBER() OVER (PARTITION BY testname,
classname ORDER BY time) as rnum FROM master_1514
{code}
I used the following query to get an overview of the situation before and after
upgrade.
{code:sql}
SELECT COUNT(*), MAX(diff), MIN(diff), AVG(diff), sum(ntime)/60/60 as
total_hours_1514 ,sum(otime)/60/60 as total_hours_1495 FROM
(SELECT n.testname, n.classname,n.time as ntime,o.time as otime,
n.time-o.time as diff
FROM (SELECT testname, classname, time, ROW_NUMBER() OVER (PARTITION BY
testname, classname ORDER BY time) as rnum FROM master_1514) n
INNER JOIN (SELECT testname, classname, time, ROW_NUMBER() OVER
(PARTITION BY testname, classname ORDER BY time) as rnum FROM master_1495) o
ON n.testname=o.testname AND n.classname = o.classname AND
n.rnum = o.rnum) compare
{code}
{noformat}
count | max | min | avg | total_hours_1514 |
total_hours_1495
-------+---------+---------+------------------------+---------------------+---------------------
47530 | 130.627 | -58.070 | 0.14675221965074689670 | 25.4390163888888889 |
23.5014794444444444
{noformat}
Observe that the total duration of the tests has increased by 8% (cumulative is
~2h) which is noticeable but maybe not problematic at this stage. The tests are
running in parallel splits so the general slowdown per split is in the order of
a few minutes. Moreover, there are tests that are much slower (see max) but
also tests that are much faster (see min) so there is nothing justifying a
revert of the Zookeeper upgrade.
Nevertheless, it may be interesting to investigate further the tests who became
much slower to see if there is anything that could be done to save some CI
resources. I used the following query to find the 1000 tests that were
seemingly affected the most after the upgrade.
{code:sql}
COPY (
SELECT n.testname, n.classname,n.time as B_1514,o.time as B_1495,
n.time-o.time as diff
FROM (SELECT testname, classname, time, ROW_NUMBER() OVER (PARTITION BY
testname, classname ORDER BY time) as rnum FROM master_1514) n
INNER JOIN (SELECT testname, classname, time, ROW_NUMBER() OVER
(PARTITION BY testname, classname ORDER BY time) as rnum FROM master_1495) o
ON n.testname=o.testname AND n.classname = o.classname AND
n.rnum = o.rnum
ORDER BY diff DESC
LIMIT 1000)
TO '/tmp/testtimes-diff-1514-1495.csv' WITH DELIMITER '@';
{code}
The results are attached in [^diff-1514-1495.csv].
> Investigate test running times before/after Zookeeper upgrade to 3.6.3
> ----------------------------------------------------------------------
>
> Key: HIVE-26807
> URL: https://issues.apache.org/jira/browse/HIVE-26807
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Attachments: diff-1514-1495.csv, test-results-1495.tgz,
> test-results-1514.tgz
>
>
> During the investigation of the CI timing out (HIVE-2686) there were some
> concerns that the Zookeeper (HIVE-26763) upgrade caused some significant
> slowdown.
> The goal of this issue is to analyse the test results from the following
> builds:
> * [Build-1495|http://ci.hive.apache.org/job/hive-precommit/job/master/1495/],
> commit just before Zookeeper upgrade;
> *
> [Builld-1514|http://ci.hive.apache.org/job/hive-precommit/job/master/1514/],
> commit after Zookeeper upgrade with skipped tests (HIVE-26796) and CI
> timeouts (HIVE-26806) fixed;
> and reason about the impact of the Zookeeper upgrade in test execution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
