Todd Lipcon has submitted this change and it was merged.
Change subject: parse_test_failure: optimization and usage improvement
......................................................................
parse_test_failure: optimization and usage improvement
This adds a couple micro-optimizations to the parse_test_failure script
which substantially increase performance. I noticed that, now that we are
running the test result server on the public internet, Google found it
and is crawling the 'diagnose' pages for failed tests. Currently, those
pages can take 10s of seconds (or worse) to load, which ends up spinning
the CPU and blocking other legit usage of the server, since it's running
single-threaded at the moment. This likely has affected other test runs
and prevented them from fetching the flaky-test-list or reporting flakes.
The micro-optimizations are simple: just using substring matches to do quick
checks before running the more expensive regular expressions. I tested by
running the script on a recent client-test.txt log I had lying around with
about 28000 lines.
I tested this in production like a good web developer. It's deployed and
seems to be working (plus verified that the 'diagnose' links load quickly
now).
before:
Performance counter stats for './build-support/parse_test_failure.py
./build/test-logs/client-test.txt':
11732.165505 task-clock (msec) # 1.001 CPUs utilized
119 context-switches # 0.010 K/sec
1 cpu-migrations # 0.000 K/sec
1,805 page-faults # 0.154 K/sec
38,840,157,152 cycles # 3.311 GHz
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
103,675,773,467 instructions # 2.67 insns per cycle
20,562,530,240 branches # 1752.663 M/sec
45,622,875 branch-misses # 0.22% of all branches
11.724291727 seconds time elapsed
after:
Performance counter stats for './build-support/parse_test_failure.py
./build/test-logs/client-test.txt' (10 runs):
73.823229 task-clock (msec) # 1.009 CPUs utilized
1 context-switches # 0.014 K/sec
1 cpu-migrations # 0.014 K/sec
1,901 page-faults # 0.026 M/sec
243,137,251 cycles # 3.332 GHz
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
513,003,074 instructions # 2.14 insns per cycle
128,224,780 branches # 1757.295 M/sec
1,149,019 branch-misses # 0.90% of all branches
0.073158565 seconds time elapsed
( +- 0.50% )
Change-Id: Ibc75aa98317ca31dbe6a0ffd7bf766732e615f94
Reviewed-on: http://gerrit.cloudera.org:8080/1756
Tested-by: Todd Lipcon <[email protected]>
Reviewed-by: Dan Burkert <[email protected]>
---
M build-support/parse_test_failure.py
1 file changed, 10 insertions(+), 3 deletions(-)
Approvals:
Dan Burkert: Looks good to me, approved
Todd Lipcon: Verified
--
To view, visit http://gerrit.cloudera.org:8080/1756
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibc75aa98317ca31dbe6a0ffd7bf766732e615f94
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Dan Burkert <[email protected]>
Gerrit-Reviewer: Todd Lipcon <[email protected]>
