Gerrrr commented on issue #137:
URL: https://github.com/apache/otava/issues/137#issuecomment-3999727884
There are `--output regressions_only` and `--output json`. Here is an
example. While, the default analysis looks something like this:
```
time commit metric1 metric2
------------------------- -------- --------- ---------
2026-01-01 02:00:00 +0000 aaa0 154023 10.43
2026-01-02 02:00:00 +0000 aaa1 138455 10.23
2026-01-03 02:00:00 +0000 aaa2 143112 10.29
2026-01-04 02:00:00 +0000 aaa3 149190 10.91
2026-01-05 02:00:00 +0000 aaa4 132098 10.34
2026-01-06 02:00:00 +0000 aaa5 151344 10.69
·········
-12.9%
·········
2026-01-07 02:00:00 +0000 aaa6 155145 9.23
2026-01-08 02:00:00 +0000 aaa7 148889 9.11
2026-01-09 02:00:00 +0000 aaa8 149466 9.13
2026-01-10 02:00:00 +0000 aaa9 148209 9.03
```
With `--output regressions_only` it will look like:
```
Computing change points for test local.sample...
Regressions in local.sample:
2026-01-07 02:00:00 +0000
metric2 : 10.5 --> 9.12 ( -12.9%)
```
If there are no regressions, it will print:
```
Computing change points for test local.sample...
No regressions found in local.sample.
```
You can do string matching in the CI script and `exit 1` if there are
regressions. For example, here is what we have in one of the pipelines:
```bash
if grep "Regressions in" otava.out; then
echo "Found performance regressions on branch ${BRANCH}. <Link to runbook>"
exit 1;
fi
```
With `--output json`:
```
Computing change points for test local.sample...
{"local.sample": [{"time": 1767751200, "attributes": {"commit": "aaa6"},
"changes": [{"metric": "metric2", "index": 6, "time": 1767751200,
"forward_change_percent": "-13", "magnitude": "0.148676", "mean_before":
"10.481667", "stddev_before": "0.241414", "mean_after": "9.125000",
"stddev_after": "0.071239", "pvalue": "0.000005"}]}]}
```
If there are no regressions:
```
Computing change points for test local.sample...
{"local.sample": []}
```
If you have `jq` available in the CI script, you can check that values are
empty lists.
I hope one of these options works for you!
Admittedly, I added `--output regressions_only` to my fork a long time ago
without giving much thought to the API. As the project is now moving toward a
1.0.0 release with a stable API, this is the best time to reconsider the
ergonomics of this and other options. Let's use this issue to discuss what the
ideal CLI args would look like.
One problem with the current `--output` option is that it ties the output
_format_ (tabular, JSON, etc.) to filtering. For example, it might be useful to
also have the regular tabular format showing only regressions.
```
time commit metric1 metric2
------------------------- -------- --------- ---------
2026-01-06 02:00:00 +0000 aaa5 151344 10.69
·········
-12.9%
·········
2026-01-07 02:00:00 +0000 aaa6 155145 9.23
```
Putting it another way, there is no way to filter for the specific
information of interest - the entire history, only regressions, only
improvements, or all change points.
Maybe, we should have 2 distinct options, one to specify the output format,
the second is what events we want to see in that output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
