henrikingo commented on issue #118:
URL: https://github.com/apache/otava/issues/118#issuecomment-3696068948
Ok, so based on your explanation,
- regressions sounds like something Piotr and Shaunak have invented and
does something very specific to Datastax internal processes. Similar to the
functionality of sending a slack notification once per week and including only
14 days of history in the notification. (As opposed to sending a notification
when a regression was found the first time, remembering that you've done so,
and possibly also tracking whether the regression was fixed or not.)
- While it is understandable that from an engineering point of view you
will not be interested in change points that happened 3 years back... I feel it
is againnst the spirit of change point detection to only focus on the most
recent tail of the timeseries. Even if you are only interested in recent
regressions, the historical data is used by the algorithm to build a model of
the noise and distribution of the data. Arbitrarily discarding most of the
historical data is a little bit wrong.
- Although the act of comparing a feature branch against a stable branch is
IMO a valid problem, it is actually a different problem and potentially
different solution. A change point detection solution would be to simply
analyze the history of the feature branch as a single, independent history. Or
rather, not completely independent, as a branch by definition shares a common
history with all other "branches" going back to the "trunk". If there are no
change points in the history of the branch, then there are no regressions. If
the main/target branch got faster for unrelated reasons, this should still be
uninteresting to the developer of the feature branch.
- I would be willing to propose we remove the regressions command entirely,
including the --branch option.
Users should instead:
- simply run otava over the history of the feature branch. There is no need
to compare it to any other branch.; OR
- GitHub-style workflow: Create a pull request, which will run benchmarks
where the feature branch is rebased and merged onto the head of the main/stable
branch. It turns out this reduces to a special case of the previous point: The
temporary branch created by a rebase+merge, is actually just a feature branch
that shares a common history with the main.
As a special concession, we can implement some kind of option to say that we
are only interested in change points that happen in the feature branch part,
But that is essentially --since already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
