Thanks, Julien, for bringing this to the mailing list, and apologies
for my late reply.
Despite the long time I needed to reply, only one other reply (by
l.mierzwa) has happened. Not sure if that means there is not much
interest in the topic, or everyone else agrees with Julien.
Anyway, here is my take:
Before we had feature flags, we already had the option of introducing
features declared as experimental and thus not covered by our semantic
versioning.
Before we had feature flags, we already had the option of hiding risky
features or those that introduced additional resource usage behind
config settings or flags (e.g. --storage.tsdb... flags for overlapping
blocks or WAL compression and many more examples). And of course,
there was nothing keeping us turning breaking changes into features
that needed to be turned on explicitly via a flag or a config setting.
Julien's idea about feature flags are the following:
> In my vision, and that's how I acted as maintainer, feature flags
> should be used when:
>
> - We change an existing behaviour. [...]
> - We introduce very risky features, that introduce additional memory /
> storage requirements.
If that's the case, why did we introduce feature flags at all? Nothing
really changed, right?
I think we introduced feature flags for more reasons, and those were
crucial for the liberating effect on our velocity:
(1) We shied away from experimental features because we got burned by
too many users using experimental features without being aware of
them being experimental and thus being angry at us if we broke
them (or, in reverse, we being reluctant to change an experimental
feature because too many users were already relying on it). And in
fairness, it is hard as a user to keep track of what features are
experimental if we have many of them. Feature flags make it very
explicit to the user if they are using an experimental feature and
which.
(2) We did not want an explosion of flags or config settings to gate
features or behaviors (as had happened in v1.x). Feature flags are
a lightweight alternative (because they are not separate flags but
just a comma separated list, which also implies that we don't have
to keep old flags around as no-ops once the experimental feature
is declared stable).
(3) We often got caught in long-winded discussions if a certain
feature is even desirable, if it perhaps encourages anti-patterns
or discourages best practices, etc. A feature flag is both light
weight but also very explicit that the feature is not yet
recommended/endorsed. It allows us to shortcut the long-winded
discussion and just try something out without throwing our users
under the bus.
(4) Even the question if a feature is actually breaking is more or
less hard to answer (obligatory reference: https://xkcd.com/1172/
). Feature flags allow us to postpone that discussion to the point
where we consider graduating a feature to stable.
In sum, it's all about "worry less, use more feature flags". But that
only works if we are liberal with using feature flags. Being
restrictive about the cases when to use feature flags will create a
whole new type of long-winded discussion (whether a particular feature
deserves a feature flag or not), and worse, it might just subtly bring
back all those blockers above (if we consciously or sub-consciously
avoid the discussion if that feature deserves a feature flag, we are
back to square one).
> In general, I think it does not benefit users to launch Prometheus with
> lots of feature flags. Our users should be able to assess the risk they
> take by using a feature, without always requiring feature flags.
> Especially for relatively small features like atan2. There is no
> intention to drop atan2 in Prometheus 2.x anyway, just we might find a
> better way to call it.
>
> I try to draw a line between what's a useful feature flag, and where
> just marking experimental in documentation is fine. Prometheus is very
> conservative anyway, and I value the continuity of our features,
> including the "experimental" ones.
>
> Just to give you an idea, if we had a very strong feature flags policy
> in Prometheus, here is what it could have looked like, based on
> https://prometheus.io/docs/prometheus/latest/stability/#api-stability-guarantees
>
> --enable-feature=promql-at-modifier
> --enable-feature=expand-external-labels
> --enable-feature=promql-negative-offset
> --enable-feature=remote-write-receiver
> --enable-feature=exemplar-storage
> --enable-feature=body-size-limit
> --enable-feature=relabel-intervals
> --enable-feature=remote-read
> --enable-feature=https-basic-auth
> --enable-feature=web-ui
> --enable-feature=service-discovery-k8s
> --enable-feature=service-discovery-consul
> --enable-feature=remote-write-retry-on-429
> --enable-feature=target-limit
>
> And that's what I want to avoid.
I think this whole line of argument is a bit of a red herrin
