On 01/03/2021 07:25, Harkishen Singh wrote:
Hi Tom,
I have tried to answer the comments. Please comment on their
satisfactoriness. I am happy for a call if required (or discussion
gets tough).
I think, the lossless nature can be controlled by the user based on
the config (limit_retries), and let the users have more control, as to
whether they are happy to compromise a bit, if the retry is too much,
since as such, if the retrying happens forever, then I don't think
that is helpful (it will never be accepted by the remote storage).
Also as Chris mentioned, some users might prefer to have few gaps and
give more priority to recent data, like for alerting. So, I think this
approach gives more flexibility to the user, at the same time, making
it optional (or by setting the retry count high enough).
Under what situations would retries happen forever?
If the receiver is available but cannot accept the data (for example due
to metric size limits or age of the samples) I would expect it to reject
with a 4XX code (permanent failure) which wouldn't trigger any retries.
Alternatively if the receiver is either unavailable or broken it could
result in "infinite" retries, but in that situation it feels like an age
based limit instead of retry limit would be better - a short retry limit
will reject samples that have just been scraped just as quickly as
samples that are days old. Instead it sounds like an age based limit
would be better - some systems have restrictions over what age can be
ingested (e.g. Timestream) or administrators could decide older data has
no usefulness (e.g. if the receiver is used for alerting or anomaly
detection. While the system should still reject such old samples once it
is working again a time based limit would at least reduce the network
impact once the receiver is back online (no need to send tons of data
that we know will be rejected).
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/cd97f615-e479-e4be-e85d-672b15c337d8%40Jahingo.com.
