Update: The PR now sends the fields over remote write from the WAL and
metadata
is also updated in the WAL when any field changes.
Now opened the PR against the primary repo:
https://github.com/prometheus/prometheus/pull/7771
I have tested this end-to-end with a modified M3 branch:
https://github.com/m3db/m3/compare/r/test-prometheus-metadata
> {... "msg":"received
series","labels":"{__name__="prometheus_rule_group_...
> iterations_total",instance="localhost:9090",job="prometheus01",role=...
> "remote"}","type":"counter","unit":"","help":"The total number of
scheduled...
> rule group evaluations, whether executed or missed."}
Tests still haven't been updated. Please any feedback on the approach /
data structures would be greatly appreciated.
Would be good to know what others thoughts are on next steps.
On Sat, Aug 8, 2020 at 11:21 AM Rob Skillington wrote:
> Here's a draft PR that builds that propagates metadata to the WAL and the
> WAL
> reader can read it back:
> https://github.com/robskillington/prometheus/pull/1/files
>
> Would like a little bit of feedback before on the datatypes and structure
> going
> further if folks are open to that.
>
> There's a few things not happening:
> - Remote write queue manager does not use or send these extra fields yet.
> - Head does not reset the "metadata" slice (not sure where "series" slice
> is
> reset in the head for pending series writes to WAL, want to do in same
> place).
> - Metadata is not re-written on change yet.
> - Tests.
>
>
> On Sat, Aug 8, 2020 at 9:37 AM Rob Skillington
> wrote:
>
>> Sounds good, I've updated the proposal with details on places in which
>> changes
>> are required given the new approach:
>>
>> https://docs.google.com/document/d/1LY8Im8UyIBn8e3LJ2jB-MoajXkfAqW2eKzY735aYxqo/edit#
>>
>>
>> On Fri, Aug 7, 2020 at 2:09 PM Brian Brazil <
>> brian.bra...@robustperception.io> wrote:
>>
>>> On Fri, 7 Aug 2020 at 15:48, Rob Skillington
>>> wrote:
>>>
True - I mean this could also be a blacklist by config perhaps, so if
you
really don't want to have increased egress you can optionally turn off
sending
the TYPE, HELP, UNIT or send them at different frequencies via config.
We could
package some sensible defaults so folks don't need to update their
config.
The main intention is to enable these added features and make it
possible for
various consumers to be able to adjust some of these parameters if
required
since backends can be so different in their implementation. For M3 I
would be
totally fine with the extra egress that should be mitigated fairly
considerably
by Snappy and the fact that HELP is common across certain metric
families and
receiving it every single Remote Write request.
>>>
>>> That's really a micro-optimisation. If you are that worried about
>>> bandwidth you'd run a sidecar specific to your remote backend that was
>>> stateful and far more efficient overall. Sending the full label names and
>>> values on every request is going to be far more than the overhead of
>>> metadata on top of that, so I don't see a need as it stands for any of this
>>> to be configurable.
>>>
>>> Brian
>>>
>>>
On Fri, Aug 7, 2020 at 3:56 AM Brian Brazil <
brian.bra...@robustperception.io> wrote:
> On Thu, 6 Aug 2020 at 22:58, Rob Skillington
> wrote:
>
>> Hey Björn,
>>
>>
>> Thanks for the detailed response. I've had a few back and forths on
>> this with
>> Brian and Chris over IRC and CNCF Slack now too.
>>
>> I agree that fundamentally it seems naive to idealistically model
>> this around
>> per metric name. It needs to be per series given what may happen
>> w.r.t.
>> collision across targets, etc.
>>
>> Perhaps we can separate these discussions apart into two
>> considerations:
>>
>> 1) Modeling of the data such that it is kept around for transmission
>> (primarily
>> we're focused on WAL here).
>>
>> 2) Transmission (and of which you allude to has many areas for
>> improvement).
>>
>> For (1) - it seems like this needs to be done per time series,
>> thankfully we
>> actually already have modeled this to be stored per series data just
>> once in a
>> single WAL file. I will write up my proposal here, but it will
>> surmount to
>> essentially encoding the HELP, UNIT and TYPE to the WAL per series
>> similar to
>> how labels for a series are encoded once per series in the WAL. Since
>> this
>> optimization is in place, there's already a huge dampening effect on
>> how
>> expensive it is to write out data about a series (e.g. labels). We
>> can always
>> go and collect a sample WAL file and measure how much extra size
>> with/without
>> HELP, UNIT and TYPE this would add, but it seems like it won't
>> fundamentally
>> change the order of magnitu
