Re: [prometheus-developers] Submitting a Tutorial for Prometheus in TrueNAS SCALE

[Ben Kochie]

Sorry, we generally do not have a place for tutorials like this on our
site. There are thousands of such tutorials out there, it would be
impossible to curate them from our site.

As a TrueNAS SCALE user, I wish SCALE itself had better Prometheus support.
I've thought about adding some to the middleware, but the number of
rejections for improvements by iXsystems means I don't want to spend the
effort just to have the PR closed.

If TrueNAS provided Prometheus metrics, it could be listed on
https://prometheus.io/docs/instrumenting/exporters.

On Tue, Sep 19, 2023 at 8:44 AM 'Mic Johnson' via Prometheus Developers <
prometheus-developers@googlegroups.com> wrote:

> We, the documentation team at iXsystems, would like to submit a user
> tutorial to help the community with instructions on deploying the
> Prometheus application in our platform.
>
> What is your process for such documentation PRs?
> Where in your repository would the file, and images it uses, be added?
> What is the approval process?
>
> Thanks in advance!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/6699079d-5ed5-437d-9d29-c8646c699295n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoWrDCWKc45Ziu5O%3D3TTXkSMC2yt2ib3xSLVf7PavyuFA%40mail.gmail.com.

Re: [prometheus-developers] Rename "start_time_seconds" to "start_timestamp_seconds"?

[Ben Kochie]

+1 to keeping consistent with the other libraries.

We should clarify the docs that `time` is an ok value in addition to
`timestamp`.

On Mon, Sep 4, 2023 at 2:44 PM Julius Volz  wrote:

> Since that's a metric that is standardized across our client libraries, we
> should keep the existing name IMO. I'm also not sure if the naming doc is
> that clear on whether a timestamp necessarily has to be named "timestamp"
> vs. just "time". We also use just "time" in other places as well, like
> "node_boot_time_seconds" in the Node Exporter, or
> "container_start_time_seconds" (although that's not us, that's cAdvisor /
> k8s.
>
> On Mon, Sep 4, 2023 at 2:20 PM 'Fabian Stäber' via Prometheus Developers <
> prometheus-developers@googlegroups.com> wrote:
>
>> Hi,
>>
>> Many Prometheus client libraries export a metric named
>> "start_time_seconds", which is the timestamp when the application was
>> started.
>>
>> However, according to https://prometheus.io/docs/practices/naming/ the
>> name should be "start_timestamp_seconds".
>>
>> So, as the Java client will have breaking changes anyway, what do you
>> think?
>>
>> * Rename to be compliant with our naming best practices?
>> * Keep the name consistent with other client libraries?
>>
>> Fabian
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/CAPX310j1ym7-uDzFix%2BhgqZNFWUExS7G3DO9-p4mRjJRMavfXQ%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6Yozndh0%3DGBYNSt9VV%2BVPhxNLE2kRaLTQgXxJf7O8J44bEw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpy4LuJ5A4WpMURVfZD81dTmFNU2gnAvgrPxL%2BcCWOpFw%40mail.gmail.com.

Re: [prometheus-developers] Potential big performance gain in common/expfmt/TextParser

[Ben Kochie]

We've done at least one rewrite of the parser in the past. We do
substantial changes to our subsystems all the time. For example, the
"stringlabels" changes were a recent substantial change to the internals of
in-memory label storage.

The only things we want to avoid is breaking existing users and reducing
the correctness of the parser.

On Fri, Jun 23, 2023 at 9:35 AM 'Antoine Pultier' via Prometheus Developers
 wrote:

> Hi,
>
> I am parsing a large number of metrics, and I noticed that the Prometheus
> expfmt.TextParser takes a significant amount of CPU time on my machine.
>
> I also noticed that VictoriaMetrics has an entirely different parsing
> implementation that is faster on my machine. I have not conducted extensive
> benchmarking; I'm unsure if I want to. But you can find a small comparison
> at the end of the email with a small string to parse and a 5MB string full
> of metrics and labels to parse.
>
> I read both implementations, both open-source with the Apache 2.0 license,
> and I guess the main difference is the extensive use of strings.IndexByte
> in the VictoriaMetrics parser. Golang provides a fast implementation to
> look for a byte in a string, which is much faster than scanning and
> comparing byte per byte (on common CPU architectures).
> Example for arm64:
> https://github.com/golang/go/blob/e45202f2154839f713b603fd6e5f8a8ad8d527e0/src/internal/bytealg/indexbyte_arm64.s
> I discovered the existence of such optimisations while reading this article
> about ripgrep: https://blog.burntsushi.net/ripgrep/#literal-optimizations
>
> I'm not a Prometheus developer, but I would guess that completely
> replacing the parser with another one is not on the table, but doing some
> changes to the existing one could be possible.
>
> However, it seems to require significant changes to gain performance. I'm
> wondering whether the Prometheus project would welcome substantial changes
> inside the parser at this point. One change would be to load more data at
> once. Perhaps the whole data into a string in memory like VictoriaMetrics
> does, which has some implications. And also the use of strings.IndexBytes
> and slices instead of constructing many strings byte by byte. These changes
> will probably make the parser less elegant, but that may or may not be
> worth it.
>
> ---
> The tiny benchmark:
> ---
> goos: darwin
> goarch: arm64
> pkg: simple-bench
> BenchmarkPrometheusTextParserMinimal-8  416382   2798 ns/op
> BenchmarkVictoriaMetricsTextParserMinimal-8   3622894   296.1 ns/op
> BenchmarkPrometheusTextParserBig-8  4287416010 ns/op
> BenchmarkVictoriaMetricsTextParserBig-8   142 8374695 ns/op
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/31a41b4f-cbcb-40c7-9df8-f1deddd15a32n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoT5-Q%3DvDqrjT9sP4w9h-c7ogMGk8vNp_16FG8nkZAJKg%40mail.gmail.com.

[prometheus-developers] Call for maintainers: Exporter team

[Ben Kochie]

Hey all,

We have a lot of exporter projects in both Prometheus and
Prometheus-Community GitHub.

There's lots of backlog of things to do.
* Triage issues.
* Review PRs.
* Improve update automation.
* Make new releases.

I've been thinking about how to handle some of this workload and had the
idea that we should have a maintainers team. We could use GitHub automatic
PR assignment in order to help spread the workload out.

If you're interested, I've created this new team:

https://github.com/orgs/prometheus/teams/exporter-maintainers

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoqNjUT4rJ8sQLKpyEvyBoQz-SvW4TepzDjKfTUJSZipA%40mail.gmail.com.

Re: [prometheus-developers] Maximum length of a time series in Prometheus

[Ben Kochie]

Although, I haven't actually checked to see if negative timestamps are
supported.

On Mon, Mar 27, 2023 at 3:38 PM Ben Kochie  wrote:

> Prometheus timestamps are in int64 (signed) milliseconds. So the maximum
> number of data points per series is: 9,223,372,036,854,775,807
>
> On Mon, Mar 27, 2023 at 3:24 PM Stuart Clark 
> wrote:
>
>> On 2023-03-27 14:10, Abdelouahab Khelifati wrote:
>> > I mean the maximum number of datapoints per time series, not the
>> > length of a specific value.
>>
>> Sorry that is what I was meaning.
>>
>> There will be loads of resource or performance limitations - such as not
>> being able to load the data for very long periods of time without huge
>> amounts of memory. However in the normal usage of scrapes around every
>> 30 seconds there are people who have multi-year storage retentions that
>> work successfully (so that would be around a million datapoints per
>> year).
>>
>> --
>> Stuart Clark
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/0f10e2d5aeb2ea7619846506fa81f1a4%40Jahingo.com
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmox54hBv6MQDBvDofmUWL9ssXwCPW0hBUv8Yh%2Bm2vXW6w%40mail.gmail.com.

Re: [prometheus-developers] Maximum length of a time series in Prometheus

[Ben Kochie]

Prometheus timestamps are in int64 (signed) milliseconds. So the maximum
number of data points per series is: 9,223,372,036,854,775,807

On Mon, Mar 27, 2023 at 3:24 PM Stuart Clark 
wrote:

> On 2023-03-27 14:10, Abdelouahab Khelifati wrote:
> > I mean the maximum number of datapoints per time series, not the
> > length of a specific value.
>
> Sorry that is what I was meaning.
>
> There will be loads of resource or performance limitations - such as not
> being able to load the data for very long periods of time without huge
> amounts of memory. However in the normal usage of scrapes around every
> 30 seconds there are people who have multi-year storage retentions that
> work successfully (so that would be around a million datapoints per
> year).
>
> --
> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/0f10e2d5aeb2ea7619846506fa81f1a4%40Jahingo.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpxXik1ZvjCkJAcXbh2tUa7xfAJ3A7_rrJh_8SvxosQsg%40mail.gmail.com.

Re: [prometheus-developers] Loading a large csv file to Prometheus

[Ben Kochie]

You will have to do this yourself.

Also, that doesn't really look like time-series data.

Prometheus is a monitoring system, not a generic database.

On Mon, Mar 27, 2023 at 11:41 AM Abdelouahab Khelifati 
wrote:

> Hello Ben and thanks for the prompt answer!
>
> Is there a tool to do such a conversion or should I implement it myself?
>
> Also, would you have a few dataset examples of time series data in the
> appropriate format for Prometheus?
>
> My data currently looks something like this:
>
> *time, id_sensor, value1, value2, ... , value n*
> *2022--01-03, 'd02', 0.54, 0.24, ... , 0.34*
> *2022--01-03, 'd04', 0.35, 0.94, ... , 0.53*
> *2022--01-04, 'd02', 0.59, 0.22, ... , 0.95*
> *...*
>
> Best,
> Abdel
>
>
> On Monday, March 27, 2023 at 11:25:43 AM UTC+2 Ben Kochie wrote:
>
>> You will want to convert the CSV to OpenMetrics format.
>>
>> See:
>> https://prometheus.io/docs/prometheus/latest/storage/#backfilling-from-openmetrics-format
>>
>> On Mon, Mar 27, 2023 at 11:18 AM Abdelouahab Khelifati <
>> ab...@exascale.info> wrote:
>>
>>> Hello all,
>>>
>>> How can I load a huge csv file into Prometheus? I tried
>>> https://github.com/stohrendorf/csv-prometheus-exporter but it doesn’t
>>> seem to be working.
>>>
>>> Thanks!
>>>
>>> Best,
>>> Abdel
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-devel...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-developers/4b580edc-8815-4ffd-9349-a8de58de5422n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-developers/4b580edc-8815-4ffd-9349-a8de58de5422n%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/71430c18-3524-403d-9a22-b4e4161e32d3n%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-developers/71430c18-3524-403d-9a22-b4e4161e32d3n%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqvTAn5hAK58e6LtQ0Hp-r0%2BemEk-sj7_%2B%3DYeD%2BUn--6g%40mail.gmail.com.

Re: [prometheus-developers] Loading a large csv file to Prometheus

[Ben Kochie]

You will want to convert the CSV to OpenMetrics format.

See:
https://prometheus.io/docs/prometheus/latest/storage/#backfilling-from-openmetrics-format

On Mon, Mar 27, 2023 at 11:18 AM Abdelouahab Khelifati 
wrote:

> Hello all,
>
> How can I load a huge csv file into Prometheus? I tried
> https://github.com/stohrendorf/csv-prometheus-exporter but it doesn’t
> seem to be working.
>
> Thanks!
>
> Best,
> Abdel
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/4b580edc-8815-4ffd-9349-a8de58de5422n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqiz7fR%2B28qdeaCHz-YtC0qyjWDdNMciJG3_UG_QAedSQ%40mail.gmail.com.

Re: [prometheus-developers] Windows Exporter License

[Ben Kochie]

It was my understanding that license changes, can be done by the copyright
holder, without consent of all contributors. Because they do not hold any
copyright to the code. IIRC this is how Grafana was able to relicense from
Apache to AGPL. They did not need to get consent from all contributors.

Of course, old versions are subject to the old license, but moving from
prometheus-community to prometheus would effectively be a fork.

In this case we could do it with permission from the original author as
stated in the LICENSE file.

On Thu, Dec 22, 2022 at 9:22 AM Julien Pivotto 
wrote:

> Yes, I will reach out to GB
>
> On 21 Dec 15:05, Julius Volz wrote:
> > Same!
> >
> > On Tue, Dec 20, 2022 at 4:15 PM Bjoern Rabenstein 
> > wrote:
> >
> > > On 13.12.22 11:19, Julien Pivotto wrote:
> > > >
> > > > In this sense, I think we should ask for an exception to the GB with
> the
> > > > following arguments:
> > >
> > > Silence means consent, right?
> > >
> > > But just in case you are wondering if anyone has read your message, I
> > > think it makes complete sense, and I fully agree with your points.
> > >
> > > --
> > > Björn Rabenstein
> > > [PGP-ID] 0x851C3DA17D748D03
> > > [email] bjo...@rabenste.in
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Prometheus Developers" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an
> > > email to prometheus-developers+unsubscr...@googlegroups.com.
> > > To view this discussion on the web visit
> > >
> https://groups.google.com/d/msgid/prometheus-developers/Y6HRoXOgbX%2BxSOuo%40mail.rabenste.in
> > > .
> > >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Prometheus Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-developers+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxV3RCVBEXRKCAC9E4xfNsrCOKwBgiL9ouG31atbL4NUg%40mail.gmail.com
> .
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/Y6QT1rc93RM/bSQg%40nixos
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqPZnq0ANfuNp2JYZwUJ4pJ%3DQiRq017nC%3Dm0wkBzSPm0w%40mail.gmail.com.

Re: [prometheus-developers] [VOTE] Promote Windows Exporter as an official exporter

[Ben Kochie]

YES

On Mon, Dec 5, 2022 at 11:44 AM Julien Pivotto 
wrote:

> Dear Prometheans,
>
> As per our governance [1], "any matter that needs a decision [...] may
> be called to a vote by any member if they deem it necessary."
>
> I am therefore calling a vote to promote Prometheus-community's Windows
> Exporter [2] to Prometheus GitHub org, to make it an official exporter.
>
> Official exporters are exporters under the Prometheus github org, listed
> as official on Prometheus.io and available under the Downloads page.
>
> This would provide recognition and credibility to the exporter and its
> contributors, which have provided a large amount of work in the last
> years, and built a huge community.
>
> It would make it easier for users to find and use the exporter, as it
> would be listed on the Prometheus website and promoted on the other
> official channels - such as our announce mailing list.
>
> Anyone interested is encouraged to participate in this vote and this
> discussion. As per our governance, only votes from the team members will
> be counted.
>
> Vote is open for 1 week - until December 12.
>
> [1] https://prometheus.io/governance/
> [2] https://github.com/prometheus-community/windows_exporter
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/Y43Lmr2%2Bb2fk8YSz%40nixos
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmokchsvdwwR%2BW0kgzseXicMqaC0s4LUZkr7TWxfPBdwug%40mail.gmail.com.

Re: [prometheus-developers] Preventing Prometheus from running out of memory

[Ben Kochie]

To add to "Nothing will break if things are missing". Prometheus alerts
depend on the existence of data for alerting to work. If you're "just
missing some data", Prometheus alerts will fail to fire. This is
unacceptable and we make specific design trade-offs in order to avoid this
situation.

On Mon, Nov 28, 2022 at 3:38 PM Ben Kochie  wrote:

> On Mon, Nov 28, 2022 at 1:40 PM l.mi...@gmail.com 
> wrote:
>
>> I agree that this is not side-effect free. With the worst possible
>> outcome of sending bogus alerts.
>> But.
>> We're talking about metrics, so from my PoV the consequences range not
>> from "annoying to dangerous" but "annoying to more annoying".
>>
>
> Actually, we're NOT talking about metrics. We're talking about monitoring.
> Prometheus is a monitoring system that also happens to be based on metrics.
>
>
>> There's no real danger in missing metrics. Nothing will break if they are
>> missing.
>>
>
> Absolutely not true. We depend on Prometheus for our critical monitoring
> behavior. Without Prometheus we don't know if our service is down. Without
> Prometheus we could have problems go undetected for millions of active
> users. If we're not serving our users this affects our company's income.
>
> To repeat, Prometheus is not some generic TSDB. It's a monitoring platform
> designed to be as robust as possible for base level alerting for everything
> it watches.
>
>
>> And if someone receives a bogus alert that's gonna be along with another
>> alert saying "you're over your limit, some metrics might be missing". Which
>> does help to limit any potential confusion.
>>
>> Again, the motivation is to prevent cardinality issues pilling up to a
>> point where Prometheus runs out of memory and crashes.
>> Maybe that's an issue that's more common in the environments I work with
>> then others and so I'm more willing to make trade-offs to fix that.
>> I do expect us to run in production with HEAD & soft limit patches (as we
>> already do) since doing so without it requires too much firefighting, so
>> I'm happy to share more experience of that later.
>> On Monday, 28 November 2022 at 11:46:09 UTC Julius Volz wrote:
>>
>>> My reaction is similar to that of Ben and Julien: if metrics within a
>>> target go partially missing, all kinds of expectations around aggregations,
>>> histograms, and other metric correlations are off in ways that you wouldn't
>>> expect in a normal Prometheus setup. Everyone expects full scrape failures,
>>> as they are common in Prometheus, and you usually have an alert on the "up"
>>> metric. Now you can say that if you explicitly choose to turn on that
>>> feature, you can expect a user to deal with the consequences of that. But
>>> the consequences are so unpredictable and can range from annoying to
>>> dangerous:
>>>
>>> * Incorrectly firing alerts (e.g. because some series are missing in an
>>> aggregation or a histogram)
>>> * Silently broken alerts that should be firing (same but the other way
>>> around)
>>> * Graphs seemingly working for an instance, but showing wrong data
>>> because some series of a metric are missing
>>> * Queries that try to correlate different sets of metrics from a target
>>> breaking in subtle ways
>>>
>>> Essentially it removes a previously strong invariant from
>>> Prometheus-based monitoring and introduces a whole new way of having to
>>> think about things like the above. Even if you have alerts to catch the
>>> underlying situation, you may get a bunch of incorrect alert notifications
>>> from the now-broken rules in the meantime.
>>>
>>> On Mon, Nov 28, 2022 at 10:58 AM l.mi...@gmail.com 
>>> wrote:
>>>
>>>>
>>>>
>>>> On Sunday, 27 November 2022 at 19:25:12 UTC sup...@gmail.com wrote:
>>>>
>>>>> Soft / partial failure modes can be very hard problems to deal with.
>>>>> You have to be a lot more careful to not end up missing partial failures.
>>>>>
>>>>
>>>> Sure, partial failures modes can be *in general* a hard problem, but
>>>> with Prometheus the worst case is that some metrics will be missing and
>>>> some won't, that's at least in my opinion, not that problematic.
>>>> What do you mean by "missing partial failures"?
>>>>
>>>>
>>>>> While it seems like having a soft sample_limt is good, and the hard
>>>>> sample_limit is bad. The &

Re: [prometheus-developers] Preventing Prometheus from running out of memory

[Ben Kochie]

On Mon, Nov 28, 2022 at 1:40 PM l.mi...@gmail.com 
wrote:

> I agree that this is not side-effect free. With the worst possible outcome
> of sending bogus alerts.
> But.
> We're talking about metrics, so from my PoV the consequences range not
> from "annoying to dangerous" but "annoying to more annoying".
>

Actually, we're NOT talking about metrics. We're talking about monitoring.
Prometheus is a monitoring system that also happens to be based on metrics.


> There's no real danger in missing metrics. Nothing will break if they are
> missing.
>

Absolutely not true. We depend on Prometheus for our critical monitoring
behavior. Without Prometheus we don't know if our service is down. Without
Prometheus we could have problems go undetected for millions of active
users. If we're not serving our users this affects our company's income.

To repeat, Prometheus is not some generic TSDB. It's a monitoring platform
designed to be as robust as possible for base level alerting for everything
it watches.


> And if someone receives a bogus alert that's gonna be along with another
> alert saying "you're over your limit, some metrics might be missing". Which
> does help to limit any potential confusion.
>
> Again, the motivation is to prevent cardinality issues pilling up to a
> point where Prometheus runs out of memory and crashes.
> Maybe that's an issue that's more common in the environments I work with
> then others and so I'm more willing to make trade-offs to fix that.
> I do expect us to run in production with HEAD & soft limit patches (as we
> already do) since doing so without it requires too much firefighting, so
> I'm happy to share more experience of that later.
> On Monday, 28 November 2022 at 11:46:09 UTC Julius Volz wrote:
>
>> My reaction is similar to that of Ben and Julien: if metrics within a
>> target go partially missing, all kinds of expectations around aggregations,
>> histograms, and other metric correlations are off in ways that you wouldn't
>> expect in a normal Prometheus setup. Everyone expects full scrape failures,
>> as they are common in Prometheus, and you usually have an alert on the "up"
>> metric. Now you can say that if you explicitly choose to turn on that
>> feature, you can expect a user to deal with the consequences of that. But
>> the consequences are so unpredictable and can range from annoying to
>> dangerous:
>>
>> * Incorrectly firing alerts (e.g. because some series are missing in an
>> aggregation or a histogram)
>> * Silently broken alerts that should be firing (same but the other way
>> around)
>> * Graphs seemingly working for an instance, but showing wrong data
>> because some series of a metric are missing
>> * Queries that try to correlate different sets of metrics from a target
>> breaking in subtle ways
>>
>> Essentially it removes a previously strong invariant from
>> Prometheus-based monitoring and introduces a whole new way of having to
>> think about things like the above. Even if you have alerts to catch the
>> underlying situation, you may get a bunch of incorrect alert notifications
>> from the now-broken rules in the meantime.
>>
>> On Mon, Nov 28, 2022 at 10:58 AM l.mi...@gmail.com 
>> wrote:
>>
>>>
>>>
>>> On Sunday, 27 November 2022 at 19:25:12 UTC sup...@gmail.com wrote:
>>>
 Soft / partial failure modes can be very hard problems to deal with.
 You have to be a lot more careful to not end up missing partial failures.

>>>
>>> Sure, partial failures modes can be *in general* a hard problem, but
>>> with Prometheus the worst case is that some metrics will be missing and
>>> some won't, that's at least in my opinion, not that problematic.
>>> What do you mean by "missing partial failures"?
>>>
>>>
 While it seems like having a soft sample_limt is good, and the hard
 sample_limit is bad. The "Fail fast" will serve you better in the long run.
 Most of the Prometheus monitoring design assumes fail fast. Partial results
 are too hard to reason about from a monitoring perspective. With fail fast
 you will know quickly and decisively that you've hit a problem. If you
 treat monitoring outages as just as bad as an actual service outage, you'll
 end up with a healthier system overall.

>>>
>>> I think I've seen the statement that partial scrapes are "too hard to
>>> reason about" a number of times
>>> From my experience they are not. A hard fail means "all or nothing",
>>> partial fail means "all or some". It's just a different level of
>>> granularity, that's not that hard to get your head around IMHO.
>>> I think that it's a hard problem from a Prometheus developers point of
>>> view, because the inner logic and handling of errors is where the
>>> (potential) complication is. But from end user perspective it's not that
>>> hard at all. Most of users I talked to asked me to implement partial
>>> failure mode, because that makes it both safer and easier for them to make
>>> changes.
>>> With hard failure any mistake 

Re: [prometheus-developers] rbac support for exporter-toolkit

[Ben Kochie]

Yes, build it in. We don't want to require sidecars for every exporter.

On Mon, Nov 28, 2022 at 12:43 PM Stuart Clark 
wrote:

> On 2022-11-28 11:40, Ben Kochie wrote:
> > It depends on if the sidecar is with Prometheus or with the target.
> >
> > If it's with Prometheus, that's probably just a docs update.
> >
> > If it's with every exporter, that's probably something we would want
> > in the exporter-toolkit.
> >
> > But, my understanding was that the typical thing here was to use mTLS
> > for securing and authorizing Prometheus.
> >
> > If it's something we need to integrate into every exporter to do some
> > kind of token auth, we might want to consider this.
> >
>
> Do you mean building in the functionality directly into the exporter
> instead of using a sidecar?
>
> --
> Stuart Clark
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrmeBX5fxbiPzDV%2BYpePy4UqYz%3DQsHJRwtPkob%2BGZ_w5Q%40mail.gmail.com.

Re: [prometheus-developers] rbac support for exporter-toolkit

[Ben Kochie]

It depends on if the sidecar is with Prometheus or with the target.

If it's with Prometheus, that's probably just a docs update.

If it's with every exporter, that's probably something we would want in the
exporter-toolkit.

But, my understanding was that the typical thing here was to use mTLS for
securing and authorizing Prometheus.

If it's something we need to integrate into every exporter to do some kind
of token auth, we might want to consider this.

On Mon, Nov 28, 2022 at 11:59 AM Stuart Clark 
wrote:

> On 2022-11-28 10:56, Jesús Samitier wrote:
> > Hi
> >
> > The idea is to integrate kube-rbac-proxy to add an extra (and
> > optional) security feature in a new exporter, so the final user can
> > rely on RBAC to assure that only Prometheus can scrape its metrics.
> > This is something you get when you install Prometheus in K8s using the
> > official helm chart - only Prometheus can scrape the Prometheus
> > metrics exposed by the K8s internals. The idea is to have something
> > similar but for any exporter.
> >
> > Any developer can integrate it in its exporter (as shown here
> >
> https://www.brancz.com/2018/02/27/using-kube-rbac-proxy-to-secure-kubernetes-workloads
> ),
> > but someone pointed out on Mastodon that we could also integrate in in
> > the exporter toolkit so it's even easier.
> >
>
> What would actually be needed in the toolkit though? Is it just some
> docs explaining how to deploy the sidecar with the exporter, or actual
> code changes?
>
> --
> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/3883faab86ed1a93de7ad2d8f8598ea5%40Jahingo.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqkO%2B5fexC%3D0674BdG83Up8rYJYmHt_ooMqGHybPv%3DYww%40mail.gmail.com.

Re: [prometheus-developers] Preventing Prometheus from running out of memory

[Ben Kochie]

Soft / partial failure modes can be very hard problems to deal with. You
have to be a lot more careful to not end up missing partial failures.

While it seems like having a soft sample_limt is good, and the hard
sample_limit is bad. The "Fail fast" will serve you better in the long run.
Most of the Prometheus monitoring design assumes fail fast. Partial results
are too hard to reason about from a monitoring perspective. With fail fast
you will know quickly and decisively that you've hit a problem. If you
treat monitoring outages as just as bad as an actual service outage, you'll
end up with a healthier system overall.

For the case of label explosions, there are some good meta metrics[0] that
can help you. The "scrape_series_added" metric can allow you to soft detect
label leaks.
In addition, there is a new feature flag[1] that adds additional target
metrics for monitoring for targets nearing their limits.

[0]:
https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series
[1]:
https://prometheus.io/docs/prometheus/latest/feature_flags/#extra-scrape-metrics

On Fri, Nov 25, 2022 at 1:27 PM l.mi...@gmail.com 
wrote:

> Hello,
>
> One of the biggest challenges we have when trying to run Prometheus with a
> constantly growing number of scraped services is keeping resource usage
> under control.
> This usually means memory usage.
> Cardinality is often a huge problem and we often end up with services
> accidentally exposing labels that are risky. One silly mistake we see every
> now and then is putting raw errors as labels, which then leads to time
> series with {error="connection from $ip:$port to $ip:$port timed out"} and
> so on.
>
> We had a lot of way of dealing with this that uses vanilla Prometheus
> features but none of it really works well for us.
> Obviously there is sample_limit that one might use here, but the biggest
> problem with it is the fact that once you hit sample_limit threshold you
> lose all metrics, and that's just not acceptable for us.
> If I have a service that exports 999 time series and it suddenly goes to
> 1001 (with sample_limit=1000) I really don't want to lose all metrics just
> because of that because losing all monitoring is bigger problem than having
> a few extra time series in Prometheus. It's just too risky.
>
> We're currently running Prometheus with patches from:
> https://github.com/prometheus/prometheus/pull/11124
>
> This gives us 2 levels of protection:
> - global HEAD limit - Prometheus is not allowed to have more than M time
> series in TSDB
> - per scrape sample_limit - but patched so that if you exceed sample_limit
> it will start rejecting time series that aren't already in TSDB
>
> This works well for us and gives us a system that:
> - gives us reassurance that Prometheus won't start getting OOM killed
> overnight
> - service owners can add new metrics without fear that a typo will cost
> them all metrics
>
> But comments on that PR suggest that it's a highly controversial feature.
> I wanted to probe this community to see what the overall feeling is and
> how likely is that vanilla Prometheus will have something like this.
> It's a small patch so I'm happy to just maintain it for our internal
> deployments but it just feels like a common problem to me, so a baked in
> solution would be great.
>
> Lukasz
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/5ab29a58-e5a4-43c5-b662-4436db61f20an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrni%3DCGPE%2BK%3DoAwrFuWcRh3Edh8cABXB8gKpCqyv5T85w%40mail.gmail.com.

Re: [prometheus-developers] Add processor metric in client_golang

[Ben Kochie]

This is available in the new Go runtime metrics, you can enable it in your
application with a runtime metrics filter.

https://github.com/prometheus/client_golang/blob/9801a4e3ceb49d24dfaf4fa6fc8c2d58aa3b2dc9/prometheus/collectors/go_collector_latest_test.go#L284

You'll probably want `/sched/gomaxprocs:threads` as your custom rule.

On Tue, Sep 13, 2022 at 11:31 AM 'damianqin' via Prometheus Developers <
prometheus-developers@googlegroups.com> wrote:

> Hello folks!
> When we maintain the golang service on k8s, we will automatically set
> the GOMAXPROCS environment variable equal to the Pod limit in the cd
> process to reduce the occurrence of cpu throttling, but other developers
> may use runtime.GOMAXPROCS  to
> override the parameters, resulting in application performance down. I think
> client_golang can use the runtime.GOMAXPROCS(0) method to expose the
> current indicator value.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/1db181d7-950a-4437-b69c-aa5cc329ae9fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpdu%2BesdcoP6EHAPGsyQfqhQYMzehFj23GOcUnxbbpHFg%40mail.gmail.com.

Re: [prometheus-developers] Memcached exporter Multi-target

[Ben Kochie]

Hey, that would be a great feature to have.

I would start by opening an issue on the exporter repo.

I also have this need, and have been starting the work to refactor the AWS
code in Prometheus to support "roles" like the kubernetes_sd_configs so we
can add direct support for something like this:

aws_sd_configs:
- role: elasticache
  region: foo

My target is to merge ec2, lightsail, and add elasticache and rds.

On Fri, Sep 2, 2022 at 12:23 PM 'Mateus Dubiela' via Prometheus Developers <
prometheus-developers@googlegroups.com> wrote:

> Hello folks!
>
> We are looking into adding support for multi-target 
>  to the memcached 
> exporter . The reasoning 
> here is to be able to use the elasticache service discovery 
>  and be able to do 
> the same thing as the redis exporter 
> .
>
> As per the CONTRIBUTING.md 
> 
>  asking the mailing list is a sensible thing to do, to have an open 
> discussion and also ask if someone else was working on this.
>
> Thank you!
>
>
> --
> Confidentiality note: This e-mail may contain confidential information
> from Nu Holdings Ltd and/or its affiliates. If you have received it by
> mistake, please let us know by e-mail reply and delete it from your system;
> you may not copy this message or disclose its contents to anyone; for
> details about what personal information we collect and why, please refer to
> our privacy policy .
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CA%2BgrcXiw_qV6Fx65v271GKfwsKDeAOuSv%3D2_6sqm0SM%2BQo-NbA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoMVwF0ZRUrgnBPpJ11kxhku2Xp-bZ-ER7uJhJA4ACEXw%40mail.gmail.com.

Re: [prometheus-developers] Move from CircleCI to GitHub action

[Ben Kochie]

Seems fine to me.

Two requirements that come to mind:
* We need a "machine" level executor, for some builds that don't work well
in Docker.
* We need a stackable containers/compose type for some integration tests,
for example MySQL and Postgres exporters.

I have not looked into how those work with GitHub actions.

On Thu, Jun 16, 2022, 10:45 AM Julien Pivotto 
wrote:

> Hello,
>
> Based on a request from CNCF, I'd like to move from CircleCI to Github
> Action.
>
> We are already using GitHub actions today, for linting and fuzzing.
>
> Compared to what we are using in CircleCI, it looks like GitHub action
> runners are comparable in size, with slightly more ram.
>
> Pros' of the move:
> - Remove a CI system
> - Better integration with Github (access to logs, etc)
> - It seems better for CNCF itself
>
> Con's of the move:
> - Threading (e.g. parallel builds) will be a bit more itchy at the
>   moment but will be doable.
>
> I plan to do the work in the coming weeks, unless we don't have lazy
> consensus.
>
> Regards,
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/YqrtnRGCHWT5f1w1%40nixos
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpDTBvVRzc5uYsDVqgmHt97XzurXnoZJJaqOzar8Hp1KA%40mail.gmail.com.

Re: [prometheus-developers] Update Prometheus Readme on docker hub

[Ben Kochie]

I looked into this a while back and didn't find a good API / tool to
automatically push updates.

Maybe things have improved since I last tried.

On Wed, Feb 2, 2022, 17:01 Julien Pivotto 
wrote:

>
> Hello,
>
> Someone pointed out on twitter that our Docker Hub readme was not
> useful to run Prometheus in Docker. They also suggested a snippet from
> our documentation to update it.
> https://twitter.com/phil_eaton/status/1488895298239877128
>
> After analyzing the claim, the README of the docker image was very
> outdated. I have therefore applied the user's advice.
>
> We can think further how we can improve this, but I think it was not
> reasonable to have a X years old readme on the docker hub.
>
> This is the new page:
>
> https://hub.docker.com/r/prom/prometheus
>
> Regards,
>
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20220202160143.GA669695%40hydrogen
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmr_0dg7kP-omQVrzeLnrQFQ%3DuoyCZBTEb_qqU6g8%2BDZkA%40mail.gmail.com.

Re: [prometheus-developers] [VOTE] Rename blackbox_exporter to prober

[Ben Kochie]

YES

On Thu, Jan 20, 2022 at 3:41 PM Julien Pivotto 
wrote:

> Dear Prometheans,
>
> As per our governance, I'd like to cast a vote to rename the Blackbox
> Exporter to Prober.
> This vote is based on the following thread:
>
> https://groups.google.com/g/prometheus-developers/c/advMjgmJ1E4/m/A0abCsUrBgAJ
>
> Any Prometheus team member is eligible to vote, and votes for the
> community are welcome too, but do not formally count in the result.
>
> Here is the content of the vote:
>
> > We want to rename Blackbox Exporter to Prober.
>
> I explicitly leave out the "how" out of this vote. If this vote passes,
> a specific issue will be created in the blackbox exporter repository
> explaining how I plan to work and communicate on this change. I will
> make sure that enough time passes so that as many people as possible can
> give their input on the "how".
>
> The vote is open until February 3rd. If the vote comes positive before
> next week's dev summit, the "how" can also be discussed during the dev
> summit, and I would use that discussion as input for the previously
> mentioned github issue.
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20220120144119.GA522055%40hydrogen
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpyFGGv5amrq4EoftRRLEbzZ5zFVpuR8CKhge5iMicmBw%40mail.gmail.com.

Re: [prometheus-developers] Add label to mysqld-exporter to show the mysql instance

[Ben Kochie]

The exporter consumes almost no resources. It should not be resource
impacting.

Just run the exporter 1:1 with your nodes, this is the correct design, It
means you are communicating over localhost with the mariadb process, which
allows you to improve security by restricting the exporter user to
localhost only.

On Tue, Jan 18, 2022 at 7:56 PM ehsan karimi 
wrote:

>
> I see. I did it with relabeling. But this option can be in the exporter
> because sometimes we have no access to the MySQL's host or some another
> reasons. The multi-target idea can be done as well. I have 3 mariadb nodes
> and I forced to run one exporter per mariadb instance and I have 3 exporter
> instances on production that is too resource consuming.
> On Monday, January 17, 2022 at 11:05:51 PM UTC+3:30 matt...@prometheus.io
> wrote:
>
>> This is not supported in the exporter and we have no plans to add it.
>> Most exporters use a different approach, which we recommend for exporters
>> in general.
>>
>> Deploy the expory as a sidecar alongside the MySQL instance. In
>> Kubernetes, this means an additional container in the MySQL pod. This
>> solves your problem by making each MySQL+exporter pod its own scrape target
>> with its own instance label.
>>
>> For most exporters, the way to think about them is not as a separate
>> service that somehow interacts with what it is translating for (in this
>> case, MySQL). Rather see the exporter as an out-of-process plugin, paired
>> 1:1 with each MySQL process. For the purposes of monitoring, they are one
>> unit, and when looking at metrics and alerts you don't need to worry
>> whether a piece of software supports Prometheus natively or through an
>> exporter.
>>
>> I hope this helps!
>> Matthias
>>
>> On Mon, Jan 17, 2022, 13:01 ehsan karimi  wrote:
>>
>>> I install the mysqld-exporter on Kubernetes and when I scrape it with
>>> Prometheus, the instance label will show the pod IP of the mysqld-exporter
>>> instance and when we saw the MySqlIsDown alert, I don't know what MySQL
>>> instance is for it. I wanna add a label to the exposed metrics to show the
>>> host of MySQL.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-devel...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-developers/0a4a1c76-3436-4e92-be4d-e90e0e6cc069n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/74b3b95c-b37f-4356-9900-4b92413fc4ffn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpV4k9%3D-yZ%3DM83bVLnys-B1H%2B0vBjapKPQbr5FvsZVyzg%40mail.gmail.com.

Re: [prometheus-developers] Add label to mysqld-exporter to show the mysql instance

[Ben Kochie]

And to add to this, for the case of managed MySQL where the sidecar is not
possible, we can add multi-target exporter support.

https://prometheus.io/docs/guides/multi-target-exporter/

There's a partial implementation of this in a PR on the exporter, but the
author has not responded to feedback requests.

On Mon, Jan 17, 2022 at 8:35 PM Matthias Rampke 
wrote:

> This is not supported in the exporter and we have no plans to add it. Most
> exporters use a different approach, which we recommend for exporters in
> general.
>
> Deploy the expory as a sidecar alongside the MySQL instance. In
> Kubernetes, this means an additional container in the MySQL pod. This
> solves your problem by making each MySQL+exporter pod its own scrape target
> with its own instance label.
>
> For most exporters, the way to think about them is not as a separate
> service that somehow interacts with what it is translating for (in this
> case, MySQL). Rather see the exporter as an out-of-process plugin, paired
> 1:1 with each MySQL process. For the purposes of monitoring, they are one
> unit, and when looking at metrics and alerts you don't need to worry
> whether a piece of software supports Prometheus natively or through an
> exporter.
>
> I hope this helps!
> Matthias
>
> On Mon, Jan 17, 2022, 13:01 ehsan karimi  wrote:
>
>> I install the mysqld-exporter on Kubernetes and when I scrape it with
>> Prometheus, the instance label will show the pod IP of the mysqld-exporter
>> instance and when we saw the MySqlIsDown alert, I don't know what MySQL
>> instance is for it. I wanna add a label to the exposed metrics to show the
>> host of MySQL.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/0a4a1c76-3436-4e92-be4d-e90e0e6cc069n%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAMV%3D_gbi7mV2UWL-n8aFAUWZY%3DVC2c3hPo8%3DfUKp7ZkefNwb2w%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpWkurNQvmbU-BEdVUmqPrKNGqRWsvTyTVC05AeSaObHg%40mail.gmail.com.

Re: [prometheus-developers] How to get all metric names from Prometheus server filtered by a particular label

[Ben Kochie]

Your second query is using not-equals. I would expect it to return all
metrics names that do not have job="prometheus". Seems to be
working correctly.

On Fri, Dec 10, 2021 at 9:10 AM Amey Barve  wrote:

> We want to get all metric names from Prometheus server filtered by a
> particular label.
>
> Step 1 : Used following query to get all metric names, query succeeded
> with all metric names.
> curl -g 'http://localhost:9090/api/v1/label/__name__/values
>
> Step 2 : Used following query to get all metrics names filtered by label,
> but query still returned all metric names without filtering those metrics
> which do not have job label equal to "prometheus"
> curl -g 
> 'http://localhost:9090/api/v1/label/__name__/values?match[]={job!="prometheus"}'
>
>
> Can somebody please help me filter all metric names by label over http?
>
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/d1b87f13-e584-49b9-9eee-4526cf6da392n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqq4-QbUeMHqznFbighZzqMadUp7utv7-0jCFTxkaeKFg%40mail.gmail.com.

Re: [prometheus-developers] amtool: passwords passed in the command line

[Ben Kochie]

There are lots of ways to easily inject secrets into configs.

Adding secrets/headers via config file is the safest way.

While I'm all for allowing sharp edges in tools if they're not default, I'm
strongly against having known unsafe things like secrets on the command
line.

On Tue, Nov 23, 2021 at 5:38 PM Augustin Husson 
wrote:

> Hello,
>
> I think having the http config file is a good idea and a safe one.
> The fact users have a rotation in the credential used only means the
> client has to authenticate themself first to get a fresher session / token
> / credentials. Maybe it's more sophisticated than that, but from my
> understanding it shouldn't be.
>
> Kubernetes is using a config file for it's kube client and it works
> nicely. The token used and stored in the file expires every 24h  and it's
> not so hard to have a fresher one.
>
> Best regards,
> Augustin.
>
> Le mar. 23 nov. 2021 à 17:15, Julien Pivotto 
> a écrit :
>
>> Hello -developers,
>>
>> In the past and still today, we have asked exporters not to use secrets
>> on the command line.
>>
>> There is a pull requests that wants to add secrets on the amtool command
>> line:
>> https://github.com/prometheus/alertmanager/pull/2764
>>
>> and users requests to pass arbitrary http headers in amtool via the
>> command line too. In the same way, users want to add arbitraty secrets
>> in HTTP headers: https://github.com/prometheus/alertmanager/issues/2597
>>
>> I am personally opposed to allow what we ask others not to do, but maybe
>> I am stubborn, so I am asking the developers community here what should
>> we do here?
>>
>> My proposal was to introduce a HTTP client configuration file to amtool,
>> so we tackle the secret issue and enable all the other HTTP client
>> options easily (oauth2, bearer token, proxy_url, ...). The community was
>> not entirely keen on it:
>>
>> https://github.com/prometheus/alertmanager/issues/2597#issuecomment-974144389
>>
>> What do the large group of developers think about all this? Note that
>> the solution we chose here could/should be applied to promtool and
>> getool later.
>>
>> Thanks!
>>
>> --
>> Julien Pivotto
>> @roidelapluie
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/20211123161546.GA696401%40hydrogen
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAOJizGcb45MwjCj3Bd6_gt9ZatS%2Bnbw%2B1QvjD8wbNdfR77eo%3DQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpuNnWrT2H6o2Vkpuuvhsa0mJ%2B5MKapUvhs2_0Vs_FZ4w%40mail.gmail.com.

Re: [prometheus-developers] Is this alerting architecture crazy?

[Ben Kochie]

On Mon, Nov 22, 2021 at 4:03 PM Tony Di Nucci  wrote:

> Thanks for the feedback Stuart, I really appreciate you taking the time
> and you've given me reason to pause and reconsider my options.
>
> I fully understand your concerns over having a new data store.  I'm not
> sure that AlertManager and Prometheus contain the state I need though and
> I'm not sure I should attempt to use Prometheus as the store for this state
> (tracking per alert latencies would end up with a metric with unbounded
> cardinality, each series would just contain a single data point and it
> would be tricky to analyse this data).
>
> On the "guaranteeing" delivery front.  You of course have a point that the
> more moving parts there are the more that can go wrong.  From the sounds of
> things though I don't think we're debating the need for another system
> (since this is what a webhook receiver would be?).
>
> Unless I'm mistaken, to hit the following requirements there'll need to be
> a system external AlertManager and this will have to maintain some state:
> * supporting complex alert enrichment (in ways that cannot be defined in
> alerting rules)
>

We actually are interested in adding this to the alertmanager, there are a
few open proposals for this. Basically the idea is that you can make an
enrichment call at alert time to do things like grab metrics/dashboard
snapshots, other system state, etc.


> * support business specific alert routing rules (which are defined outside
> of alerting rules)
>

The alertmanager routing rules are pretty powerful already. Depending on
what you're interested in adding, this is something we could support
directly.


> * support detailed alert analysis (which includes per alert latencies)
>

This is, IMO, more of a logging problem. I think this is something we could
add. You ship the alert notifications to any kind of BI system you like,
ELK, etc.

Maybe something to integrate into
https://github.com/yakshaving-art/alertsnitch.


>
> I think this means that the question is limited to; is it better in my
> case to push or pull from AlertManager.  BTW, I'm sorry for the way I
> worded my original post because I now realise how important it was to make
> explicit the requirements that (I think) necessitate the majority of the
> complexity.
>

Honestly, most of what you want is stuff we could support in Alertmanager
without a lot of trouble. And are things that other users would want as
well. Rather than build a whole new system, why not contribute improvements
directly to the Alertmanager.


>
> As I still see it, the problems with the push approach (which are not
> present with the pull approach are):
> * It's only possible to know that an alert cannot be delivered after
> waiting for *group_interval *(typically many minutes)
> * At a given moment it's not possible to determine whether a specific
> active alert has been delivered (at least I'm not aware of a way to
> determine this)
> * It is possible for alerts to be dropped (e.g.
> https://github.com/prometheus/alertmanager/blob/b2a4cacb95dfcf1cc2622c59983de620162f360b/cluster/delegate.go#L277
> )
>
> The tradeoffs for this are:
> * I'd need to discover the AlertManager instances.  This is pretty
> straight forward in k8s.
> * I may need to dedupe alert groups across AlertManager instances.  I
> think this would be pretty straight forward too, esp. since AlertManager
> already populates fingerprints.
>
>
>
>
> On Sunday, November 21, 2021 at 10:28:49 PM UTC Stuart Clark wrote:
>
>> On 20/11/2021 23:42, Tony Di Nucci wrote:
>> > Yes, the diagram is a bit of a simplification but not hugely.
>> >
>> > There may be multiple instances of AlertRouter however they will share
>> > a database.  Most likely things will be kept simple (at least
>> > initially) where each instance holds no state of its own.  Each active
>> > alert in the DB will be uniquely identified by the alert fingerprint
>> > (which the AlertManager API provides, i.e. a hash of the alert groups
>> > labels).  Each non-active alert will have a composite key (where one
>> > element is the alert group fingerprint).
>> >
>> > In this architecture I see AlertManager having the responsibilities of
>> > capturing, grouping, inhibiting and silencing alerts.  The AlertRouter
>> > will have the responsibilities of; enriching alerts, routing based on
>> > business rules, monitoring/guaranteeing delivery and enabling analysis
>> > of alert history.
>> >
>> > Due to my requirements, I think I need something like the
>> > AlertRouter.  The question is really, am I better to push from
>> > AlertManager to AlertRouter, or to have AlertRouter pull from
>> > AlertManager.  My current opinion is that pulling comes with more
>> > benefits but since I've not seen anyone else doing this I'm concerned
>> > there could be good reasons (I'm not aware of) for not doing this.
>>
>> If you really must have another system connected to Alertmanager having
>> it respond to webhook notifications would be the much 

Re: [prometheus-developers] Is this alerting architecture crazy?

[Ben Kochie]

Also, the alertmanager does have an "even store", it's a shared state
between all instances.

If you're interested in changing some of the behavior of the retry
mechanisms or how this works, feel free to open specific issues. You don't
need to build an entirely new system, we can add new features to the
existing Alertmanager clustering framework.

On Sat, Nov 20, 2021 at 11:29 AM Ben Kochie  wrote:

> What gives you the impression that the Alertmanager is "best effort"?
>
> The alertmanager provides a reasonably robust HA solution (gossip
> clustering). The only thing best-effort here is actually deduplication. The
> Alertmanager design is "at least once" delivery, so it's robust against
> network split-brain issues. So in the event of a failure, you may get
> duplicate alerts, not none.
>
> When it comes to delivery, the Alertmanager does have retries. If a
> connection to PagerDuty or other receivers has an issue, it will retry.
> There are also metrics for this, so you can alert on failures to alternate
> channels.
>
> What you likely need is a heartbeat setup. Because services like PagerDuty
> and Slack do have outages, you can't guarantee delivery if they're down.
>
> The method here is to have an end-to-end "always firing heartbeat" alert,
> which goes to a system/service like healthchecks.io or deadmanssnitch.com.
> These will trigger an alert in the absence of your heartbeat. Letting you
> know that some part of the pipeline has failed.
>
> On Sat, Nov 20, 2021 at 11:02 AM Tony Di Nucci 
> wrote:
>
>> Cross-posted from
>> https://discuss.prometheus.io/t/is-this-alerting-architecture-crazy/610
>>
>> In relation to alerting, I’m looking for a way to get strong alert
>> delivery guarantees (and if delivery is not possible I want to know about
>> it quickly).
>>
>> Unless I’m mistaken AlertManager only offers best-effort delivery. What’s
>> puzzled me though is that I’ve not found anyone else speaking about this,
>> so I worry I’m missing something obvious. Am I?
>>
>> Assuming I’m not mistaken I’ve been thinking of building a system with
>> the architecture shown below.
>>
>> [image: alertmanager-alertrouting.png]
>>
>> Basically rather than having AlertManager try and push to destinations
>> I’d have an AlertRouter which polls AlertManager. On each polling cycle the
>> steps would be (neglecting any optimisations):
>>
>>- All active alerts are fetched from AlertManager.
>>- The last known set of active alerts is read from the Alert Event
>>Store.
>>- The set of active alerts is compared with the last known state.
>>- New alerts are added to an “active” partition in the Alert Event
>>Store.
>>- Resolved alerts are removed from the “active” partition and added
>>to a “resolved” partition.
>>
>> A secondary process within AlertRouter would:
>>
>>- Check for alerts in the “active” partition which do not have a
>>state of “delivered = true”.
>>- Attempt to send each of these alerts and set the “delivered” flag.
>>- Check for alerts in the “resolved” partition which do not have a
>>state of “delivered = true”.
>>- Attempt to send each of these resolved alerts and set the
>>“delivered” flag.
>>- Move all alerts in the “resolved” partition where “delivered=true”
>>to a “completed” partition.
>>
>> Among other metrics, the AlertRouter would emit one called
>> “undelivered_alert_lowest_timestamp_in_seconds” and this could be used to
>> alert me to cases where any alert could not be delivered quickly enough.
>> Since the alert is still held in the Alert Event Store it should be
>> possible for me to resolve whatever issue is blocking and not lose the
>> alert.
>>
>> I think there are other benefits to this architecture too, e.g. similar
>> to the way Prometheus scrapes, natural back-pressure is a property of the
>> system.
>>
>> Anyway, as mentioned I’ve not found anyone else doing something like this
>> and this makes me wonder if there’s a very good reason not to. If anyone
>> knows that this design is crazy I’d love to hear!
>>
>> Thanks
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/be2f9bfd-ba4d-46ea-9816-f19ebef499d6n

Re: [prometheus-developers] Is this alerting architecture crazy?

[Ben Kochie]

What gives you the impression that the Alertmanager is "best effort"?

The alertmanager provides a reasonably robust HA solution (gossip
clustering). The only thing best-effort here is actually deduplication. The
Alertmanager design is "at least once" delivery, so it's robust against
network split-brain issues. So in the event of a failure, you may get
duplicate alerts, not none.

When it comes to delivery, the Alertmanager does have retries. If a
connection to PagerDuty or other receivers has an issue, it will retry.
There are also metrics for this, so you can alert on failures to alternate
channels.

What you likely need is a heartbeat setup. Because services like PagerDuty
and Slack do have outages, you can't guarantee delivery if they're down.

The method here is to have an end-to-end "always firing heartbeat" alert,
which goes to a system/service like healthchecks.io or deadmanssnitch.com.
These will trigger an alert in the absence of your heartbeat. Letting you
know that some part of the pipeline has failed.

On Sat, Nov 20, 2021 at 11:02 AM Tony Di Nucci 
wrote:

> Cross-posted from
> https://discuss.prometheus.io/t/is-this-alerting-architecture-crazy/610
>
> In relation to alerting, I’m looking for a way to get strong alert
> delivery guarantees (and if delivery is not possible I want to know about
> it quickly).
>
> Unless I’m mistaken AlertManager only offers best-effort delivery. What’s
> puzzled me though is that I’ve not found anyone else speaking about this,
> so I worry I’m missing something obvious. Am I?
>
> Assuming I’m not mistaken I’ve been thinking of building a system with the
> architecture shown below.
>
> [image: alertmanager-alertrouting.png]
>
> Basically rather than having AlertManager try and push to destinations I’d
> have an AlertRouter which polls AlertManager. On each polling cycle the
> steps would be (neglecting any optimisations):
>
>- All active alerts are fetched from AlertManager.
>- The last known set of active alerts is read from the Alert Event
>Store.
>- The set of active alerts is compared with the last known state.
>- New alerts are added to an “active” partition in the Alert Event
>Store.
>- Resolved alerts are removed from the “active” partition and added to
>a “resolved” partition.
>
> A secondary process within AlertRouter would:
>
>- Check for alerts in the “active” partition which do not have a state
>of “delivered = true”.
>- Attempt to send each of these alerts and set the “delivered” flag.
>- Check for alerts in the “resolved” partition which do not have a
>state of “delivered = true”.
>- Attempt to send each of these resolved alerts and set the
>“delivered” flag.
>- Move all alerts in the “resolved” partition where “delivered=true”
>to a “completed” partition.
>
> Among other metrics, the AlertRouter would emit one called
> “undelivered_alert_lowest_timestamp_in_seconds” and this could be used to
> alert me to cases where any alert could not be delivered quickly enough.
> Since the alert is still held in the Alert Event Store it should be
> possible for me to resolve whatever issue is blocking and not lose the
> alert.
>
> I think there are other benefits to this architecture too, e.g. similar to
> the way Prometheus scrapes, natural back-pressure is a property of the
> system.
>
> Anyway, as mentioned I’ve not found anyone else doing something like this
> and this makes me wonder if there’s a very good reason not to. If anyone
> knows that this design is crazy I’d love to hear!
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/be2f9bfd-ba4d-46ea-9816-f19ebef499d6n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmq%2BL7My_srC-pe8BGg0wnQqTq_TF%2B7BFE-0Lt9jPdAeHw%40mail.gmail.com.

Re: [prometheus-developers] Feature request: Setting a custom lookback delta for a query

[Ben Kochie]

Thanos automatically deals with downsample lookback with no changes. This
was fixed at least 2 years ago.

On Mon, Nov 15, 2021, 18:54 Julien Pivotto 
wrote:

> We are willing to have downsampling in Prometheus too in the future.
>
> I would merge that pull request.
>
> Le lun. 15 nov. 2021, 15:19, Vilius Pranckaitis 
> a écrit :
>
>> *TL;DR*: ability to set a custom lookback delta for a query would be
>> useful when time series have drastically different resolutions.
>>
>> ---
>>
>> Currently, Prometheus allows configuring lookback delta through
>> configuration flag. This works well if the resolutions/scrape intervals of
>> time series are similar, but is inconvenient if they differ a lot. E.g. if
>> we had two time series, one with a datapoint each hour and another one with
>> a datapoint each minute, a lookback of 2h would work for the first one, but
>> would be way too big for the second one.
>>
>> This impacts looking at downsampled data greater than 5 minutes (the
>> default lookback period) which both Thanos and M3 frequently keep
>> (depending on user configuration). Both support downsampling data to user
>> configured resolutions, thus lookback delta should be selected according to
>> what resolution is being queried when querying downsampled data. Both
>> Thanos and M3 depend on Prometheus and use `promql.Engine` for executing
>> queries, which doesn't allow setting custom lookback for a particular query.
>>
>> While I understand that this might be more relevant for M3 and Thanos
>> than for Prometheus, just wanted to check what the maintainers of
>> Prometheus think about this. If this seems useful, someone from M3 should
>> be able to submit a PR (we have already looked at the code and it doesn't
>> look like it needs huge changes).
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/84c03b13-25d7-479c-b678-44cd8625f2c0n%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAFJ6V0p-_k_Yjy_gUdvHaogxZz%3D7DeNEMM7zz7uBYD%2BYBA%2B6bw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrjPhnrWFm441MnH86qVy1mxhE4U8F6ZDPt8Hg-oRjEsA%40mail.gmail.com.

Re: [prometheus-developers] Consider using more strict formatters for Go code like gci and gofumpt

[Ben Kochie]

+1, the formatting changes look nice.

On Thu, Oct 21, 2021 at 10:36 AM Julien Pivotto 
wrote:

> On 21 Oct 01:33, Mateusz Gozdek wrote:
> > Hi all,
> >
> > While recently contributing for the first time to Prometheus code base,
> > I've noticed that the Go source files could be more consistently
> formatted.
> > My IDE is configured for a long time to use "gofumpt" and "gci"
> formatters
> > on file save, so "git diff" was producing a lot of unrelated changes to
> me,
> > which is a bit annoying if one wants to produce clean commits without
> mixed
> > changes, so some "git add -p" dance was required almost all the time.
> >
> > Considering that both of mentioned formatters are backward compatible
> with
> > core "gofmt" and that golangci-lint supports enforcing both of them, I'd
> > like to propose enabling those linters on the repositories.
> >
> > I believe this can also resolve some possible discussions on formatting,
> > saving time and focus on more important areas of the code.
> >
> > You can see suggested formatting changes
> > here:
> https://github.com/prometheus/prometheus/commit/282990880615ab698fe9583b21ae52e19c333f46
> >
> > This suggestion was originally posted here:
> > https://github.com/prometheus/prometheus/issues/9557
> >
> > Let me know what you think.
> >
> > Regards,
> > Mateusz Gozdek
>
> I like this idea.
>
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20211021083605.GA130008%40hydrogen
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmo%3Dx_4PYap8W8xbVKrnQSHUH%3Dn3VnxBH-5zT5R2w6ECiQ%40mail.gmail.com.

Re: [prometheus-developers] Adding timestamps to Gauge Metric

[Ben Kochie]

Yup, Prometheus is a monitoring system, not a general use time-series
database.

Specifically, Prometheus has a "look back window" where Prometheus will
take the timestamp of your query and look back in time for samples to match.

This is fundamental to how PromQL works.

So no, what you are asking for is not possible. PromQL graph queries base
the X-axis on the input parameters of the range query. The start, end, and
step.

See the documentation on the API:
https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries

On Sun, Aug 29, 2021 at 1:09 PM Stuart Clark 
wrote:

> That very much depends on whatever tool you are using to display graphs.
>
> However it is sounding like Prometheus may not be the right system
> depending on what you are trying to do. Prometheus is a metric system,
> which works by sampling the current state of a system at regular time
> periods, meaning the exact timestamp doesn't generally matter.
>
> It sounds like you are instead wanting to record events - things that
> happen at a specific period of time, not at a regular frequency. For that
> use case you should look at an event store - something like Elasticsearch,
> InfluxDB or a standard relational or no-SQL database.
>
> On 29 August 2021 11:15:44 BST, Prince  wrote:
>>
>> Is it possible to give a custom timestamp in Prometheus for X-axis
>> 
>> ?
>> Example: I* am getting **my_metric_name *152.401 163013412 *at
>> metric endpoint but in Prometheus graph, I am getting the value *152.401* 
>> when
>> it is scraped, but I want it should be displayed at *163013412(Saturday,
>> August 28, 2021, 7:00:00.012 AM) *this time in Prometheus graph.*
>>
>> Is it possible? if yes, can you please let me know how?
>> Thank you.
>>
>> On Sunday, August 29, 2021 at 11:06:27 AM UTC+5:30 Prince wrote:
>>
>>> Thank you, understood. I have used the following:
>>>
>>> in Collect()
>>>  *  t := time.Date(2021, time.August, 28, 07, 0, 0, 12345678, time.UTC)*
>>> *s := prometheus.MustNewConstMetric(c.metric, prometheus.GaugeValue,
>>> float64(s.value))*
>>> *ch<- prometheus.NewMetricWithTimestamp(t,s)*
>>>
>>> *I am getting **my_metric_name 152.401 163013412 *(both things
>>> the value and timestamp), but I am not getting this timestamp in the x-axis
>>> of prometheus graph. Can You please let me how can I get that timestamps in
>>> The x-axis of prometheus graph?
>>>
>>> On Wednesday, August 25, 2021 at 12:49:26 PM UTC+5:30
>>> juliu...@promlabs.com wrote:
>>>
 So NewMetricWithTimestamp() returns a Metric interface object that you
 can then emit from a Collector's Collect() method. See this example from
 cadvisor:
 https://github.com/google/cadvisor/blob/19df107fd64fa31efc90e186af91b97f38d205e9/metrics/prometheus.go#L1931-L1934

 You can see more usage example here:
 https://sourcegraph.com/search?q=context:global+prometheus.NewMetricWithTimestamp=literal

 In general, it seems like you are building an exporter (a process
 that proxies/translates existing values into the Prometheus format, in your
 case those existing values are coming from a file), so you are not
 instrumenting the exporting process itself, and thus you probably don't
 want to use the "NewGauge()" / "mygauge.WithLabelValues().Set()" functions
 that are for direct instrumentation of a process. Instead, you'll want to
 implement a Collector interface that just returns a set of proxied metrics,
 as outlined here:

 *
 https://pkg.go.dev/github.com/prometheus/client_golang/prometheus#hdr-Custom_Collectors_and_constant_Metrics
 *
 https://prometheus.io/docs/instrumenting/writing_exporters/#collectors

 On Tue, Aug 24, 2021 at 7:43 PM Prince  wrote:

> Thank you, As I understood the  NewMetricWithTimestamp() takes two
> parameters one the time and the other one is metric. So as my metric name
> is go_duration, so I did this :
>
> *1st way:*
> *go func(){*
> go_duration.WithLabelValues("type").Set(12345.678)
> prometheus.NewMetricWithTimestamp(time_var, go_duration )
> *}()*
>
> *2nd way: *
> *go func(){*
>
> prometheus.NewMetricWithTimestamp(time_var,go_duration.WithLabelValues("type").Set(12345.678))
> *}()*
>
>
> Using 1st way not getting the timestamp only values are getting
> scarped.
> Using 2nd way getting error as: 
> "go_duration.WithLabelValues("type").Set(12345.678)
> used as a value"
> On Tuesday, August 24, 2021 at 10:15:57 PM UTC+5:30
> juliu...@promlabs.com wrote:
>
>> Hi,
>>
>> You should be able to use the NewMetricWithTimestamp() function for
>> this:
>> https://pkg.go.dev/github.com/prometheus/client_golang/prometheus?utm_source=godoc#NewMetricWithTimestamp
>>
>> Note that client-side timestamps should 

Re: [prometheus-developers] Deprecating https://github.com/prometheus/nagios_plugins (?)

[Ben Kochie]

The fork owner can make a manual request to github to mark the repo as not
a fork. It's pretty easy to do, I've done this for several other projects.

On Wed, Aug 18, 2021 at 12:54 PM Bjoern Rabenstein 
wrote:

> On 18.08.21 12:03, Julien Pivotto wrote:
> > We should graveyard it. If you are concerned about redirecting people to
> > the fork, we should transfer the ownership of the repo.
>
> Yeah, but transferring the ownership will require quite a risky dance
> (because the fork already exists for a while, see my original mail).
>
> In any case, we now have diverging opinions about merely archiving
> vs. graveyarding. I'll archive the repo now in any case, because that
> doesn't exclude graveyarding later, but "stops the bleeding".
>
> Personally, I'm not very concerned about the redirect, but if we don't
> do the redirect, we should graveyard only after a good while (if we
> want to do it at all).
>
> Bottom line, I guess: Let's discuss the graveyarding in a few months
> time.
>
> --
> Björn Rabenstein
> [PGP-ID] 0x851C3DA17D748D03
> [email] bjo...@rabenste.in
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20210818105417.GT3669%40jahnn
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpTk87HMcOtsd638PcfhMC1fa4i9qhC7orO6uB4%2Bz0UFA%40mail.gmail.com.

Re: [prometheus-developers] Deprecating https://github.com/prometheus/nagios_plugins (?)

[Ben Kochie]

+1 to mark archived.

Manually graveyarding things isn't really needed anymore unless we really
want to break use of a thing.

On Tue, Aug 17, 2021 at 10:54 AM Matthias Rampke 
wrote:

> I think the no-magic route is better. You can also archive the repo[0] to
> make it clear that it's read only (with this GitHub feature, do we still
> need to graveyard anything ourselves?
>
> /MR
>
>
>
> [0]
> https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/archiving-a-github-repository/archiving-repositories
>
> On Fri, Aug 13, 2021, 17:57 Bjoern Rabenstein  wrote:
>
>> Hi,
>>
>> More than a year ago, I added a pointer from
>> https://github.com/prometheus/nagios_plugins (the "old repo") to its
>> fork https://github.com/magenta-aps/check_prometheus_metric (the "new
>> repo"), see https://github.com/prometheus/nagios_plugins/pull/26 .
>>
>> I've never heard any complaints about the new plugin, so I think it's
>> about time to properly deprecate the old repo.
>>
>> First of all: Does anyone have any objections?
>>
>> Assuming we can go forward with it: What do you think is the best
>> procedure? Ideally, we would redirect from the old to the new
>> repo. However, that's not as easy as it looks. So far, I think this
>> would require the following gymnastics:
>>
>> - Delete the new repo.
>> - Transfer the ownership of the old repo to magenta-aps with
>>   the same name as the (deleted) new repo.
>> - Replay all the commits that happened in the new repo to the
>>   transfered repo to make it appear like the new repo before,
>>   just not as a fork.
>>
>> Does anyone have a better idea?
>>
>> And if not, should we really do that or would it be better to apply less
>> magic, just put a big and fat deprecation warning onto the old repo,
>> and graveyard it after another half year or so?
>>
>> Any feedback welcome.
>> --
>> Björn Rabenstein
>> [PGP-ID] 0x851C3DA17D748D03
>> [email] bjo...@rabenste.in
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/20210813155716.GE3669%40jahnn
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAMV%3D_gYjKWMns7DgkhBXr3g%3DKVonp4fyGG%2B6GhnhCiO5tEkAjA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpsBcOwf3Y710xGirVsuTywdRiGK07H5n4VGvmgK4b3Xw%40mail.gmail.com.

Re: [prometheus-developers] increasing prometheus data retention time from 7days to 30 days

[Ben Kochie]

Please use the users email list for help requests. This list is for
developer topics.

On Thu, Jul 15, 2021 at 1:24 PM Mithun Batabyal 
wrote:

> Hi Team,
>
> we are using Prometheus and grafana combination for monitoring in out
> toyota project.
> we want to increase data retention time from 7days to 30 days.
> we have deployed grafana and proemtheus using helm chart from github.
> how to increase it?
> Please guide me.
>
> Thanks and Regards,
> Mithun Batabyal
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/238a7929-94b2-415b-9854-4cd9f9d3eb10n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqgVf%3DoNoKf1MJvoPc9p4in48oBS3K0O7pPXtm6P8%2B56g%40mail.gmail.com.

Re: [prometheus-developers] How about adding some anomaly detection functions in the promql/function.go file

[Ben Kochie]

We've talked about making build-time plugins possible. Specifically for
service discovery methods because some of them are very large/bloated due
to vendored code.

A build time interface like Caddy or CoreDNS for other functions would be
interesting.

On Tue, Jul 13, 2021 at 6:50 AM Darshan Chaudhary 
wrote:

> In a similar vein to what Levi is proposing, how about exposing a plugin
> interface in prometheus that external projects can implement?
> We can provide a UI to select the plugins that the user needs to bake into
> the prometheus binary before downloading
>
> Something like - https://caddyserver.com/download
>
>
> On Thursday, 1 July 2021 at 11:28:45 UTC+5:30 tcol...@gmail.com wrote:
>
>> Just as a point of personal experience on use of AD in production. Having
>> tried in a few places:
>>
>> Seasonality of human activity rarely lines up well with the data slicing
>> needed for AD, You get *a lot* of false positives. It's almost always
>> basically unusable for pagable alerting.
>>
>> You need a lot of traffic. Doing stats on infrequent requests, or on
>> systems that he state thier own traffic spikes (like batch jobs that cause
>> traffic of any size close to typical usage), just doesn't work well.
>>
>> A lot of simpler AD models have a loopback problem where you normal
>> traffic looks anomalous sone time after an actual event , this causes
>> another source of annoying false positive that result in people ignoring
>> alerts.
>>
>> There are AD models now that avoid some of the false positive issues, but
>> I think it remains true that they would be useless without sufficient
>> traffic volumes (last paper I read was from twitter I think).
>>
>> Personally I think any features along these lines will be good fodder for
>> blog posts, but unlikely to end up useful for real world monitoring. (I
>> speak as an author of related posts).
>>
>>
>> https://medium.com/qubit-engineering/using-seasonality-in-prometheus-alerting-d90e68337a4c
>>
>> On Wed, 30 Jun 2021, 22:24 Bjoern Rabenstein,  wrote:
>>
>>> On 25.06.21 01:27, Shandong Dong wrote:
>>> > Ok, I will try the PR first. Can I know what‘s the concern of
>>> "Personally,
>>> > I'm still not sure if that's a sustainable approach. "?
>>>
>>> We had a handful of requests in the past to add specific advanced
>>> statistics functions. In one case, a function was actually added, see
>>>
>>> https://prometheus.io/docs/prometheus/latest/querying/functions/#holt_winters
>>>
>>> The problem with the latter is that it was actually not the variety of
>>> Holt-Winters that most people wanted. A lot of misunderstanding
>>> happened because of that. My impression is (but I might be proven
>>> wrong) is that this is a rarely used PromQL function. But now we have
>>> to support it at least until the next major release.
>>>
>>> That latter problem will be avoided by feature flags. But if we now
>>> each of the five to ten persons that requested new functions will add
>>> on average two to three new functions, we end up with about 20 new
>>> functions, all with the same potential of being misunderstand. Many
>>> might be overlapping, so any new function needs to be reviewed for
>>> overlap with existing ones. Even if they are all behind feature flags,
>>> they will require a lot of code with potential interaction with
>>> existing code and with each other, so there is some maintenance
>>> overhead.
>>>
>>> Eventually, reviewing and acceptance of even more functions behind
>>> feature flags will slow down. So we are back at square one. And the
>>> multitude of experimental functions will make it harder for users to
>>> find the right one to try out. Which in turn will make it harder to
>>> identify the actually generally useful functions and "graduate"
>>> them. Realistically, there will be small groups of users liking
>>> subsets of functions, but rarely functions that aither everyone needs
>>> and likes or nobody.
>>>
>>> It feels a bit like the attempt to create a Python interpreter for
>>> data science that doesn't understand modules and instead tries to have
>>> all required functions built-in. That's hardly a reasonably
>>> approach. And that's why my personal idea is that Prometheus either
>>> has to keep stating that it is only meant for basic mathematical
>>> operations on metrics, or it has to provide some kind of "scripting
>>> interface" to allow custom mathematical "libraries" for users with
>>> special requirements.
>>>
>>> --
>>> Björn Rabenstein
>>> [PGP-ID] 0x851C3DA17D748D03
>>> [email] bjo...@rabenste.in
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-devel...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-developers/20210630212438.GO11559%40jahnn
>>> .
>>>
>> --
> You received this message because you are 

Re: [prometheus-developers] How about adding some anomaly detection functions in the promql/function.go file

[Ben Kochie]

Is anyone using this kind of thing in production? I've seen lots of demos
and promises over the years. Every anomaly detection project seems to be
abandoned after 6 months.

I'd like to avoid adding functionality that is "cool" but has no practical
application.

For example: https://github.com/etsy/skyline

On Fri, Jun 25, 2021 at 9:16 PM Levi Harrison 
wrote:

> We could even use something like Yaegi ,
> an interpreter, which would be slower than using compiled Go code, but
> makes the creation process a lot easier and more stable. This is even
> closer to having our own scripting language, but still is distinguished by
> the ability to simply drop this in with little extra work, and the
> adoption of Go as a language (Yaegi completely supports the Go
> specification).
>
> On Fri, Jun 25, 2021 at 7:02 AM Levi Harrison <
> levisamuelharri...@gmail.com> wrote:
>
>> Yeah, no Windows support is less than desirable. Also after more
>> research, the library seems to be really fraught with issues. Hashicorp has 
>> its
>> plugin library , but I don't
>> think we can afford to take the performance hit of going over RPC. I did
>> find this library  though, which I
>> believe supports Windows and looks really promising. We would have to
>> subject it to further testing though because I don't think it's broadly
>> adopted.
>>
>> It is the case that a scripting engine as Björn suggested would probably
>> the cleanest way to allows users to define their own functions, but that
>> approach brings a lot more complexity so I think this is a good solution
>> for now, also having the added benefit of language that's already widely
>> known.
>>
>> On Fri, Jun 25, 2021 at 5:30 AM Julien Pivotto <
>> roidelapl...@prometheus.io> wrote:
>>
>>> On 24 Jun 14:53, Levi Harrison wrote:
>>> > Maybe we could utilize Golang's plugin library  <
>>> https://pkg.go.dev/plugin>to
>>> > provide a way for users to write their own functions in compliance
>>> with the defined
>>> > format
>>> > <
>>> https://github.com/prometheus/prometheus/blob/7cb55d57328c60e4a69e741c4953b97e41bf0be3/promql/functions.go#L46>
>>>
>>> > and then dynamically load them into PromQL.
>>>
>>> That approach would ban windows users. I am not sure if that is an
>>> acceptable trade-off.
>>>
>>> >
>>> > On Thursday, June 24, 2021 at 5:14:51 PM UTC-4 bjo...@rabenste.in
>>> wrote:
>>> >
>>> > > On 24.06.21 00:05, 董善东 wrote:
>>> > > > hi,all
>>> > > > In the existing prometheus version, the anomaly detection still
>>> relies
>>> > > > fully on the rules setting. We find that it is inconvenient to set
>>> and
>>> > > hard
>>> > > > to maintain in practical use.
>>> > > > So I propose to add some statistical analysis functions to provide
>>> > > better
>>> > > > and stronger AD ability.
>>> > >
>>> > > Yeah, that's a frequent request. Unfortunately, there are so many
>>> > > statistical analysis functions that we can hardly just add them all.
>>> > >
>>> > > So far, the usual recommendation is to extract data from Prometheus
>>> > > via the HTTP API and feed it to a fully-fledged statistics tool.
>>> > >
>>> > > Obviously, that doesn't help you with alerts (which you probably want
>>> > > to keep within Prometheus).
>>> > >
>>> > > At the previous to last dev-summit (2021-05-27), we discussed the use
>>> > > case.
>>> > >
>>> > > Outcome was the following:
>>> > > * We want to explore supporting analytics use cases within PromQL
>>> behind
>>> > > a feature flag
>>> > > * We are open to wrapping other languages, e.g. R, Fortran,
>>> SciPython,
>>> > > given an accepted design doc
>>> > >
>>> > > See alse notes here:
>>> > >
>>> > >
>>> https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit?ts=6036b8e0=1#heading=h.sa2f6aem9wdt
>>> > >
>>> > > So I guess you could just implement the functions you like and put
>>> > > them into a PR, locked behind a feature flag.
>>> > >
>>> > > Personally, I'm still not sure if that's a sustainable
>>> > > approach. Perhaps integrating some scripting engine to allow
>>> > > user-defined functions might be better. But we'll see…
>>> > >
>>> > > --
>>> > > Björn Rabenstein
>>> > > [PGP-ID] 0x851C3DA17D748D03
>>> > > [email] bjo...@rabenste.in
>>> > >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-developers+unsubscr...@googlegroups.com.
>>> > To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-developers/8fb5d7fc-30f8-4a68-aa27-05fb66b4c2e7n%40googlegroups.com
>>> .
>>>
>>>
>>> --
>>> Julien Pivotto
>>> @roidelapluie
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving 

Re: [prometheus-developers] Prometheus Developers

[Ben Kochie]

This is not a development topic. Please use our support community.

https://prometheus.io/community/

On Thu, Jun 24, 2021 at 1:29 PM Mithun Batabyal 
wrote:

> Hi Team,
>
> im using Prometheus and grafana  as part of monitoring and deployed
> through helm charts.
>
> currently we have default data retention for Prometheus data to 15 days.
> How to increase this limit to 30days?
> Please guide me.
>
> Thanks and Regards,
> Mithun Batabyal
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/1ec4204d-99e3-4d1b-8cfb-d749ba568706n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqJ_fy-6%2B5xf61VGHODEb8gZ4CgHX_b8vTfhxZq-z%2Br5Q%40mail.gmail.com.

Re: [prometheus-developers] Hey

[Ben Kochie]

Check out the "help wanted" label on github.

https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Aprometheus+label%3A%22help+wanted%22

On Fri, Jun 4, 2021 at 8:38 AM Rs coding  wrote:

> I am a beginner in this project anyone can suggest to me beginner-friendly
> issues.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/23c35bc4-bf48-4a0c-80aa-738d0e4b43f8n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrdjoG%3D1%3D_MsdUBdoeiWREhG%2BVxPGkh4hFFW2dxnbya9g%40mail.gmail.com.

Re: [prometheus-developers] [VOTE] Allow environment variable expansion on external label values

[Ben Kochie]

YES

On Thu, Mar 25, 2021 at 11:07 PM Julien Pivotto 
wrote:

> Hereby I am calling a vote to allow the expansion on environment
> variables in the prometheus configuration file.
> Because it can be seen as an override of a previous vote[1], I am calling a
> new vote for this specific part.
>
> The consensus in the dev summit is:
>
> We will allow substitution of ENV variables into label values in the
> external_label configuration block only, behind an experimental feature
> flag.
>
> The vote is open for a week (until April 2nd), or until we have 9 ayes or
> 9 noes.
> Any Prometheus team member is eligible to vote[2].
>
> 1:
> https://groups.google.com/g/prometheus-developers/c/tSCa4ukhtUw/m/J-j0bSEYCQAJ
> 2: https://prometheus.io/governance/
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20210325220654.GA1317022%40oxygen
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmraBQ2zzmyP6cUfKkqvub6r-OSS_oK-ujv7Zfc9uKfNdw%40mail.gmail.com.

Re: [prometheus-developers] Add collector for database/sql#DBStats

[Ben Kochie]

I like the idea of making client_golang have more "APM" like features. I'm
not sure this would be something to enable by default, but having a central
place for these things makes sense to me.



On Tue, Mar 23, 2021 at 11:24 AM Matthias Rampke 
wrote:

> Ah, even before following the link I had the same questions as Björn:
> Could this be a separate thing? Does it benefit significantly from being
> part of the standard client_golang, or could it just as well be something
> people pull in on demand?
>
> I see at least two  libraries
>  that already do this, do we
> need a third? What is different about your approach?
>
> /MR
>
> On Mon, Mar 22, 2021 at 11:06 AM Mitsuo Heijo 
> wrote:
>
>> Hi Prometheus Developers.
>>
>> I was guided here from
>> https://github.com/prometheus/client_golang/pull/848
>>
>> There are several agenda items.
>>
>> 1. Whether client_golang should include DBStatsCollector.
>> 2. Should the metrics be prefixed with go_... ? While the database/sql
>> package is Go-specific, those metrics aren't really coming from the Go
>> runtime.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/ecb8b635-8e88-4cd2-880a-c1f00a76d41an%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAMV%3D_gbvhRNuKHBJC%2BqRMeT-vWZUsfYoUj%2BM_xdPDN%3Dcoob-%3DA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmot-EQziwcepriON4ATxQaY5nHR7vboq%3DYZtX780oi8pw%40mail.gmail.com.

Re: [prometheus-developers] Prometheus builds: Try Earthly?

[Ben Kochie]

Sorry, but this doesn't seem like any real improvement over the tooling we
already have. We're already CI vendor portable. We converted from Travis to
Circle long ago, our build steps just aren't that complicated. Everything
is in a standardized Makefile system. Adding more abstraction wrappers
isn't an improvement over GNU make.

On Fri, Mar 19, 2021 at 2:21 AM Vlad A. Ionescu  wrote:

> Hi Bjoern,
>
> Thanks for the pointers! I don’t think Earthly needs to necessarily
> replace promu. Or at least not entirely, and not all at the same time.
> Conceivably, it’s possible to set up Earthly to work kinda like a
> pass-through for the promu commands.
>
> I’m not familiar with the challenges of the Prometheus build, so I’ll try
> to list a few *possible* benefits. I’m shooting in the dark here -
> emphasis on “possible” :-)
>
>- In general, build scripts not captured by promu run containerized
>when ran through Earthly. This helps with keeping everything reproducible.
>(Earthly can also run things natively
>,
>if that is needed - but it’s not the default). If promu is ran in Earthly,
>then it can optionally run containerized, eliminating any
>environment-specific variance.
>- Changes to the CI scripts can be tested locally, thus helping with
>the iteration speed when changing the build in any way.
>- Integration tests for client libraries can execute in a
>containerized environment. This helps with putting together the right
>language-specific dependencies, and also possibly set up matrix testing, to
>test against multiple versions of the client’s language. Such tests can
>execute isolated from one-another and in parallel.
>- For setups where many languages and repos interact with each other,
>Earthly has very powerful cross-repo import mechanisms if needed to bring
>everything together for integration testing. For example, if another
>project needs promu, it can be as easy as something like COPY
>github.com/prometheus/promu+promu/promu ./. You can also control
>versioning, tags, branches, etc. through build args.
>- Multi-platform support allows testing against other architectures
>(e.g. arm64). In Earthly, this works really well via QEMU under the hood.
>- If migrating from one CI vendor to another, migrating to Earthly
>first helps abstract away the CI. This helps reduce CI vendor lock-in, in
>case Circle decides to do what Travis did.
>- Compute intensive steps (if there are any) can be cached and not
>repeated if the inputs have not changed. In Earthly, this can work with a 
> remote
>cache  too.
>
> If any of this seems interesting, I’d be happy to put together a demo PR
> to POC a particular benefit. Also open to hearing of any specific
> challenges the community has with the build.
>
> Cheers,
> Vlad.
>
> On Thu, Mar 18, 2021 at 8:43 AM Bjoern Rabenstein 
> wrote:
>
>> On 17.03.21 11:43, Vlad A. Ionescu wrote:
>> >
>> > Not sure if this is the right place for this question. Wondering if
>> anyone
>> > is interested in trying https://github.com/earthly/earthly for the
>> > Prometheus build.
>> >
>> > Earthly could help with reproducing CI failures locally (via containers)
>> > and for performing multiple isolated integration tests in parallel.
>> >
>> > It works well on top of Circle CI.
>>
>> The Prometheus project has its own organically grown build system, see
>> https://github.com/prometheus/promu , and rather elaborate CircleCI
>> setup. Not saying that's a perfect solution, but any new solution
>> needs to meet that bar, plus justify the effort and friction of
>> changing by some added relevant value.
>>
>> From that perspective, perhaps the first step should be to clarify
>> what would improve and what would change, and then convince
>> stakeholders that they actually want and need the improvements.
>>
>> --
>> Björn Rabenstein
>> [PGP-ID] 0x851C3DA17D748D03
>> [email] bjo...@rabenste.in
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAO_OUmMq9h4HAWxqOCnoWQjM5Eqb753iEpL%2B%3D%3DATfmx6mdHzBQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

Re: [prometheus-developers] Updating client_golang to use the new runtime/metrics package

[Ben Kochie]

This sounds awesome to me. I'm glad to see some Prometheus love from the Go
team.

If you would please file an issue on the client_golang project, with some
more details on the specific changes you want to make, that would be
perfect.

Looking forward to seeing this added.

On Sat, Mar 6, 2021 at 12:37 AM 'Michael Knyszek' via Prometheus Developers
 wrote:

> Hey Prometheus devs,
>
> My name is Michael and I work on the Go runtime at Google.
>
> I wanted to send out a message to the mailing list regarding my intent to
> add support for reading metrics exported by the runtime/metrics package
>  in the Go client
> . The goal of this new
> package is to allow the runtime to export implementation-specific metrics.
> This is useful both for users to gain greater insight into their
> application's behavior, and for the Go team in understanding performance
> issues, bugs, and places where we can improve. The core API works a lot
> like ReadMemStats, but the set of metrics available is significantly more
> dynamic.
>
> I took a look at the codebase, and AFAICT this switch should be relatively
> straightforward, and pretty much just amounts to providing a second
> goCollector
> 
>  that's
> used for Go 1.16+. I decided to send a message out to the mailing list
> because this involves a little bit of refactoring and a medium amount of
> code, and I didn't want to just show up with a big ol' PR. Does this plan
> of action sound reasonable to y'all?
>
> I figure a second goCollector should be OK, because the new one will
> generally be simpler than the existing one. For instance, a lot of the
> hand-coded data about what each metric means is now available at runtime,
> like metric descriptions. Metric names can also be programmatically
> generated, because the runtime/metrics keys are already namespaced.
> Furthermore, the runtime/metrics package's Read method does not
> stop-the-world, so all the logic that's working around ReadMemStats' high
> latency can pretty much disappear in the new goCollector. I figure it will
> still be worthwhile to re-export some of the metrics under their old names
> too to avoid breaking dashboards, but I don't know how much of a concern
> that is here.
>
> Anyway, please let me know what you think!
>
> Thank you,
> Michael
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/14eb46d5-5bae-4823-afba-c8aeac32c504n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmodP7qz9kZuM2xPPsLEzPpAZxk4hZuypT7O1xfSnQbaMQ%40mail.gmail.com.

Re: [prometheus-developers] Standard path for mixins

[Ben Kochie]

* Make it easier to find in the repo.
* Make testing easier.

https://github.com/prometheus/circleci/pull/15

On Wed, Mar 3, 2021 at 4:34 PM 'Matthias Loibl' via Prometheus Developers <
prometheus-developers@googlegroups.com> wrote:

> Hi,
> agreed to what has been said so far.
>
> Do you simply want more consistency or is it for discoverability too, Ben?
> The latter should be solved with https://monitoring.mixins.dev.
>
>
> fbra...@gmail.com schrieb am Mittwoch, 3. März 2021 um 16:08:32 UTC+1:
>
>> In newer versions of jsonnet-bundler this is not strictly necessary
>> anymore, the module name has been only a “legacy” import option for the
>> last year (I just checked the version allowing it went out almost to the
>> day exactly a year ago). If there are conflicts the absolute (go import
>> style) full module name must be specified, which might be nice way to move
>> the ecosystem forward anyways.
>>
>> I don’t think this should stop us from doing this, I don’t feel strongly
>> one way or the other about the directory structure itself though I agree we
>> could be more consistent.
>>
>> On Wed 3. Mar 2021 at 15:59, Tom Wilkie  wrote:
>>
>>> @Frederic Branczyk should weigh in on this, but I believe the module
>>> name == the directory name in jsonnet bundler, so if we call them all
>>> monitoring-mixin then it will make it tricky to import multiple ones into
>>> the same project.  So with that in mind I like "-mixin".
>>>
>>> But we could be more consistent about root-of-repo vs docs vs
>>> documentation directories...
>>>
>>>
>>> Tom
>>>
>>> On Wed, Mar 3, 2021 at 2:47 PM Ben Kochie  wrote:
>>>
>>>> We currently have mixins spread out over various paths within each
>>>> repository. It would be nice if there was a standard path for this.
>>>>
>>>> What do people think of having all projects keep their mixins in
>>>> `/monitoring-mixin`?
>>>>
>>>> alertmanager/doc/alertmanager-mixin
>>>> mysqld_exporter/mysqld-mixin
>>>> node_exporter/docs/node-mixin
>>>> prometheus/documentation/prometheus-mixin
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Prometheus Developers" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to prometheus-devel...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrhhG-%3DEQcb2zm%3Dkc2%3DuqX7nxHb2%3D6YG8cgTNec-fqdRQ%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrhhG-%3DEQcb2zm%3Dkc2%3DuqX7nxHb2%3D6YG8cgTNec-fqdRQ%40mail.gmail.com?utm_medium=email_source=footer>
>>>> .
>>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/661e30ac-8f39-4123-997f-b6537358f0edn%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-developers/661e30ac-8f39-4123-997f-b6537358f0edn%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpAcW1L7csyY2OxS%2BCrDPCqeR0WZDhQZ8FbgSBBPtsRxw%40mail.gmail.com.

[prometheus-developers] Standard path for mixins

[Ben Kochie]

We currently have mixins spread out over various paths within each
repository. It would be nice if there was a standard path for this.

What do people think of having all projects keep their mixins in
`/monitoring-mixin`?

alertmanager/doc/alertmanager-mixin
mysqld_exporter/mysqld-mixin
node_exporter/docs/node-mixin
prometheus/documentation/prometheus-mixin

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrhhG-%3DEQcb2zm%3Dkc2%3DuqX7nxHb2%3D6YG8cgTNec-fqdRQ%40mail.gmail.com.

Re: [prometheus-developers] Remote-write drop samples | design doc

[Ben Kochie]

If a remote write receiver is unable to ingest, wouldn't this be something
to fix on the receiver side? The receiver could have a policy where it
drops data rather than returning an error.

This way Prometheus sends, but doesn't have to need to know or deal with
ingestion policies. It sends a bit more data over the wire, but that part
is cheap compared to the ingestion costs.

On Mon, Mar 1, 2021 at 11:13 AM Stuart Clark 
wrote:

> On 01/03/2021 07:25, Harkishen Singh wrote:
> > Hi Tom,
> >
> > I have tried to answer the comments. Please comment on their
> > satisfactoriness. I am happy for a call if required (or discussion
> > gets tough).
> >
> > I think, the lossless nature can be controlled by the user based on
> > the config (limit_retries), and let the users have more control, as to
> > whether they are happy to compromise a bit, if the retry is too much,
> > since as such, if the retrying happens forever, then I don't think
> > that is helpful (it will never be accepted by the remote storage).
> > Also as Chris mentioned, some users might prefer to have few gaps and
> > give more priority to recent data, like for alerting. So, I think this
> > approach gives more flexibility to the user, at the same time, making
> > it optional (or by setting the retry count high enough).
> >
> Under what situations would retries happen forever?
>
> If the receiver is available but cannot accept the data (for example due
> to metric size limits or age of the samples) I would expect it to reject
> with a 4XX code (permanent failure) which wouldn't trigger any retries.
>
> Alternatively if the receiver is either unavailable or broken it could
> result in "infinite" retries, but in that situation it feels like an age
> based limit instead of retry limit would be better - a short retry limit
> will reject samples that have just been scraped just as quickly as
> samples that are days old. Instead it sounds like an age based limit
> would be better - some systems have restrictions over what age can be
> ingested (e.g. Timestream) or administrators could decide older data has
> no usefulness (e.g. if the receiver is used for alerting or anomaly
> detection. While the system should still reject such old samples once it
> is working again a time based limit would at least reduce the network
> impact once the receiver is back online (no need to send tons of data
> that we know will be rejected).
>
> --
> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/cd97f615-e479-e4be-e85d-672b15c337d8%40Jahingo.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpOC8EPAnHsj0Zyh5JSworYLciDL6nCXyzSSnHAX981RA%40mail.gmail.com.

Re: [prometheus-developers] Difference in scope between smokeping_prober and blackbox_exporter?

[Ben Kochie]

On Mon, Feb 22, 2021 at 3:48 AM Marcelo Magallón <
marcelo.magal...@grafana.com> wrote:

> Thanks Ben.
>
> On Thu, Feb 18, 2021 at 1:37 PM Ben Kochie  wrote:
>
>> The problem with what you're proposing is you're getting an invalid
>> picture of data over time. This is the problem with the original smokeping
>> program that the smokeping prober is trying to solve.
>>
>> The original smokeping software does exactly what you're talking about.
>> It sends out a burst of 10 packets at the configured interval (in your
>> example, 1 minute). The problem is this does not give you a real picture,
>> because the packets are not evenly spaced.
>>
>> This is why I made the smokeping_prober work the way it does. It sends a
>> regular stream, but captures the data in a smarter way, as a histogram.
>>
>> From the histogram data you can only collect the metrics every minute,
>> and generate the same "min / max / avg / dev / loss" values that you're
>> looking for. But the actual values are much more statistically valid, as
>> it's measuring evenly over time.
>>
>
> That's fair. I do understand the argument for preferring continuous
> observations.
>
> The problem I have with the histogram approach (and this is partly due to
> the current way histograms work in Prometheus) is that I don't know the
> distribution a priori.
>
> I let smokeping_prober run for a few days against several IP addresses.
> For a particular one, after 250+ thousand observations, it's telling me
> that the round trip time is somewhere between 51.2 ms and 102.4 ms. Using
> the sum and the count from histogram data I can derive an average (not
> mean) over a short window and it's giving me ~ 60 ms. I happen to know
> (from the individual observations) that the 95th percentile is also ~ 60
> ms, and that's pretty much the 50th percentile (the spread of the
> observations is very small). The actual min/max/avg from observations is
> something like 59.1 / 59.7 / 59.4 ms. If I use the data from the histogram
> the 50th percentile comes out as ~ 77 ms and the 95th percentile as ~ 100
> ms. I must be missing something, because I don't see how I would extract
> the min / max / dev from the available data. I do understand that the
> standard deviation for this data is unusually small (compared to what you'd
> expect to see in the wild), but still...
>

The default histogram buckets in the smokeping_prober cover latency
durations from localhost to the moon and back. It's relatively easy to
adjust the buckets, and easy enough to get within a reasonable range for
your network expectations.

Without knowing exactly what queries you're running, it's hard to say what
you're doing. If you're using the histogram count/sum, this will give you
the mean value.

There is one known issue with the smokeping_prober right now that I need to
fix, the ping library handling of sequence numbers is broken and doesn't
wrap correctly.


>
> I also have to think of the data size. For 1 ICMP packet every 1 second,
> I'm at (order of magnitude) 100 MB of data per target per month. Reducing
> this to 5 packets every 60 seconds I'm down to 10 MB (order of magnitude).
> This doesn't sound like much for a single target but it does add up.
>

Yes, this is going to be an issue no matter what you do. I don't see how
this relates to any mode of operation.


>
> As a side note, I noticed that smokeping_prober resolves the IP address
> once. With BBE this happens everytime the probe runs, so I don't have to do
> anything if I'm monitoring a host where IP addresses might change every now
> and then.
>

Yes, this is currently intentional, but re-resolving is something I'm
planning to do eventually.


>
> Thanks again,
>
> Marcelo
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmr9Vrz5%2Buoeo6vJtPfPO3Z18dKC82XBsFYRbAYLxGx6Gw%40mail.gmail.com.

Re: [prometheus-developers] Re: auto discover of targets in Prometheus

[Ben Kochie]

Prometheus absolutely does do dynamic discovery.

See the scrape_config section of the docs:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

There are 16 available XXX_sd_configs options there. If your source of
truth isn't listed, you can either write your own adapter using
file_sd_configs as an API. Or if you think your system is popular enough,
contribute it to the codebase.

There is a growing list of adapters here:
https://prometheus.io/docs/operating/integrations/#file-service-discovery

On Fri, Feb 19, 2021 at 11:19 PM Gajendra Ambi  wrote:

> Hi,
> I too just discovered that it does not do dynamic discovery. I have to
> tell prometheus to see it by using file based discovery in prometheus. This
> is very unfortunate where we have 100s of nodes and growing and each group
> of devs who own them get to add them when they want. Currently it seems We
> have to do it manually. Thanks for the quick info.
>
> On Saturday, February 20, 2021 at 3:33:47 AM UTC+5:30 Julien Pivotto wrote:
>
>> On 19 Feb 14:00, Gajendra Ambi wrote:
>> > Everyone gives the same link
>> >
>> https://prometheus.io/docs/prometheus/latest/configuration/configuration
>> when
>> > asked about auto discovery of physical nodes but the entire page has
>> only 2
>> > occurences of the word auto-discovery.
>> > We have 700+ physical servers and Prometheus is running on k8s (targets
>> are
>> > not part of k8s). Can someone please point to a blog or the actual part
>> of
>> > the documentation which has auto discovery of physical nodes, not the
>> same
>> > link please. I could not find anything there. It says, you can do it
>> using
>> > 1 of the auto discovery methods but the link does not have a single
>> method
>> > of auto discovery listed there.
>>
>> Hello,
>>
>> Prometheus does not do *auto* discovery. We do *service* discovery. You
>> will need a source of truth where your register your nodes.
>>
>>
>> >
>> > On Monday, September 3, 2018 at 7:44:49 PM UTC+5:30 HARSH AGARWAL
>> wrote:
>> >
>> > > You can use Prometheus-service-discovery(SD) to automatically
>> discover the
>> > > targets.
>> > >
>> https://prometheus.io/docs/prometheus/latest/configuration/configuration
>> > >
>> > > Using relabel-configs
>> > > <
>> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Crelabel_config%3E>,
>>
>> > > you can specify which targets you want to keep/discard/change-labels,
>> etc.
>> > >
>> > > Prometheus supports SD for many providers. You can also use your
>> > > own custom-service-discovery mechanisms using file_sd
>> > > <
>> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cfile_sd_config%3E>
>>
>> > > .
>> > >
>> > >
>> > > On Monday, September 3, 2018 at 7:36:28 PM UTC+5:30, HJS wrote:
>> > >>
>> > >> For a particular job in Prometheus, it seems like the typical config
>> is
>> > >> something like this:
>> > >>
>> > >> static_configs:
>> > >> - targets: ['localhost:9090']
>> > >>
>> > >> But in the case where I want a dynamic list of hosts((auto discover
>> of
>> > >> targets)), this option is available in prometheus and if available
>> how can
>> > >> config??
>> > >>
>> > >>
>> > >> Thank you in advance!
>> > >>
>> > >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to prometheus-devel...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/cfc35736-9321-438e-8692-c3591ef2e550n%40googlegroups.com.
>>
>>
>>
>> --
>> Julien Pivotto
>> @roidelapluie
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/be0fbf93-357f-403a-8f8f-b70140ead2den%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqHj_SHfATx2DcHrxCB0mu56d0gwrx5cQsG23mcm%3DdSmg%40mail.gmail.com.

Re: [prometheus-developers] Difference in scope between smokeping_prober and blackbox_exporter?

[Ben Kochie]

The problem with what you're proposing is you're getting an invalid picture
of data over time. This is the problem with the original smokeping program
that the smokeping prober is trying to solve.

The original smokeping software does exactly what you're talking about. It
sends out a burst of 10 packets at the configured interval (in your
example, 1 minute). The problem is this does not give you a real picture,
because the packets are not evenly spaced.

This is why I made the smokeping_prober work the way it does. It sends a
regular stream, but captures the data in a smarter way, as a histogram.

>From the histogram data you can only collect the metrics every minute, and
generate the same "min / max / avg / dev / loss" values that you're looking
for. But the actual values are much more statistically valid, as it's
measuring evenly over time.


On Thu, Feb 18, 2021 at 3:17 PM Marcelo Magallón <
marcelo.magal...@grafana.com> wrote:

> On Wed, Feb 17, 2021 at 5:15 PM Stuart Clark 
> wrote:
>
>> > Thoughts?
>>
>> Is there any reason you can't use the Blackbox Exporter as it currently
>> is, just decreasing the scrape interval? Prometheus can scrape as
>> infrequently as every 2 minutes or as frequently as several times a
>> second.
>>
>
> Thanks Stuart,
>
> Decreasing the scope interval with blackbox_exporter would cause it to
> behave more like smokeping_prober, which is not exactly the thing I'm after.
>
> Also, smokeping_prober creates histograms from the observations, so even
> if you set up an ICMP check with blackbox_exporter using a very short
> interval, you wouldn't get the same information from it.
>
> To provide a concrete example, in terms of timestamps, with
> smokeping_prober you send packets at 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
> ... (1 packet every second, with the scrape interval be say 60 seconds).
>
> With current blackbox_exporter you could do something similar by setting
> the scrape interval to 1 second, but you'd get slightly different
> information.
>
> With what I'm proposing you'd send packets at say 0, 1, 2, 3, 4, 60, 61,
> 62, 63, 64, 120, 121, 122, 123, 123, ... (5 packets within 5 seconds, then
> wait for 55 seconds, send another 5 packets in 5 seconds, etc; the scrape
> interval would be 60 seconds)
>
> --
> Marcelo Magallón
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqeHRJDX_2VC-AYib85uJm378OCd4EX66TYoe7%3D_nng0w%40mail.gmail.com.

Re: [prometheus-developers] RFC: better histograms for Prometheus

[Ben Kochie]

http://gph.is/1TFACgU

On Wed, Feb 10, 2021 at 7:21 PM Bjoern Rabenstein 
wrote:

> Hi Prometheans,
>
> tl;dr: I have just published a draft of a design document about
> better histograms for Prometheus and would appreciate your feedback:
>
> https://docs.google.com/document/d/1cLNv3aufPZb3fNfaJgdaRBZsInZKKIHo9E6HinJVbpM/edit?usp=sharing
>
>
> As many of you might know, I have been (in-)famously working on
> histograms for Prometheus from the very beginning of the Promethean
> era. As Goutham has recently found out, I even mentioned histograms as
> my favorite Prometheus topic during the very first conference talk
> about Prometheus ever! (To be precise: It was SRECon Europe on
> 2015-05-14, during the Q, when none less than Brendan Burns asked
> about the topic.)
>
> What we currently have in Prometheus was only ever a prototype, at
> least from my perspective. (o:
>
> In an ideal world, I would have sat down back in 2015 and created the
> document linked above. Too many distractions by other interesting or
> even urgent things, I guess. To get the whole narrative, you could
> check out my recent "histogram trilogy of talks" (which will also give
> you the gist of the design document):
>
> https://fosdem.org/2020/schedule/event/histograms/
>
> https://promcon.io/2019-munich/talks/prometheus-histograms-past-present-and-future/
> https://www.youtube.com/watch?v=HG7uzON-IDM
>
> The bad news is that even after all those years, most of the work
> still has to be done. Every layer of the Prometheus stack has to
> change, which needs a coordinated effort. That's precisely the reason
> why I created the design document, which you could also call an
> RFC. After collecting your feedback, I hope to be able to evolve it
> into something we can agree on as the way forward, serving as a master
> plan to align the many detailed efforts that will have to follow.
>
> I hope you will enjoy the read, at least somewhat...
> --
> Björn Rabenstein
> [PGP-ID] 0x851C3DA17D748D03
> [email] bjo...@rabenste.in
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20210210182124.GP2725%40jahnn
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqWbf9YPrV0hyrs3eUn0BRFDMV%2BgHfpD%2B53JjG82jGiPQ%40mail.gmail.com.

Re: [prometheus-developers] SNMP monitoring | PUSH Method feasibility

[Ben Kochie]

I think the point is that commvault only implements traps, and has no
metrics that can be polled via the snmp_exporter.

In this case, Prometheus is not really a possible monitoring solution, as
it's not an event store.

On Thu, Jan 28, 2021 at 11:43 PM Stuart Clark 
wrote:

> On 28/01/2021 13:02, s.saurab...@gmail.com wrote:
>
>
> Hi Everyone ...
>
> We have commvault solution in our environment which we have to integrate
> with Prometheus for Hardware monitoring. However it is found that commvault
> don't support PULL mechanism on which prometheus works. Is there any other
> alternate through which we can integrate prometheus with commvault. Pls
> suggest
>
> The email subject mentions SNMP. That is fully supported using the SNMP
> Exporter: https://github.com/prometheus/snmp_exporter
>
> --
> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/cca55c97-d7b6-a994-cbbf-1495a5693e0e%40Jahingo.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrx%2BDV%3DPhwKKuU3%2Bhh7bHvPKU0c3A3UVCEOyTNCyTfC8Q%40mail.gmail.com.

Re: [prometheus-developers] Docker images from scratch

[Ben Kochie]

That's proposal #4.

On Sun, Jan 31, 2021 at 5:31 PM Gabriel Cavalcante <
gabriel.cavalcant...@gmail.com> wrote:

> Is it possible to use the scratch image with Prometheus binary inside
> only? That would reduce the surface entirely.
>
> On Sun, 31 Jan 2021 at 13:26 Julien Pivotto 
> wrote:
>
>> Hello,
>>
>> From time to time we get users reporting that the docker image we use to
>> build Prometheus contain a Busybox vulnerability:
>>
>> https://github.com/prometheus/node_exporter/issues/1937
>> https://github.com/prometheus/prometheus/issues/8277
>> https://github.com/prometheus/prometheus/issues/7794
>>
>> We have a few options here:
>>
>> 1. ignoring those reports as there is no evidence that this can be used
>> without first getting shell access into the container.
>>
>> 2. removing wget from the container
>>
>> 3. switching to a base image that does not contain the fix, e.g. alpine
>>
>> 4. only shipping our binaries and a few other files (from scratch or
>> from distroless-static
>>
>> https://github.com/GoogleContainerTools/distroless/blob/master/base/README.md
>> )
>>
>> My thinking:
>>
>> 1. This is (was) the current strategy. And clearly, scanners do not care
>> that Prometheus uses or does not use the said binaries.
>> However, in security, less attack surface is always positive.
>>
>> 2. Even if we remove /bin/wget, it can still be invoked by calling
>> /bin/busybox wget
>>
>> 3. Alpine etc would increase the surface, require rebuild a lot more
>> often than busybox.
>>
>> 4. Distroless static seems to be what we have now (takes certs etc from
>> debian), without busybox. The advantage here would be that we can simply
>> stop using prometheus/busybox, and we would have updated upstreams
>> images all the time.
>>
>> So I'd go and investigate distroless base image in the future.
>>
>> --
>> Julien Pivotto
>> @roidelapluie
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/20210131162630.GA13747%40oxygen
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAHaDZeRGK49OQUE8NmYCQs4pfdWbOf3DPEYOfJkYUALSzU71qQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqn5Fr16umtizJzyTqZkRi3u5HyEkhP53scky%2BNsVLDkQ%40mail.gmail.com.

Re: [prometheus-developers] Docker images from scratch

[Ben Kochie]

Another option is we could fully build our own busybox binary, with the
necessary fixes.

I'm somewhat in favor of going distroless. With a large number of users
using our container images in Kubernetes, it's less necessary to include
busybox, as they can attach userspace sidecar containers.

On Sun, Jan 31, 2021 at 5:26 PM Julien Pivotto 
wrote:

> Hello,
>
> From time to time we get users reporting that the docker image we use to
> build Prometheus contain a Busybox vulnerability:
>
> https://github.com/prometheus/node_exporter/issues/1937
> https://github.com/prometheus/prometheus/issues/8277
> https://github.com/prometheus/prometheus/issues/7794
>
> We have a few options here:
>
> 1. ignoring those reports as there is no evidence that this can be used
> without first getting shell access into the container.
>
> 2. removing wget from the container
>
> 3. switching to a base image that does not contain the fix, e.g. alpine
>
> 4. only shipping our binaries and a few other files (from scratch or
> from distroless-static
>
> https://github.com/GoogleContainerTools/distroless/blob/master/base/README.md
> )
>
> My thinking:
>
> 1. This is (was) the current strategy. And clearly, scanners do not care
> that Prometheus uses or does not use the said binaries.
> However, in security, less attack surface is always positive.
>
> 2. Even if we remove /bin/wget, it can still be invoked by calling
> /bin/busybox wget
>
> 3. Alpine etc would increase the surface, require rebuild a lot more
> often than busybox.
>
> 4. Distroless static seems to be what we have now (takes certs etc from
> debian), without busybox. The advantage here would be that we can simply
> stop using prometheus/busybox, and we would have updated upstreams
> images all the time.
>
> So I'd go and investigate distroless base image in the future.
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20210131162630.GA13747%40oxygen
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmo397CuKSLpGjcU0fivFkRprbw04g3ys_F_yKJhP-hA_A%40mail.gmail.com.

Re: [prometheus-developers] Lazy consensus: Merging options

[Ben Kochie]

I agree, I tend to avoid squashing unless the PR author generates a really
messy set of "fix typo" commits, rather than maintaining a clean PR branch.

On Thu, Dec 3, 2020 at 4:55 PM Sylvain Rabot  wrote:

> I also don’t like squashing.
>
> I don’t count the hours lost because I thought a commit referenced in a
> merged PR was not in a tag because the squash generated a new commit id.
>
> On 3 Dec 2020, at 15:27, Frederic Branczyk  wrote:
>
> 
> I don’t like squash merging, don’t think I’ve ever used rebase merging but
> don’t feel too strongly about it. Merge commit is my preference.
>
> On Thu 3. Dec 2020 at 15:06, Julien Pivotto 
> wrote:
>
>> On 03 Dec 14:59, Bartłomiej Płotka wrote:
>> > I am ok with this proposal.
>> >
>> > Long term I would even vote for squash only, but we discussed this in
>> the
>> > past.
>>
>> How would you merge release branches in master?
>>
>> >
>> > Kind Regards,
>> > Bartek Płotka (@bwplotka)
>> >
>> >
>> > On Thu, 3 Dec 2020 at 14:20, Brian Brazil <
>> brian.bra...@robustperception.io>
>> > wrote:
>> >
>> > > On Thu, 3 Dec 2020 at 13:15, Ben Kochie  wrote:
>> > >
>> > >> I'd like to adjust our defaults for GitHub merging settings:
>> > >>
>> > >> Right now, we allow all three modes for PR merges.
>> > >> * Merge commits
>> > >> * Squash merging
>> > >> * Rebase merging
>> > >>
>> > >> Proposal: Remove rebase merging (aka fast-forward merges) so that we
>> > >> stick to merge/squash and merge.
>> > >>
>> > >
>> > > I use rebase merges sometimes to keep the history clean from
>> > > unnecessary merge commits, so I'd like it to hang around.
>> > >
>> > > Brian
>> > >
>> > >
>> > >>
>> > >> [image: image.png]
>> > >>
>> > >> --
>> > >> You received this message because you are subscribed to the Google
>> Groups
>> > >> "Prometheus Developers" group.
>> > >> To unsubscribe from this group and stop receiving emails from it,
>> send an
>> > >> email to prometheus-developers+unsubscr...@googlegroups.com.
>> > >> To view this discussion on the web visit
>> > >>
>> https://groups.google.com/d/msgid/prometheus-developers/CABbyFmp0X26pjfvyATvaUxH9p_nwBh0QSMgtJGNzfDLnZJjdMQ%40mail.gmail.com
>> > >> <
>> https://groups.google.com/d/msgid/prometheus-developers/CABbyFmp0X26pjfvyATvaUxH9p_nwBh0QSMgtJGNzfDLnZJjdMQ%40mail.gmail.com?utm_medium=email_source=footer
>> >
>> > >> .
>> > >>
>> > >
>> > >
>> > > --
>> > > Brian Brazil
>> > > www.robustperception.io
>> > >
>> > > --
>> > > You received this message because you are subscribed to the Google
>> Groups
>> > > "Prometheus Developers" group.
>> > > To unsubscribe from this group and stop receiving emails from it,
>> send an
>> > > email to prometheus-developers+unsubscr...@googlegroups.com.
>> > > To view this discussion on the web visit
>> > >
>> https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLpwuPY6iE0k7zRP8PFAGTrEx9hYzx6j%3DQT8p4hLQVF6-w%40mail.gmail.com
>> > > <
>> https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLpwuPY6iE0k7zRP8PFAGTrEx9hYzx6j%3DQT8p4hLQVF6-w%40mail.gmail.com?utm_medium=email_source=footer
>> >
>> > > .
>> > >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to prometheus-developers+unsubscr...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/CAMssQwahWTP3uPQuEDcu8jQB_EBDe5AOKXrJYd6%2Bad-wqOpEFQ%40mail.gmail.com
>> .
>>
>>
>>
>> --
>> Julien Pivotto
>> @roidelapluie
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the

[prometheus-developers] Lazy consensus: Merging options

[Ben Kochie]

I'd like to adjust our defaults for GitHub merging settings:

Right now, we allow all three modes for PR merges.
* Merge commits
* Squash merging
* Rebase merging

Proposal: Remove rebase merging (aka fast-forward merges) so that we stick
to merge/squash and merge.

[image: image.png]

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmp0X26pjfvyATvaUxH9p_nwBh0QSMgtJGNzfDLnZJjdMQ%40mail.gmail.com.

Re: [prometheus-developers] Stop building for 32 bit

[Ben Kochie]

I think that PR link is incorrect.

Yes, I think we should consider dropping 32-bit official builds. Unless we
can reduce the amount of mmap'd data.

One thing I noticed when running on a 32-bit machine a while back was that
the mmap'd data is mapped multiple times for different OS threads.
Setting GOMAXPROCS=1
helped, but only a little.

There was some discussion about doing mmap on demand, but there was not
enough interest in 32-bit to consider spending the time.

On Fri, Aug 28, 2020 at 3:05 PM Julien Pivotto 
wrote:

> Dear developers,
>
> Based on this comment from Brian, should we stop providing official 32 bit
> builds
> of Prometheus?
>
> > 2.19 requires more virtual memory than previous versions, you need a
> > 64bit machine. 32bit system are generally not supported as we use mmap
> > a lot, if you want to try to get it to work you can but we don't
> > promise anything there as trying to support 32bit systems would
> > cripple things for the vast majority of our users on 64bit systems. I
> > can only suggest looking at Prometheus 1.x, which did not use mmap.
>
> https://github.com/prometheus/prometheus/pull/7686#issuecomment-682534007
>
> If we build them but not recommend them, that seems like a waste of time
> and resources.
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20200828130516.GA1428921%40oxygen
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrkuKc-VHAxydAHEO3PwA95V_tBu0WuuUQ-7c654mS%2Bzg%40mail.gmail.com.

Re: [prometheus-developers] Docker Hub's upcoming 100 pulls per user/IP limit

[Ben Kochie]

Yes, we should apply and get more details on the new FOSS plans.

Anyone volunteer to take that on?

On Wed, Aug 26, 2020 at 10:59 AM Julius Volz  wrote:

> On Wed, Aug 26, 2020 at 10:16 AM Johannes Ziemke 
> wrote:
>
>> Hi everyone,
>>
>> as mentioned in chat yesterday, Docker will limit the number of layers a
>> user can pull within 6 hours:
>>
>>
>> https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress
>>
>> This will certainly cause problems for people pulling our images from the
>> Docker Hub. Especially in Kubernetes scenarios where ImagePullPolicy=Always
>> is common practice and often even enforced by running the AlwaysPullImages
>> admission controller. This will likely lead to outages in dynamic
>> environments (autoscaling, spot instances etc). I'd expect especially the
>> node-exporter to be affected since it's probably the thing people run the
>> most instances of in their infrastructure.
>>
>> There is not much we can do. We could beg Docker to void the limits for us
>>
>
> I mean as the article states "Finally, as part of Docker’s commitment to
> the open source community, before November 1 we will be announcing
> availability of new open source plans. To apply for an open source plan,
> please complete the short form here.".
>
> I wonder when exactly "before November 1" will be and how much time that
> will give us to decide things. In case their OSS plans allow unlimited
> pulls again, then we should be fine?
>
> But also fine switching to quay.io everywhere. But the general problem is
> services that are expensive to run, but offered for free... wondering if
> something similar will happen to quay.io at some point. But now that it
> belongs to Red Hat, err, IBM, they might be big enough to not care about
> the costs it produces.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6Yoy6qBfYQdEyidbs3MsAbvnKJZ6bDDr54Ha_rFLi%3DYP4ZQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpMog6LHpqBoAkObYh4kaoeKz2kbZaKrE4q5t0KnQGaag%40mail.gmail.com.

Re: [prometheus-developers] node_exporter on Solaris/Illumos

[Ben Kochie]

I would start with an issue on the repo.

On Fri, Jul 31, 2020 at 2:51 PM Danny Smith  wrote:

> I've recently had to build node_exporter for an Oracle Solaris system, and
> noticed that the 'Solaris' support is actually currently
> Illumos/OpenIndiana support. Both systems have different sets of ZFS kstat
> information available, and hence the zfs collector doesn't work on Oracle
> Solaris as-is.
>
> I've modified the collector to properly support both flavours, and enabled
> the 'uname' collector (bsd variety) for Solaris/Illumos. I've also added a
> filesystem collector for Solaris/Illumos based on the Linux filesystem
> collector, but using /etc/mnttab instead. I realise this strictly breaks
> the rules, namely 'A Collector may only read /proc or /sys files', however
> under Solaris the kernel mount table is mounted via the 'mnttab' FS as
> /etc/mnttab.
>
> I also had to tweak Makefile.common to enable a build on both systems, as
> by default the build expects to be able to download a 'promu' binary, and
> solaris/illumos are not part of the standard releases.
>
> Could someone take a look please and let me know if this is considered to
> be in a suitable state for a PR?
>
> https://github.com/dsnt02518/node_exporter
>
> Danny
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/de5f4571-2c49-4d89-a2ea-38c110aa8ec9n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpRuDriRh3uLib6bt-1hA_igFCA_4ppKyfS%3DOZ3VNevPg%40mail.gmail.com.

Re: [prometheus-developers] Introduce the concept of scrape Priority for Targets

[Ben Kochie]

I'm with Brian and Julian on this.

Multi-tenancy is not really something we want to solve in Prometheus. This
is a concern for higher level systems like Kubernetes. Prometheus is
designed to be distributed. If you have targets with different needs, they
need to have separate Prometheus instances.

This is also why we have things like Thanos and Cortex as aggregation
layers.

Similar to why we have said we don't plan to implement IO limits, this is a
scheduling concern, out of scope for Prometheus.

On Thu, Jul 30, 2020, 10:31 Frederic Branczyk  wrote:

> That's only effective in limiting the number of targets, the point here is
> that selectively scraping those with a higher priority based on
> backpressure of the system as a whole.
>
> On Wed, 22 Jul 2020 at 17:00, Julien Pivotto 
> wrote:
>
>> On 22 Jul 16:47, Frederic Branczyk wrote:
>> > In practice even that can still be problematic. You only know that
>> > Prometheus has a problem when everything fails, the point is to keep
>> things
>> > alive well enough for more critical components.
>> >
>> > On Wed, 22 Jul 2020 at 16:38, Julien Pivotto <
>> roidelapl...@prometheus.io>
>> > wrote:
>> >
>> > > On 22 Jul 16:36, Frederic Branczyk wrote:
>> > > > It's unclear how that helps, can you help me understand?
>> > >
>> > > - job: highprio
>> > >   relabel_configs:
>> > >   - target_label: job
>> > > replacement: pods
>> > >   - source_labels: [__meta_pod_priority]
>> > > regex: high
>> > > action: keep
>>
>> highprio job will always be scraped.
>>
>> > > - job: lowprio
>> > >   relabel_configs:
>> > >   - target_label: job
>> > > replacement: pods
>> > >   - source_labels: [__meta_pod_priority]
>> > > regex: high
>> > > action: drop
>> > >   target_limit: 1000
>> > >
>> > > >
>> > > > On Wed, 22 Jul 2020 at 16:34, Julien Pivotto <
>> roidelapl...@prometheus.io
>> > > >
>> > > > wrote:
>> > > >
>> > > > > On 22 Jul 16:32, Frederic Branczyk wrote:
>> > > > > > Can you explain what you mean by two jobs? Do you mean two
>> scrape
>> > > > > configs?
>> > > > >
>> > > > > Yes.
>> > > > >
>> > > > > >
>> > > > > > On Wed, 22 Jul 2020 at 11:40, Julien Pivotto <
>> > > roidelapl...@prometheus.io
>> > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > On 22 Jul 02:35, Lili Cosic wrote:
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Wednesday, 22 July 2020 11:23:00 UTC+2, Brian Brazil
>> wrote:
>> > > > > > > > >
>> > > > > > > > > On Wed, 22 Jul 2020 at 10:18, Julien Pivotto <
>> > > > > roidel...@prometheus.io
>> > > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> On 22 Jul 02:14, Lili Cosic wrote:
>> > > > > > > > >> > Only now seen in the docs that I am supposed to start
>> any
>> > > > > > > discussions
>> > > > > > > > >> here
>> > > > > > > > >> > first before opening an issue, sorry about that! :)
>> > > > > > > > >> >
>> > > > > > > > >> > Currently there is no way of a target to have higher
>> scrape
>> > > > > > > priority
>> > > > > > > > >> over
>> > > > > > > > >> > another, but if you have a setup and even if you set
>> target
>> > > > > limits
>> > > > > > > and
>> > > > > > > > >> > sample limits you can still overestimate your setup,
>> you
>> > > still
>> > > > > want
>> > > > > > > to
>> > > > > > > > >> have
>> > > > > > > > >> > a higher priority targets that are preferred over the
>> entire
>> > > > > > > Prometheus
>> > > > > > > > >> to
>> > > > > > > > >> > fail. It would need to be based on the inability to
>> ingest
>> > > into
>> > > > > > > tsdb on
>> > > > > > > > >> the
>> > > > > > > > >> > current rate we are scrapping, if that is hit the
>> priority
>> > > class
>> > > > > > > would
>> > > > > > > > >> take
>> > > > > > > > >> > affect and only the highest priority targets would be
>> > > scrapped
>> > > > > in
>> > > > > > > > >> favour of
>> > > > > > > > >> > lower priority. Another option which might be simpler
>> would
>> > > be
>> > > > > to
>> > > > > > > have
>> > > > > > > > >> a
>> > > > > > > > >> > global limit on how much prometheus can handle based
>> on perf
>> > > > > > > testing.
>> > > > > > > > >> >
>> > > > > > > > >> > This would be treated as a last resort, and there would
>> > > > > definitely
>> > > > > > > be a
>> > > > > > > > >> > need for a high severity alert to inform the admin that
>> > > > > something
>> > > > > > > went
>> > > > > > > > >> > terribly wrong, but because we would still be able to
>> ingest
>> > > > > > > Prometheus
>> > > > > > > > >> > metrics for example if they are higher priority class
>> > > alerting
>> > > > > > > would be
>> > > > > > > > >> > possible.
>> > > > > > > > >>
>> > > > > > > > >> Hi,
>> > > > > > > > >>
>> > > > > > > > >> I think that limiting the number of targets you scrape is
>> > > already
>> > > > > a
>> > > > > > > last
>> > > > > > > > >> resort. I don't think we would need a second line of
>> defense.
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > > > > I agree with Julien here. If you've gotten to this 

[prometheus-developers] Linux Memory experts needed

[Ben Kochie]

Hey all.

In order to help make the node_exporter more useful, sometimes we like to
reduce the number of metrics it outputs by default. This helps by reducing
the load on Prometheus, especially for users with large numbers of nodes.

To help with this, I'm proposing[0] we add an include/exclude filter for
the `node_memory_...` metrics. I'm looking for feedback from the community
for help deciding what would be useful defaults.

[0]: https://github.com/prometheus/node_exporter/issues/1781

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmp1ePL39%3DWT2Oe9Uzfj0%3DFi9k_iVauFDWgrmxbg8AYD2w%40mail.gmail.com.

Re: [prometheus-developers] Changing `master` to `main` across the org?

[Ben Kochie]

Nice.

On Thu, Jul 2, 2020 at 12:21 PM Julien Pivotto 
wrote:

> MySQL had decided
>
> https://mysqlhighavailability.com/mysql-terminology-updates/
>
>
> Le mer. 1 juil. 2020 à 09:54, Matthias Rampke  a
> écrit :
>
>> +1 for changing the default branch, +1 to seeing what GitHub does to help
>> us, but if there is no movement on that we can also work out a plan
>> ourselves.
>>
>> For the MySQL case, *if* we are willing to break compatibility and
>> consistency, we could change the metric names even without waiting for the
>> underlying commands to change. A risk is that we choose one terminology,
>> and upstream ends up choosing another in a year. A chance is that we set a
>> precedent and upstream eventually follows :)
>>
>> /MR
>>
>> On Wed, Jun 24, 2020 at 4:38 PM Frederic Branczyk 
>> wrote:
>>
>>> Tobias’ idea sounds great, and I’m +1 with this!
>>>
>>> On Wed 24. Jun 2020 at 17:17, Tobias Schmidt  wrote:
>>>
 +1

 As Github seems to be working on it already, I'd wait to see what they
 can provide to simplify the transition. Would it make sense to tweet from
 our Twitter account to let them know we're interested in such 
 functionality?

 I looked at other problematic terminologies across our code bases, and
 it'll be hard to do much about it until third-parties have changed it on
 their side, e.g.
 https://github.com/search?q=org%3Aprometheus+slave=Code

 On Wed, Jun 24, 2020 at 4:15 PM Bartłomiej Płotka 
 wrote:

> +1 on this from Prometheus side, but also cc Thanos Team, I think we
> should do that everywhere.
>
> Kind Regards
> Bartek
>
> On Wed, 24 Jun 2020 at 16:06, Richard Hartmann <
> richih.mailingl...@gmail.com> wrote:
>
>> Dear all,
>>
>> I talked about this with a few of you already and the general feeling
>> of the room was "this is worthwhile, but it carries an opportunity
>> cost". So: What do you think? Should this be a goal?
>>
>> CNCF is on board and assigned tech writer resources to switching over,
>> and I suggested making CI/CD migrations etc part of Community Bridge
>> outreach as it's a great intern project.
>>
>> It definitely makes sense to wait for GitHub to decide on a name and
>> to provide tooling to minimize toil.
>>
>> Thoughts?
>>
>>
>> Richard
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgTerwefdtSU5PB0vPmYA%2BW-20VoDy2GG7AH6F%2BG2RoL3Q%40mail.gmail.com
>> .
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAMssQwZPF36eS-Sz0c%3DTArmdy7scuZ7FGv2KQjaF2CzwnkLF1g%40mail.gmail.com
> 
> .
>
 --
 You received this message because you are subscribed to the Google
 Groups "Thanos Team" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to thanos-io+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/thanos-io/CAChBsdC01oRuGXRmTk-gJVwB1MmbwF5nGfj4OD732Oswfbp1dA%40mail.gmail.com
 
 .

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-developers+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-developers/CAOs1Umwj%3DtUA8RzN2z2bUGd7uoopf9yidh01K1qLMUyngvB%3DKw%40mail.gmail.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> 

Re: [prometheus-developers] Integrate Push Notifications with AlertManager

[Ben Kochie]

Similar, GitLab supports webhooks directly into Issues, and a new incident
management system.

On Tue, Jun 30, 2020 at 5:06 PM Julien Pivotto 
wrote:

> On 28 Jun 07:17, Josh Wolff wrote:
> > Oh, great, thank you for pointing me to these links!
> >
> > On Sunday, June 28, 2020 at 12:42:09 AM UTC-7, Stuart Clark wrote:
> > >
> > > On 28/06/2020 00:08, Josh Wolff wrote:
> > > > Hello,
> > > >
> > > > I want to integrate push notifications with Prometheus's
> AlertManager
> > > >
> > > > I am the founder of Spontit (https://api.spontit.com), and we
> enable
> > > > people to send limitless, free push notifications. I was reading
> about
> > > > AlertManager and I think an integration of our API would definitely
> > > > provide a great option for developers.
> > > >
> > > > Does anyone have any thoughts on this, how I should go about this,
> or
> > > > any ideas in general?
> > > >
> > > Take a look at the webhook API details:
> > >
> https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
> > >
> > > If you create a receiver which then forwards alerts to your service it
> > > can be added here:
> > >
> > >
> https://prometheus.io/docs/operating/integrations/#alertmanager-webhook-receiver
>
> However as you are the owner of the service you might be interested in
> baking Webhook receiver support directly in the service, instead of
> forcing the users to use an intermediate webhook forwarder. See how
> Squadcast is doing it: https://support.squadcast.com/docs/prometheus
>
> > >
> > >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Prometheus Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-developers+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/e3f72c05-3287-4dbb-a5f9-5a5e09f40c2co%40googlegroups.com
> .
>
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20200630150624.GA597177%40oxygen
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoQVJ79Sk7oTp%2B%3DKL-aZQ-Wd47qLUfYRj%2BAh5MGewKaMA%40mail.gmail.com.

[prometheus-developers] Google Season of Docs

[Ben Kochie]

It was mentioned[0] that we could apply for Google Season of Docs[1]. The
idea would be to add additional documentation about exposed metrics for
various exporters. Especially common things like the blackbox_exporter, and
node_exporter.

I'm interested in this, is anyone else?

[0]:
https://github.com/prometheus/blackbox_exporter/issues/166#issuecomment-650726803
[1]: https://developers.google.com/season-of-docs/docs/get-started

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmr5ZY5vKdn6WHtFOrvTUNXOq1pYO3bEcPnyqEao_bWZAw%40mail.gmail.com.

Re: [prometheus-developers] PromQL formatting/prettifying support in promtool

[Ben Kochie]

One subject that I don't see covered in the doc is spacing. There are
several places where whitespace is optional, and it would be good to have a
consistent opinion on it.

* Between label selectors {foo="foo",bar="bar"} vs {foo="foo", bar="bar"}
* Between params: histogram_quantile(0.9, rate(...))
* Between aggregation operator modifiers: sum without(label)(metric_name)
vs sum without(label) (metric_name) vs sum without (label) (metric_name)

My personal prefernce is
* No whitespace between label selectors.
* A single whitespace between params.
* Whitespace on both sides of operator modifiers. IE sum without (label)
(metric_name)

On Sat, Jun 20, 2020 at 1:22 PM Tobias Schmidt  wrote:

> Thanks a lot for your great work! Expression formatting will likely
> require dozens of detailed rules in order to get things consistent, and
> style discussions are the perfect case for bikeshedding. I really
> appreciate your efforts and can't wait for a `promtool fmt` on-save editor
> integration. The proverb of Go has arguably held true: Gofmt's style is
> no one's favorite, yet gofmt is everyone's favorite
> .
>
> I hope we can get it right without having to make (large) changes in later
> releases. The most annoying thing with auto-formatters is changing rules
> with every release creating constant diff noise (looking at you rubocop).
>
> 
>
> On Fri, Jun 19, 2020 at 3:19 PM Harkishen Singh <
> harkishensingh...@gmail.com> wrote:
>
>> Hello everyone!
>>
>> As part of the GSoC 2020, I am working on designing a Promql expression
>> formatting/prettifying tool whose support will be as an extension in the
>> current promtool. A design document related to the same has been made and
>> it would be great for some comments/views/suggestions, etc.
>>
>> Here is the link to the document: PromQL prettier
>> 
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/0e1b1867-b818-4afe-a970-1bbc21046844o%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAChBsdDHBjphxKUc_%3D7bcKuPoorGPxiy5duYqzvXM%2B3jNoNC%3Dw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmocCQV0z9DiQ%2BDODmp0tC-ZNBGWgcf8GzsqaXJdGzc0Zg%40mail.gmail.com.

Re: [prometheus-developers] Change language on blackbox vs whitebox to closedbox vs openbox

[Ben Kochie]

+1 for "prober". We could call it simply "prober". It more directly
describes what it's doing.



On Thu, Jun 11, 2020 at 12:35 PM Julius Volz  wrote:

> Btw. in case we do end up going for a rename, I think Probe Exporter or
> Prober Exporter would be a nice and descriptive name for what it's doing
> (active probing of things).
>
> On Thu, Jun 11, 2020, 12:03 Julius Volz  wrote:
>
>> Hi,
>>
>> Good motivation, personally I think that kind of replacement is a good
>> thing to do with blacklist/whitelist (as is happening with flag names in
>> the Node Exporter). But I don't know how people feel about blackbox /
>> whitebox yet, as it doesn't carry the same negative/positive association
>> with colors. It would mean more invasive changes, like renaming the entire
>> repository and related documentation web pages, and potential Google
>> findability confusion (old blog posts etc. mentioning the old name).
>>
>> I haven't heard blackbox/whitebox crop up in the same way as e.g.
>> blacklist/whitelist yet, but would be interested to affected Black people's
>> opinion on whether this seems troublesome as well. If this turns out to be
>> offensive to affected people, that'd be a good argument for renaming it IMO.
>>
>> Julius
>>
>> On Thu, Jun 11, 2020 at 11:29 AM Frederic Branczyk 
>> wrote:
>>
>>> Hi all,
>>>
>>> I would like to propose to change all occurrences and namings within the
>>> Prometheus project of whitebox and blackbox monitoring. In itself the term 
>>> Black
>>> box  doesn't seem to come from
>>> a racist background, but I think it's problematic when the opposite is
>>> "white", in particular as this has a connotation in relation to
>>> whitelist/blacklist namings which are undoubtedly problematic. I would like
>>> to propose to replace them with open/closed box monitoring, which not only
>>> removes any potential of being offensive, it actually conveys much more
>>> clearly what is meant without having to explain.
>>>
>>> The biggest impact this would have would be renaming of the
>>> blackbox_exporter  to
>>> closedbox_exporter, all other occurrences of this language seem to be
>>> limited to documentation.
>>>
>>> Best regards,
>>> Frederic
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-developers+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-developers/98b3d620-d0d6-4436-bf15-3d90915f7909o%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6Yoyawu8gP7XU95jFyO24Dk_bZ8VnKy-7wUwHE%2BVDnRTcNQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrju%2BxYo6pvjEOhNh-gFXa9%2BB5rKyHwXsSjgSy-KCSFSA%40mail.gmail.com.

Re: [prometheus-developers] Change language on blackbox vs whitebox to closedbox vs openbox

[Ben Kochie]

I'm planning to do the flag name change to the node_exporter soon.

I also am not sure about blackbox/whitebox.

How about this proposal: replace "whitebox" with "internal". This better
describes what we really mean, removes the black vs white connotations, and
simplifies the change.

On Thu, Jun 11, 2020 at 12:03 PM Julius Volz  wrote:

> Hi,
>
> Good motivation, personally I think that kind of replacement is a good
> thing to do with blacklist/whitelist (as is happening with flag names in
> the Node Exporter). But I don't know how people feel about blackbox /
> whitebox yet, as it doesn't carry the same negative/positive association
> with colors. It would mean more invasive changes, like renaming the entire
> repository and related documentation web pages, and potential Google
> findability confusion (old blog posts etc. mentioning the old name).
>
> I haven't heard blackbox/whitebox crop up in the same way as e.g.
> blacklist/whitelist yet, but would be interested to affected Black people's
> opinion on whether this seems troublesome as well. If this turns out to be
> offensive to affected people, that'd be a good argument for renaming it IMO.
>
> Julius
>
> On Thu, Jun 11, 2020 at 11:29 AM Frederic Branczyk 
> wrote:
>
>> Hi all,
>>
>> I would like to propose to change all occurrences and namings within the
>> Prometheus project of whitebox and blackbox monitoring. In itself the term 
>> Black
>> box  doesn't seem to come from
>> a racist background, but I think it's problematic when the opposite is
>> "white", in particular as this has a connotation in relation to
>> whitelist/blacklist namings which are undoubtedly problematic. I would like
>> to propose to replace them with open/closed box monitoring, which not only
>> removes any potential of being offensive, it actually conveys much more
>> clearly what is meant without having to explain.
>>
>> The biggest impact this would have would be renaming of the
>> blackbox_exporter  to
>> closedbox_exporter, all other occurrences of this language seem to be
>> limited to documentation.
>>
>> Best regards,
>> Frederic
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/98b3d620-d0d6-4436-bf15-3d90915f7909o%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6Yoz5EvGmJx-LdkEgra09SupXFAvhuUAyO8jjxFy6yyRqLA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqr5O-W_F7uNvdejTJf%2B8fH9P_FLLtVRE91GXsXuwunLQ%40mail.gmail.com.

[prometheus-developers] Enable Dependabot on prometheus repos

[Ben Kochie]

I'd like to enable Dependabot on prometheus/prometheus, specifically to
automatically handle the React UI version bumps. This will deal with the
small version bumps for all the random security vulnerabilities that pop up.

The bot will create PRs like this one:

https://github.com/prometheus-community/monaco-promql/pull/12

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrdAwW8Qp0%2BemgwQQKWNpPVUA1RrRVuOXAndbM3VpPHLQ%40mail.gmail.com.

Re: [prometheus-developers] [VOTE] Allow Kelvin as temperature unit in some cases

[Ben Kochie]

YES

On Thu, May 28, 2020 at 8:52 PM Bjoern Rabenstein 
wrote:

> Dear Prometheans,
>
> So far, we have recommended Celsius as the base unit for temperatures,
> despite Kelvin being the SI unit. That was well justified by the
> overwhelming majority of use cases, where Kelvin would be just
> weird. I'd really like to see more scientific usage of Prometheus, so
> I was never super happy with that recommendation, but since it was
> just a recommendation, I could live with it.
>
> Now Matt Layher came up with another, more technical use case: color
> temperature. Here, using Celsius would be even weirder. So there is a
> case where you clearly do not want to follow the suggestion of the
> linter, which is more in line with typical Prometheus use cases than
> my arguably somewhat far fetched time series for low-temperature
> experiments.
>
> Therefore, Matt suggested to make the metrics linter not complain
> about kelvin.
>
> I think this is a clearly defined problem with clear arguments and a
> clear disagreement between Brian Brazil on the one side and Matt and
> myself on the other side. The appropriate amount of effort has been
> spent to find a consensus. All arguments can be found in
> https://github.com/prometheus/client_golang/pull/761 and
> https://github.com/prometheus/docs/pull/1648 .
>
> I hereby call a vote for the following proposal:
>
> Allow Kelvin as a base unit in certain cases and update our
> documented recommendation and the linter code accordingly.
>
>
> (The changes may take the form of the two PRs out there, but the vote
> in about the general idea above, not the implementation detail.)
>
>
> The vote closes on 2020-06-04 20:00 UTC.
> --
> Björn Rabenstein
> [PGP-ID] 0x851C3DA17D748D03
> [email] bjo...@rabenste.in
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20200528185216.GK2326%40jahnn
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrynK5YGgw10%2BdwoA1_2kQO0RegD3AY27oNRZwJy9oqYg%40mail.gmail.com.

Re: [prometheus-developers] [VOTE] Allow listing non-SNMP exporters for devices that can already be monitored via the SNMP Exporter

[Ben Kochie]

YES

On Thu, May 28, 2020 at 9:30 PM Julius Volz  wrote:

> Dear Prometheans,
>
> https://github.com/prometheus/docs/pull/1640 proposes to list an exporter
> for Fortigate devices that uses a REST API. Brian's stance so far has been
> to only add exporters for metrics that cannot be already produced by
> another exporter on the list at
> https://prometheus.io/docs/instrumenting/exporters/, with the intention
> of focusing community efforts rather than spreading them over multiple
> competing exporters for the same thing.
>
> Since the Fortigate device can also be monitored via SNMP, Brian's
> argument is that the SNMP Exporter (
> https://github.com/prometheus/snmp_exporter) already covers this use case
> and a more specialized REST-API-based exporter for Fortigate should not be
> listed. On the other hand, many people feel that SNMP is painful to work
> with, and monitoring via a REST API is much preferrable. (Further, and not
> directly relevant to this vote, the Fortigate REST API also reports some
> information that is not available via SNMP (see
> https://github.com/prometheus/docs/pull/1640#issuecomment-633303249)).
>
> In general I like the guideline of not adding too many competing exporters
> for the same thing to our public integrations list, but I believe that it
> should only be a guideline. I believe that SNMP is enough of an operational
> pain that we should allow exceptions for more specific, non-SNMP-based
> exporters like the linked one. It seems that the discussion has been had
> and is laid out in the issue.
>
> I therefore call a vote for the following proposal:
>
> Allow adding exporters to
> https://prometheus.io/docs/instrumenting/exporters/ although the devices
> or applications that they export data for can already be monitored via SNMP
> (and thus via the SNMP Exporter). This proposal does not affect other
> criteria that we may use in deciding whether to list an exporter or not.
>
> The vote closes on 2020-06-04 20:30 UTC.
>
> Cheers,
> Julius
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YozJK6v%2B9HtYC_n1F7fMH_XPVJqd%2BtU_shAihc70jp%3Dn2g%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmr2e%3DuUhsjPCP6VBLnDUXWMNBv5V68Kou5PAXpKcE32QQ%40mail.gmail.com.

Re: [prometheus-developers] move https/ package to client_golang

[Ben Kochie]

I was thinking about building an "exporter kit" repo that would include
some helpful functions to reduce the amount of boilerplate needed to write
exporters.

Perhaps we could star this project, have the https code moved there?

github.com/prometheus/exporter/https

On Wed, May 27, 2020 at 12:00 AM Julien Pivotto 
wrote:

> On 26 May 21:15, Bjoern Rabenstein wrote:
> > On 24.05.20 21:17, Julien Pivotto wrote:
> > >
> > > I think that for the benefit of the whole community, it might be better
> > > to move it to our "public code" repository,
> github.com/prometheus/client_golang.
> >
> > I don't think that github.com/prometheus/client_golang is our "public
> > code" repository. It's the Go instrumentation client, which happens to
> > share its repository with the experimental and not properly maintained
> > HTTP API client. The latter is essentially an accident, a result of a
> > habit of certain core contributors back then to "just do" things
> > without discussing them in depth with their peers. Go Modules weren't
> > a thing back then, otherwise I would have resisted more strongly.
> >
> > With Go Modules in the game, we should either avoid putting multiple
> > more or less independent libraries into the same repo, or we need to
> > embrace multi-module repos after all. I haven't developed an educated
> > opinion about the latter yet. I just noticed a number of fairly strong
> > statements that multi-module repos are supposed to be very painful.
> >
> > Having said all that, github.com/prometheus/common is of course just
> > as problematic as a location. It contains many independent libraries
> > in the same repo. It would essentially be impossible to have
> > meaningful post-1.0 semver in that repo without turning it into a
> > multi-module repo. Plus, the common repo is (by now) explicitly marked
> > as an internal one.
> >
> > The most straight forward solution right now is to create a new repo
> > "github.com/prometheus/tls" or similar. (And I would then advocate to
> > move the HTTP API client out of client_golang into its own repo,
> > should we ever see that it is steering towards a v1.0 release.)
>
> I would be in favor of creating a new repository. As I said, it is
> important to be able to release that quickly and independently, and also
> trying to follow go semver "for real" in this repo.
>
> > If people feel strongly about the proliferation of repos in the
> > Prometheus GH org, I'm willing to educate myself about multi-module
> > repositories so that I can say if I will support or resist a move to a
> > multi-module client_golang repo (if that's possible in a non-breaking
> > fashion at all - if not, we should probably plan to make the common
> > repo multi-module).
>
> This is code that is going to be used and share across many
> repositories, I think if we are willing to put it in a separate repo we
> should do it.
>
> > --
> > Björn Rabenstein
> > [PGP-ID] 0x851C3DA17D748D03
> > [email] bjo...@rabenste.in
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Prometheus Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-developers+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20200526191532.GQ2326%40jahnn
> .
>
> --
> Julien Pivotto
> @roidelapluie
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20200526192156.GA1130273%40oxygen
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoh49%2BrvMpFaM8fwTEpQS%2BFyHHtuNhS_C%2B%3DKOwe7SkDUw%40mail.gmail.com.

Re: [prometheus-developers] Changing Prometheus from lazy to rough consensus?

[Ben Kochie]

+1 for IETF Rough consensus

On Mon, May 25, 2020 at 2:04 PM Richard Hartmann <
richih.mailingl...@gmail.com> wrote:

> Dear all,
>
> as most of you know, the Prometheus governance[1] has three levels of
> formalized decision making. In increasing order of strength and
> requirements, they are:
> * Consensus
> * Majority vote
> * Supermajority vote
>
>
> The consensus definition is taken from CouchDB[2] and goes deeply into
> its intention to be lightweight and easy to reverse. Yet, a careful
> and literal reading of it enshrines what is factually a veto
> mechanism. Vetos are easy to utter but hard to unblock short of voting
> or an in-between secondary consensus mechanism we have come to call
> "Call for Consensus" on the mailing lists.
>
> In practice, it sometimes feels as if consensus is heavier a process
> than outright voting, an order which goes against the initial
> intention behind our governance.
>
> Contrast this to the IETF definition[3] which is more
> formal, but explicitly acknowledges that "Rough consensus is when all
> issues are addressed, but not necessarily accommodated". IETF has
> decades of experience with highly contentious topics which are,
> sometimes, discussed by directly opposed parties. It's important to
> note that the IETF rough consensus mechanism presumes the existence of
> Chairs, but I think we would not need to rely on such heavyweight
> process.
>
> Personally, I am more used to the IETF definition and have not seen
> the CouchDB version seen wide adoption, which is part of the reason
> why I adopted rough IETF consensus for the Grafana governance[4]
> recently released.
>
> In order to make Prometheus move more quickly and more smoothly, I
> would like to suggest adoption the following change:
>
> https://github.com/prometheus/docs/commit/f7ee6ac5a9b841be6be25d92e2c403de15949491
>
>
> Best,
> Richard
>
> [1] https://prometheus.io/governance/
> [2] https://couchdb.apache.org/bylaws.html#lazy
> [3] https://tools.ietf.org/html/rfc7282
> [4] https://github.com/grafana/grafana/blob/master/GOVERNANCE.md
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgRvwnU%2BdFRMQ2n%2Bqf6D0oyej-btzsQYh8LjBV9FJ1a78w%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrFYZ5sE-j0DngyZjKE9vkwGfbKQS%2BK4q-vsyh-wVFmuA%40mail.gmail.com.

Re: [prometheus-developers] Looking for a maintainer for the collectd_exporter

[Ben Kochie]

Adding Prometheus users as well.



On Fri, May 8, 2020, 16:10 Julius Volz  wrote:

> Hi everyone,
>
> By some historic accident I became the maintainer of the collectd exporter
> (https://github.com/prometheus/collectd_exporter), although I have never
> used collectd myself and know very little about it. This has led to some
> neglect from my side...
>
> Does anyone feel invested in collectd and feels qualified to maintain it?
>
> Cheers,
> Julius
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YozbJRnOYNi_oaBhmSmCeSSaxtJ-fi4qds9kc5NLMkSsuA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmryfqXJ9_hLObK75eH6krXOLjBP7FSpWpYTQzt-64Y96A%40mail.gmail.com.

Re: [prometheus-developers] Chef Server SD

[Ben Kochie]

For Chef users, people typically implement Chef search based templates and
file_sd_configs.

IMO, there's no real point into adding Chef support directly into
Prometheus. Chef searches are too slow/expensive to add directly to
something that might send multiple queries every 30-60 seconds.

On Sat, May 2, 2020 at 8:00 PM José Antonio Abraham Palomo <
abraham...@gmail.com> wrote:

> Hi, I hope all people here be fine, in my current job we are working with 
> *chef
> server*, and I want to develop that Service Discovery for prometheus, but
> before I want to ask the community if there is a PR or someone working on
> it.
>
> Thanks regards.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/ebe13d9f-fc40-4870-bf2d-8514354eb884%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoxpmwjhLnaFy3%2B69UE46kpXZr5oT4uApu91xo8tA2%2BWg%40mail.gmail.com.

Re: [prometheus-developers] Call for Consensus: node_exporter 1.0.0 release

[Ben Kochie]

Yes, let's release right away.

On Thu, Apr 23, 2020 at 1:41 PM Richard Hartmann <
richih.mailingl...@gmail.com> wrote:

> Dear all,
>
> This is a call for consensus within Prometheus-team on releasing
> node_exporter 1.0.0 as-is.
>
> node_exporter 1.0.0-rc.0 has been cut on 2020-02-20[1]. It features
> experimental TLS support[2]. We are planning to use this TLS support
> as a template for all other exporters within and outside of Prometheus
> proper. To make sure we didn’t build a footgun nor that we’re holding
> it wrong, CNCF is sponsoring an external security review by Cure53. We
> have not been giving a clear timeline but work should start in week 22
> (May 25th) at the latest with no time to completion stated.
>
> There are two positions:
> * Wait for the security review to finish before cutting 1.0.0
> * Release ASAP, given that this feature is clearly marked as
> experimental and it will not see wider testing until we cut 1.0.0
>
> I am asking Prometheus-team to establish rough consensus with a hum.
>
> Should the maintainers (Ben & Fish) be allowed to release without
> waiting for the audit to finish?
>
>
> Best,
> Richard
>
> [1] https://github.com/prometheus/node_exporter/releases/tag/v1.0.0-rc.0
> [2] https://github.com/prometheus/node_exporter/pull/1277
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgRRHN%2BXfAxXeVitomS9Gz1Nx78ZGw1p%3D6xhtfQmqPJcXg%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoA0uyzRaLRQ0pNgEbmxiTMkL1Og8YVuJZn2DRfcuUm0g%40mail.gmail.com.

Re: [prometheus-developers] split prometheus api and go tests

[Ben Kochie]

https://about.gitlab.com/solutions/github/

On Thu, Apr 9, 2020 at 10:05 AM Bartłomiej Płotka 
wrote:

> Nice! I guess you cannot use GitLab CI while having repo on GitHub? 樂
>
> Bartek
>
> On Thu, 9 Apr 2020 at 08:50, Ben Kochie  wrote:
>
>> Something, something, gitlabci[0]. ;-)
>>
>> react-test:
>>   only:
>> changes:
>> - web/ui/react-app/**
>>
>> [0]: https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges
>>
>> On Thu, Apr 9, 2020 at 9:15 AM Bartłomiej Płotka 
>> wrote:
>>
>>> Yes, please!
>>>
>>> Also, we can think later on crafting some different matchers for which
>>> files should trigger which job! Thanks for proposing this (:
>>>
>>> Something simple like this
>>> https://discuss.circleci.com/t/how-to-run-a-job-or-a-step-only-when-a-specified-file-changed/29751
>>>  but
>>> some step that actually
>>> check what file has changed or something.
>>>
>>> Bartek
>>>
>>> On Thu, 9 Apr 2020 at 08:01, Ben Kochie  wrote:
>>>
>>>> Seems like a good idea to me. It can also run the tests in parallel.
>>>>
>>>> On Wed, Apr 8, 2020 at 11:23 PM Julien Pivotto 
>>>> wrote:
>>>>
>>>>>
>>>>> Hi there,
>>>>>
>>>>> What would you think of splitting React and Golang tests in 2 different
>>>>> circleci jobs?
>>>>>
>>>>> React is not predictable as we don't really pin dependencies, and a
>>>>> failure there implies that go tests don't run.
>>>>>
>>>>> --
>>>>>  (o-Julien Pivotto
>>>>>  //\Open-Source Consultant
>>>>>  V_/_   Inuits - https://www.inuits.eu
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Prometheus Developers" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to prometheus-developers+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/prometheus-developers/20200408212352.GA5655%40oxygen
>>>>> .
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Prometheus Developers" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to prometheus-developers+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/prometheus-developers/CABbyFmonc4n2R02t7N%2BO7tBuD6jOqdWUELEv6%3DShg7ZUHsCdXg%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/prometheus-developers/CABbyFmonc4n2R02t7N%2BO7tBuD6jOqdWUELEv6%3DShg7ZUHsCdXg%40mail.gmail.com?utm_medium=email_source=footer>
>>>> .
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmqZ-cumOb-aUZwPWu-n6HdzYA%2BwkzhRUhaDc5GXvx-8iw%40mail.gmail.com.

Re: [prometheus-developers] split prometheus api and go tests

[Ben Kochie]

Something, something, gitlabci[0]. ;-)

react-test:
  only:
changes:
- web/ui/react-app/**

[0]: https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges

On Thu, Apr 9, 2020 at 9:15 AM Bartłomiej Płotka  wrote:

> Yes, please!
>
> Also, we can think later on crafting some different matchers for which
> files should trigger which job! Thanks for proposing this (:
>
> Something simple like this
> https://discuss.circleci.com/t/how-to-run-a-job-or-a-step-only-when-a-specified-file-changed/29751
>  but
> some step that actually
> check what file has changed or something.
>
> Bartek
>
> On Thu, 9 Apr 2020 at 08:01, Ben Kochie  wrote:
>
>> Seems like a good idea to me. It can also run the tests in parallel.
>>
>> On Wed, Apr 8, 2020 at 11:23 PM Julien Pivotto 
>> wrote:
>>
>>>
>>> Hi there,
>>>
>>> What would you think of splitting React and Golang tests in 2 different
>>> circleci jobs?
>>>
>>> React is not predictable as we don't really pin dependencies, and a
>>> failure there implies that go tests don't run.
>>>
>>> --
>>>  (o-Julien Pivotto
>>>  //\Open-Source Consultant
>>>  V_/_   Inuits - https://www.inuits.eu
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-developers+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-developers/20200408212352.GA5655%40oxygen
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/CABbyFmonc4n2R02t7N%2BO7tBuD6jOqdWUELEv6%3DShg7ZUHsCdXg%40mail.gmail.com
>> <https://groups.google.com/d/msgid/prometheus-developers/CABbyFmonc4n2R02t7N%2BO7tBuD6jOqdWUELEv6%3DShg7ZUHsCdXg%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmpWo1zYTk5-GG0Q%3D9svQW1CsrxV%2BRuCeLPdEFW2r-wsgw%40mail.gmail.com.

Re: [prometheus-developers] split prometheus api and go tests

[Ben Kochie]

Seems like a good idea to me. It can also run the tests in parallel.

On Wed, Apr 8, 2020 at 11:23 PM Julien Pivotto 
wrote:

>
> Hi there,
>
> What would you think of splitting React and Golang tests in 2 different
> circleci jobs?
>
> React is not predictable as we don't really pin dependencies, and a
> failure there implies that go tests don't run.
>
> --
>  (o-Julien Pivotto
>  //\Open-Source Consultant
>  V_/_   Inuits - https://www.inuits.eu
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20200408212352.GA5655%40oxygen
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmonc4n2R02t7N%2BO7tBuD6jOqdWUELEv6%3DShg7ZUHsCdXg%40mail.gmail.com.

Re: [prometheus-developers] New function join_label_sorted()

[Ben Kochie]

Are you sure you're not looking for inhibit rules?

https://prometheus.io/docs/alerting/configuration/#inhibit_rule

This allows one alert to inhibit another alert from firing.

On Fri, Mar 13, 2020 at 12:35 AM D  wrote:

> Hi All,
>
> I am interested in adding (or having) a new function join_label_sorted()
> which will be similar to join_label(). It is described as:
>
> For each timeseries in v, label_join_sorted(v instant-vector, dst_label,
> string, separator string, src_label1 string, src_label2 string...) joins
> all the values of all the src_labels using separator and returns the
> timeseries with the label dst_label contained the joined value such that
> joined value is concatenation of original values in sorted order. There can
> be any number of src_labels in this function.
>
> The following example will return a vector with each time series having a
> foo label with the value 7_9_12 added to it.
>
> label_join_sorted(up{job="api-server",src1="12",src2="7",src3="9"}, "foo",
> "_", "src1", "src2", "src3")
>
> It will help me write alarming rules where I can do group by based on
> the composite label-value. I am not sure if we can achieve it today using
> UNLESS clause or so. Let me elaborate with an example to understand why
> this is a requirement. Let's consider that are two services or devices
> which are connected as follows:
>
>  A--B
>
> Suppose the logical/physical connection between A or B goes down, metrics
> collected from/for each end-point can potentially trigger an alarm. Metric
> from A will generate an alarm with labels - local node is A and remote node
> is B. Metric from B will generate an alarm with swapped values. Since the
> underlying issue could be same, we don't want to generate both the alarms
> because JIRA fails to de-dup such issues more often than not.
>
>
> Thanks,
> Dhiman
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAB2OGjzJY8hHBUXWRsTqQ4EuDt4kFRSiordgfdS3BD1-S15-Tg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrDwLtXd0GSfNVOvL0%3D05P1-wwZA4FQZJCjM4BOd4Uppw%40mail.gmail.com.

Re: [prometheus-developers] Proposal: Deprecate prometheus/nagios-plugins in lieu of a 3rd party fork

[Ben Kochie]

I would prefer to start with an offer to give someone else access to
maintain the code, rather than shutdown an existing codebase.

If this fork is better, maybe we can get them to merge everything and they
take over?

Another option, we move it to prometheus-community.

On Wed, Feb 26, 2020 at 5:00 PM Bjoern Rabenstein 
wrote:

> Hi Prometheans,
>
> If you are not interested at all in integrating Nagios with
> Prometheus, you can stop reading now.
>
> Otherwise, you are probably aware of the little plugin I maintain in
> the https://github.com/prometheus/nagios_plugins repository. (The
> plural is a lie, I guess when the repo was created by Joshua Hoffman
> back in 2013, we had bolder plans with the integration than just one
> measly shell script.)
>
> Having an own repo in the prometheus GH org kind of suggests that
> the Nagios integration is important and 1st class. Which is simply not
> true. If we didn't have that repo today, we would certainly encourage
> such an integrations to be maintained in a 3rd party repository, like
> the hundreds of other integrations. The reason why the repo exists in
> the prometheus GH org is purely historical (i.e. SoundCloud needed the
> integration back then in 2013, and with essentially no other users
> than SoundCloud itself, it was just natural to put it into the
> prometheus GH org).
>
> On top of that, I am not using Nagios anymore. One of my
> accomplishments at SoundCloud was to remove Nagios/Icinga from the
> tech stack. And Grafana Labs, my current employer, doesn't use
> Nagios/Icinga either.
>
> For the time being, I have been happy to keep maintaining the
> prometheus/nagios_plugins repo, including reviewing the occasional bug
> fix or minor improvement contributed to the project. However,
> inevitably more involved feature requests or even contributions will
> happen, and I don't feel qualified to review those. One of them can be
> seen here: https://github.com/prometheus/nagios_plugins/issues/25
> (The discussion on that issue is what lead to this mail of mine.)
>
> I can see two options to go from here:
>
> 1. We find somebody that is qualified to maintain
> prometheus/nagios_plugins in place.
>
> 2. We deprecate prometheus/nagios_plugins and refer to a 3rd party
> integration (possibly based on forking prometheus/nagios_plugins) that
> is properly maintained and developed. In this case, I'd keep the
> existing repo around until further notice, but not accepting any
> contributions besides important bug fixes.
>
> Emil 'Skeen' Madsen, @Skeen on GH, filed above mentioned issue #25,
> and he is already playing around with a feature-enriched fork. I would
> now like to encourage Nagios/Icinga users to try out his fork, to be
> found under https://github.com/magenta-aps/check_prometheus_metric and
> give Emil feedback. If that turns out well, I suggest to go for
> option (2), deprecate prometheus/nagios_plugins in lieu of
> agenta-aps/check_prometheus_metric and list the latter as an
> integration on https://prometheus.io
>
> Please let me know what you think.
> --
> Björn Rabenstein
> [PGP-ID] 0x851C3DA17D748D03
> [email] bjo...@rabenste.in
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/20200226160040.GJ27526%40jahnn
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrdYyk3X27EV9NpV5woZ30hODuzDX5ZzknA%2B3ngQZMNAQ%40mail.gmail.com.

Re: [prometheus-developers] [VOTE] New governance document

[Ben Kochie]

Yes

On Wed, Feb 19, 2020 at 9:44 PM Richard Hartmann <
richih.mailingl...@gmail.com> wrote:

> Dear all,
>
> I am hereby calling for a vote on merging
> https://github.com/prometheus/docs/pull/1552 at commit
> de2266c36d8a2ea1f139f97632808e12b354bb76.
>
> References and discussion can be found in said PR and in the email
> thread "Pre-vote feedback on proposed governance changes" on
> prometheus-team (not visible to the public).
>
> This vote will run until 2020-02-26 23:59 UTC or when supermajority
> has been reached, whichever comes first.
>
>
> Please note that while we are voting in public, only Prometheus team
> members are eligible to vote.
>
>
> Best,
> Richard
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgTHNk%3DORUiU%3DKucKtgKDUtSEfGv%2BQVLFKB-d0sVc%2ByBNQ%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoNWZaWGMFg1mdAnkCbux7L2Ege3FnAAOLOXBS86kCp4Q%40mail.gmail.com.

Re: [prometheus-developers] prometheus/prometheus Changelog Management

[Ben Kochie]

But we don't have to block on changelog entries like we do for DCO legal
requirements.

For example, in GitLab codebases, we have a CI test that warns if a
changelog entry is missing. Unless you tag the review with a label.

On Fri, Feb 14, 2020 at 2:22 PM Brian Brazil <
brian.bra...@robustperception.io> wrote:

> On Fri, 14 Feb 2020 at 13:15, Matthias Rampke  wrote:
>
>> The friction is real – DCO is not a submission quality issue but a
>> roundtrip one. This would be even more difficult with wording.
>>
>
> I'm with Matthias on this one. We've often have PRs which are ready to
> merge, but have been sitting there months waiting for a DCO.
>
> Brian
>
>
>>
>>  I agree that in *many* cases contributors can write the changelog
>> entry; having the field in the PR template would encourage them to do so
>> proactively.
>>
>> /MR
>>
>> On Fri, Feb 14, 2020 at 1:12 PM Ben Kochie  wrote:
>>
>>> DCO is a strawman argument. We're always going to have issues with
>>> submission quality.
>>>
>>> I've had very good luck asking for changelog entries on the
>>> node_exporter.
>>>
>>> On Fri, Feb 14, 2020 at 8:22 AM Brian Brazil <
>>> brian.bra...@robustperception.io> wrote:
>>>
>>>> On Fri, 14 Feb 2020 at 07:10, Frederic Branczyk 
>>>> wrote:
>>>>
>>>>> I recall Simon having a tool that would largely generate the changelog
>>>>> automatically, that worked pretty well last time I was release shepherd.
>>>>> Otherwise I'm also happy to discuss a process like in Kubernetes where the
>>>>> changelog item is written into the PR. On Thanos we have in the PR 
>>>>> template
>>>>> that people have ensured that the changelog item was added respective to
>>>>> the change. Seems like there are options,
>>>>>
>>>>
>>>>
>>>>
>>>>> I personally would favor something that would be done at contribution
>>>>> time, so not all the responsibility falls on the release shepherd as it
>>>>> does today, and more generally it seems like the person contributing the
>>>>> change probably is also a good candidate to describe it in the changelog.
>>>>>
>>>>
>>>> This is additional friction to contributions, we already have enough
>>>> fun getting the DCO signed. It's also an additional burden on every single
>>>> PR, we need to individually figure out if it's worth mentioned in the
>>>> changelog (many PRs aren't) and then get it in the right category, with
>>>> good wording, and handling the regular conflicts as everyone would be
>>>> touching the same lines in the same file.
>>>>
>>>> Even with all that the release shepard would still need to go through
>>>> all the commits and double check that nothing was missed, plus fixing poor
>>>> wording. I don't think saving 2-3 minutes off a release is worth all these
>>>> downsides.
>>>>
>>>> Brian
>>>>
>>>>
>>>>>
>>>>> On Fri, 14 Feb 2020 at 08:05, Callum Styan 
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'd like to start a discussion around changing how we manage the
>>>>>> prometheus/prometheus changelog, specifically the fact that the changelog
>>>>>> is generated manually by the release shepherd as part of the release
>>>>>> process.
>>>>>>
>>>>>> We can discuss options for what the new process would look like, such
>>>>>> as requiring PR's to include changelog entries before merging or the next
>>>>>> release shepherd periodically updating the changelog prior to the 
>>>>>> release,
>>>>>> in more detail later. However I'd first like to get a sense of whether
>>>>>> anyone else feels strongly about either changing or not changing this 
>>>>>> part
>>>>>> of the release process.
>>>>>>
>>>>>> Thanks,
>>>>>> Callum.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Prometheus Developers" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to prometheu

Re: [prometheus-developers] prometheus/prometheus Changelog Management

[Ben Kochie]

The way I've been doing it in the node_exporter is to ask for the changelog
entry, but I don't block on it. If the code is ok, but they don't add a
changelog entry, I'll merge and do it myself.

Usually what saves the round trip time is I include a copy-paste text of
exactly what I want in the changelog, rather than leave it up to them to
have to figure it out.

For example:
https://github.com/prometheus/node_exporter/pull/1580#issuecomment-569954208

On Fri, Feb 14, 2020 at 2:15 PM Matthias Rampke  wrote:

> The friction is real – DCO is not a submission quality issue but a
> roundtrip one. This would be even more difficult with wording.
>
>  I agree that in *many* cases contributors can write the changelog entry;
> having the field in the PR template would encourage them to do so
> proactively.
>
> /MR
>
> On Fri, Feb 14, 2020 at 1:12 PM Ben Kochie  wrote:
>
>> DCO is a strawman argument. We're always going to have issues with
>> submission quality.
>>
>> I've had very good luck asking for changelog entries on the node_exporter.
>>
>> On Fri, Feb 14, 2020 at 8:22 AM Brian Brazil <
>> brian.bra...@robustperception.io> wrote:
>>
>>> On Fri, 14 Feb 2020 at 07:10, Frederic Branczyk 
>>> wrote:
>>>
>>>> I recall Simon having a tool that would largely generate the changelog
>>>> automatically, that worked pretty well last time I was release shepherd.
>>>> Otherwise I'm also happy to discuss a process like in Kubernetes where the
>>>> changelog item is written into the PR. On Thanos we have in the PR template
>>>> that people have ensured that the changelog item was added respective to
>>>> the change. Seems like there are options,
>>>>
>>>
>>>
>>>
>>>> I personally would favor something that would be done at contribution
>>>> time, so not all the responsibility falls on the release shepherd as it
>>>> does today, and more generally it seems like the person contributing the
>>>> change probably is also a good candidate to describe it in the changelog.
>>>>
>>>
>>> This is additional friction to contributions, we already have enough fun
>>> getting the DCO signed. It's also an additional burden on every single PR,
>>> we need to individually figure out if it's worth mentioned in the changelog
>>> (many PRs aren't) and then get it in the right category, with good wording,
>>> and handling the regular conflicts as everyone would be touching the same
>>> lines in the same file.
>>>
>>> Even with all that the release shepard would still need to go through
>>> all the commits and double check that nothing was missed, plus fixing poor
>>> wording. I don't think saving 2-3 minutes off a release is worth all these
>>> downsides.
>>>
>>> Brian
>>>
>>>
>>>>
>>>> On Fri, 14 Feb 2020 at 08:05, Callum Styan 
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'd like to start a discussion around changing how we manage the
>>>>> prometheus/prometheus changelog, specifically the fact that the changelog
>>>>> is generated manually by the release shepherd as part of the release
>>>>> process.
>>>>>
>>>>> We can discuss options for what the new process would look like, such
>>>>> as requiring PR's to include changelog entries before merging or the next
>>>>> release shepherd periodically updating the changelog prior to the release,
>>>>> in more detail later. However I'd first like to get a sense of whether
>>>>> anyone else feels strongly about either changing or not changing this part
>>>>> of the release process.
>>>>>
>>>>> Thanks,
>>>>> Callum.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Prometheus Developers" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to prometheus-developers+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/prometheus-developers/CAN2d5OTjOrCfpRF_NXGcQB5nOz%3DVPgnz3LdEk15ucV4PFz%2B4BQ%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/prometheus-developers/CAN2d5OTjOrCfpRF_NXGcQB5nOz%3DVPgnz3LdEk15ucV4PFz%2B4BQ%40mail.gmail.com?utm_medium=email_source=footer>
>>>>&g

Re: [prometheus-developers] prometheus/prometheus Changelog Management

[Ben Kochie]

DCO is a strawman argument. We're always going to have issues with
submission quality.

I've had very good luck asking for changelog entries on the node_exporter.

On Fri, Feb 14, 2020 at 8:22 AM Brian Brazil <
brian.bra...@robustperception.io> wrote:

> On Fri, 14 Feb 2020 at 07:10, Frederic Branczyk 
> wrote:
>
>> I recall Simon having a tool that would largely generate the changelog
>> automatically, that worked pretty well last time I was release shepherd.
>> Otherwise I'm also happy to discuss a process like in Kubernetes where the
>> changelog item is written into the PR. On Thanos we have in the PR template
>> that people have ensured that the changelog item was added respective to
>> the change. Seems like there are options,
>>
>
>
>
>> I personally would favor something that would be done at contribution
>> time, so not all the responsibility falls on the release shepherd as it
>> does today, and more generally it seems like the person contributing the
>> change probably is also a good candidate to describe it in the changelog.
>>
>
> This is additional friction to contributions, we already have enough fun
> getting the DCO signed. It's also an additional burden on every single PR,
> we need to individually figure out if it's worth mentioned in the changelog
> (many PRs aren't) and then get it in the right category, with good wording,
> and handling the regular conflicts as everyone would be touching the same
> lines in the same file.
>
> Even with all that the release shepard would still need to go through all
> the commits and double check that nothing was missed, plus fixing poor
> wording. I don't think saving 2-3 minutes off a release is worth all these
> downsides.
>
> Brian
>
>
>>
>> On Fri, 14 Feb 2020 at 08:05, Callum Styan  wrote:
>>
>>> Hi all,
>>>
>>> I'd like to start a discussion around changing how we manage the
>>> prometheus/prometheus changelog, specifically the fact that the changelog
>>> is generated manually by the release shepherd as part of the release
>>> process.
>>>
>>> We can discuss options for what the new process would look like, such as
>>> requiring PR's to include changelog entries before merging or the next
>>> release shepherd periodically updating the changelog prior to the release,
>>> in more detail later. However I'd first like to get a sense of whether
>>> anyone else feels strongly about either changing or not changing this part
>>> of the release process.
>>>
>>> Thanks,
>>> Callum.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-developers+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-developers/CAN2d5OTjOrCfpRF_NXGcQB5nOz%3DVPgnz3LdEk15ucV4PFz%2B4BQ%40mail.gmail.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/CAOs1UmyOfHbC75bdk55frFQt-KYgD6cg7vh%2BCPSmVmMnSV3sng%40mail.gmail.com
>> 
>> .
>>
>
>
> --
> Brian Brazil
> www.robustperception.io
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLrFL_kN28EiagWYFbKMr5XWC%2Bk7h8n9D8VijvmOnX_5Tw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoOTL3BvNguLk%3D%3DpNJi52zP458DugtpOTz0HAdQanyXgQ%40mail.gmail.com.

Re: [prometheus-developers] Reduce list of GOOS/GOARCH crossbuilt for each PR

[Ben Kochie]

Part of the problem is that we run the builds in serial. We have a lot more
compute capacity in our CircleCI for running more docker tasks in parallel,
but promu doesn't know how to distribute the work.

But I also think we could cut the build step down for PRs. But I think we
should keep the full build in master.

On Wed, Feb 12, 2020 at 3:50 PM Chris Marchbanks 
wrote:

> I also support this, waiting 2-3 hours for the build job to finish is
> frustrating. I know that building on 32 bit architectures does not catch
> all issues, specifically the alignment bug using the atomic package.
> Perhaps add at least one 32 bit build on the pull request though?
>
> Is it worth it to build everything on every master, or should a build all
> job be added to the nightly build? I agree that we should build everything
> on a cadence more frequent than a release.
>
> On Wed, Feb 12, 2020 at 1:58 AM 'Matthias Rampke' via Prometheus
> Developers  wrote:
>
>> I would build everything on master, that way we catch *before* starting
>> a release if there is something wrong.
>>
>> /MR
>>
>> On Wed, Feb 12, 2020 at 8:37 AM Sylvain Rabot 
>> wrote:
>>
>>> I did not say it but I was speaking of prometheus/prometheus, I haven't
>>> checked others repos for their full cross-building time.
>>>
>>> I think we can come up with a minimal list of GOOS/GOARCH for PRs but,
>>> if you think building the complete list on tags only is not enough, we
>>> could do it on tags & master.
>>>
>>> If we were to choose to build the complete list for tags only I would
>>> suggest to build this for PRs:
>>>
>>> - linux/amd64
>>> - linux/386
>>> - linux/arm
>>> - linux/arm64
>>> - darwin/amd64
>>> - windows/amd64
>>> - freebsd/amd64
>>> - openbsd/amd64
>>> - netbsd/amd64
>>> - dragonfly/amd64
>>>
>>> If we were to choose to build the complete list for tags & master then I
>>> would suggest an even more reduced one:
>>>
>>> - linux/amd64
>>> - darwin/amd64
>>> - windows/amd64
>>> - freebsd/amd64
>>>
>>> Regards.
>>>
>>> On Tue, 11 Feb 2020 at 23:17, Matthias Rampke  wrote:
>>>
 There are some exceptions like node exporter where it's important that
 all variants at least build, but that has a custom setup already.

 What would be a sufficient subset? Do we need to worry about endianness
 and 32 bit architectures, or would just building not catch issues specific
 to these anyway?

 /MR

 On Tue, 11 Feb 2020, 22:50 Krasimir Georgiev, 
 wrote:

> I think that is a very good idea.
>
> On Feb 11 2020, at 11:19 pm, Sylvain Rabot 
> wrote:
>
> Hi,
>
>
> I'm wondering if we could reduce the list of GOOS/GOARCH that are 
> crossbuilt for every PR by circle-ci.
>
>
> The building of the complete list seems like a waste of time & resources 
> to me.
>
>
> Maybe we could select a few and only build the complete list when 
> building tags ?
>
>
> Regards.
>
>
> --
> Sylvain Rabot 
>
> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CADjtP1FJKyVj_gq-hgVgyyVbJ%3D-pECFqcPK-QviXmKB1R-oAgg%40mail.gmail.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/2508CDF1-CC2A-4AC6-B9EE-D68B53AFF166%40getmailspring.com
> 
> .
>

>>>
>>> --
>>> Sylvain Rabot 
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/CAFU3N5V6MiMGH32a4OB0KMfJmV7FBZbJWEe6HZ-z9%2BmiOkqusQ%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
>