Another option if your security will allow opening one port to Prometheus data, there is the "exporter_exporter"
https://github.com/QubitProducts/exporter_exporter On the subject of external labels, you should have those anyway on your Prometheus instances otherwise you can't identify them properly. On Thu, Mar 25, 2021 at 2:59 PM robbe vaes <robbetj...@gmail.com> wrote: > Alright, did not know all of that. We are now setting up the Prometheus > Proxy and are hoping that it does what we want it to do. VictoriaMetrics > would be a last case scenario anyway > > Op donderdag 25 maart 2021 om 14:54:22 UTC+1 schreef juliu...@promlabs.com > : > >> What should be noted when bringing up VictoriaMetrics in the Prometheus >> context is that it is deliberately incompatible with Prometheus in multiple >> ways: >> >> - VM's MetricsQL behaves differently from PromQL in a multitude of ways >> and is *not* backwards-compatible with PromQL (see also >> https://promlabs.com/promql-compliance-tests). >> - The VMAgent does remote_write without the necessary pre-processing of >> series staleness handling, and it throws away datapoints like special float >> values (like NaN). At least last time I checked. >> - The TSDB has lossy compression, so it throws away part of your data >> even further. Prometheus and Thanos store and retrieve float sample values >> exactly as sent, which depending on the use case, may matter. >> >> All that is to say that you can use VictoriaMetrics, but you are "on your >> own" then in terms of proper Prometheus ecosystem support and compatibility. >> >> On Thu, Mar 25, 2021 at 2:31 PM robbe vaes <robbe...@gmail.com> wrote: >> >>> It is correct that Thanos' deduplication works when using just 2 or more >>> Prometheus servers. The thing is, Thanos cannot use existing labels to do >>> deduplication. I want it to deduplicate using the label called >>> exported_instance, but i cannot predefine this label in the external labels >>> for thanos because it is not from prometheus but from the collectd >>> exporter. We are now looking into Prometheus Proxy, since we already tried >>> PushProx and it did not do what we wanted it to do. VictoriaMetrics might >>> be interesting to look into as well! >>> >>> Regards, >>> Robbe >>> >>> Op donderdag 25 maart 2021 om 14:25:17 UTC+1 schreef val...@gmail.com: >>> >>>> Probably VictoriaMetrics could help with this setup. It supports data >>>> push via various protocols >>>> <https://victoriametrics.github.io/#how-to-import-time-series-data>, >>>> it supports data deduplication >>>> <https://victoriametrics.github.io/#deduplication> and it is compatible >>>> with Prometheus datasource in Grafana >>>> <https://victoriametrics.github.io/#grafana-setup>. >>>> >>>> On Thu, Mar 25, 2021 at 3:20 PM robbe vaes <robbe...@gmail.com> wrote: >>>> >>>>> The thing is we can implement multiple prometheus instances as well, >>>>> thats no issue and will probably happen anyway. The thing is, we tried >>>>> using Thanos to manage multiple prometheus servers and do deduplication, >>>>> but the deduplication does not work for the collectd exporters. The >>>>> problem >>>>> its having i think is that we need to filter out duplicate data by >>>>> exported >>>>> instance, but this is not possible for Thanos since it needs predefined >>>>> external labels within prometheus. >>>>> >>>>> Regards, >>>>> >>>>> Op donderdag 25 maart 2021 om 14:15:24 UTC+1 schreef Julien Pivotto: >>>>> >>>>>> The way this is usually solved is by duplicating prometheus - it >>>>>> seems >>>>>> that now you have moved the SPOF from the exporter to prometheus. >>>>>> >>>>>> Regards, >>>>>> >>>>>> On 25 Mar 06:01, robbe vaes wrote: >>>>>> > Okay so, we want to have an environment using Prometheus, where we >>>>>> can >>>>>> > monitor our servers etc with a push method rather then pull due to >>>>>> network >>>>>> > security aspects. As of now, we managed to set up collectd together >>>>>> with >>>>>> > collectd exporter for prometheus. This way we can have the clients >>>>>> or >>>>>> > servers push their data to the exporter and have prometheus scrape >>>>>> the data >>>>>> > from the exporter to minimize the amount of pulling that has to be >>>>>> done. >>>>>> > The issue now is, we want to have multiple collectd exporters to >>>>>> improve >>>>>> > HA, but the problem now is that in prometheus, when scraping both >>>>>> > exporters, it takes the data from both. It shows for example for >>>>>> server A >>>>>> > the cpu load but it shows it twice instead of only once because its >>>>>> being >>>>>> > pulled from both exporters. We want to have something like this but >>>>>> it >>>>>> > needs to have a form of deduplication so we dont have the same data >>>>>> twice. >>>>>> > >>>>>> > >>>>>> > Op donderdag 25 maart 2021 om 13:27:44 UTC+1 schreef >>>>>> sup...@gmail.com: >>>>>> > >>>>>> > > Can you describe more about what your network topology is >>>>>> exactly? There >>>>>> > > are a number of solutions for dealing with distributed >>>>>> monitoring. >>>>>> > > >>>>>> > > On Thu, Mar 25, 2021 at 12:45 PM robbe vaes <robbe...@gmail.com> >>>>>> wrote: >>>>>> > > >>>>>> > >> Hi, >>>>>> > >> >>>>>> > >> I am trying to setup a monitoring environment with Prometheus, >>>>>> but it has >>>>>> > >> to be using a push mechanism instead of the standard pull >>>>>> mechanism >>>>>> > >> Prometheus uses. I was wondering what options there are to >>>>>> create an >>>>>> > >> environment like this. It would also have to perfom data >>>>>> deduplication. The >>>>>> > >> main issue is that I don't want Prometheus to scrape the clients >>>>>> itself, >>>>>> > >> but rather that it scrapes a certain location for all the >>>>>> metrics and that >>>>>> > >> the clients push their metrics to that location automatically. >>>>>> > >> >>>>>> > >> Suggestions are very welcome! >>>>>> > >> >>>>>> > >> -- >>>>>> > >> You received this message because you are subscribed to the >>>>>> Google Groups >>>>>> > >> "Prometheus Users" group. >>>>>> > >> To unsubscribe from this group and stop receiving emails from >>>>>> it, send an >>>>>> > >> email to prometheus-use...@googlegroups.com. >>>>>> > >> To view this discussion on the web visit >>>>>> > >> >>>>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com >>>>>> > >> < >>>>>> https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> >>>>>> > >> . >>>>>> > >> >>>>>> > > >>>>>> > >>>>>> > -- >>>>>> > You received this message because you are subscribed to the Google >>>>>> Groups "Prometheus Users" group. >>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to prometheus-use...@googlegroups.com. >>>>>> > To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/prometheus-users/3325df03-3ad7-4d24-be81-f0e6c4c5b07fn%40googlegroups.com. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Julien Pivotto >>>>>> @roidelapluie >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to prometheus-use...@googlegroups.com. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> >>>> Aliaksandr Valialkin, CTO VictoriaMetrics >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to prometheus-use...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Julius Volz >> PromLabs - promlabs.com >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/e717b6cf-775c-4662-a1d4-e45157805a1bn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/e717b6cf-775c-4662-a1d4-e45157805a1bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmoXowBqQaKz0%3DBD5pBxrcs710xKBD0OFpZJ_%2BLMd8y3Xw%40mail.gmail.com.
