[prometheus-developers] Thanos sidecar is not discovered in Thanos Querier by DNS servcie discovery
Thanos sidecar was not discovered by DNS service discovery in Thanos querier: I was using stable/prometheus-operator helm chart to deploy prometheus and Thanos Sidecar. below is my config in Thanos querier deployment file for discovering all thanos sidecars… But I don’t see any reporting.. All Thanos and Prometheus related components were deployed in namespace called “monitoring”. I had to hard code in the ip and port for each of sidecar for reporting to Thanos querier for now. Any advice? --store=dnssrv+_grpc._tcp.thanos-store.monitoring.svc.cluster.local -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/0ba73a8a-f6aa-44d8-a995-796bdbb7745a%40googlegroups.com.
[prometheus-developers] Prometheus alert tracking
Hi all, We have seen an alert getting triggered couple of days ago and it got resolved after 5 seconds. After checking container logs & grafana screenshots, no evidence was seen in support of the alert which was created and resolved. System does not show any symptom relevant to the particular alert since last week. So wanted to know if there is anything which can be tracked from prometheus side to know why alert was generated? Please let me know if someone has any inputs about these kind of suspicious alerts. Thanks. -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/b82f7538-212a-4ea4-8899-c0cd234d3d65%40googlegroups.com.
Re: [prometheus-developers] Changing Prometheus from lazy to rough consensus?
On 27.05.20 16:13, Richard Hartmann wrote: > On Wed, May 27, 2020 at 12:54 AM Bjoern Rabenstein wrote: > > > - The changes as suggested in > > > > https://github.com/prometheus/docs/commit/f7ee6ac5a9b841be6be25d92e2c403de15949491 > > are _not_ following the spirit of RFC7282. > > I disagree with the above, and with the longer-form text not quoted, as > * rough consensus obviously tries to go the happy path of full consensus first Weird. My reading is that "obviously" rough consensus is what you do if you cannot reach "smooth" consensus but want to avoid voting. > * rough consensus as per the commit contains its mechanisms strictly > below the stronger mechanisms of majority and supermajority vote So you are saying that, if you say "rough consensus", you mean a wider concept around the idea of rough consensus, as described in RFC7282, and not just rough consensus itself as a step during decision making. > > - To truly adopt rough consensus, we need to introduce a chair and get > > rid of decision finding by voting (with change of governance, > > admitting and removing team members, and electing the chair as > > notable exceptions). But that would be a very invasive change of the > > governance. > > If "truly adopting rough consensus" was the proposal, I would agree. > That's _not_ the proposal, though. OK, but then the proposal is really difficult to understand. You don't want to adopt RFC7282 completely, but you also say that "rough consensus" is a whole method or procedure and more than just one step during decision making. I'd say we need a proper definition then and not just a link to RFC7282, of which some parts are relevant and some not, and the reader has to guess which are which. > The proposal is to improve the self-contained "consensus" bit while > keeping both voting mechanisms intact. As such, there's an easy way > out of any impassé. If you want to give a clearer definition of "consensus", fine. So far, things have just become less clear to me. > * consensus: > * 100% agreement by team members who care to join the discussion That's not how I understand consensus. As I have learned much earlier, consensus may include many flavors of disagreement. "Disagree but commit" is perhaps the strongest (and arguably a borderline case). The excellent RFC7282 describes many other "good" varieties of consensus (but that's not the first time I read about them). Example quotes: "Consensus doesn't require that everyone is happy and agrees that the chosen solution is the best one. Consensus is when everyone is sufficiently satisfied with the chosen solution, such that they no longer have specific objections to it." "Coming to consensus is when everyone (including the person making the objection) comes to the conclusion that either the objections are valid, and therefore make a change to address the objection, or that the objection was not really a matter of importance, but merely a matter of taste." It also describes varieties of "not really consensus": "People might very well agree that a solution is sufficient and have no objection to it, but if they also don't actively think it's a good and correct outcome, it's absurd to declare that the group has consensus." Or the "capitulation" antipattern, when one party just has "simply given up by trying to appease the others". or the "horse-trading" variety. Note that none of the consensus examples above, neither the proper ones nor the false ones, are called "rough consensus" in the RFC: "Of course, coming to full consensus like that does not always happen. That's why in the IETF, we talk about 'rough consensus'." (Strictly speaking, you could argue that "rough consensus" is the superset of the happy case and the rough case, but it's still very confusing if the governance says "our default way of deciding is rough consensus". It should be "our default way of deciding is by consensus, if need be by a rough one".) Circling back: If all you want is to define "consensus" better, then let's just do it. I can see how the "as long as nobody objects" in the governance can be read as including some of the "not really consensus" cases from RFC7282, but as excluding some of the "proper consensus" cases. Here is a rough suggestion how a minimal governance change could look like: https://github.com/prometheus/docs/commit/66dcf05d9d8fa246ef7c557b4a1978c718455fd2 _However,_ I have no high hopes that such a governance change will on its own change anything in our decision making. For one, I'm confident that most of us were alredy aware that consensus does not equal "100% agreement by team members". Furthermore, as has been said multiple times by now, most of the difficult decision making had its root cause in precisely the problem that not all participants saw their objections addressed. If we cannot agree that all objections have been addressed, and we have no chair that has the power to make that assessment, there will still be no decision. Th
Re: [prometheus-developers] Changing Prometheus from lazy to rough consensus?
On Wed, May 27, 2020 at 8:35 AM Stuart Clark wrote: > In a lot of organisations with chair based systems which work well the > opposite is often true. > > The chair is there to facilitate the debate and therefore doesn't > express their own opinion. As a result, if the judgement call has to be > made (such as calling rough consensus) their is trust from all sides > that it is a reasonable decision not just following their personal belief. > > (I've also seen and had similar advice for face-to-face meetings - it > works best if the chair/facilitator is not one of the active participants) This matches my experience 100%. From taking part in those groups, from shadowing Chairs in the same office, to Chairing myself. It's also how I hope people would agree the dev summits are structured. In practice, our group is too small and too specialized to have much in choice of external chairs so it means switching hats, but I believe that there's trust in this group that this is happening properly. The Call for Consensus during the dev summits is founded on all this. Richard -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgS2fjKmXgmBb01guUq_907i42ypw%3Do7aUD_h4JL7rESOA%40mail.gmail.com.
Re: [prometheus-developers] Changing Prometheus from lazy to rough consensus?
On Wed, May 27, 2020 at 12:54 AM Bjoern Rabenstein wrote: > - The changes as suggested in > > https://github.com/prometheus/docs/commit/f7ee6ac5a9b841be6be25d92e2c403de15949491 > are _not_ following the spirit of RFC7282. I disagree with the above, and with the longer-form text not quoted, as * rough consensus obviously tries to go the happy path of full consensus first * rough consensus as per the commit contains its mechanisms strictly below the stronger mechanisms of majority and supermajority vote > - To truly adopt rough consensus, we need to introduce a chair and get > rid of decision finding by voting (with change of governance, > admitting and removing team members, and electing the chair as > notable exceptions). But that would be a very invasive change of the > governance. If "truly adopting rough consensus" was the proposal, I would agree. That's _not_ the proposal, though. The proposal is to improve the self-contained "consensus" bit while keeping both voting mechanisms intact. As such, there's an easy way out of any impassé. The reason why this is the proposal is the imbalance we have between the three decision making mechanisms: * consensus: * 100% agreement by team members who care to join the discussion * not time-bound * can contain an unlimited amount of explicit and implicit topics in whatever freeform text and places * no social pressure to voice an opinion * majority vote: * 50% agreement by team members who care to join the discussion * time-bound * can contain an explicit list of options on one single topic in a specific place * social pressure to voice an opinion * supermajority vote: * 66% agreement amongst all team members * time-bound * can contain an explicit list of options on one single topic in a specific place * I will track you down and make you vote (whichever way you prefer) to make sure we reach a result one way or the other Consensus should have the weakest, not the strongest, requirements for passing within this system. > - Before we do that, I would prefer to not start a discussion with a > possible solution, but with the problem we are trying to solve. I > think we first have to understand the apparent or actual problems in > decision making much better. Then we are in much better shape to > find a solution. Personally, I feel as if I have a good grasp on the problem domain and the thread seems to reflect that sentiment in others. This is not to say that having a distinct wider discussion in parallel, or instead, wouldn't be an option. Again, personally, I am content with the proposal as is, but I am biased. Richard -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgRFqFVYGS3iBO5Z%3DwT1t9DBwov8tJ9k9vGoYyTZBOS%2BFw%40mail.gmail.com.
Re: [prometheus-developers] Changing Prometheus from lazy to rough consensus?
On Tue, May 26, 2020 at 11:08 PM Julius Volz wrote: > Even if there's already an outspoken (relative) majority against that > minority? How would that be rough consensus then? If 100 people wanted to support traces natively in Prometheus, it would not need much of discussion to dismiss them. It's relatively common on RIPE's address-WG that people are vocal about IPv4/IPv6 and whatever they came up with after 20 whole minutes of thought, and then they bring their friends. Overall, it's a possible, but not very likely, that this would happen within Prometheus, imo. -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgSA36J57pSz3-UA3S1TGBT5g7uwBWtAUz68k2Z%2BOYA4SQ%40mail.gmail.com.
Re: [prometheus-developers] move https/ package to client_golang
On 27 May 08:05, Brian Brazil wrote: > On Wed, 27 May 2020 at 07:52, Stuart Clark wrote: > > > On 27/05/2020 07:50, Brian Brazil wrote: > > > > On Wed, 27 May 2020 at 07:05, Ben Kochie wrote: > > > >> I was thinking about building an "exporter kit" repo that would include > >> some helpful functions to reduce the amount of boilerplate needed to write > >> exporters. > >> > > > > I've thought such a thing would be useful for a long time, though my > > presumption was always that it would end up in client_golang as it's not > > too far from instrumentation. > > > > In general I'm not a big fan of widespread proliferation of repos, > > particularly if it's lots of tiny repos. Even in the previous cases where > > we managed to get the layering largely right, it still was quite a pain in > > terms of overhead and release management if the repos were being actively > > developed. A single toolkit-y repo I could live with, I'd be concerned if > > we were talking repos beyond that. > > > > > > How does the release management/overhead differ between several single > > purpose repos and a single repo containing independent things in different > > directories? > > > I don't think we've really had that particular situation come up yet, but I > imagine there'd be similar challenges either way - likely with the > multiple-repo case being a bit trickier to debug through. We would need to adapt our Makefiles; and stop vendoring if we go for multi repo in one. The benefits for the users is to be able to only pull the https/package. But now that I think about it, of we make a toolkit repo, it will probably ALSO include the http server logic, so the exporters would NOT need to use http directly. At the end, I then think we then can do a single toolkit repo. Next to the toolkit repo (in prometheus org) we could also create a template repo [1] github.com/prometheus-community/exemple_exporter which reuses that and that people can just "fork" [2]. [1] https://help.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-template-repository [2] https://help.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-from-a-template > > -- > Brian Brazil > www.robustperception.io > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLrCUHOYnd3u%2B%2B8KNexFr0qdrT_0e69eeC8ZMz--r4WymQ%40mail.gmail.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200527072640.GA334861%40oxygen.
Re: [prometheus-developers] move https/ package to client_golang
On Wed, 27 May 2020 at 07:52, Stuart Clark wrote: > On 27/05/2020 07:50, Brian Brazil wrote: > > On Wed, 27 May 2020 at 07:05, Ben Kochie wrote: > >> I was thinking about building an "exporter kit" repo that would include >> some helpful functions to reduce the amount of boilerplate needed to write >> exporters. >> > > I've thought such a thing would be useful for a long time, though my > presumption was always that it would end up in client_golang as it's not > too far from instrumentation. > > In general I'm not a big fan of widespread proliferation of repos, > particularly if it's lots of tiny repos. Even in the previous cases where > we managed to get the layering largely right, it still was quite a pain in > terms of overhead and release management if the repos were being actively > developed. A single toolkit-y repo I could live with, I'd be concerned if > we were talking repos beyond that. > > > How does the release management/overhead differ between several single > purpose repos and a single repo containing independent things in different > directories? > I don't think we've really had that particular situation come up yet, but I imagine there'd be similar challenges either way - likely with the multiple-repo case being a bit trickier to debug through. -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLrCUHOYnd3u%2B%2B8KNexFr0qdrT_0e69eeC8ZMz--r4WymQ%40mail.gmail.com.
