Re: [prometheus-developers] Protobuf support
Disclaimer: I'm not the expert for remote-write/-read, and only do a little with the protobuf stuff. But since you didn't get an answer in a week, I'm trying now. On 29.02.24 20:38, Clément Jean wrote: > > I finally go around to do this and the unit tests seem to be passing. You > can find what I did here: > https://github.com/Clement-Jean/prometheus-move-to-proto. Thank you. This looks very unconventional with separate patch files. Wouldn't it be more straight forward to just fork prometheus/prometheus and work in the normal git style in that fork? > I still have few questions: > - For the benchmarking, should I try running the benchmark on the current > version of prometheus and then run it on my version, or is there a better > way to do this? That sounds like the way to go. > - Is `make bench_tsdb` enough or should I run other benchmarks? `make bench_tsdb` is a rather special benchmark for certain TSDB aspects. I doubt it touches protobuf at all. We usually use the benchmarking framework as built into the Go toolchain, i.e. you run `go test -bench NameOfBenchmarkToRun`, usually also using the `-benchmem` flag to see memary allocation stats. Bryan referred to https://github.com/prometheus/prometheus/blob/122f9506e9c6/storage/remote/queue_manager_test.go#L872 previously. You would run that benchmark in the storage/remote directory by typing `go test -bench BenchmarkSampleSend -benchmem` > - For forward/backward compat, I'm not sure how I should go about it. Could > you clarify a little bit? It's important that the protobuf messages on the wire are still encodable and decodable by older and newer versions of the code. So I believe what Brian was referring to is that you set up a Prometheus server with and without your changes and let them send remote write and remote read to each other. In different news: On this same mailing list, somebody else (mircodezo...@gmail.com) is also working on the same topic, see thread titled "Migrating away from github.com/gogo/protobuf". They have run a bunch of benchmarks already, and their approach to replace gogo-protobuf has run into some issues. I suggest you two join forces and exchange your experiences. If you want a chat-like medium to discuss things, you should know that most of the dev conversation happens on the CNCF Slack these days (channel #prometheus-dev). (There is also a #prometheus-dev on IRC, but I'm afraid Slack has successfully sucked most devs into its black hole.) Hope that helps a bit. -- Björn Rabenstein [PGP-ID] 0x851C3DA17D748D03 [email] bjo...@rabenste.in -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/ZehYOmNqb7R%2BsNFb%40mail.rabenste.in.
Re: [prometheus-developers] Protobuf support
Hey, I finally go around to do this and the unit tests seem to be passing. You can find what I did here: https://github.com/Clement-Jean/prometheus-move-to-proto. I still have few questions: - For the benchmarking, should I try running the benchmark on the current version of prometheus and then run it on my version, or is there a better way to do this? - Is `make bench_tsdb` enough or should I run other benchmarks? - For forward/backward compat, I'm not sure how I should go about it. Could you clarify a little bit? Thank you On Sunday, October 22, 2023 at 11:34:38 PM UTC+8 Bryan Boreham wrote: > > - why not moving to the official generated code? It seems it's the only > stable one on the long term > > At least because nobody has put the work in to do the conversion and > demonstrate it is no worse than today. > I don't like the extra fields in generated structs, but I didn't measure > the impact. > > > - Is there a benchmark that I could run for prometheus using different > plugins? > > Yes, here for instance: > https://github.com/prometheus/prometheus/blob/122f9506e9c6/storage/remote/queue_manager_test.go#L872 > > For completeness one should check the 4 combinations: remote-read > sender/receiver and remote-write sender/receiver. > > There are some unit tests but I don't think they will help to check > forwards/backwards compatibility. > You could set up another Prometheus and have them to talk to each other. > Some discussion about how to check at > https://github.com/prometheus/prometheus/pull/9991 > > Bryan > > On Friday, 20 October 2023 at 03:36:13 UTC+1 Clément Jean wrote: > >> That actually sounds interesting to me! I have few questions though: >> >> - why not moving to the official generated code? It seems it's the only >> stable one on the long term >> - Is there a benchmark that I could run for prometheus using different >> plugins? >> - How can I test if my code changes didn't break the project? go test >> ./... ? >> - Are there some platform specific tests? >> >> Hope we can work on that >> >> On Wednesday, October 18, 2023 at 8:42:21 PM UTC+8 Bjoern Rabenstein >> wrote: >> >>> On 05.10.23 17:48, Clément Jean wrote: >>> > I'm not entirely sure yet because I'm new to contributing to >>> prometheus. If >>> > there is any use cases that you guys already discussed around >>> Protobuf, I'd >>> > be happy to help. >>> >>> If you are really deep into protobuf, there is definitely one big and >>> fat issue to solve: We are still using gogo-protobuf in >>> prometheus/prometheus, which has good performance properties, but is >>> unmaintained. The plan has been for a while to migrate to another >>> protobuf implementation that performs similarly well. Here is the >>> discussion on this mailing list: >>> >>> https://groups.google.com/g/prometheus-developers/c/uFWRyqZaQis/m/1OOGT7s5AwAJ >>> >>> >>> And here is a branch that contains a PoC of migrating to the vitess >>> protoc plugin: >>> https://github.com/austince/prometheus/tree/feat/drop-gogo >>> (Note that it is more than two years old, and I'm not sure if the >>> vitess plugin performs well enough. But it could be a starting point.) >>> >>> -- >>> Björn Rabenstein >>> [PGP-ID] 0x851C3DA17D748D03 >>> [email] bjo...@rabenste.in >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6d2cc6c3-9c40-42f7-be0a-77660aa9082en%40googlegroups.com.
Re: [prometheus-developers] Protobuf support
> - why not moving to the official generated code? It seems it's the only stable one on the long term At least because nobody has put the work in to do the conversion and demonstrate it is no worse than today. I don't like the extra fields in generated structs, but I didn't measure the impact. > - Is there a benchmark that I could run for prometheus using different plugins? Yes, here for instance: https://github.com/prometheus/prometheus/blob/122f9506e9c6/storage/remote/queue_manager_test.go#L872 For completeness one should check the 4 combinations: remote-read sender/receiver and remote-write sender/receiver. There are some unit tests but I don't think they will help to check forwards/backwards compatibility. You could set up another Prometheus and have them to talk to each other. Some discussion about how to check at https://github.com/prometheus/prometheus/pull/9991 Bryan On Friday, 20 October 2023 at 03:36:13 UTC+1 Clément Jean wrote: > That actually sounds interesting to me! I have few questions though: > > - why not moving to the official generated code? It seems it's the only > stable one on the long term > - Is there a benchmark that I could run for prometheus using different > plugins? > - How can I test if my code changes didn't break the project? go test > ./... ? > - Are there some platform specific tests? > > Hope we can work on that > > On Wednesday, October 18, 2023 at 8:42:21 PM UTC+8 Bjoern Rabenstein wrote: > >> On 05.10.23 17:48, Clément Jean wrote: >> > I'm not entirely sure yet because I'm new to contributing to >> prometheus. If >> > there is any use cases that you guys already discussed around Protobuf, >> I'd >> > be happy to help. >> >> If you are really deep into protobuf, there is definitely one big and >> fat issue to solve: We are still using gogo-protobuf in >> prometheus/prometheus, which has good performance properties, but is >> unmaintained. The plan has been for a while to migrate to another >> protobuf implementation that performs similarly well. Here is the >> discussion on this mailing list: >> >> https://groups.google.com/g/prometheus-developers/c/uFWRyqZaQis/m/1OOGT7s5AwAJ >> >> >> And here is a branch that contains a PoC of migrating to the vitess >> protoc plugin: >> https://github.com/austince/prometheus/tree/feat/drop-gogo >> (Note that it is more than two years old, and I'm not sure if the >> vitess plugin performs well enough. But it could be a starting point.) >> >> -- >> Björn Rabenstein >> [PGP-ID] 0x851C3DA17D748D03 >> [email] bjo...@rabenste.in >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6b56e75d-7fb5-4ab8-a0bd-3d140c32ec7fn%40googlegroups.com.
Re: [prometheus-developers] Protobuf support
That actually sounds interesting to me! I have few questions though: - why not moving to the official generated code? It seems it's the only stable one on the long term - Is there a benchmark that I could run for prometheus using different plugins? - How can I test if my code changes didn't break the project? go test ./... ? - Are there some platform specific tests? Hope we can work on that On Wednesday, October 18, 2023 at 8:42:21 PM UTC+8 Bjoern Rabenstein wrote: > On 05.10.23 17:48, Clément Jean wrote: > > I'm not entirely sure yet because I'm new to contributing to prometheus. > If > > there is any use cases that you guys already discussed around Protobuf, > I'd > > be happy to help. > > If you are really deep into protobuf, there is definitely one big and > fat issue to solve: We are still using gogo-protobuf in > prometheus/prometheus, which has good performance properties, but is > unmaintained. The plan has been for a while to migrate to another > protobuf implementation that performs similarly well. Here is the > discussion on this mailing list: > > https://groups.google.com/g/prometheus-developers/c/uFWRyqZaQis/m/1OOGT7s5AwAJ > > And here is a branch that contains a PoC of migrating to the vitess > protoc plugin: > https://github.com/austince/prometheus/tree/feat/drop-gogo > (Note that it is more than two years old, and I'm not sure if the > vitess plugin performs well enough. But it could be a starting point.) > > -- > Björn Rabenstein > [PGP-ID] 0x851C3DA17D748D03 > [email] bjo...@rabenste.in > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/fbd81c3a-182b-4d05-984f-fdc702d4b809n%40googlegroups.com.
Re: [prometheus-developers] Protobuf support
On 05.10.23 17:48, Clément Jean wrote: > I'm not entirely sure yet because I'm new to contributing to prometheus. If > there is any use cases that you guys already discussed around Protobuf, I'd > be happy to help. If you are really deep into protobuf, there is definitely one big and fat issue to solve: We are still using gogo-protobuf in prometheus/prometheus, which has good performance properties, but is unmaintained. The plan has been for a while to migrate to another protobuf implementation that performs similarly well. Here is the discussion on this mailing list: https://groups.google.com/g/prometheus-developers/c/uFWRyqZaQis/m/1OOGT7s5AwAJ And here is a branch that contains a PoC of migrating to the vitess protoc plugin: https://github.com/austince/prometheus/tree/feat/drop-gogo (Note that it is more than two years old, and I'm not sure if the vitess plugin performs well enough. But it could be a starting point.) -- Björn Rabenstein [PGP-ID] 0x851C3DA17D748D03 [email] bjo...@rabenste.in -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/ZS/SqHfCPP9Cag2m%40mail.rabenste.in.
Re: [prometheus-developers] Protobuf support
I'm not entirely sure yet because I'm new to contributing to prometheus. If there is any use cases that you guys already discussed around Protobuf, I'd be happy to help. On Wednesday, October 4, 2023 at 9:22:05 PM UTC+8 Bjoern Rabenstein wrote: > On 03.10.23 18:06, Clément Jean wrote: > > > > I met with Richard Hartmann at KubeCon Shanghai and he mentioned that > the > > team is interested in adding support for Protobuf. I'm here to see what > I > > can do to help. If you have any recommendation on how to get started, > I'd > > be happy to start contributing. > > Thanks for your interest in contributing to Prometheus. > > Could you provide more details about what kind of protobuf support you > would like to add? Prometheus already supports protobuf for scraping, > remote write, and remote read. > > In general, you could check out issues on the many prometheus > repositories to get inspiration what's needed. Ideally, you pick an > issue that also scratches one of your own itches. If it is tagged as > "good first issue", even better. I would avoid the issues tagged as > "not as easy as it looks" until you have gathered more experience. > > Cheers, > -- > Björn Rabenstein > [PGP-ID] 0x851C3DA17D748D03 > [email] bjo...@rabenste.in > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/00f7711e-75e3-4960-9e07-e588579f181cn%40googlegroups.com.
Re: [prometheus-developers] Protobuf support
On 03.10.23 18:06, Clément Jean wrote: > > I met with Richard Hartmann at KubeCon Shanghai and he mentioned that the > team is interested in adding support for Protobuf. I'm here to see what I > can do to help. If you have any recommendation on how to get started, I'd > be happy to start contributing. Thanks for your interest in contributing to Prometheus. Could you provide more details about what kind of protobuf support you would like to add? Prometheus already supports protobuf for scraping, remote write, and remote read. In general, you could check out issues on the many prometheus repositories to get inspiration what's needed. Ideally, you pick an issue that also scratches one of your own itches. If it is tagged as "good first issue", even better. I would avoid the issues tagged as "not as easy as it looks" until you have gathered more experience. Cheers, -- Björn Rabenstein [PGP-ID] 0x851C3DA17D748D03 [email] bjo...@rabenste.in -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/ZR1m%2BPFlN3H/nnXJ%40mail.rabenste.in.
[prometheus-developers] Protobuf support
Hey everyone, I met with Richard Hartmann at KubeCon Shanghai and he mentioned that the team is interested in adding support for Protobuf. I'm here to see what I can do to help. If you have any recommendation on how to get started, I'd be happy to start contributing. -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/e9054b91-f527-4a66-90db-3a6fd90cabd9n%40googlegroups.com.
