Re: GRe: [prometheus-developers] Migrating away from github.com/gogo/protobuf

> Any reason samples are still encoded as a list of pairs? Using two arrays would both reduce the number of objects and allow using varint encoding for timestamps. Just to note that Prometheus is coded to send just one sample per series in a remote-write message. In normal usage we want to send

Re: GRe: [prometheus-developers] Migrating away from github.com/gogo/protobuf

On 02.03.24 08:00, Mirco wrote: > Any reason samples are still encoded as a list of pairs? Using two arrays > would both reduce the number of objects and allow using varint encoding for > timestamps. I assume you are referring to the protobuf messages for TimeSeries and Sample? I'm not an exper

GRe: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Given the benchmarks it seems like moving away from gogo is not feasible with the current protobuf schema. I've only now looked at the remote write 2.0 branch, and that might fix the issues with label allocations. Any reason samples are still encoded as a list of pairs? Using two arrays would b

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

I tried benchmarking BenchmarkSampleSend and the results seem awful, from ~15 B/op and ~250 allocs/op to ~140 B/op to ~17 allocs/op. Profiling the code, it seems like labelsToLabelsProto is the offender, as it allocates a *prompb.Label per label. I experimented with using a map[str

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Do you have one for sending remote-write data? The one you linked is RemoteWritehandler which is for receiving. >From what I've seen elsewhere, Proto3 structs have extra data members which I expect to show significantly increased memory allocation. Unfortunately the remote-write structure with

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

> I'm wondering how easily we'll introduce bugs in that way I've just realized that using enableunsafedecode=true would prevent us from pooling the buffers, as whenever using unsafe.Pointer to convert between []byte and string one cannot modify the original []byte array. I'll update the pooling b

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Thanks for doing this. Beyond benchmarks, two general concerns: - How unsafe is `enableunsafedecode=true`? I spot-checked the csproto code, and the risk seems to be on the side of the user code, i.e. luckily there isn't any unsafe input, but I'm wondering how easily we'll introduce bugs in

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Would the already present benchmarks in the code be sufficient? If so, here are the remote read , remote write , and a cheeky

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

No need to apologise! Do you have benchmarks using the Prometheus remote-read and remote-write protobufs ? On Wednesday 14 February 2024 at 08:08:37 UTC mircod...@gmail.com wrote: > Hi all, sorry to disrupt this discussion. > > Before stumbling upon this thread, I had worked on a separate fork

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Hi all, sorry to disrupt this discussion. Before stumbling upon this thread, I had worked on a separate fork to deprecate gogo in favor of csproto, as compiling it using enableunsafedecode=true seems

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Issue for reference: https://github.com/prometheus/prometheus/issues/11908 Kind Regards, Bartek Płotka On Saturday, February 3, 2024 at 12:56:09 PM UTC Bartłomiej Płotka wrote: > We did a bit of testing for remote write 2.0 work (e.g. here > ) for gog

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

We did a bit of testing for remote write 2.0 work (e.g. here ) for gogo vs different plugins, and vtproto is the most promising even with more pointers. We have to get rid of nullables, yes (more pointers, pore separate objects on heap, generally more

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

As part of all the remote write proto changes I've been working on I tried out moving us off of gogoproto, cherry picking Austin's original changes into a new branch off of the current main branch. As Tom mentioned, the main reason for using gogoproto is that `repeated SomeMessageType = n;` fie

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

I think I'd be most useful to rebase, and create a PR from this, then we can see whether tests pass and we can run prombench (although I don't think there are any perf tests that involve the proto parts). Then we can discuss on there and figure out where to take this. Thank you so much for the wor

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

I've updated my branch ( https://github.com/austince/prometheus/tree/feat/drop-gogo) to use both the vitess plugin and the buf tool, which indeed fit very nicely together. I've only updated the code enough for it to compile, have not investigated the semantic differences. This is likely the furthe

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

I have heard great thing, but haven’t used it. Wrongfully thought that they are mutually exclusive but turns out they are actually complementary: https://twitter.com/fredbrancz/status/1405192828049838080?s=21 We should probably do an investigation of the combination. On Thu 17. Jun 2021 at 18:26,

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Just saw this on the CNCF blog as well, seems like a promising library. Tangentially, have you heard of https://github.com/bufbuild/buf? It seems much nicer than compiling with shell scripts and protoc. -- You received this message because you are subscribed to the Google Groups "Prometheus Dev

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

This looks promising for us to potentially move off of gogo proto without a performance hit (maybe even an improvement): https://vitess.io/blog/2021-06-03-a-new-protobuf-generator-for-go/ On Fri 30. Apr 2021 at 00:01, Austin Cawley-Edwards wrote: > Totally on the benchmarking, that would be grea

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Totally on the benchmarking, that would be great! I've got a work in progress branch here , but it looks like it will be rather intensive, unfo

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Sounds good! As this is pretty performance sensitive, I think we'd want to benchmark any changes to this code before we merge it. Let me know if I can help there.. > You mentioned there are others for Cortex @tom.wil...@gmail.com? Bringing this up at the next Cortex community is probably the bes

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Looks like Thanos is having a similar discussion internally: https://github.com/thanos-io/thanos/pull/4086#discussion_r617274400 On Wed, Apr 28, 2021 at 1:19 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Here's also some nice history on how Protobuf in Go has changed: > https://git

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

Here's also some nice history on how Protobuf in Go has changed: https://github.com/protocolbuffers/protobuf-go#historical-legacy And just for clarification, I'm recommending we switch to google.golang.org/protobuf (which is now the official implementation), sorry for the confusion/typo with golan

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

> > This surprised me a little, and sounds awesome - but I can't find any more > information about it. Does v1.4.0 generate code for the serialisation / > deserialisation function or still rely on Golang reflection? Does it allow > for the neat tricks to get rid of pointers and all the allocation

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

> Along with that, many of the performance boosts that are provided by gogo have been addressed in the official Go lib, golang/protobuf , as of v1.4.0 . This surprised me a little, and sounds awesome - but I can't find any

Re: [prometheus-developers] Migrating away from github.com/gogo/protobuf

I'd be very happy with this. One consideration that we should take (certainly not blocking this but should keep in mind) is how this would affect immediate downstream users. Off the top of my head, I would think checking in with the Cortex and Thanos projects is probably a good idea, I know both ha

[prometheus-developers] Migrating away from github.com/gogo/protobuf

Hi all, The protobuf library used in Prometheus, gogo/protobuf , is no longer actively maintained (activity largely stopped pre-2020, looking for new ownership ). Along with that, many of the performance boosts tha