Lets move this discussion to the associated GH Issue that you opened here https://github.com/protocolbuffers/protobuf/issues/22075
Thanks! On Tue, Jun 3, 2025 at 10:21 PM Yordis Prieto <[email protected]> wrote: > To add extra context, I am speaking purely from the end-user perspective > experience, I am sure they are technical concerns to take into > consideration as someone pointed out, I am bringing my naive experience > with the intent to figure out whatever optimal strategy would be. > > On Tuesday, June 3, 2025 at 1:12:29 PM UTC-4 Yordis Prieto wrote: > >> Personally, I do not mind the builder pattern or dealing with >> getters/setters; if that is the answer, I am OK with it, my primary concern >> is when things are combined and I can construct broken messages (bypassing >> builder, and setters) and the builder pattern. >> >> Extra context, I am optimizing for LLM effectiveness, after doing some >> polyglot work around TypeScript, Go, and Rust. I found that Rust type >> system avoid hallucination to a point that isn't easy to ignore. For that >> to happen, the types coming out of the protobuf and its API needs to avoid >> silly mistakes. The same mistakes that some developer make based on >> suboptimal assumptions (context). >> >> Without sidetracking too much, the biggest pain in the Go version is that >> everything is a pointer; we can not know if the reason for the pointer is >> because we actually want `nil` values or because of memory layout or >> technical concern. I wish an Option type existed in Go or added to the >> protobuf for this same reason. And well, defaults .... I digress. >> >> I can strategize around the package, in the worst case, hoping that the >> protoc plugin allows some policies to change the behaviour. The interesting >> situation is that, reading about the required deprecation a bit more, >> definitely wouldn't like stale proto files to blow up if new information is >> added but my Processor does't care since it doesn't make use of the info, >> and therefore, it doesn't need the new proto file. >> Purely speaking about Event Sourcing so my events are immutable, never >> breaking change messages. >> >> What do you recommend me to do here? I need to find a way to test the v4 >> asap, I feel it is prudent to move to it >> On Tuesday, June 3, 2025 at 8:29:06 AM UTC-4 Em Rauch wrote: >> >>> > but I would expect two types based on my needs >>> (PartialMessage<Message> and Message). >>> > I guess I am going against what the philosophy is, which it is OK by >>> me, just feel suboptimal, I want to get the most I can out of the type >>> system without requiring that much discipline from developers. >>> >>> This does make perfect sense in general: if required was newly created >>> in 2025 it maybe could try to have semantics of always having Builders, >>> being able to parsePartial into Builders. Note that not many of the >>> official implementations of Protobuf do not use builder pattern though, so >>> anything like that would be very difficult to retrofit. >>> >>> But the context is more that the idea of "blanket enforcement of >>> presence of a field at parse time" is considered a misfeature (as discussed >>> in the "Required is Strongly Deprecated" section here >>> <https://protobuf.dev/programming-guides/proto2/#:~:text=Required%20is%20Strongly%20Deprecated>). >>> Under the expected usecase patterns of Protobuf, its considered a big >>> footgun since many people believe they have a field that is obviously >>> always strictly required, but actually a few years later they will want to >>> replace it with something different, and hard enforcement at parse time >>> makes that evolution almost impossible. >>> >>> Zooming out a bit: for the usecase you're describing, for the official >>> Google supported implementations, I think just doing the normal / natural >>> thing should be able to achieve the effective behavior you're thinking >>> about. Make a normal message, make it have normal message-typed fields, >>> treat it as normal child structs which are just "always there". Treat the >>> maybe-lazy-allocation of those inner structs as an implementation detail, >>> and treat the has-bit tracking as an implementation detail. If the >>> default-initialized child struct isn't a valid value, check in your >>> application code against that condition, rather than checking against the >>> hassers (it would practically never matter for "this is a valid value of >>> the child" if the child message field was set but in turn none of the >>> fields on it were set anyway). >>> >>> With Prost specifically, it may be harder to achieve that pattern >>> naturally since all of the message-typed fields are typed as Option<>, but >>> thats the nature of it and there's some tradeoffs Prost makes in turn for >>> other ergonomic benefits. The V4 official Rust impl is in beta right now, >>> it is an opaque api instead of open structs and so doesn't have the same >>> implication, but it does end up being a design choice that will come with a >>> number of other tradeoffs (some of which are already detailed in our beta >>> documentation here <https://protobuf.dev/reference/rust/>). >>> >>> On Mon, Jun 2, 2025 at 5:05 PM Yordis Prieto <[email protected]> >>> wrote: >>> >>>> > Can you clarify a bit more what it is you are trying to achieve, and >>>> if there are any official implementations where Option/nullability is a >>>> pain point? >>>> >>>> To be clear, I haven't use the official Rust package. I would love to, >>>> the conversation started in Prost, and the short comings there. I am trying >>>> to figure out where to find the official v4 rust version to figure out the >>>> situation there. >>>> >>>> > I think it could not omit the Option<> on required message fields >>>> because the semantic of required is inherently only on wire and not >>>> in-memory: >>>> >>>> I figured, that is where I wish I couldn't even construct a "broken" >>>> type, if a given field is required, then it must be set in order to >>>> construct the message. Otherwise, the type system isn't helping much and we >>>> must figure out the issues at runtime. I am unsure of the details about >>>> parsePartial and memory vs. wire; but I would expect two types based on my >>>> needs (PartialMessage<Message> and Message). >>>> >>>> I guess I am going against what the philosophy is, which it is OK by >>>> me, just feel suboptimal, I want to get the most I can out of the type >>>> system without requiring that much discipline from developers. >>>> >>>> On Monday, June 2, 2025 at 4:00:07 PM UTC-4 Em Rauch wrote: >>>> >>>>> Can you clarify a bit more what it is you are trying to achieve, and >>>>> if there are any official implementations where Option/nullability is a >>>>> pain point? >>>>> >>>>> So in general the official Google Protobuf implementations do not >>>>> expose Option<SomeMessage> or nullable-SomeMessage* in almost any of our >>>>> implementations (as per the page I mentioned on the github issue: >>>>> https://protobuf.dev/design-decisions/nullable-getters-setters/) >>>>> >>>>> If I'm following collectly, on all official implementations you should >>>>> be able to get the behavior you're describing by just declaring the child >>>>> messages and just don't check their has-bits. As far as the observable >>>>> behavior is concerned all of the submessages being eagerly allocated or >>>>> lazily allocated should be an implementation detail. >>>>> >>>>> proto2 `required` is considered strongly regretted and we discourage >>>>> its use, but just considering the semantics of required with Prost shape >>>>> of >>>>> API, I think it could not omit the Option<> on required message fields >>>>> because the semantic of required is inherently only on wire and not >>>>> in-memory: its the normal design that you would first construct a message >>>>> without the required fields set, set them, and then eventually serialize >>>>> it. Even in terms of data that was from parsed data, most Google official >>>>> Protobuf implementations support a "parsePartial" which means "parse these >>>>> bytes but don't enforce required, and then me write the logic against >>>>> hassers directly". >>>>> >>>>> >>>>> On Mon, Jun 2, 2025 at 3:46 PM Yordis Prieto <[email protected]> >>>>> wrote: >>>>> >>>>>> As requested by [@esrauchg](https://github.com/esrauchg), moving the >>>>>> conversation here: >>>>>> https://github.com/tokio-rs/prost/issues/1031#issuecomment-2932003751 >>>>>> . >>>>>> >>>>>> Although this originated in a `prost` discussion, the underlying >>>>>> issue concerns >>>>>> how the official protobuf toolchain treats field presence. The intent >>>>>> here is to >>>>>> clarify the expected stance and behavior of the generator. >>>>>> >>>>>> **What language does this apply to?** >>>>>> >>>>>> Primarily **Rust**, but also interested in **Go** and **TypeScript** >>>>>> as they >>>>>> relate to protobuf code generation. >>>>>> >>>>>> **Describe the problem you are trying to solve.** >>>>>> >>>>>> The generated types are structurally imprecise: **all fields are >>>>>> wrapped in `Option<T>`**, >>>>>> even those marked as `required` (explicitly or implicitly) in the >>>>>> proto >>>>>> definition. >>>>>> >>>>>> This leads to: >>>>>> >>>>>> - Verbose boilerplate (`Some(...)` wrappers) >>>>>> - Increased cognitive overhead >>>>>> - Risk of runtime errors when unwrapping values that should be >>>>>> guaranteed >>>>>> >>>>>> ```rust >>>>>> fn calculate_minimum_bid_increment(cmd: &StartAuctionRun) -> Result< >>>>>> MoneyAmount, Error> { >>>>>> match &cmd.minimum_bid_increment_policy { >>>>>> Some(policy) => match &policy.policy { >>>>>> Some(minimum_bid_increment_policy::Policy::Fixed(fixed_policy)) => { >>>>>> //... >>>>>> } >>>>>> // ... >>>>>> // NOTE: this should never happen, it is required >>>>>> None => Err(Error::MinimumBidIncrementPolicyRequired), >>>>>> }, >>>>>> // NOTE: this should never happen, it is required >>>>>> None => Err(Error::MinimumBidIncrementPolicyRequired), >>>>>> } >>>>>> } >>>>>> ``` >>>>>> >>>>>> Despite the domain clearly requiring this field, the type system does >>>>>> not >>>>>> enforce it. Structurally speaking. >>>>>> >>>>>> We're using protobuf as the canonical schema for all: >>>>>> >>>>>> - Commands >>>>>> - Events >>>>>> - Aggregate Snapshot >>>>>> >>>>>> That applies across multiple language runtimes (via WASM modules) and >>>>>> is >>>>>> critical for: >>>>>> >>>>>> - Schema evolution and change detection >>>>>> - Consistent interop across services >>>>>> - Serialization correctness >>>>>> - Avoiding runtime reflection or manual encoders/decoders >>>>>> >>>>>> We treat protobuf-generated types as the source of truth and only >>>>>> validate >>>>>> `commands` post-deserialization (or via protovalidate extension). >>>>>> >>>>>> Here is the existing callbacks (thus far): >>>>>> >>>>>> ```rust >>>>>> pub type InitialState<State> = fn() -> State; >>>>>> pub type IsTerminal<State> = fn(state: State) -> bool; >>>>>> pub type Decide<State, Command, Event, Error> = >>>>>> fn(state: &State, command: Command) -> Result<Decision<State, Command, >>>>>> Event, Error>, Error>; >>>>>> pub type Evolve<State, Event> = fn(state: &State, event: Event) -> >>>>>> State; >>>>>> >>>>>> pub type GetStreamId<Command> = fn(Command) -> String; >>>>>> pub type IsOrigin<Event> = fn(Event) -> bool; >>>>>> pub type GetEventID<Event> = fn(Event) ->String; >>>>>> ``` >>>>>> >>>>>> **Describe the solution you'd like** >>>>>> >>>>>> The code generator should respect the `required` field modifier in >>>>>> `.proto` >>>>>> definitions, emitting non-optional Rust fields where appropriate. >>>>>> >>>>>> That would: >>>>>> >>>>>> - Better align with schema intent >>>>>> - Eliminate unnecessary `Option<T>` wrappers >>>>>> - Improve safety and ergonomics >>>>>> >>>>>> **Describe alternatives you've considered** >>>>>> >>>>>> Creating application-level copies of protobuf types, but such types >>>>>> have very little value in our context. The distintion between >>>>>> application >>>>>> and serialization matters better little to us, especially when >>>>>> protobuffer >>>>>> files can not break change. >>>>>> >>>>>> **Additional context** >>>>>> >>>>>> - https://github.com/tokio-rs/prost/pull/1286 >>>>>> - https://github.com/tokio-rs/prost/issues/1031 >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Protocol Buffers" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion visit >>>>>> https://groups.google.com/d/msgid/protobuf/b5386744-e770-430e-bccb-959409a05641n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/protobuf/b5386744-e770-430e-bccb-959409a05641n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Protocol Buffers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion visit >>>> https://groups.google.com/d/msgid/protobuf/41ab96b4-4af9-4df2-8d88-fdb9f469f70dn%40googlegroups.com >>>> <https://groups.google.com/d/msgid/protobuf/41ab96b4-4af9-4df2-8d88-fdb9f469f70dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/protobuf/06d1891f-f82a-4ab8-b668-f51c0020a6fbn%40googlegroups.com > <https://groups.google.com/d/msgid/protobuf/06d1891f-f82a-4ab8-b668-f51c0020a6fbn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/protobuf/CAKRmVH98ame%3DHOdaP%3DGszFXegeGrcChoDF3s%2BCXEALzXhp0dpg%40mail.gmail.com.
