I regret opening the issue ... my intent was to discuss things, what happened is that I created the post here and it disappeared, well turns out that eventually it was here all along ... anyway, moving the conversation there then
On Wednesday, June 4, 2025 at 9:01:47 AM UTC-4 Em Rauch wrote: > Lets move this discussion to the associated GH Issue that you opened here > https://github.com/protocolbuffers/protobuf/issues/22075 > > Thanks! > > On Tue, Jun 3, 2025 at 10:21 PM Yordis Prieto <[email protected]> wrote: > >> To add extra context, I am speaking purely from the end-user perspective >> experience, I am sure they are technical concerns to take into >> consideration as someone pointed out, I am bringing my naive experience >> with the intent to figure out whatever optimal strategy would be. >> >> On Tuesday, June 3, 2025 at 1:12:29 PM UTC-4 Yordis Prieto wrote: >> >>> Personally, I do not mind the builder pattern or dealing with >>> getters/setters; if that is the answer, I am OK with it, my primary concern >>> is when things are combined and I can construct broken messages (bypassing >>> builder, and setters) and the builder pattern. >>> >>> Extra context, I am optimizing for LLM effectiveness, after doing some >>> polyglot work around TypeScript, Go, and Rust. I found that Rust type >>> system avoid hallucination to a point that isn't easy to ignore. For that >>> to happen, the types coming out of the protobuf and its API needs to avoid >>> silly mistakes. The same mistakes that some developer make based on >>> suboptimal assumptions (context). >>> >>> Without sidetracking too much, the biggest pain in the Go version is >>> that everything is a pointer; we can not know if the reason for the pointer >>> is because we actually want `nil` values or because of memory layout or >>> technical concern. I wish an Option type existed in Go or added to the >>> protobuf for this same reason. And well, defaults .... I digress. >>> >>> I can strategize around the package, in the worst case, hoping that the >>> protoc plugin allows some policies to change the behaviour. The interesting >>> situation is that, reading about the required deprecation a bit more, >>> definitely wouldn't like stale proto files to blow up if new information is >>> added but my Processor does't care since it doesn't make use of the info, >>> and therefore, it doesn't need the new proto file. >>> Purely speaking about Event Sourcing so my events are immutable, never >>> breaking change messages. >>> >>> What do you recommend me to do here? I need to find a way to test the v4 >>> asap, I feel it is prudent to move to it >>> On Tuesday, June 3, 2025 at 8:29:06 AM UTC-4 Em Rauch wrote: >>> >>>> > but I would expect two types based on my needs >>>> (PartialMessage<Message> and Message). >>>> > I guess I am going against what the philosophy is, which it is OK by >>>> me, just feel suboptimal, I want to get the most I can out of the type >>>> system without requiring that much discipline from developers. >>>> >>>> This does make perfect sense in general: if required was newly created >>>> in 2025 it maybe could try to have semantics of always having Builders, >>>> being able to parsePartial into Builders. Note that not many of the >>>> official implementations of Protobuf do not use builder pattern though, so >>>> anything like that would be very difficult to retrofit. >>>> >>>> But the context is more that the idea of "blanket enforcement of >>>> presence of a field at parse time" is considered a misfeature (as >>>> discussed >>>> in the "Required is Strongly Deprecated" section here >>>> <https://protobuf.dev/programming-guides/proto2/#:~:text=Required%20is%20Strongly%20Deprecated>). >>>> >>>> Under the expected usecase patterns of Protobuf, its considered a big >>>> footgun since many people believe they have a field that is obviously >>>> always strictly required, but actually a few years later they will want to >>>> replace it with something different, and hard enforcement at parse time >>>> makes that evolution almost impossible. >>>> >>>> Zooming out a bit: for the usecase you're describing, for the official >>>> Google supported implementations, I think just doing the normal / natural >>>> thing should be able to achieve the effective behavior you're thinking >>>> about. Make a normal message, make it have normal message-typed fields, >>>> treat it as normal child structs which are just "always there". Treat the >>>> maybe-lazy-allocation of those inner structs as an implementation detail, >>>> and treat the has-bit tracking as an implementation detail. If the >>>> default-initialized child struct isn't a valid value, check in your >>>> application code against that condition, rather than checking against the >>>> hassers (it would practically never matter for "this is a valid value of >>>> the child" if the child message field was set but in turn none of the >>>> fields on it were set anyway). >>>> >>>> With Prost specifically, it may be harder to achieve that pattern >>>> naturally since all of the message-typed fields are typed as Option<>, but >>>> thats the nature of it and there's some tradeoffs Prost makes in turn for >>>> other ergonomic benefits. The V4 official Rust impl is in beta right now, >>>> it is an opaque api instead of open structs and so doesn't have the same >>>> implication, but it does end up being a design choice that will come with >>>> a >>>> number of other tradeoffs (some of which are already detailed in our beta >>>> documentation here <https://protobuf.dev/reference/rust/>). >>>> >>>> On Mon, Jun 2, 2025 at 5:05 PM Yordis Prieto <[email protected]> >>>> wrote: >>>> >>>>> > Can you clarify a bit more what it is you are trying to achieve, and >>>>> if there are any official implementations where Option/nullability is a >>>>> pain point? >>>>> >>>>> To be clear, I haven't use the official Rust package. I would love to, >>>>> the conversation started in Prost, and the short comings there. I am >>>>> trying >>>>> to figure out where to find the official v4 rust version to figure out >>>>> the >>>>> situation there. >>>>> >>>>> > I think it could not omit the Option<> on required message fields >>>>> because the semantic of required is inherently only on wire and not >>>>> in-memory: >>>>> >>>>> I figured, that is where I wish I couldn't even construct a "broken" >>>>> type, if a given field is required, then it must be set in order to >>>>> construct the message. Otherwise, the type system isn't helping much and >>>>> we >>>>> must figure out the issues at runtime. I am unsure of the details about >>>>> parsePartial and memory vs. wire; but I would expect two types based on >>>>> my >>>>> needs (PartialMessage<Message> and Message). >>>>> >>>>> I guess I am going against what the philosophy is, which it is OK by >>>>> me, just feel suboptimal, I want to get the most I can out of the type >>>>> system without requiring that much discipline from developers. >>>>> >>>>> On Monday, June 2, 2025 at 4:00:07 PM UTC-4 Em Rauch wrote: >>>>> >>>>>> Can you clarify a bit more what it is you are trying to achieve, and >>>>>> if there are any official implementations where Option/nullability is a >>>>>> pain point? >>>>>> >>>>>> So in general the official Google Protobuf implementations do not >>>>>> expose Option<SomeMessage> or nullable-SomeMessage* in almost any of our >>>>>> implementations (as per the page I mentioned on the github issue: >>>>>> https://protobuf.dev/design-decisions/nullable-getters-setters/) >>>>>> >>>>>> If I'm following collectly, on all official implementations you >>>>>> should be able to get the behavior you're describing by just declaring >>>>>> the >>>>>> child messages and just don't check their has-bits. As far as the >>>>>> observable behavior is concerned all of the submessages being eagerly >>>>>> allocated or lazily allocated should be an implementation detail. >>>>>> >>>>>> proto2 `required` is considered strongly regretted and we discourage >>>>>> its use, but just considering the semantics of required with Prost shape >>>>>> of >>>>>> API, I think it could not omit the Option<> on required message fields >>>>>> because the semantic of required is inherently only on wire and not >>>>>> in-memory: its the normal design that you would first construct a >>>>>> message >>>>>> without the required fields set, set them, and then eventually serialize >>>>>> it. Even in terms of data that was from parsed data, most Google >>>>>> official >>>>>> Protobuf implementations support a "parsePartial" which means "parse >>>>>> these >>>>>> bytes but don't enforce required, and then me write the logic against >>>>>> hassers directly". >>>>>> >>>>>> >>>>>> On Mon, Jun 2, 2025 at 3:46 PM Yordis Prieto <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> As requested by [@esrauchg](https://github.com/esrauchg), moving >>>>>>> the conversation here: >>>>>>> https://github.com/tokio-rs/prost/issues/1031#issuecomment-2932003751 >>>>>>> . >>>>>>> >>>>>>> Although this originated in a `prost` discussion, the underlying >>>>>>> issue concerns >>>>>>> how the official protobuf toolchain treats field presence. The >>>>>>> intent here is to >>>>>>> clarify the expected stance and behavior of the generator. >>>>>>> >>>>>>> **What language does this apply to?** >>>>>>> >>>>>>> Primarily **Rust**, but also interested in **Go** and **TypeScript** >>>>>>> as they >>>>>>> relate to protobuf code generation. >>>>>>> >>>>>>> **Describe the problem you are trying to solve.** >>>>>>> >>>>>>> The generated types are structurally imprecise: **all fields are >>>>>>> wrapped in `Option<T>`**, >>>>>>> even those marked as `required` (explicitly or implicitly) in the >>>>>>> proto >>>>>>> definition. >>>>>>> >>>>>>> This leads to: >>>>>>> >>>>>>> - Verbose boilerplate (`Some(...)` wrappers) >>>>>>> - Increased cognitive overhead >>>>>>> - Risk of runtime errors when unwrapping values that should be >>>>>>> guaranteed >>>>>>> >>>>>>> ```rust >>>>>>> fn calculate_minimum_bid_increment(cmd: &StartAuctionRun) -> Result< >>>>>>> MoneyAmount, Error> { >>>>>>> match &cmd.minimum_bid_increment_policy { >>>>>>> Some(policy) => match &policy.policy { >>>>>>> Some(minimum_bid_increment_policy::Policy::Fixed(fixed_policy)) => { >>>>>>> //... >>>>>>> } >>>>>>> // ... >>>>>>> // NOTE: this should never happen, it is required >>>>>>> None => Err(Error::MinimumBidIncrementPolicyRequired), >>>>>>> }, >>>>>>> // NOTE: this should never happen, it is required >>>>>>> None => Err(Error::MinimumBidIncrementPolicyRequired), >>>>>>> } >>>>>>> } >>>>>>> ``` >>>>>>> >>>>>>> Despite the domain clearly requiring this field, the type system >>>>>>> does not >>>>>>> enforce it. Structurally speaking. >>>>>>> >>>>>>> We're using protobuf as the canonical schema for all: >>>>>>> >>>>>>> - Commands >>>>>>> - Events >>>>>>> - Aggregate Snapshot >>>>>>> >>>>>>> That applies across multiple language runtimes (via WASM modules) >>>>>>> and is >>>>>>> critical for: >>>>>>> >>>>>>> - Schema evolution and change detection >>>>>>> - Consistent interop across services >>>>>>> - Serialization correctness >>>>>>> - Avoiding runtime reflection or manual encoders/decoders >>>>>>> >>>>>>> We treat protobuf-generated types as the source of truth and only >>>>>>> validate >>>>>>> `commands` post-deserialization (or via protovalidate extension). >>>>>>> >>>>>>> Here is the existing callbacks (thus far): >>>>>>> >>>>>>> ```rust >>>>>>> pub type InitialState<State> = fn() -> State; >>>>>>> pub type IsTerminal<State> = fn(state: State) -> bool; >>>>>>> pub type Decide<State, Command, Event, Error> = >>>>>>> fn(state: &State, command: Command) -> Result<Decision<State, >>>>>>> Command, Event, Error>, Error>; >>>>>>> pub type Evolve<State, Event> = fn(state: &State, event: Event) -> >>>>>>> State; >>>>>>> >>>>>>> pub type GetStreamId<Command> = fn(Command) -> String; >>>>>>> pub type IsOrigin<Event> = fn(Event) -> bool; >>>>>>> pub type GetEventID<Event> = fn(Event) ->String; >>>>>>> ``` >>>>>>> >>>>>>> **Describe the solution you'd like** >>>>>>> >>>>>>> The code generator should respect the `required` field modifier in >>>>>>> `.proto` >>>>>>> definitions, emitting non-optional Rust fields where appropriate. >>>>>>> >>>>>>> That would: >>>>>>> >>>>>>> - Better align with schema intent >>>>>>> - Eliminate unnecessary `Option<T>` wrappers >>>>>>> - Improve safety and ergonomics >>>>>>> >>>>>>> **Describe alternatives you've considered** >>>>>>> >>>>>>> Creating application-level copies of protobuf types, but such types >>>>>>> have very little value in our context. The distintion between >>>>>>> application >>>>>>> and serialization matters better little to us, especially when >>>>>>> protobuffer >>>>>>> files can not break change. >>>>>>> >>>>>>> **Additional context** >>>>>>> >>>>>>> - https://github.com/tokio-rs/prost/pull/1286 >>>>>>> - https://github.com/tokio-rs/prost/issues/1031 >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Protocol Buffers" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion visit >>>>>>> https://groups.google.com/d/msgid/protobuf/b5386744-e770-430e-bccb-959409a05641n%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/protobuf/b5386744-e770-430e-bccb-959409a05641n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Protocol Buffers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion visit >>>>> https://groups.google.com/d/msgid/protobuf/41ab96b4-4af9-4df2-8d88-fdb9f469f70dn%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/protobuf/41ab96b4-4af9-4df2-8d88-fdb9f469f70dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion visit >> https://groups.google.com/d/msgid/protobuf/06d1891f-f82a-4ab8-b668-f51c0020a6fbn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/protobuf/06d1891f-f82a-4ab8-b668-f51c0020a6fbn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/protobuf/e6f6c04d-f3d7-4de1-af82-13fd7991f16dn%40googlegroups.com.
