To add extra context, I am speaking purely from the end-user perspective experience, I am sure they are technical concerns to take into consideration as someone pointed out, I am bringing my naive experience with the intent to figure out whatever optimal strategy would be.
On Tuesday, June 3, 2025 at 1:12:29 PM UTC-4 Yordis Prieto wrote: > Personally, I do not mind the builder pattern or dealing with > getters/setters; if that is the answer, I am OK with it, my primary concern > is when things are combined and I can construct broken messages (bypassing > builder, and setters) and the builder pattern. > > Extra context, I am optimizing for LLM effectiveness, after doing some > polyglot work around TypeScript, Go, and Rust. I found that Rust type > system avoid hallucination to a point that isn't easy to ignore. For that > to happen, the types coming out of the protobuf and its API needs to avoid > silly mistakes. The same mistakes that some developer make based on > suboptimal assumptions (context). > > Without sidetracking too much, the biggest pain in the Go version is that > everything is a pointer; we can not know if the reason for the pointer is > because we actually want `nil` values or because of memory layout or > technical concern. I wish an Option type existed in Go or added to the > protobuf for this same reason. And well, defaults .... I digress. > > I can strategize around the package, in the worst case, hoping that the > protoc plugin allows some policies to change the behaviour. The interesting > situation is that, reading about the required deprecation a bit more, > definitely wouldn't like stale proto files to blow up if new information is > added but my Processor does't care since it doesn't make use of the info, > and therefore, it doesn't need the new proto file. > Purely speaking about Event Sourcing so my events are immutable, never > breaking change messages. > > What do you recommend me to do here? I need to find a way to test the v4 > asap, I feel it is prudent to move to it > On Tuesday, June 3, 2025 at 8:29:06 AM UTC-4 Em Rauch wrote: > >> > but I would expect two types based on my needs >> (PartialMessage<Message> and Message). >> > I guess I am going against what the philosophy is, which it is OK by >> me, just feel suboptimal, I want to get the most I can out of the type >> system without requiring that much discipline from developers. >> >> This does make perfect sense in general: if required was newly created in >> 2025 it maybe could try to have semantics of always having Builders, being >> able to parsePartial into Builders. Note that not many of the official >> implementations of Protobuf do not use builder pattern though, so anything >> like that would be very difficult to retrofit. >> >> But the context is more that the idea of "blanket enforcement of presence >> of a field at parse time" is considered a misfeature (as discussed in the >> "Required is Strongly Deprecated" section here >> <https://protobuf.dev/programming-guides/proto2/#:~:text=Required%20is%20Strongly%20Deprecated>). >> >> Under the expected usecase patterns of Protobuf, its considered a big >> footgun since many people believe they have a field that is obviously >> always strictly required, but actually a few years later they will want to >> replace it with something different, and hard enforcement at parse time >> makes that evolution almost impossible. >> >> Zooming out a bit: for the usecase you're describing, for the official >> Google supported implementations, I think just doing the normal / natural >> thing should be able to achieve the effective behavior you're thinking >> about. Make a normal message, make it have normal message-typed fields, >> treat it as normal child structs which are just "always there". Treat the >> maybe-lazy-allocation of those inner structs as an implementation detail, >> and treat the has-bit tracking as an implementation detail. If the >> default-initialized child struct isn't a valid value, check in your >> application code against that condition, rather than checking against the >> hassers (it would practically never matter for "this is a valid value of >> the child" if the child message field was set but in turn none of the >> fields on it were set anyway). >> >> With Prost specifically, it may be harder to achieve that pattern >> naturally since all of the message-typed fields are typed as Option<>, but >> thats the nature of it and there's some tradeoffs Prost makes in turn for >> other ergonomic benefits. The V4 official Rust impl is in beta right now, >> it is an opaque api instead of open structs and so doesn't have the same >> implication, but it does end up being a design choice that will come with a >> number of other tradeoffs (some of which are already detailed in our beta >> documentation here <https://protobuf.dev/reference/rust/>). >> >> On Mon, Jun 2, 2025 at 5:05 PM Yordis Prieto <[email protected]> wrote: >> >>> > Can you clarify a bit more what it is you are trying to achieve, and >>> if there are any official implementations where Option/nullability is a >>> pain point? >>> >>> To be clear, I haven't use the official Rust package. I would love to, >>> the conversation started in Prost, and the short comings there. I am trying >>> to figure out where to find the official v4 rust version to figure out the >>> situation there. >>> >>> > I think it could not omit the Option<> on required message fields >>> because the semantic of required is inherently only on wire and not >>> in-memory: >>> >>> I figured, that is where I wish I couldn't even construct a "broken" >>> type, if a given field is required, then it must be set in order to >>> construct the message. Otherwise, the type system isn't helping much and we >>> must figure out the issues at runtime. I am unsure of the details about >>> parsePartial and memory vs. wire; but I would expect two types based on my >>> needs (PartialMessage<Message> and Message). >>> >>> I guess I am going against what the philosophy is, which it is OK by me, >>> just feel suboptimal, I want to get the most I can out of the type system >>> without requiring that much discipline from developers. >>> >>> On Monday, June 2, 2025 at 4:00:07 PM UTC-4 Em Rauch wrote: >>> >>>> Can you clarify a bit more what it is you are trying to achieve, and if >>>> there are any official implementations where Option/nullability is a pain >>>> point? >>>> >>>> So in general the official Google Protobuf implementations do not >>>> expose Option<SomeMessage> or nullable-SomeMessage* in almost any of our >>>> implementations (as per the page I mentioned on the github issue: >>>> https://protobuf.dev/design-decisions/nullable-getters-setters/) >>>> >>>> If I'm following collectly, on all official implementations you should >>>> be able to get the behavior you're describing by just declaring the child >>>> messages and just don't check their has-bits. As far as the observable >>>> behavior is concerned all of the submessages being eagerly allocated or >>>> lazily allocated should be an implementation detail. >>>> >>>> proto2 `required` is considered strongly regretted and we discourage >>>> its use, but just considering the semantics of required with Prost shape >>>> of >>>> API, I think it could not omit the Option<> on required message fields >>>> because the semantic of required is inherently only on wire and not >>>> in-memory: its the normal design that you would first construct a message >>>> without the required fields set, set them, and then eventually serialize >>>> it. Even in terms of data that was from parsed data, most Google official >>>> Protobuf implementations support a "parsePartial" which means "parse these >>>> bytes but don't enforce required, and then me write the logic against >>>> hassers directly". >>>> >>>> >>>> On Mon, Jun 2, 2025 at 3:46 PM Yordis Prieto <[email protected]> >>>> wrote: >>>> >>>>> As requested by [@esrauchg](https://github.com/esrauchg), moving the >>>>> conversation here: >>>>> https://github.com/tokio-rs/prost/issues/1031#issuecomment-2932003751. >>>>> >>>>> Although this originated in a `prost` discussion, the underlying >>>>> issue concerns >>>>> how the official protobuf toolchain treats field presence. The intent >>>>> here is to >>>>> clarify the expected stance and behavior of the generator. >>>>> >>>>> **What language does this apply to?** >>>>> >>>>> Primarily **Rust**, but also interested in **Go** and **TypeScript** >>>>> as they >>>>> relate to protobuf code generation. >>>>> >>>>> **Describe the problem you are trying to solve.** >>>>> >>>>> The generated types are structurally imprecise: **all fields are >>>>> wrapped in `Option<T>`**, >>>>> even those marked as `required` (explicitly or implicitly) in the >>>>> proto >>>>> definition. >>>>> >>>>> This leads to: >>>>> >>>>> - Verbose boilerplate (`Some(...)` wrappers) >>>>> - Increased cognitive overhead >>>>> - Risk of runtime errors when unwrapping values that should be >>>>> guaranteed >>>>> >>>>> ```rust >>>>> fn calculate_minimum_bid_increment(cmd: &StartAuctionRun) -> Result< >>>>> MoneyAmount, Error> { >>>>> match &cmd.minimum_bid_increment_policy { >>>>> Some(policy) => match &policy.policy { >>>>> Some(minimum_bid_increment_policy::Policy::Fixed(fixed_policy)) => { >>>>> //... >>>>> } >>>>> // ... >>>>> // NOTE: this should never happen, it is required >>>>> None => Err(Error::MinimumBidIncrementPolicyRequired), >>>>> }, >>>>> // NOTE: this should never happen, it is required >>>>> None => Err(Error::MinimumBidIncrementPolicyRequired), >>>>> } >>>>> } >>>>> ``` >>>>> >>>>> Despite the domain clearly requiring this field, the type system does >>>>> not >>>>> enforce it. Structurally speaking. >>>>> >>>>> We're using protobuf as the canonical schema for all: >>>>> >>>>> - Commands >>>>> - Events >>>>> - Aggregate Snapshot >>>>> >>>>> That applies across multiple language runtimes (via WASM modules) and >>>>> is >>>>> critical for: >>>>> >>>>> - Schema evolution and change detection >>>>> - Consistent interop across services >>>>> - Serialization correctness >>>>> - Avoiding runtime reflection or manual encoders/decoders >>>>> >>>>> We treat protobuf-generated types as the source of truth and only >>>>> validate >>>>> `commands` post-deserialization (or via protovalidate extension). >>>>> >>>>> Here is the existing callbacks (thus far): >>>>> >>>>> ```rust >>>>> pub type InitialState<State> = fn() -> State; >>>>> pub type IsTerminal<State> = fn(state: State) -> bool; >>>>> pub type Decide<State, Command, Event, Error> = >>>>> fn(state: &State, command: Command) -> Result<Decision<State, Command, >>>>> Event, Error>, Error>; >>>>> pub type Evolve<State, Event> = fn(state: &State, event: Event) -> >>>>> State; >>>>> >>>>> pub type GetStreamId<Command> = fn(Command) -> String; >>>>> pub type IsOrigin<Event> = fn(Event) -> bool; >>>>> pub type GetEventID<Event> = fn(Event) ->String; >>>>> ``` >>>>> >>>>> **Describe the solution you'd like** >>>>> >>>>> The code generator should respect the `required` field modifier in >>>>> `.proto` >>>>> definitions, emitting non-optional Rust fields where appropriate. >>>>> >>>>> That would: >>>>> >>>>> - Better align with schema intent >>>>> - Eliminate unnecessary `Option<T>` wrappers >>>>> - Improve safety and ergonomics >>>>> >>>>> **Describe alternatives you've considered** >>>>> >>>>> Creating application-level copies of protobuf types, but such types >>>>> have very little value in our context. The distintion between >>>>> application >>>>> and serialization matters better little to us, especially when >>>>> protobuffer >>>>> files can not break change. >>>>> >>>>> **Additional context** >>>>> >>>>> - https://github.com/tokio-rs/prost/pull/1286 >>>>> - https://github.com/tokio-rs/prost/issues/1031 >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Protocol Buffers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion visit >>>>> https://groups.google.com/d/msgid/protobuf/b5386744-e770-430e-bccb-959409a05641n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/protobuf/b5386744-e770-430e-bccb-959409a05641n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Protocol Buffers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion visit >>> https://groups.google.com/d/msgid/protobuf/41ab96b4-4af9-4df2-8d88-fdb9f469f70dn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/protobuf/41ab96b4-4af9-4df2-8d88-fdb9f469f70dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/protobuf/06d1891f-f82a-4ab8-b668-f51c0020a6fbn%40googlegroups.com.
