Re: [protobuf] Re: Compatibility Issue + Max value for the indices/field numbers + are high field number slower?
> Separate embedded messages would involve switches for the code generation (official build vs modded build) but could be doable, and maybe it is even a bit cleaner. > The CRC thing would have meant an uniform solution, but maybe namespacing the modded messages isn't the worst idea. I'd be really surprised if a CRC ended up being more performant than a single branch to see if the mod message exists if I'm understanding your situation correctly. On Mon, Jun 20, 2016 at 3:36 PM, a_teammate wrote: > Am Montag, 20. Juni 2016 20:50:17 UTC+2 schrieb Jeremy Ong: > >> https://developers.google.com/protocol-buffers/docs/encoding#structure >> >> Protobuf messages are associative arrays of key value pairs, where the >> key is a union of the field number encoded as a varint and the value wire >> type (union operator being a left shift of the field number by 3 bits). >> Because the field number is variable width, it's theoretical size is >> unbounded but is likely implementation dependent as some programming >> languages super arbitrarily large numbers, and other implementations might >> use fixed width types to represent the field for convenience. >> > > Ah! thank you for pointing that out, that lets me understand the structure > of protobuf much better, and especially thanks for the link! > > Am Montag, 20. Juni 2016 21:47:39 UTC+2 schrieb Feng Xiao: >> >> >> On Sun, Jun 19, 2016 at 7:56 AM, a_teammate wrote: >>> >>> The problem is however that we don't only want forward backward >>> compability between server and client, but also sideways: >>> e.g. person A introduced message index 2 and also did person B, both >>> meaning totally different things, but it should be recognized and >>> ignored(/or maybe even accepted?! if we find a way to do this) >>> >> If person A already added a field with field number 2, how can person B >> add another one with the same field number? Do you have multiple copies of >> the .proto files and they are not synced? >> > > Well we're an open-source multiplayer-game and highly encourage modding: > So it would be cool if a modded client meeting a modded server could work > together (could be doable since our scripting uses a similar/the same API, > if that's smart security-wise is another question ofc) > The protobuf code gets generated here from code reflection, so people > don't need to deal with syncing themselves. That's where I meant would the > CRCing would come into play, > well I assume I wasn't quite clear about that initially. > > Am Montag, 20. Juni 2016 20:50:17 UTC+2 schrieb Jeremy Ong: > >> If you are trying to prevent collisions between two people modifying the >> key space, I recommend making separate embedded messages so there is no >> chance of collision. CRC-ing field numbers is just too heavy weight for >> what it is you're trying to do in my opinion. >> > > Separate embedded messages would involve switches for the code generation > (official build vs modded build) but could be doable, and maybe it is even > a bit cleaner. > The CRC thing would have meant an uniform solution, but maybe namespacing > the modded messages isn't the worst idea. > > Another alternative would be to sync the metadata initially (so the modded > server deals with the input according to the clients description of its own > protocol, not on the servers assumption), well we've got the choice :) > > Am Montag, 20. Juni 2016 20:50:17 UTC+2 schrieb Jeremy Ong: > >> Regarding performance, varint encoding/decoding time is O(n) in the byte >> length of the result. Whether this is important depends on your application >> of course, but you're really better off understanding how the encoding >> works so you can do a quick back of the envelope guess to see if it >> matters, followed by actually benchmarking if performance is really that >> important to you. >> >> > Yeah I see, well benchmarking will come into play sooner or later thats > for sure! > > Am Montag, 20. Juni 2016 21:47:39 UTC+2 schrieb Feng Xiao: >> >> >> >> On Sun, Jun 19, 2016 at 7:56 AM, a_teammate wrote: >> >>> 1) what is the maximum value of the protobuf field numbers? >>> >> The range of valid field numbers is 1 to 2^29 - 1: >> https://developers.google.com/protocol-buffers/docs/proto#assigning-tags >> >> Some field numbers in this range are reserved so you will need to account >> for those as well. >> > > Ah nice! so if our benchmarks suggests a negligible performane impact, > hashing could be doable, since that area see
Re: [protobuf] Compatibility Issue + Max value for the indices/field numbers + are high field number slower?
all this stuff :) >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+unsubscr...@googlegroups.com. >> To post to this group, send email to protobuf@googlegroups.com. >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- Jeremy Ong PlexChat CTO 650.400.6453 -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Compatibility Issue + Max value for the indices/field numbers + are high field number slower?
https://developers.google.com/protocol-buffers/docs/encoding#structure
Protobuf messages are associative arrays of key value pairs, where the key
is a union of the field number encoded as a varint and the value wire type
(union operator being a left shift of the field number by 3 bits). Because
the field number is variable width, it's theoretical size is unbounded but
is likely implementation dependent as some programming languages super
arbitrarily large numbers, and other implementations might use fixed width
types to represent the field for convenience.
If you are trying to prevent collisions between two people modifying the
key space, I recommend making separate embedded messages so there is no
chance of collision. CRC-ing field numbers is just too heavy weight for
what it is you're trying to do in my opinion.
Regarding performance, varint encoding/decoding time is O(n) in the byte
length of the result. Whether this is important depends on your application
of course, but you're really better off understanding how the encoding
works so you can do a quick back of the envelope guess to see if it
matters, followed by actually benchmarking if performance is really that
important to you.
On Sun, Jun 19, 2016 at 7:56 AM, a_teammate wrote:
> Hey there,
>
> This might a stupid question, but i haven't found anything certain in the
> docs/specs about that:
>
> Our main goal is actually to keep compatibility while syncing a tree.
>
> The protocol is actually just one giant oneof containing all possible
> paths for the tree:
>
> message TreeNodeChanged {
>
>oneof key {
>
> sometype path_to_node1 = 1;
>
> sometype path_to_node2 = 2;
>
> ...
>
> }
>
> }
>
>
> and thats already working.
> The problem is however that we don't only want forward backward
> compability between server and client, but also sideways:
> e.g. person A introduced message index 2 and also did person B, both
> meaning totally different things, but it should be recognized and
> ignored(/or maybe even accepted?! if we find a way to do this)
>
>
> So the idea of my mate was to make the field number a hash of the
> path_to_node!
> Candidates are e.g. a 32bit FNV-hash or maybe an adapted CRC32. This would
> both mean field numbers in a pretty high area.
>
> Maybe we're going the totally wrong way here but following this path leads
> to the following issues:
>
>
> 1) what is the maximum value of the protobuf field numbers?
>
> In the proto language specification
> <https://developers.google.com/protocol-buffers/docs/reference/proto3-spec#fields>
> it simply says its of type "intLit" and intLit is:
>
> intLit = decimalLit | octalLit | hexLit
>> decimalLit = ( "1" … "9" ) { decimalDigit }
>> octalLit = "0" { octalDigit }
>> hexLit = "0" ( "x" | "X" ) hexDigit { hexDigit }
>>
>>
> So this means only decimal and hexadecimal values are actually allowed
> doesnt it?
> Then however given:
>
> decimalDigit = "0" … "9"
> hexDigit = "0" … "9" | "A" … "F" | "a" … "f"
>
> Means it has different limits for hex and int notation, is that correct?
>
> I mean:
>
> the max value for decimalLit is one billion-1 : "999 999 999"
> according to this specs, which fits fine in a 32bit integer (with 30bits
> set)
>
> but for base 16 its allowed length is 16! which would be awesome cause that
> would mean an allowed integer size of 64bit.
>
> So which one is true? Both?
>
> which leads to issue 2:
>
> 1) are there issues with high field numbers
>
> And are they even tested at all?
>
> I've red elsewhere
> <https://developers.google.com/protocol-buffers/docs/proto#customoptions>
> that *"we have used field numbers in the range 5-9. This range is
> reserved for internal use within individual organizations"*
> which would suggest that even values above 50 000 are uncommen ..
>
> Furthermore some people mentioned high values would suffer from beeing
> less performant, but: in how far is that relevant? Only because the index
> number consumes slightly more memory?
>
>
> Well: Maybe we totally ask the wrong questions here and theres a much
> simpler logic already introduced or invented to better make protobuf
> message version independent, if yes we would be happy to hear them!
>
> Thanks in advance and for reading all this stuff :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receivin
Re: [protobuf] sanitizer/asan_interface.h: No such file or directory when building with -fsanitize=address
Those debug sanitizers are part of the LLVM suite of tools. http://compiler-rt.llvm.org/ On Wed, Jun 1, 2016 at 12:06 PM, Benjamin Sapp wrote: > Hi, I ran the following: > > $ git clone https://github.com/google/protobuf.git > $ cd protobuf > $ bazel build --copt -fsanitize=address --linkopt -fsanitize=address --copt > -DADDRESS_SANITIZER=1 --compilation_mode=fastbuild --verbose_failures -- > curses=no :protobuf > INFO: Loading... > INFO: Found 1 target... > INFO: Building... > ERROR: /tmp/protobuf/BUILD:71:1: C++ compilation of rule > '//:protobuf_lite' failed: namespace-sandbox failed: error executing > command > (cd /home/bensapp/.cache/bazel/_bazel_bensapp/ > 9d77888f7e298040819668f1b7f626a8/protobuf && \ > exec env - \ > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ > /home/bensapp/.cache/bazel/_bazel_bensapp/ > 9d77888f7e298040819668f1b7f626a8/protobuf/_bin/namespace-sandbox @/home/ > bensapp/.cache/bazel/_bazel_bensapp/9d77888f7e298040819668f1b7f626a8/ > protobuf/bazel-sandbox/7e30b972-a4b9-427a-add1-c307b1088902-0.params -- / > usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector - > Wall -Wunused-but-set-parameter -Wno-free-nonheap-object > -fno-omit-frame-pointer > '-fsanitize=address' '-DADDRESS_SANITIZER=1' '-std=c++0x' -iquote . -iquote > bazel-out/local_linux-fastbuild/genfiles -iquote external/bazel_tools -iquote > bazel-out/local_linux-fastbuild/genfiles/external/bazel_tools -isystem > src -isystem bazel-out/local_linux-fastbuild/genfiles/src -isystem > external/bazel_tools/tools/cpp/gcc3 -DHAVE_PTHREAD -Wall -Wwrite-strings - > Woverloaded-virtual -Wno-sign-compare '-Wno-error=unused-function' -no- > canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined > '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' > '-D__TIME__="redacted"' > '-frandom-seed=bazel-out/local_linux-fastbuild/bin/_objs/protobuf_lite/src/google/protobuf/arena.pic.o' > -MD -MF bazel-out/local_linux-fastbuild/bin/_objs/protobuf_lite/src/google > /protobuf/arena.pic.d -fPIC -c src/google/protobuf/arena.cc -o bazel-out/ > local_linux-fastbuild/bin/_objs/protobuf_lite/src/google/protobuf/arena. > pic.o). > src/google/protobuf/arena.cc:35:38: fatal error: sanitizer/asan_interface. > h: No such file or directory > #include > ^ > compilation terminated. > INFO: Building complete. > Target //:protobuf failed to build > INFO: Elapsed time: 0.525s, Critical Path: 0.34s > > The only place I have sanitizer/asan_interface.h on my machine is > /usr/lib/llvm-3.6/lib/clang/3.6.0/include/sanitizer/asan_interface.h > > but I'm using (and would like to keep using) gcc. > > Any advice, or is there a proper place to file a bug? > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- Jeremy Ong PlexChat CTO 650.400.6453 -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] can protobuf3 be used with protobuf2?
The handling of unknown fields is precisely where the difference lies and sticking to proto2 is indeed the plan. However, because you have many clients forced to stick with proto2, you inevitably bifurcate the users, and implementations are now forced to support two standards. Proto3 was designed supposedly to *simplify* the implementation of the protocol, but this is moot when the old implementation is necessary anyways. Honestly, it feels like a repeat of a python2 and python3 thingy (although admittedly perhaps not as serious). As for JSON fields not being keyed to field number, that was a choice made with the encoding and has nothing to do with JSON itself. Do those JSON blobs also lose forward compatibility when the schema changes? Thanks for the discussion, J On Thu, May 19, 2016 at 7:01 PM, Tim Kientzle wrote: > > > On May 18, 2016, at 10:01 PM, Jeremy Ong wrote: > > > > Why does adding JSON require dropping unknown fields? So long as fields > are keyed to field number, I don't see why the JSON encoding requires > special treatment with respect to the binary one. > > JSON fields aren’t keyed to field number. They’re keyed to field name. > > Even apart from field naming, JSON and protobuf wire formats don’t > correspond 1:1, so you can’t even correctly translate the primitive values > without the schema. > > > However, proto3 makes breaks in compatibility with the underlying data > (proto2 encoded), which is where I find myself in disagreement. > > What do you think is different? Having decoded (by hand) a fair bit of > proto2 and proto3 data, they look exactly the same to me. > > As I mentioned before, if preserving unknown fields is essential for you, > you should stick with proto2. It’s still around and will be for a long > time. > Cheers, > > Tim > > -- Jeremy Ong PlexChat CTO 650.400.6453 -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] can protobuf3 be used with protobuf2?
Why does adding JSON require dropping unknown fields? So long as fields are keyed to field number, I don't see why the JSON encoding requires special treatment with respect to the binary one. I can understand how transitioning between major versions may require breaks in compatibility. However, proto3 makes breaks in compatibility with the underlying data (proto2 encoded), which is where I find myself in disagreement. Why not preserve data compatibility so that overtime, proto2 users can migrate? Unknown field handling (or lack thereof) is honestly the one I find most egregious. On Wed, May 18, 2016 at 9:51 PM, Tim Kientzle wrote: > After studying proto3 pretty carefully, I’ve come around quite a bit on > these changes: > > I believe adding JSON requires dropping unknown fields. You simply cannot > preserve unknown fields and properly support multiple encodings. > > I’m less sure about replacing extension support with Any. Extensions have > some ugly problems, but I feel the current spec for Any also has some real > drawbacks. > > Removing field presence is a subtle issue, but I’m starting to suspect it > was actually a very good change. It reduces the generated code and the > workaround of using a single-element oneof is cleaner than it might sound. > In essence, a single-element oneof is just a way to explicitly declare that > you want to track presence for that field. And oneof is supported by > proto2 now, so you can use that technique there as well. > > Finally, remember that proto2 is not going away: If proto2 assumptions > are deeply baked into your systems, you can keep using it. protoc will > continue to support it for a very long time. > > Cheers, > > Tim > > > > > On May 18, 2016, at 1:33 PM, Jeremy Ong wrote: > > > > Big fan of 4, 5, 6, and 7. Huge un-fan of 2, and 3. I am mixed on 1 > because I love the removal of required fields, hate the removal of field > presence. All the changes I dislike are significant losses in functionality > and break compatibility with existing users of proto2 and I'd be interested > to understand why "ease of implementation" is good justification for this > break in compatibility and what I perceive to be a loss in functionality. > > > > On Wed, May 18, 2016 at 11:18 AM, 'Feng Xiao' via Protocol Buffers < > protobuf@googlegroups.com> wrote: > > > > > > On Wed, May 18, 2016 at 9:27 AM, Artem Kazakov > wrote: > > +1 > > Yes, a checklist would be extremely helpful. > > > > > > On Friday, April 29, 2016 at 5:04:56 PM UTC-4, Kostiantyn Shchepanovskyi > wrote: > > It would be nice to have a migration guide (checklist) somewhere, like: > > > > 1. All fields should be optional. > > 2. Do not use custom defailt values. > > 3. All enums should have first element with tag = 0. > > 4. Do not use extension for anything except custom options. > > > > Something else? > > In 3.0.0-alpha-1 release note there is a list of main proto3 changes: > > The following are the main new features in language version 3: > > > > • Removal of field presence logic for primitive value fields, > removal of required fields, and removal of default values. This makes > proto3 significantly easier to implement with open struct representations, > as in languages like Android Java, Objective C, or Go. > > • Removal of unknown fields. > > • Removal of extensions, which are instead replaced by a new > standard type called Any. > > • Fix semantics for unknown enum values. > > • Addition of maps. > > • Addition of a small set of standard types for representation of > time, dynamic data, etc. > > • A well-defined encoding in JSON as an alternative to binary > proto encoding. > > > > > > > > > > On Friday, April 29, 2016 at 1:18:12 AM UTC+3, Feng Xiao wrote: > > > > > > On Tue, Apr 26, 2016 at 7:04 PM, Bo Gao wrote: > > suppose server side is updating into protobuf3, but client side still > use protobuf2, can then communicate will? > > Yes, as long as you only use proto3 features, they are wire compatible. > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Protocol Buffers" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to protobuf+u...@googlegroups.com. > > To post to this group, send email to prot...@googlegroups.com. > > Visit this group at https://groups.google.com/group/protobuf. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You receiv
Re: [protobuf] can protobuf3 be used with protobuf2?
Big fan of 4, 5, 6, and 7. Huge un-fan of 2, and 3. I am mixed on 1 because I love the removal of required fields, hate the removal of field presence. All the changes I dislike are significant losses in functionality and break compatibility with existing users of proto2 and I'd be interested to understand why "ease of implementation" is good justification for this break in compatibility and what I perceive to be a loss in functionality. On Wed, May 18, 2016 at 11:18 AM, 'Feng Xiao' via Protocol Buffers < protobuf@googlegroups.com> wrote: > > > On Wed, May 18, 2016 at 9:27 AM, Artem Kazakov wrote: > >> +1 >> Yes, a checklist would be extremely helpful. >> >> >> On Friday, April 29, 2016 at 5:04:56 PM UTC-4, Kostiantyn Shchepanovskyi >> wrote: >>> >>> It would be nice to have a migration guide (checklist) somewhere, like: >>> >>> 1. All fields should be optional. >>> 2. Do not use custom defailt values. >>> 3. All enums should have first element with tag = 0. >>> 4. Do not use extension for anything except custom options. >>> >>> Something else? >>> >> In 3.0.0-alpha-1 release note > <https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1> there is > a list of main proto3 changes: > > The following are the main new features in language version 3: > >1. Removal of field presence logic for primitive value fields, removal >of required fields, and removal of default values. This makes proto3 >significantly easier to implement with open struct representations, as in >languages like Android Java, Objective C, or Go. >2. Removal of unknown fields. >3. Removal of extensions, which are instead replaced by a new standard >type called Any. >4. Fix semantics for unknown enum values. >5. Addition of maps. >6. Addition of a small set of standard types for representation of >time, dynamic data, etc. >7. A well-defined encoding in JSON as an alternative to binary proto >encoding. > > > > > >> >>> On Friday, April 29, 2016 at 1:18:12 AM UTC+3, Feng Xiao wrote: >>>> >>>> >>>> >>>> On Tue, Apr 26, 2016 at 7:04 PM, Bo Gao wrote: >>>> >>>>> suppose server side is updating into protobuf3, but client side still >>>>> use protobuf2, can then communicate will? >>>>> >>>> Yes, as long as you only use proto3 features, they are wire compatible. >>>> >>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Protocol Buffers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to protobuf+u...@googlegroups.com. >>>>> To post to this group, send email to prot...@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+unsubscr...@googlegroups.com. >> To post to this group, send email to protobuf@googlegroups.com. >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- Jeremy Ong PlexChat CTO -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: Forced serialization/deserialization of unknown fields in proto3 messages
Neither are appropriate in my use case unfortunately. I want to be able to tag any message with data in a field range special within the organization. The point is that I don't want to add fields to the existing hundreds and hundreds of message types we have already. For the time being, I have switched everything back to proto2, which despite some inconvenience I believe is more feature complete and superior to proto3 at this time. On Fri, Mar 18, 2016 at 4:20 PM, 'Feng Xiao' via Protocol Buffers wrote: > > > On Tuesday, March 15, 2016 at 12:32:07 PM UTC-7, Jeremy Ong wrote: >> >> Hi google pb, >> >> I was wondering if an interface exists for specifying that I do not want >> the proto3 serialization or deserialization to discard unknown fields. My >> understanding was that this change was made from proto2 to proto3, and is a >> pretty severe restriction if there are no ways around it. The motivating >> example in my case is to potentially decorate a message with fields in the >> extension ranges that are not part of the message body. The meaning is >> purely semantic and I do not want the data therein to be contained in the >> protobuf format itself. If unknown fields are not an option, are there other >> options or suggestions to handle this? > > Some alternatives to consider: > 1. use a bytes field to store these data, and decode it manually if it's a > proto. > 2. use an google.protobuf.Any field for it if the data is still a protobuf > message. > >> >> >> Best, >> Jeremy > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: Forced serialization/deserialization of unknown fields in proto3 messages
Thanks for the suggestion but this would mean that every message would need to be packed in this structure including nested messages. It's simply too much for a hack for what I believe was a simple and useful feature that was part of proto2 and removed with no explanation or way to disable it. On Fri, Mar 18, 2016 at 7:06 PM, Steven Parkes wrote: > You can do something like this by serializing the normal message and the > added fields separately and then concatenating the byte strings. > > Not helpful if you need to do this above the byte string level. And may > have other disadvantages, e.g., extra copies ... > > On Fri, Mar 18, 2016 at 6:51 PM, Jeremy Ong wrote: > >> Neither are appropriate in my use case unfortunately. I want to be >> able to tag any message with data in a field range special within the >> organization. The point is that I don't want to add fields to the >> existing hundreds and hundreds of message types we have already. >> >> For the time being, I have switched everything back to proto2, which >> despite some inconvenience I believe is more feature complete and >> superior to proto3 at this time. >> >> On Fri, Mar 18, 2016 at 4:20 PM, 'Feng Xiao' via Protocol Buffers >> wrote: >> > >> > >> > On Tuesday, March 15, 2016 at 12:32:07 PM UTC-7, Jeremy Ong wrote: >> >> >> >> Hi google pb, >> >> >> >> I was wondering if an interface exists for specifying that I do not >> want >> >> the proto3 serialization or deserialization to discard unknown fields. >> My >> >> understanding was that this change was made from proto2 to proto3, and >> is a >> >> pretty severe restriction if there are no ways around it. The >> motivating >> >> example in my case is to potentially decorate a message with fields in >> the >> >> extension ranges that are not part of the message body. The meaning is >> >> purely semantic and I do not want the data therein to be contained in >> the >> >> protobuf format itself. If unknown fields are not an option, are there >> other >> >> options or suggestions to handle this? >> > >> > Some alternatives to consider: >> > 1. use a bytes field to store these data, and decode it manually if >> it's a >> > proto. >> > 2. use an google.protobuf.Any field for it if the data is still a >> protobuf >> > message. >> > >> >> >> >> >> >> Best, >> >> Jeremy >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Protocol Buffers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to protobuf+unsubscr...@googlegroups.com. >> > To post to this group, send email to protobuf@googlegroups.com. >> > Visit this group at https://groups.google.com/group/protobuf. >> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+unsubscr...@googlegroups.com. >> To post to this group, send email to protobuf@googlegroups.com. >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- Jeremy Ong PlexChat CTO 301.648.8260 -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Forced serialization/deserialization of unknown fields in proto3 messages
Hi google pb, I was wondering if an interface exists for specifying that I do not want the proto3 serialization or deserialization to discard unknown fields. My understanding was that this change was made from proto2 to proto3, and is a pretty severe restriction if there are no ways around it. The motivating example in my case is to potentially decorate a message with fields in the extension ranges that are not part of the message body. The meaning is purely semantic and I do not want the data therein to be contained in the protobuf format itself. If unknown fields are not an option, are there other options or suggestions to handle this? Best, Jeremy -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
