Re: [protobuf] How to get all dependency .proto files of a given .proto file hierarchically?

2019-12-01 Thread Marc Gravell
Untested, but can't you just use protoc with the descriptor-set mode to build a FileDescriptorSet (IIRC) which will be the processed output of protoc's efforts? Then parse it using the details from descriptor.proto, and see what descriptors are in the FileDescriptorSet? -- You received this

Re: [protobuf] com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has be

2020-02-03 Thread Marc Gravell
The most likely thing is that it was truncated during your transmission, meaning: a bug in your code that serializes and/or sends the data. Can we see any of that code? Note that you can use "protoc" (decode-raw) or https://protogen.marcgravell.com/decode to validate a payload for integrity. On

Re: [protobuf] Enum field encoding question

2020-01-16 Thread Marc Gravell
Hi; default values *are not sent*, especially in proto3 where zero is default and default is zero. Likewise, the root object in a message is not wrapped in any way - only fields *on* the root object. This means that the binary encoding of a CommandRequest with Code.RESET is: zero bytes, which is

Re: [protobuf] Re: "Architectural" question - Java -> proto -> C#

2020-01-04 Thread Marc Gravell
The context here seems to be: "I have .proto file/files, and (for some reason) I need to make additional tweaks to them in java/c#", right? In the case of C#, "partial classes" may be your friend, especially if all you are trying to do is add annotations/attributes. The tooling may also generate

Re: [protobuf] C# Official Supported .NET Version

2020-04-29 Thread Marc Gravell
The google protobuf package ( https://www.nuget.org/packages/Google.Protobuf/) supports .NET 4.5 upwards; protobuf-net (an independent / unaffiliated implementation of the serializer) supports .NET 2.0 and upwards for v2.*, but: from v3.* onwards that will change to .NET 4.6.1 Frankly, you are

Re: [protobuf] How to control the order of field while converting a protobuf message to string using TextFormat in java?

2020-03-22 Thread Marc Gravell
I would imagine that most implementations are still going to use field number order. Is there a specific reason you need to control it? Most JSON tools won't care, and if you want to control the JSON, frankly you shouldn't be using an opinionated serializer like protobuf - you should be using a

Re: [protobuf] Adding metadata to an RPC

2020-10-14 Thread Marc Gravell
"custom options" exist, but frankly they don't do much unless you have custom code generation that is going to emit something useful from them, and that "something" itself does something useful at runtime - which may itself be tricky between runtimes/platforms. Does "add a comment" suffice? On

Re: [protobuf] How to Assign encoding number to the field inside message

2020-06-30 Thread Marc Gravell
See https://developers.google.com/protocol-buffers/docs/overview#assigning_field_numbers Basically: - positive integers - unique within each message (can reuse between different messages) - lower is cheaper - avoid some reserved ranges On Tue, 30 Jun 2020, 20:12 rohit nv, wrote: > > I am new

Re: [protobuf] Re: [protobuf-net] Unknow wire-type 7 when serializing List data structure

2020-07-27 Thread Marc Gravell
Hi; it is pretty hard to comment without code. Do you have a minimal repro that shows what you're seeing? The fact that you mention Encoding suggests that you're treating protobuf data as though it were text, which is never a good idea. Also, since this is library specific, you may wish to

Re: [protobuf] key/tag encoding when message types > 31

2021-01-08 Thread Marc Gravell
The composed wire-type and field number are treated as a varint. And since the MSB is reserved for continuation, after the 3-bit wire type that only leaves 4 bits of field number, not 5, for single-byte field headers. On Fri, 8 Jan 2021, 12:47 'emaz...@cisco.com' via Protocol Buffers, <

Re: [protobuf] Use of non-standard values for index

2021-01-07 Thread Marc Gravell
100 isn't "non-standard" as such, and shouldn't cause anything to fail. What exactly are you seeing? The valid range is 1-536870911, omitting 19000-1 (and any reserved areas in your specific messages) smaller numbers are cheaper (fewer bytes) to encode, so are usually preferred - but: that's

Re: [protobuf] key/tag encoding when message types > 31

2021-01-08 Thread Marc Gravell
Field number 76 with which wire type? The lowest 3 bits are the wire-type, then the field number is whacked on the end. Then split into groups of 7 bits for varint encoding. On Fri, 8 Jan 2021, 14:05 'emaz...@cisco.com' via Protocol Buffers, < protobuf@googlegroups.com> wrote: > Hi Marc, > > OK

Re: [protobuf] How to decode the data below,thanks

2021-05-05 Thread Marc Gravell
What language/framework? On Wed, 5 May 2021, 09:58 Danny Lee, wrote: > Hello everyone, > > There is a proto file as below. > And it is required not to use the .options file. > > How to decode the data below: > MsgInfo.data.msg2_data.data.msg22_data.jobs.job_data.job_attr.age > > Any useful

Re: [protobuf] Protobuf for Enum

2021-05-21 Thread Marc Gravell
This is specific to protobuf-net. If you're using v3, the main usage here is to provide explicit names for the enum and (via ProtoEnumAttribute) the defined values; the names only matter if you are generating a .proto from a code-first model (GetSchema(), GetProto(), etc) In v2, you can *also*

Re: [protobuf] Protobuf Compiler in a C# Shared Library

2021-04-19 Thread Marc Gravell
The tooling package you need here is Grpc.Tools, which should work fine in a class library. On Sun, 18 Apr 2021 at 23:08, Chris Langlois wrote: > Thanks for the recommendations. I can add .proto files to the Shared > projects. What I cannot do is add a build action for "protobuf compiler" to >

Re: [protobuf] protoc v3.6 and v3.15 compatibility for different languages

2021-04-21 Thread Marc Gravell
The actual serialization format hasn't changed since protobuf was released; as long as *your schemas* haven't changed in that time (at least, not in incompatible ways; adding fields etc is pretty safe as long as they aren't "required" in proto2), you should be absolutely fine. It is expected and

Re: [protobuf] Protobuf Compiler in a C# Shared Library

2021-04-18 Thread Marc Gravell
What is stopping you from adding them to the shared project? What TFM is it targeting? This should JustWorkTM. You could also look at the csproj changes in the client and server, and try to apply the same changes in the shared project - it might tell you why it isn't happy. On Sun, 18 Apr 2021,

Re: [protobuf] Feature Request: make protocol buffer AST/parser SDK

2021-02-16 Thread Marc Gravell
You can use protoc to output the compiled schema (one of the command-line-options - something "file descriptor set"), and deserialize the resultant binary file as a FileDescriptorSet instance, deserializing via your choice of language via descriptor.proto - any use? As an aside, I also have a

Re: [protobuf] Unable to get Client Service to work

2021-02-21 Thread Marc Gravell
I'm a little confused here... Normally, assuming that this is using the Google implementation, you would have, in your generated code: - a generated concrete client proxy - a generated abstract server stub The client code would "new" the client proxy (the yellow code). Your server code would

Re: [protobuf] Extra dot in descriptor data (protoc java)

2021-12-22 Thread Marc Gravell
IIRC, the leading dot means that the name is absolute rather than relative. I'm not sure it represents an error. On Wed, 22 Dec 2021, 19:00 'Venkat Duddu' via Protocol Buffers, < protobuf@googlegroups.com> wrote: > We are seeing extra dot for external referenced variables in > descriptorData in

Re: [protobuf] If i have a 209 byte size data of about 50 parameters, if i use protobuf to serialize it,what would the size be like?

2022-01-30 Thread Marc Gravell
You say "209 byte size" - using what measure is it 209 bytes? And what is the data? The payload size in protobuf often depends on the specific data, for example: - for any fields: whether it is the default / unset value, vs whether it is a non-detault / explicitly set value, can change whether

Re: [protobuf] [ACTION REQUIRED] Protobuf Java users, please update to our latest release

2022-01-06 Thread Marc Gravell
I notice that the advisory is scant on details at the moment; is there any mechanism for non-Google protobuf library authors to request additional details to see whether our own implementations may be vulnerable to the attack? Thanks On Thu, 6 Jan 2022 at 17:15, 'Derek Perez' via Protocol Buffers

Re: [protobuf] Parse .proto files

2023-10-04 Thread Marc Gravell
If .NET is an option, protobuf-net has this (in the protobuf-net.Reflection package): var schema = """ syntax = "proto3"; package helloworld; // My cool new message message MyMessage { string some_string = 1; repeated int64 some_numbers = 2; } """; var set =

[protobuf] Re: Documentation about endianness support

2023-10-21 Thread Marc Gravell
protobuf ensures that the *value of primitives* will be preserved; so if you send the uint32 23433451, it will be decoded as 23433451 regardless of the CPU/process endianness of sender and receiver. The details are here: https://protobuf.dev/programming-guides/encoding/ - but they **don't

Re: [protobuf] Protobuff

2022-06-15 Thread Marc Gravell
Is this really a gRPC question? protobuf is just a payload serialization format If it *is* gRPC, then: *how* is the API key expected to be transmitted? Is it a header? a payload item? Honestly, linking to the API specification would be a good idea. Otherwise this is very vague. On Wed, 15 Jun

Re: [protobuf] protocol buffers for inter-language IPC

2022-05-05 Thread Marc Gravell
Without a long-running process, what would this look like? what would cause the remote / out-of-process execution? On Thu, 5 May 2022 at 00:00, 'Bill Thorp' via Protocol Buffers < protobuf@googlegroups.com> wrote: > Inter-language interop is hard, but protocol buffers have good code > generation

Re: [protobuf] using

2022-09-03 Thread Marc Gravell
No On Sat, 3 Sep 2022, 09:48 berge, wrote: > Hi all! > > in C++(>=11) I can do: > ``` > using TMyLittleType = int64_t; > > TMyLittleType xyz; > ``` > > Is there a trick I can use to do the same in proto3? > like > ``` > magictrick TMyLittleType = sint64; > > message XYZ > { > >

Re: [protobuf] ParseDelimited from unreliable comm channel

2022-10-10 Thread Marc Gravell
No, protobuf does not have any resync capability. You're going to need to wrap that yourself. On Mon, 10 Oct 2022, 18:19 David L, wrote: > When serializing a message with SerializeDelimitedToOstream, is the > expectation that parsing the resulting serial stream will re-sync if some > data is

Re: [protobuf] What does singular mean in ProtoBuf >=3.15?

2022-12-07 Thread Marc Gravell
An 'optional' field is still singular (optional having been re-added to proto3 relatively recently); singular just means: not 'repeated' With regards to "wrappers.proto": all of the inner values in the wrapper types are singular; as stated in the text part of wrappers.proto, it isn't intended to

Re: [protobuf] What does singular mean in ProtoBuf >=3.15?

2022-12-07 Thread Marc Gravell
(oh, and not "map"; "map" is just a specialized example of the semantics from "repeated", under the hood) On Wed, 7 Dec 2022 at 10:11, Marc Gravell wrote: > An 'optional' field is still singular (optional having been re-added to > proto3 relatively recently)

Re: [protobuf] Skip certain indices while encoding

2022-11-28 Thread Marc Gravell
For 1 options: a) don't assign a value to salary (unless it is "required" in proto2, which doesn't appear to be the case), or b) create a new EmployeeDataMinimal message that *doesn't have a salary*, and serialize that (I can't comment on 2; not a C++ person) On Mon, 28 Nov 2022 at 08:14,

Re: [protobuf] Is Resulting Binary from Changing Proto Definition Source (From Nested to Non-Nested) for Fields initialization Backward-Compatible?

2023-04-20 Thread Marc Gravell
As long as you aren't using `Any`: then at the binary payload data it won't be visible. Any existing code that uses the generated types will need to be updated, obviously. On Thu, 20 Apr 2023, 07:00 'Felik' via Protocol Buffers, < protobuf@googlegroups.com> wrote: > Hello, > > I came to

Re: [protobuf] Compatibility error on adding sub message type in existing proto file

2023-02-04 Thread Marc Gravell
That is ... contextual, though; an unexpected sub-message can still, depending on the implementation, be round-tripped and potentially even inspected via reflection. It won't usually *break* code, just like adding any new field. It really depends how strict you're being in you'd definitions of

Re: [protobuf] Is there a method to not utilise the 3bit Tag so the full fixed32 is used.

2024-05-13 Thread Marc Gravell
minimize serialization code? > > On Monday, May 13, 2024 at 3:52:56 PM UTC+10 Marc Gravell wrote: > >> A tag isn't on a value - it is part of the *field header* which is the >> combination of field-number and wire-type ; a fixed32 *value* is 32 bits. >> >> The sizes of

Re: [protobuf] Is there a method to not utilise the 3bit Tag so the full fixed32 is used.

2024-05-12 Thread Marc Gravell
A tag isn't on a value - it is part of the *field header* which is the combination of field-number and wire-type ; a fixed32 *value* is 32 bits. The sizes of values aren't directly controllable by you; protobuf-net isn't a general purpose binary descriptor format and cannot usually be used to

<    1   2   3   4