Re: [protobuf] Re: protobuf with swift3

[Tim Kientzle]

> On Jan 18, 2017, at 11:24 PM, Kanishka Vedi  wrote:
> 
> Thanks Done that now. 
> 
> now the issue is different --
> 
> Making install in plugin/compiler
> 
>  ../.././install-sh -c -d '/usr/local/bin'
> 
>   /bin/sh ../../libtool   --mode=install /usr/bin/install -c protoc-gen-swift 
> '/usr/local/bin'
> 
> libtool: install: /usr/bin/install -c protoc-gen-swift 
> /usr/local/bin/protoc-gen-swift
> 
> make[2]: Nothing to be done for `install-data-am'.
> 
> make[2]: Nothing to be done for `install-exec-am'.
> 
> make[2]: Nothing to be done for `install-data-am'.
> 
> 

This looks like it worked.

What is your concern?

Tim



-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Problem encoding between C# string on windows and C++ on unix

[Tim Kientzle]

> On Jan 4, 2017, at 11:57 AM, Jamie Sherman  wrote:
> 
> So I forced the message to have fixed values:
>  
> C#
> xsetHeader.TotalXicSets = 10; 
> xsetHeader.WiffName = "myWiffNameHolder";
> 
> hexdump of message: 
> 
> 080a12106d7957694e616d65486f6c646572
> 
> C++
> 
> XicHeader test;
> test.set_totalxics(10);
> test.set_wiffname("myWiffNameHolder");
> test.SerializeToOstream(of);
>  
> hexdump of message:
> 
> 080a1a106d7957694e616d65486f6c646572
> 
> 

It looks like you have two incompatible versions of your proto file
floating around.  You should look through your source code very
carefully to see how that happened.

Your C# code thinks the `wiffName` field is field #2 (the third byte there
is 0x12 = 2 * 8 + 2).

Your C++ code thinks the `wiffName` field is field #3 (the third byte is
0x1a = 3 * 8 + 2).

Also, both are correctly writing UTF-8 strings into the output.

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf::Any Message vs MessageLite interface

[Tim Kientzle]

> On Oct 7, 2016, at 9:54 AM, 'Adam Cozzette' via Protocol Buffers 
>  wrote:
> 
> Here's the thing, though: we are actually thinking about removing the 
> MessageLite::GetTypeName method at some point in the future. Lite protos are 
> intended primarily for mobile and so in that environment it's important to 
> keep the binary as small as possible and also avoid including symbols in the 
> code that's distributed, whereas currently that method requires us to 
> generate code that includes the message type names. So I think it would be 
> best to avoid creating another dependency on that GetTypeName() method when 
> it might be going away in the future.
> 

This seems to suggest that you don’t intend to support Any for mobile?

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Apple's Swift Protobuf Plugin

[Tim Kientzle]

> On Sep 22, 2016, at 3:23 PM, chai2010  wrote:
> 
> cool！
> do you plan support grpc?

Not directly in this project, no.

I think it would be better for swift-protobuf-plugin to include insertion 
points that another plugin could use to add grpc support.

   https://developers.google.com/protocol-buffers/docs/reference/other

Tim



> 
> thanks!
> 
> 2016年9月23日星期五， > 写道：
> If you’re interested in protobuf support for Swift, you might want to take a 
> look at the protoc plugin that Apple just open-sourced.
> 
> You can find more details at
> https://github.com/apple/swift-protobuf-plugin/ 
> 
> 
> As one of the maintainers, I’m eager to get your help in making this as good 
> as it can be.  We plan to develop it completely in the open, so please feel 
> free to file github Issues with your questions and ideas and send us github 
> Pull Requests with your proposed improvements.
> 
> I look forward to working with you,
> 
> Tim
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to protobuf@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/protobuf 
> .
> For more options, visit https://groups.google.com/d/optout 
> .
> 
> 
> -- 
> https://github.com/golang-china/gopl-zh 
> 
> https://github.com/golang-china 
> https://github.com/chai2010 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to protobuf@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/protobuf 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Can I assume ByteSize() always return same size?

[Tim Kientzle]

> On Aug 19, 2016, at 10:45 AM, job4char...@gmail.com wrote:
> 
> The question is:
> 
> My message looks like:
> 
> Message data
> {
> optional uint32 object_id = 1;
> optional uint32 next_id = 2;
> }
> 
> I might define my message object like below:
> 
> data;
> size_t msg_size_before = data.ByteSize();
> 
> uint32 id = CreateNewId();
> uint32 next_id = GetCurNextId();
> 
> data.set_object_id(id);
> data.set_next_id(next_id);
> 
> size_t msg_size_after = data.ByteSize();
> 
> 
> Since I defined only integers in my message type, can I always assume assert 
> "msg_size_before == msg_size_after" 

No.

Protobuf uses variable-length integer encodings, so the size will change 
depending on the exact integer values.

If you need the size to be the same, you can use "fixed32" field type instead 
of "uint32".

Tim



-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] can protobuf3 be used with protobuf2?

[Tim Kientzle]

> On May 18, 2016, at 10:01 PM, Jeremy Ong  wrote:
> 
> Why does adding JSON require dropping unknown fields? So long as fields are 
> keyed to field number, I don't see why the JSON encoding requires special 
> treatment with respect to the binary one.

JSON fields aren’t keyed to field number.  They’re keyed to field name.

Even apart from field naming, JSON and protobuf wire formats don’t correspond 
1:1, so you can’t even correctly translate the primitive values without the 
schema.

> However, proto3 makes breaks in compatibility with the underlying data 
> (proto2 encoded), which is where I find myself in disagreement.

What do you think is different?  Having decoded (by hand) a fair bit of proto2 
and proto3 data, they look exactly the same to me.

As I mentioned before, if preserving unknown fields is essential for you, you 
should stick with proto2.  It’s still around and will be for a long time.

Cheers,

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] can protobuf3 be used with protobuf2?

[Tim Kientzle]

After studying proto3 pretty carefully, I’ve come around quite a bit on these 
changes:

I believe adding JSON requires dropping unknown fields.  You simply cannot 
preserve unknown fields and properly support multiple encodings.

I’m less sure about replacing extension support with Any.  Extensions have some 
ugly problems, but I feel the current spec for Any also has some real drawbacks.

Removing field presence is a subtle issue, but I’m starting to suspect it was 
actually a very good change.  It reduces the generated code and the workaround 
of using a single-element oneof is cleaner than it might sound.  In essence, a 
single-element oneof is just a way to explicitly declare that you want to track 
presence for that field.  And oneof is supported by proto2 now, so you can use 
that technique there as well.

Finally, remember that proto2 is not going away:   If proto2 assumptions are 
deeply baked into your systems, you can keep using it.  protoc will continue to 
support it for a very long time.

Cheers,

Tim



> On May 18, 2016, at 1:33 PM, Jeremy Ong  wrote:
> 
> Big fan of 4, 5, 6, and 7. Huge un-fan of 2, and 3. I am mixed on 1 because I 
> love the removal of required fields, hate the removal of field presence. All 
> the changes I dislike are significant losses in functionality and break 
> compatibility with existing users of proto2 and I'd be interested to 
> understand why "ease of implementation" is good justification for this break 
> in compatibility and what I perceive to be a loss in functionality.
> 
> On Wed, May 18, 2016 at 11:18 AM, 'Feng Xiao' via Protocol Buffers 
>  wrote:
> 
> 
> On Wed, May 18, 2016 at 9:27 AM, Artem Kazakov  wrote:
> +1
> Yes, a checklist would be extremely helpful. 
> 
> 
> On Friday, April 29, 2016 at 5:04:56 PM UTC-4, Kostiantyn Shchepanovskyi 
> wrote:
> It would be nice to have a migration guide (checklist) somewhere, like:
> 
> 1. All fields should be optional.
> 2. Do not use custom defailt values.
> 3. All enums should have first element with tag = 0.
> 4. Do not use extension for anything except custom options.
> 
> Something else?
> In 3.0.0-alpha-1 release note there is a list of main proto3 changes:
> The following are the main new features in language version 3:
> 
>   • Removal of field presence logic for primitive value fields, removal 
> of required fields, and removal of default values. This makes proto3 
> significantly easier to implement with open struct representations, as in 
> languages like Android Java, Objective C, or Go.
>   • Removal of unknown fields.
>   • Removal of extensions, which are instead replaced by a new standard 
> type called Any.
>   • Fix semantics for unknown enum values.
>   • Addition of maps.
>   • Addition of a small set of standard types for representation of time, 
> dynamic data, etc.
>   • A well-defined encoding in JSON as an alternative to binary proto 
> encoding.
> 
> 
>  
> 
> On Friday, April 29, 2016 at 1:18:12 AM UTC+3, Feng Xiao wrote:
> 
> 
> On Tue, Apr 26, 2016 at 7:04 PM, Bo Gao  wrote:
> suppose server side is updating into protobuf3, but client side still use 
> protobuf2, can then communicate will?
> Yes, as long as you only use proto3 features, they are wire compatible.
>  
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+u...@googlegroups.com.
> To post to this group, send email to prot...@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 
> -- 
> Jeremy Ong
> PlexChat CTO
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to 

Re: [protobuf] Default Values vs Missing Values

[Tim Kientzle]

> On Mar 26, 2016, at 11:43 AM, Yoav H  wrote:
> 
> Hi,
> 
> I wanted ask regarding the decision to populate fields with default values, 
> even if they do not appear in the encoded message.
> If I want to send a "patch" message, where I want to update just the provided 
> fields, how can I do that with protobuf (without adding IsXXXSet for every 
> field)?
> 
> Why not add another type, representing a default value? 
> So the schematics would be, if the field is missing, it is null, and if the 
> field exists, but with this "missing value" type, it will get the default 
> value?

As Ilia pointed out, proto2 still exists, is still supported, and can be used 
for
cases where you require these particular semantics.

For proto3, you might look at google.protobuf.FieldMask, which is a new
standard message (one of the "well-known types") specifically designed
to store a set of field names.  You might be able to achieve what you want by
providing a FieldMask with your data listing the specific fields to
be updated.

Tim


-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any JSON coding question

[Tim Kientzle]

> On Mar 17, 2016, at 10:04 AM, Feng Xiao <xiaof...@google.com> wrote:
> 
> On Wed, Mar 16, 2016 at 7:49 PM, Tim Kientzle <kient...@gmail.com> wrote:
> 
> > On Mar 14, 2016, at 7:07 PM, Feng Xiao <xiaof...@google.com> wrote:
> >
> > In google, we are using a front-end proxy to convert JSON to/from protobuf, 
> > and everything else behind it is using protobuf only. The current Any 
> > design works with this approach very well.
> 
> That approach works well my proposed Any design as well.  Your proxies can 
> still walk the object tree and validate and/or convert Any fields as 
> necessary for your particular systems.
> 
> But that approach is not appropriate for everyone.  Your Any design cannot be 
> used by people that require different policies.

> Can you elaborate a bit more? You mentioned the middle system does not know 
> google.profile and has no way to get it. Why is that? I think whether the 
> middle system knows the type info is the only thing that makes a difference. 
> In the google scenario, the middle system always knows all types it will be 
> converting from JSON to protobuf and rejects inputs it doesn't know about.

One example I’ve been considering is a local logging service that needs to 
decode the messages received from various local agents, set standard fields 
(system time, etc), and then recode and store it for later forwarding.  The 
service has no need to know the various attached data for the log entries it is 
forwarding.  Your current design supports this gracefully for protobuf coding 
(Any fields can be decoded and recoded in protobuf without knowing the schema 
for the embedded data) but not for JSON.

You can certainly not assume that a local service of this sort always has 
network access.

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any JSON coding question

[Tim Kientzle]

> On Mar 18, 2016, at 10:37 AM, Feng Xiao  wrote:
> 
> In a lot of places, you can't simply switch from protobuf binary format to 
> JSON format and expect it to work exactly the same way

Actually, I'm more interested in the other direction:  Switching existing 
JSON-based systems (currently using various ad hoc JSON parsers) to use fast, 
safe protoc-generated JSON parsers.

>From what I understand so far, it sounds like Any was designed for systems 
>that are primarily using protobuf binary support and would like occasional 
>JSON interoperability (and are willing to invest in transcoding proxies to 
>enforce this).

But your design is not symmetric:  people who are primarily invested in JSON 
but might like to interoperate with protobuf binary for a few 
performance-critical areas can't really use Any in the same ways.  This is odd, 
since every other part of the proto3 design works equally well for systems that 
are primarily JSON or primarily protobuf.

Tim


-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any JSON coding question

[Tim Kientzle]

> On Mar 14, 2016, at 7:07 PM, Feng Xiao  wrote:
> 
> In google, we are using a front-end proxy to convert JSON to/from protobuf, 
> and everything else behind it is using protobuf only. The current Any design 
> works with this approach very well.

That approach works well my proposed Any design as well.  Your proxies can 
still walk the object tree and validate and/or convert Any fields as necessary 
for your particular systems.

But that approach is not appropriate for everyone.  Your Any design cannot be 
used by people that require different policies.

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any JSON coding question

[Tim Kientzle]

> On Mar 14, 2016, at 11:50 AM, Feng Xiao <xiaof...@google.com> wrote:
> 
> 
> 
> On Thu, Mar 3, 2016 at 6:37 PM, Tim Kientzle <kient...@gmail.com> wrote:
> 
> I think your current Any JSON design requires that protobuf serialization be 
> able to fail.  This is definitely a major change from proto2 and something 
> the conformance test does not explore.
> 
> 1. Decode a valid JSON-encoded message with an Any field in an unknown schema.
> 
> 2. Attempt to encode it in protobuf format.
> 
> If the descriptor cannot be fetched, then step #2 here must fail.
> 
> (The alternative would be for #1 to fail in this case, but that would imply 
> that JSON-to-JSON transcoding must fail in some circumstances.)
> With the current Any JSON design, #1 must fail. It's the same for the other 
> direction. Encoding an Any message with an unknown schema to JSON must fail 
> as well.

Interesting.  I knew that was true of the current C++ implementation; I did not 
realize that was an intentional part of the coding design.

This should certainly be carefully documented in any.proto, since I expect some 
people will be quite surprised to find that decoding valid JSON and encoding 
the same data back to JSON will fail unless you have schema for the contents of 
every Any field.


> If Any had been defined slightly differently, then transcoding would be 
> trivial.  If you are unable to change the current definition of Any, please 
> consider introducing an alternative:
> 
> package google.protobuf;
> 
> message Other {
>   string type_url = 1;
>   oneof encoded {
>  bytes protobuf = 2;
>  google.protobuf.Value json = 3;
>   }
> }
> This and similar proposals (like storing JSON in the value field and append a 
> ":json" suffix to type URL to tell consumers the payload is JSON) had been 
> considered in the design process, and it's decided to stay with the current 
> Any representation. +liujisi may be able to provide more context around this 
> decision.
> 
> I think the advantage of the current Any representation is that the JSON to 
> protobuf is done once in the proxy without the end users to worry about 
> whether the Any payload is protobuf or JSON (i.e., it simplifies the end user 
> APIs). With your proposal, the conversion from JSON to protobuf will spread 
> out to all places. Every reader of the data must be able to do the conversion 
> in order to read the data. That will complicate things quite a bit in a large 
> system.

With my proposal, the decision of how Any gets encoded is a contract between 
the originator of the data and the final consumer.  Intermediate systems can 
store, transcode, and forward the data without having any knowledge of that 
schema.

I was considering Any for cases such as logging, data storage, and other 
systems where intermediaries do not need to access or understand the additional 
data.  This was a common use of extensions in proto2:  Intermediate handlers 
would decode and act on a limited set of well-known fields; additional data 
would be simply carried through until a final consumer decoded and acted on it. 
 Any satisfies this use case for protobuf-only systems but not for systems that 
use JSON.

Tim



>  This is trivial to transcode:  it only requires that the final consumer be 
> able to decode whatever format the originator attached.  Intermediate systems 
> can freely transcode without needing to understand the schema at all, since 
> the ‘protobuf’ field here can be represented in JSON and the ‘json’ field can 
> be encoded to protobuf.  The actual JSON serialization is very similar to the 
> current Any serialization (with the addition of the extra “json” tag).
> 
> Cheers,
> 
> Tim
> 
> 
> 
> > On Feb 5, 2016, at 1:53 PM, Josh Haberman <haber...@google.com> wrote:
> >
> > Hi Tim,
> >
> > I think your analysis is correct. The answer to your question is that it's 
> > not possible to transcode Any from JSON <-> protobuf unless you have the 
> > schema. That is one of the reasons why the type URL exists as a way to 
> > fetch schemas for types you otherwise wouldn't know about.
> >
> > I'm glad the conformance suite was helpful!
> >
> > Best,
> > Josh
> >
> > On Wednesday, February 3, 2016 at 1:37:40 PM UTC-8, Tim Kientzle wrote:
> > I’m trying to implement Any, including proper JSON and protobuf encoding, 
> > and the following scenario has me stumped:
> >
> >
> >
> > I’m looking at the any.proto file, which includes this example:
> >
> >
> >
> > Sample proto file:
> >
> > // package google.profile;
> >
> > // message Person {
> >
> > //   string first_nam

[protobuf] JSON field naming inconsistency

[Tim Kientzle]

What should the JSON coding be for the following?

message Foo {
   string _field_name4 = 7;
}

Currently, the protoc tests say that the JSON field name should be 
“fieldName4”, the conformance test says that the JSON field name should be 
“FieldName4”.  Both seem reasonable, but only one can be right.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: C++ JsonToBinaryStream losing large uint32 values

[Tim Kientzle]

On Friday, February 19, 2016 at 3:56:34 PM UTC-8, Josh Haberman wrote:
>
> Thanks a lot for tracking this down. For reference, here it is in our bug 
> tracker: https://github.com/google/protobuf/issues/1243
>
> On Thursday, February 11, 2016 at 3:28:33 AM UTC-8, Ron Ben-Yosef wrote:
>>
>> Hi,
>>
>> I'm using the proto3 C++ library compiled in MSVC, and noticed some 
>> pretty strange behavior - using the JsonToBinaryString or 
>> JsonToBinaryStream functions, values in the upper half of the uint32 range 
>> get lost in the transition from JSON to binary.
>>
>> After some investigation, it seems the issue's root cause is in the 
>> MathUtil::Sign function (called from NumberConvertAndCheck, which is called 
>> from DataPiece::GenericConvert, which gets called 
>> from DataPiece::ToUint32...)
>>
>>
>>

Actually, there's a more basic problem here:  MathUtil::Sign should return 
1 or -1 (if the number is positive or negative, respectively), not value or 
-value.  This bug breaks the following conformance tests for C++ (the first 
two are broken for all languages because the C++ protobuf used by the 
conformance test can't parse a correct JSON response):

 JsonInput.Int32FieldMinFloatValue.JsonOutput
 JsonInput.Int32FieldMinValue.JsonOutput
 JsonInput.Int32FieldMinFloatValue.ProtobufOutput
 JsonInput.Int32FieldMinValue.ProtobufOutput


-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any JSON coding question

[Tim Kientzle]

I think your current Any JSON design requires that protobuf serialization be 
able to fail.  This is definitely a major change from proto2 and something the 
conformance test does not explore.

1. Decode a valid JSON-encoded message with an Any field in an unknown schema.

2. Attempt to encode it in protobuf format.

If the descriptor cannot be fetched, then step #2 here must fail.

(The alternative would be for #1 to fail in this case, but that would imply 
that JSON-to-JSON transcoding must fail in some circumstances.)

If Any had been defined slightly differently, then transcoding would be 
trivial.  If you are unable to change the current definition of Any, please 
consider introducing an alternative:

package google.protobuf;

message Other {
  string type_url = 1;
  oneof encoded {
 bytes protobuf = 2;
 google.protobuf.Value json = 3;
  }
}

This is trivial to transcode:  it only requires that the final consumer be able 
to decode whatever format the originator attached.  Intermediate systems can 
freely transcode without needing to understand the schema at all, since the 
‘protobuf’ field here can be represented in JSON and the ‘json’ field can be 
encoded to protobuf.  The actual JSON serialization is very similar to the 
current Any serialization (with the addition of the extra “json” tag).

Cheers,

Tim



> On Feb 5, 2016, at 1:53 PM, Josh Haberman <haber...@google.com> wrote:
> 
> Hi Tim,
> 
> I think your analysis is correct. The answer to your question is that it's 
> not possible to transcode Any from JSON <-> protobuf unless you have the 
> schema. That is one of the reasons why the type URL exists as a way to fetch 
> schemas for types you otherwise wouldn't know about.
> 
> I'm glad the conformance suite was helpful!
> 
> Best,
> Josh
> 
> On Wednesday, February 3, 2016 at 1:37:40 PM UTC-8, Tim Kientzle wrote:
> I’m trying to implement Any, including proper JSON and protobuf encoding, and 
> the following scenario has me stumped:
> 
> 
> 
> I’m looking at the any.proto file, which includes this example:
> 
> 
> 
> Sample proto file:
> 
> // package google.profile;
> 
> // message Person {
> 
> //   string first_name = 1;
> 
> //   string last_name = 2;
> 
> // }
> 
> 
> 
> Resulting JSON encoding:
> 
> // {
> 
> //   "@type": "type.googleapis.com/google.profile.Person",
> 
> //   "firstName": ,
> 
> //   "lastName": 
> 
> // }
> 
> 
> 
> The scenario I’m trying to understand involves three systems:
> 
> 
> 
> Originating system:
> 
> * Knows google.profile and writes out the JSON above
> 
> 
> 
> Middle system:
> 
> * Reads the JSON example above
> 
> * Does *not* know google.profile (doesn’t have the descriptor and has no way 
> to get it)
> 
> * Writes out protobuf
> 
> 
> 
> Receiving system:
> 
> * Knows google.profile and reads the protobuf
> 
> 
> 
> In particular:  What is the protobuf serialization between the middle and 
> receiving systems?
> 
> 
> 
> Without the descriptor, the middle system cannot poduce the “real" protobuf 
> binary format (the field numbers aren’t in that format), so the only option 
> would seem to be to serialize the JSON data as a protobuf Struct 
> representation (which in turn has interesting implications for the receiving 
> system). Am I missing some other option?
> 
> 
> 
> Tim
> 
> 
> 
> P.S.  Nice work on the conformance test suite, by the way!  That clarified a 
> *lot* of details about the expected handling for protobuf JSON format.
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Fwd: Protobuf question for mailing list

[Tim Kientzle]

I’m trying to implement Any, including proper JSON and protobuf encoding,
and the following scenario has me stumped:

>
I’m looking at the any.proto file, which includes this example:

>
Sample proto file:

// package google.profile;

// message Person {

//   string first_name = 1;

//   string last_name = 2;

// }

>
Resulting JSON encoding:

// {

//   "@type": "type.googleapis.com/google.profile.Person",

//   "firstName": ,

//   "lastName": 

// }

>
The scenario I’m trying to understand involves three systems:

>
Originating system:

* Knows google.profile and writes out the JSON above

>
Middle system:

* Reads the JSON example above

* Does *not* know google.profile (doesn’t have the descriptor and has no
way to get it)

* Writes out protobuf

>
Receiving system:

* Knows google.profile and reads the protobuf

>
In particular:  What is the protobuf serialization between the middle and
receiving systems?

>
Without the descriptor, the middle system cannot poduce the “real" protobuf
binary format (the field numbers aren’t in that format), so the only option
would seem to be to serialize the JSON data as a protobuf Struct
representation (which in turn has interesting implications for the
receiving system). Am I missing some other option?

>
Tim


P.S.  Nice work on the conformance test suite, by the way!  That clarified
a *lot* of details about the expected handling for protobuf JSON format.

>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] JSON serialization can fail?

[Tim Kientzle]

According to the conformance suite, there are a handful of cases for which 
JSON serialization should fail:

  = Duration value in memory is outside of valid range for JSON
  = Timestamp value in memory is outside of valid range for JSON
  = Fieldmask value in memory is not representable in JSON

This would seem to violate one of the stated goals of proto3 — to simplify 
implementations.  In particular, requiring that JSON serialization be able 
to fail requires additional boilerplate for every single call to a JSON 
serialization method.  This seems somewhat burdensome just to handle the 
remote possibility that someone might try to accidentally process a date 
thousands of years from now.  (It’s especially burdensome for those users 
who aren’t actually using any of these types so have to write boilerplate 
error-handling logic that will never be called.)

It would be far simpler to instead truncate Durations and Timestamps to the 
representable range and to redefine Fieldmask to drop unrepresentable 
fields.  This would eliminate a lot of boilerplate from implementations 
(and users) of the JSON code.

Cheers,

Tim


-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] JSON serialization for 'Any'

[Tim Kientzle]

I’m trying to implement Any, including proper JSON and protobuf encoding, 
and the following scenario has me stumped:

Consider the following example from the google/protobuf/any.proto source 
file:

Sample proto file:
// package google.profile;
// message Person {
//   string first_name = 1;
//   string last_name = 2;
// }

Resulting JSON encoding:
// {
//   "@type": "type.googleapis.com/google.profile.Person",
//   "firstName": ,
//   "lastName": 
// }

The scenario I’m trying to understand involves three systems:

Originating system:
* Knows google.profile and writes out the JSON above

Middle system:
* Reads the JSON example above
* Does *not* know google.profile (doesn’t have the descriptor and has no 
way to get it)
* Writes out protobuf

Receiving system:
* Knows google.profile and reads the protobuf

In particular:  What is the protobuf serialization between the middle and 
receiving systems?

Without the descriptor, the middle system cannot poduce the “real" protobuf 
binary format (it doesn't have the field numbers), so the only option would 
seem to be to serialize the JSON data as a protobuf Struct, which would 
have interesting implications for the receiving system. Am I missing some 
other option?

Tim

P.S.  Nice work on the conformance test suite, by the way!  That answered a 
*lot* of questions about the expected handling for protobuf JSON format.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Any JSON coding question

[Tim Kientzle]

 

I’m trying to implement Any, including proper JSON and protobuf encoding, 
and the following scenario has me stumped:


I’m looking at the any.proto file, which includes this example:


Sample proto file:

// package google.profile;

// message Person {

//   string first_name = 1;

//   string last_name = 2;

// }


Resulting JSON encoding:

// {

//   "@type": "type.googleapis.com/google.profile.Person",

//   "firstName": ,

//   "lastName": 

// }


The scenario I’m trying to understand involves three systems:


Originating system:

* Knows google.profile and writes out the JSON above


Middle system:

* Reads the JSON example above

* Does *not* know google.profile (doesn’t have the descriptor and has no 
way to get it)

* Writes out protobuf


Receiving system:

* Knows google.profile and reads the protobuf


In particular:  What is the protobuf serialization between the middle and 
receiving systems?


Without the descriptor, the middle system cannot poduce the “real" protobuf 
binary format (the field numbers aren’t in that format), so the only option 
would seem to be to serialize the JSON data as a protobuf Struct 
representation (which in turn has interesting implications for the 
receiving system). Am I missing some other option?


Tim


P.S.  Nice work on the conformance test suite, by the way!  That clarified 
a *lot* of details about the expected handling for protobuf JSON format.


-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.