Re: [protobuf] Compilation warning with "arenas" option enabled for protobuf-3.1

[changchunzhao . tnsi]

Adam, thank you for debugging and quick response!

So is any fix planned in next 3.1 patch? Or need to wait little bit further?

Regards.
Changchun

On Friday, November 18, 2016 at 8:18:36 PM UTC-5, Adam Cozzette wrote:
>
> I think I see what's going on. Your oneof field gets stored as a union 
> that looks like this in the generated code:
>
> union MsgDataUnion {
> MsgDataUnion() {}
> ::google::protobuf::internal::ArenaStringPtr first_;
> ::google::protobuf::uint32 second_;
>   } msg_data_;
>
> In DemoMessage::mutable_first() we call 
> msg_data_.first_.UnsafeSetDefault(), but this is technically incorrect 
> because we are calling a method on one of the types in the union without 
> first initializing it. For example it could be that the union is actually 
> in the uint32 form and yet we are attempting to call an ArenaStringPtr 
> method on the data. I think this works out OK in practice but is probably 
> technically undefined behavior and so we should fix it.
>
> On Mon, Nov 14, 2016 at 10:05 AM,  
> wrote:
>
>> Adam, yes, forgot providing my setup, here it is:
>>
>> $ g++ --version
>> g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-3)
>> Copyright (C) 2010 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions.  There is 
>> NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR 
>> PURPOSE.
>>
>> $ uname -a
>> Linux demo-host 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 
>> 2013 x86_64 x86_64 x86_64 GNU/Linux
>>
>> *And generated code around the problematic line*:
>> 262 inline ::std::string* DemoMessage::mutable_first() {
>> 263   if (!has_first()) {
>> 264 clear_msg_data();
>> 265 set_has_first();
>> 266 msg_data_.first_.UnsafeSetDefault(&::google::protobuf::internal::
>> GetEmptyStringAlreadyInited());
>> 267   }
>>
>> I have also attached the generated files from my system for your 
>> reference.
>>
>> Thanks again!
>>
>> Regards,
>> Changchun
>>
>> On Monday, November 14, 2016 at 12:50:56 PM UTC-5, Adam Cozzette wrote:
>>>
>>> Thanks, Changchun. I have just a few more questions to try to debug the 
>>> problem:
>>> - What version of g++ are you using? (g++ --version should tell you)
>>> - What architecture are you building for?
>>> - Could you send the generated code around line 266 in demo.pb.h just so 
>>> that it's clear what lines the compiler is warning about?
>>>
>>> On Mon, Nov 14, 2016 at 7:55 AM,  wrote:
>>>
 Adam, Thanks for your reply.

 Here I attached a simple .proto file to demonstrate this warning 
 message:

 $ protobuf-3.1/bin/protoc --cpp_out=. demo.proto 
  
 $ /usr/bin/g++ -Iprotobuf-3.1/include -c -Wall -O2 -Wno-long-long demo.
 pb.cc -o demo.pb.o
 In member function 'virtual bool 
 demo::DemoMessage::MergePartialFromCodedStream(google::protobuf::io::CodedInputStream*)'
 :
 cc1plus: warning: dereferencing pointer 'prephitmp.1630' does break
  strict-aliasing rules
 demo.pb.h:266: note: initialized from here
 cc1plus: note: initialized from here

 Please let me know if anything else you need to debug further.

 Thanks,
 Changchun

 On Friday, November 11, 2016 at 4:45:03 PM UTC-5, Adam Cozzette wrote:
>
> What version of g++ are you using? Also, do you have a way of 
> determining what actual line of code the compiler is warning about there?
>
> I tried building with -Wstrict-aliasing=2 at commit df8390790a and saw 
> a few warnings like this:
>
> ./google/protobuf/generated_message_util.h:94:66: warning: 
> dereferencing type-punned pointer might break strict-aliasing rules 
> [-Wstrict-aliasing]
>const T& get() const { return reinterpret_cast(union_); }
>
> But that doesn't look like it's quite the same warning as what you are 
> seeing.
>
> On Fri, Nov 11, 2016 at 12:39 PM,  wrote:
>
>> I have below option in my .proto file:
>>
>> *option cc_enable_arenas = true;*
>>
>> But g++ for Linux gives below warning:
>>
>> *cc1plus: warning: dereferencing pointer 'prephitmp.4925' does break 
>> strict-aliasing rules*
>>
>> Will this cause issue and how to solve it?
>>
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, 
>> send an email to protobuf+u...@googlegroups.com.
>> To post to this group, send email to prot...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> -- 
 You received this message because you are subscribed to the Google 
 Groups "Protocol Buffers" group.
 To unsubscribe from this group and stop receiving emails from it, 

Re: [protobuf] Compilation warning with "arenas" option enabled for protobuf-3.1

['Adam Cozzette' via Protocol Buffers]

I think I see what's going on. Your oneof field gets stored as a union that
looks like this in the generated code:

union MsgDataUnion {
MsgDataUnion() {}
::google::protobuf::internal::ArenaStringPtr first_;
::google::protobuf::uint32 second_;
  } msg_data_;

In DemoMessage::mutable_first() we call
msg_data_.first_.UnsafeSetDefault(), but this is technically incorrect
because we are calling a method on one of the types in the union without
first initializing it. For example it could be that the union is actually
in the uint32 form and yet we are attempting to call an ArenaStringPtr
method on the data. I think this works out OK in practice but is probably
technically undefined behavior and so we should fix it.

On Mon, Nov 14, 2016 at 10:05 AM,  wrote:

> Adam, yes, forgot providing my setup, here it is:
>
> $ g++ --version
> g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-3)
> Copyright (C) 2010 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE
> .
>
> $ uname -a
> Linux demo-host 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013
> x86_64 x86_64 x86_64 GNU/Linux
>
> *And generated code around the problematic line*:
> 262 inline ::std::string* DemoMessage::mutable_first() {
> 263   if (!has_first()) {
> 264 clear_msg_data();
> 265 set_has_first();
> 266 msg_data_.first_.UnsafeSetDefault(&::google::protobuf::internal::
> GetEmptyStringAlreadyInited());
> 267   }
>
> I have also attached the generated files from my system for your reference.
>
> Thanks again!
>
> Regards,
> Changchun
>
> On Monday, November 14, 2016 at 12:50:56 PM UTC-5, Adam Cozzette wrote:
>>
>> Thanks, Changchun. I have just a few more questions to try to debug the
>> problem:
>> - What version of g++ are you using? (g++ --version should tell you)
>> - What architecture are you building for?
>> - Could you send the generated code around line 266 in demo.pb.h just so
>> that it's clear what lines the compiler is warning about?
>>
>> On Mon, Nov 14, 2016 at 7:55 AM,  wrote:
>>
>>> Adam, Thanks for your reply.
>>>
>>> Here I attached a simple .proto file to demonstrate this warning message:
>>>
>>> $ protobuf-3.1/bin/protoc --cpp_out=. demo.proto
>>>
>>> $ /usr/bin/g++ -Iprotobuf-3.1/include -c -Wall -O2 -Wno-long-long demo.
>>> pb.cc -o demo.pb.o
>>> In member function 'virtual bool demo::DemoMessage::MergePartia
>>> lFromCodedStream(google::protobuf::io::CodedInputStream*)':
>>> cc1plus: warning: dereferencing pointer 'prephitmp.1630' does break
>>>  strict-aliasing rules
>>> demo.pb.h:266: note: initialized from here
>>> cc1plus: note: initialized from here
>>>
>>> Please let me know if anything else you need to debug further.
>>>
>>> Thanks,
>>> Changchun
>>>
>>> On Friday, November 11, 2016 at 4:45:03 PM UTC-5, Adam Cozzette wrote:

 What version of g++ are you using? Also, do you have a way of
 determining what actual line of code the compiler is warning about there?

 I tried building with -Wstrict-aliasing=2 at commit df8390790a and saw
 a few warnings like this:

 ./google/protobuf/generated_message_util.h:94:66: warning:
 dereferencing type-punned pointer might break strict-aliasing rules
 [-Wstrict-aliasing]
const T& get() const { return reinterpret_cast(union_); }

 But that doesn't look like it's quite the same warning as what you are
 seeing.

 On Fri, Nov 11, 2016 at 12:39 PM,  wrote:

> I have below option in my .proto file:
>
> *option cc_enable_arenas = true;*
>
> But g++ for Linux gives below warning:
>
> *cc1plus: warning: dereferencing pointer 'prephitmp.4925' does break
> strict-aliasing rules*
>
> Will this cause issue and how to solve it?
>
> --
> You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+u...@googlegroups.com.
> To post to this group, send email to prot...@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

 --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and 

Re: [protobuf] Protobuf and Schema Registry

['Adam Cozzette' via Protocol Buffers]

Protocol buffer schemas are represented with what are called descriptors,
and descriptors themselves can be serialized as protocol buffers so that
you can save them on disk or send them over the network or whatever you
like. If you want to have a sort of schema registry for protocol buffers
then I would look into the DescriptorPool

and DescriptorDatabase

and related classes. These don't provide a full-blown schema server, but
they provide all the core functionality that you would need to create one
fairly easily. Those classes I linked to are in C++, but we have similar
functionality for Java (I'm not sure offhand about other languages).

I don't think schema discovery is actually that useful for protocol buffers
most of the time, though. I'm not very familiar with Avro but from reading
about it today it appears that when you parse a message you need to have
more-or-less the exact schema that was used to serialize it. In that case I
can see why you might want to have a versioned schema registry. But with
Protobuf it's generally not a problem at all if your schema is older or
newer than that of the peer you're communicating with or the data you're
reading from disk. The worst that can happen is that you come across some
new fields that your schema doesn't know about, and when that happens they
are just treated as "unknown fields" and are invisible to you unless you
want to go out of your way to inspect them. With protobuf you almost always
just use the code generated by the protocol compiler (protoc), and there's
generally no need to even think about descriptors unless you're doing
something unusual. (The main example would be needing to manipulate message
types that are unknown at compile time.)

On Fri, Nov 18, 2016 at 7:59 AM, Ryan Morton  wrote:

> m conducting an investigation into the use of Protobuf for my organization
> and have a need to perform schema discovery. I've seen a number of articles
> related to Confluent Schema Registry with Avro. What are folks using for
> Protobuf?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Protobuf and Schema Registry

[Ryan Morton]

m conducting an investigation into the use of Protobuf for my organization 
and have a need to perform schema discovery. I've seen a number of articles 
related to Confluent Schema Registry with Avro. What are folks using for 
Protobuf?

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.