Re: [protobuf] protobuf::Any Message vs MessageLite interface

[Mohamed Koubaa]

Hello,

This works as long as my test project links against the full protobuf
runtime.  Attempting to link against the lite runtime produces the
following unresolved symbols:

error LNK2001: unresolved external symbol "public: __cdecl
google::protobuf::Any::Any(void)" (??0Any@protobuf@google@@QEAA@XZ)

error LNK2001: unresolved external symbol "void __cdecl
google::protobuf::protobuf_AddDesc_google_2fprotobuf_2fany_2eproto(void)"
(?protobuf_AddDesc_google_2fprotobuf_2fany_2eproto@protobuf@google@@YAXXZ)

error LNK2001: unresolved external symbol "public: virtual bool __cdecl
google::protobuf::Any::MergePartialFromCodedStream(class
google::protobuf::io::CodedInputStream *)" (?MergePartialFromCodedStream@Any
@protobuf@google@@UEAA_NPEAVCodedInputStream@io@23@@Z)

error LNK2001: unresolved external symbol "public: virtual int __cdecl
google::protobuf::Any::ByteSize(void)const " (?ByteSize@Any@protobuf@google
@@UEBAHXZ)

error LNK2001: unresolved external symbol "public: void __cdecl
google::protobuf::Any::MergeFrom(class google::protobuf::Any const &)"
(?MergeFrom@Any@protobuf@google@@QEAAXAEBV123@@Z)

error LNK2001: unresolved external symbol "public: static class
google::protobuf::Any const & __cdecl
google::protobuf::Any::default_instance(void)" (?default_instance@Any
@protobuf@google@@SAAEBV123@XZ)


Thanks,
Mohamed Koubaa
Software Developer
ANSYS Inc

On Thu, Dec 1, 2016 at 1:36 PM, Mohamed Koubaa 
wrote:

> Hello,
>
> FWIW, this is the boilerplate I use for my proto3.0.0 project.  It depends
> on GetTypeName() whose future is uncertain in the lite runtime.  It appears
> to work in one of my tests but I am not sure if I am missing something
> subtle.  I'm using SerializeWithCachedSizesToArray because I learned that
> it is faster for large messages because it does not compute the size twice.
>
> static void PackInto(google::protobuf::Any* target, const
> google::protobuf::MessageLite& msg)
> {
> int msg_size = msg.ByteSize();
> char* msg_buffer = new char[msg_size];
>
> msg.SerializeWithCachedSizesToArray((google::protobuf::uint8*)msg_buffer);
> //avoids double
> target->set_type_url(msg.GetTypeName());
> target->set_value(msg_buffer,msg_size);
>
> delete[] msg_buffer;
> }
>
> static void UnpackFrom(const google::protobuf::Any& source,
> google::protobuf::MessageLite* msg)
> {
> EXPECT_EQ(source.type_url(), msg->GetTypeName()); //Could be converted
> to an assert or CHECK style macro in a non-test project
> msg->ParseFromArray(source.value().c_str(), source.value().size());
> }
>
> Thanks,
> Mohamed Koubaa
> Software Developer
> ANSYS Inc
>
> On Tue, Nov 29, 2016 at 8:22 PM, Adam Cozzette 
> wrote:
>
>> Right now there doesn't seem to be a consensus around adding built-in
>> support for Any in the lite runtime, so I suspect that the status quo will
>> probably remain for now. If you would like to use Any with the lite
>> runtime, I think it's probably best to just manually serialize and parse to
>> and from your Any fields, since that will work even if it involves a little
>> extra boilerplate.
>>
>> On Tue, Nov 29, 2016 at 11:20 AM, Mohamed Koubaa <
>> mohamed.kou...@ansys.com> wrote:
>>
>>> Hello,
>>>
>>> I am sorry to bring back an old thread, but the outcome is not clear.
>>> Is there either an intent or any ongoing work to support Any types with the
>>> lite runtime?
>>>
>>> Best Regards,
>>> Mohamed Koubaa
>>> Software Developer
>>> ANSYS Inc
>>>
>>> On Mon, Oct 10, 2016 at 3:00 PM, 'Adam Cozzette' via Protocol Buffers <
>>> protobuf@googlegroups.com> wrote:
>>>

 On Fri, Oct 7, 2016 at 2:16 PM, Arpit Baldeva 
 wrote:

> Thanks for the info.
>
> I feel like without pack/unpack/Is method, the utility of Any will
> diminish. For example, the rpc status proto (
> https://github.com/googleapis/googleapis/blob/master/google
> /rpc/status.proto) uses repeated Any field. It'd not be possible to
> write code like one described here - https://developers.google.co
> m/protocol-buffers/docs/proto3#any because you won't know if it is
> safe to convert value to a give message. I also came across this post 
> after
> my post which marks the request as a bug currently -
> https://github.com/google/protobuf/issues/1974
>

 What you're saying makes sense, we might want to consider just updating
 Any to have first-class support for MessageLite. In C++ this would be
 straightforward but in Java, for example, we would need to think carefully
 about how to do it because in Java lite we don't currently have the message
 names available at runtime.

 Regarding the future of GetTypeName, though it has overhead, feel like
> it could have many utilities outside of the Any support as well. I don't
> have concrete use case in mind though as I am just starting on protobuf.
> This brings another important question that 

Re: [protobuf] protobuf::Any Message vs MessageLite interface

[Mohamed Koubaa]

Hello,

FWIW, this is the boilerplate I use for my proto3.0.0 project.  It depends
on GetTypeName() whose future is uncertain in the lite runtime.  It appears
to work in one of my tests but I am not sure if I am missing something
subtle.  I'm using SerializeWithCachedSizesToArray because I learned that
it is faster for large messages because it does not compute the size twice.

static void PackInto(google::protobuf::Any* target, const
google::protobuf::MessageLite& msg)
{
int msg_size = msg.ByteSize();
char* msg_buffer = new char[msg_size];

msg.SerializeWithCachedSizesToArray((google::protobuf::uint8*)msg_buffer);
//avoids double
target->set_type_url(msg.GetTypeName());
target->set_value(msg_buffer,msg_size);

delete[] msg_buffer;
}

static void UnpackFrom(const google::protobuf::Any& source,
google::protobuf::MessageLite* msg)
{
EXPECT_EQ(source.type_url(), msg->GetTypeName()); //Could be converted
to an assert or CHECK style macro in a non-test project
msg->ParseFromArray(source.value().c_str(), source.value().size());
}

Thanks,
Mohamed Koubaa
Software Developer
ANSYS Inc

On Tue, Nov 29, 2016 at 8:22 PM, Adam Cozzette  wrote:

> Right now there doesn't seem to be a consensus around adding built-in
> support for Any in the lite runtime, so I suspect that the status quo will
> probably remain for now. If you would like to use Any with the lite
> runtime, I think it's probably best to just manually serialize and parse to
> and from your Any fields, since that will work even if it involves a little
> extra boilerplate.
>
> On Tue, Nov 29, 2016 at 11:20 AM, Mohamed Koubaa  > wrote:
>
>> Hello,
>>
>> I am sorry to bring back an old thread, but the outcome is not clear.  Is
>> there either an intent or any ongoing work to support Any types with the
>> lite runtime?
>>
>> Best Regards,
>> Mohamed Koubaa
>> Software Developer
>> ANSYS Inc
>>
>> On Mon, Oct 10, 2016 at 3:00 PM, 'Adam Cozzette' via Protocol Buffers <
>> protobuf@googlegroups.com> wrote:
>>
>>>
>>> On Fri, Oct 7, 2016 at 2:16 PM, Arpit Baldeva 
>>> wrote:
>>>
 Thanks for the info.

 I feel like without pack/unpack/Is method, the utility of Any will
 diminish. For example, the rpc status proto (
 https://github.com/googleapis/googleapis/blob/master/google
 /rpc/status.proto) uses repeated Any field. It'd not be possible to
 write code like one described here - https://developers.google.co
 m/protocol-buffers/docs/proto3#any because you won't know if it is
 safe to convert value to a give message. I also came across this post after
 my post which marks the request as a bug currently -
 https://github.com/google/protobuf/issues/1974

>>>
>>> What you're saying makes sense, we might want to consider just updating
>>> Any to have first-class support for MessageLite. In C++ this would be
>>> straightforward but in Java, for example, we would need to think carefully
>>> about how to do it because in Java lite we don't currently have the message
>>> names available at runtime.
>>>
>>> Regarding the future of GetTypeName, though it has overhead, feel like
 it could have many utilities outside of the Any support as well. I don't
 have concrete use case in mind though as I am just starting on protobuf.
 This brings another important question that I was wondering if somebody
 already has data around. There are two options for reducing code bloat. One
 is Lite and another is code_size. I understand that lite reduces code bloat
 by removing descriptors/reflections related code (thereby reducing the
 library size) and code_size reduces the code bloat by generating less code
 per message but puts descriptors/reflectors back in(shared code). And the
 recommendation is to choose code_size option if number of message are much
 higher compared to the overhead caused by the size of lib. Are there any
 benchmarks around what the size of the library is (and lite version) and
 what is the per message overhead saved by the code_size option? And the
 performance drop with code_size option?

>>>
>>> Here's one way to break it down.
>>>
>>> SPEED:
>>> - Fixed overhead of full runtime (e.g. the Message class)
>>> - Per-message overhead of generated parsing/serialization code
>>> - Per-message overhead of generated descriptors
>>>
>>> LITE_RUNTIME:
>>> - Fixed overhead of lite runtime (e.g. includes MessageLite but not
>>> Message)
>>> - Per-message overhead of generated parsing/serialization code
>>>
>>> CODE_SIZE:
>>> - Fixed overhead of full runtime (e.g. the Message class)
>>> - Per-message overhead of generated descriptors
>>>
>>> SPEED and LITE_RUNTIME should be about the same speed because they both
>>> benefit from the fast generated code for parsing and serialization, while
>>> CODE_SIZE is much slower because it relies on reflection instead of
>>> generated code.