Re: combine protobuf messages

[Kenton Varda]

If you can just write code specific to your message type which deals with
merging items with the same ID, I'd suggest doing that.  If you really need
to write a totally generic algorithm, you can use reflection as documented
here:
http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.message.html#Reflection

On Fri, Aug 14, 2009 at 4:46 PM, George Georgiev  wrote:

>  Hi,
>
> MergeFrom sounds good.
>
> The only issue that I still will have is with the repeated fields. For
> some of them I will have an Id attribute. So what I would like to achieve
> is when I merge the messages instead of adding new partial message in the
> list it to combine those of the repeated messages that have the same Id.
>
> >> custom algorithm based on reflection
> I suppose this is what I need in this case. Could you please give me a bit
> more information what you mean.
>
> Thanks
> -George
>
>  --
> *From:* Kenton Varda [mailto:ken...@google.com]
> *Sent:* Friday, August 14, 2009 2:01 PM
> *To:* George Georgiev
> *Cc:* Protocol Buffers
> *Subject:* Re: combine protobuf messages
>
> On Fri, Aug 14, 2009 at 1:18 PM, George Georgiev <
> georgi.georg...@citrix.com> wrote:
>
>>  1. How to serialize parts from the message without validation
>>
>
> Use the "Partial" serialization and parsing methods, e.g.
> SerializePartialToString() and ParsePartialFromString().  These do not check
> required fields.
>
>
>>  2. How to combine two parts. Or how to parse second part which may
>> overlap some of the already existing parts
>>
>
> Use MergeFrom():
>
>   MyMessage combined;
>   // Iterate over messages from lowest to highest priority.
>   for (int priority = 0; priority < max_priority; ++priority) {
> combined.MergeFrom(messages_by_priority[priority]);
>   }
>
>   // Check
>   if (!combined.IsInitialized()) {
> cerr << "Missing required fields: "
>  << combined.InitializationErrorString() << endl;
> return false;
>   }
>
> As documented, the semantics of MergeFrom() are:  "Singular fields will be
> overwritten, except for embedded messages which will be merged. Repeated
> fields will be concatenated."
>
> So, if you merge the highest-priority message last, then its values will
> end up taking precedence.
>
> If MergeFrom() isn't quite what you want, then you'll have to write a
> custom algorithm based on reflection.
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

RE: combine protobuf messages

[George Georgiev]

Hi,

MergeFrom sounds good.

The only issue that I still will have is with the repeated fields. For some of 
them I will have an Id attribute. So what I would like to achieve is when I 
merge the messages instead of adding new partial message in the list it to 
combine those of the repeated messages that have the same Id.

>> custom algorithm based on reflection
I suppose this is what I need in this case. Could you please give me a bit more 
information what you mean.

Thanks
-George


From: Kenton Varda [mailto:ken...@google.com]
Sent: Friday, August 14, 2009 2:01 PM
To: George Georgiev
Cc: Protocol Buffers
Subject: Re: combine protobuf messages

On Fri, Aug 14, 2009 at 1:18 PM, George Georgiev 
mailto:georgi.georg...@citrix.com>> wrote:
1. How to serialize parts from the message without validation

Use the "Partial" serialization and parsing methods, e.g. 
SerializePartialToString() and ParsePartialFromString().  These do not check 
required fields.

2. How to combine two parts. Or how to parse second part which may overlap some 
of the already existing parts

Use MergeFrom():

  MyMessage combined;
  // Iterate over messages from lowest to highest priority.
  for (int priority = 0; priority < max_priority; ++priority) {
combined.MergeFrom(messages_by_priority[priority]);
  }

  // Check
  if (!combined.IsInitialized()) {
cerr << "Missing required fields: "
 << combined.InitializationErrorString() << endl;
return false;
  }

As documented, the semantics of MergeFrom() are:  "Singular fields will be 
overwritten, except for embedded messages which will be merged. Repeated fields 
will be concatenated."

So, if you merge the highest-priority message last, then its values will end up 
taking precedence.

If MergeFrom() isn't quite what you want, then you'll have to write a custom 
algorithm based on reflection.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: combine protobuf messages

[Kenton Varda]

On Fri, Aug 14, 2009 at 1:18 PM, George Georgiev  wrote:

>  1. How to serialize parts from the message without validation
>

Use the "Partial" serialization and parsing methods, e.g.
SerializePartialToString() and ParsePartialFromString().  These do not check
required fields.


> 2. How to combine two parts. Or how to parse second part which may overlap
> some of the already existing parts
>

Use MergeFrom():

  MyMessage combined;
  // Iterate over messages from lowest to highest priority.
  for (int priority = 0; priority < max_priority; ++priority) {
combined.MergeFrom(messages_by_priority[priority]);
  }

  // Check
  if (!combined.IsInitialized()) {
cerr << "Missing required fields: "
 << combined.InitializationErrorString() << endl;
return false;
  }

As documented, the semantics of MergeFrom() are:  "Singular fields will be
overwritten, except for embedded messages which will be merged. Repeated
fields will be concatenated."

So, if you merge the highest-priority message last, then its values will end
up taking precedence.

If MergeFrom() isn't quite what you want, then you'll have to write a custom
algorithm based on reflection.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

combine protobuf messages

[George Georgiev]

Hi,

I have a following use case and I'm not sure what will be the best way to go.

I have a relatively complex protobuf message that I'm using in my application. 
It should be correctly initialized my application to works.

The problem I have is that the information that makes this complete message is 
provided from several different sources. I have no control over them (how much 
and what information they will provide). The only requirement is at the end 
after all of the information is collected from all of the sources the message 
is full. The sources will have priorities so if there are overlaps of the 
information the information from the source with high priority will be used.

What most probably I need is:
1. How to serialize parts from the message without validation
2. How to combine two parts. Or how to parse second part which may overlap some 
of the already existing parts

Any other ideas?

thanks,
George

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: protobuf HPUX support

[Kenton Varda]

Thanks.  In order to accept this I'll need you to sign the Contributor
License Agreement:
http://code.google.com/legal/individual-cla-v1.0.html -- If you own
copyright on this patch.
http://code.google.com/legal/corporate-cla-v1.0.html -- If your employer
does.

You can sign the individual agreement via the web page.

Let me know when you've signed and I'll submit this.

2009/8/13 COFF 

>
> New patch avoid this sent.
> On 14 авг, 10:54, COFF  wrote:
> > Thank you for answer. I will try doing this way too.
> >
> > On 14 авг, 10:18, Kenton Varda  wrote:
> >
> > > Instead of changing everything that calls EXPECT_EQ on strings, can we
> > > change the implementation of EXPECT_EQ itself so that doesn't trigger
> this
> > > error?  For example, could it be overloaded with an explicit
> implementation
> > > for non-const strings?
> > > Any changes to gtest will have to be submitted back to the gtest
> project.
> > >  Protocol Buffers ships with the verbatim gtest code -- we don't
> maintain
> > > local modifications.
> >
> > > On Thu, Aug 13, 2009 at 7:47 PM, COFF  wrote:
> >
> > > > I sent patches directly to you.
> > > > Note on adding  also VC 7.1 support (macro EXPECT_EQ produse Internal
> > > > Compiller Error on non const std::string)
> >
> > > > On 14 авг, 02:24, Kenton Varda  wrote:
> > > > > Please send a patch.  Use "diff -u" to create it (see the man
> page), or
> > > > if
> > > > > you made your changes against the SVN sources, just use "svn diff".
> >
> > > > > On Wed, Aug 12, 2009 at 10:08 PM, COFF  wrote:
> >
> > > > > > Hello Kenton!
> >
> > > > > > I succesfully merge my project on new version of protobuf.
> > > > > > I made patches that enable HPUX (and partialy STLPORT) support in
> > > > > > protobuf (only .cc and .h - I not use autoconf stuff).
> > > > > > Can and how I send it to you?
> >
> > > > > > Best regards,
> > > > > > Alex.
> >
> >
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Core Dump with c++ 4.1.1

[Kenton Varda]

I assume that when you say "C++ 4.1.1", what you mean is "G++ 4.1.1", i.e.
GCC 4.1.1's C++ compiler?  Or do you mean some other compiler?  C++ itself
does not have version numbers.
I have tested protocol buffers on many different versions of GCC, including
both the 3.x and 4.x range.  It seems very unlikely that 4.1.1 in particular
doesn't work.

Can you run the tests?  I.e. do "make check" in the top-level protobuf
directory.  If the tests pass, then the problem is certainly something with
the way you are installing the libraries.  Maybe you installed them to the
wrong location, and then when you build the examples you are actually
building them against the old version?  The crash looks like something that
would happen if you mixed compiler versions.

There's not much more I can say with only one line of a stack trace.  If you
can't figure it out, it might help if you provided the whole stack.

On Fri, Aug 14, 2009 at 8:52 AM, Sushil Shelly  wrote:

> Kenton and Team,
>
> We recently moved to using c++ 4.1.1 and are getting a segmentation fault
> as shown below. We are simply building the tutorial code and then run
> 'add_person' (This same test runs fine when built with C++ 3.4.0).
>
> Partial core dump is shown below!!
> -
> Core was generated by `add_person'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x0809dbc7 in std::_Rb_tree std::pair >, std::_Select1st std::pair > >, std::less,
> std::allocator > >
> >::insert_unique (this=0x4,
> _...@0xbfbed9c8)
> -
>
> This is fairly urgent for us and is a show-stopper - could you please
> comment.
>
> thanks,
>
> Sushil.
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Endian

[Kenton Varda]

The encoding documentation:
http://code.google.com/apis/protocolbuffers/docs/encoding.html

specifies
that fixed-width numbers are encoded in little-endian order.  However, you
only have to know that if you are writing a protobuf parser or encoder.  As
a user, this is all handled automatically.

On Fri, Aug 14, 2009 at 8:28 AM, DaveP  wrote:

>
> I quickly reviewed the documentation and the faq and found no
> reference to the protocol demanding big endian or little endian or
> associated byte order conversion.  Does the protocol require one or
> the other?  Alternately, does the compiled have an "endian" switch/
> conversion option in the resulting code?  Specifically, I'll be
> generating binary data on a big endian and sending to a little endian.
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Endian

[Henner Zeller]

On Fri, Aug 14, 2009 at 8:28 AM, DaveP wrote:
>
> I quickly reviewed the documentation and the faq and found no
> reference to the protocol demanding big endian or little endian or
> associated byte order conversion.  Does the protocol require one or
> the other?  Alternately, does the compiled have an "endian" switch/
> conversion option in the resulting code?  Specifically, I'll be
> generating binary data on a big endian and sending to a little endian.

The generated code of protocol buffers guarantee that you get read the
data out in the correct byte order for the local machine. The _binary_
encoding of the protocol buffers is independent of the platform it is
sent from, so you don't have to care about that. So you're all good ;)

-h

(the binary format is doing little endian because of slightly better
performance on the typical machines Google uses .. but as I said, you
don't have to care about that).

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Core Dump with c++ 4.1.1

[Monty Taylor]

Sushil Shelly wrote:
> Yea we do a make clean and rebuild to get new libraries, Is there any
> one actually using C++ 4.1.1?

By C++ 4.1.1 I'm assuming you are using GCC? If so, yes, I use this on
several of the machines in my build farm for Drizzle (which uses
Protobuf quite heavily) with no problems.

If you do not mean GCC, then what compiler are you talking about?

Monty

> On Fri, Aug 14, 2009 at 11:59 AM, Monty Taylor  > wrote:
> 
> Sushil Shelly wrote:
> > Kenton and Team,
> >
> > We recently moved to using c++ 4.1.1 and are getting a segmentation
> > fault as shown below. We are simply building the tutorial code and
> then
> > run 'add_person' (This same test runs fine when built with C++ 3.4.0).
> 
> Did you re-build protobuf after upgrading to 4.1.1? C++ doesn't
> guarantee binary compatibility in this particular case - so if your
> library was built with the old compiler version and the test program
> with the new, it may be an issue.
> 
> > Partial core dump is shown below!!
> > -
> > Core was generated by `add_person'.
> > Program terminated with signal 11, Segmentation fault.
> > #0  0x0809dbc7 in std::_Rb_tree > const, std::pair >,
> > std::_Select1st const*, int>
> >> >, std::less, std::allocator const,
> > std::pair > > >::insert_unique (this=0x4,
> > _...@0xbfbed9c8)
> > -
> >
> > This is fairly urgent for us and is a show-stopper - could you please
> > comment..
> >
> > thanks,
> >
> > Sushil.
> >
> > > >
> 
> 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Core Dump with c++ 4.1.1

[Sushil Shelly]

Yea we do a make clean and rebuild to get new libraries, Is there any one
actually using C++ 4.1.1?


On Fri, Aug 14, 2009 at 11:59 AM, Monty Taylor  wrote:

> Sushil Shelly wrote:
> > Kenton and Team,
> >
> > We recently moved to using c++ 4.1.1 and are getting a segmentation
> > fault as shown below. We are simply building the tutorial code and then
> > run 'add_person' (This same test runs fine when built with C++ 3.4.0).
>
> Did you re-build protobuf after upgrading to 4.1.1? C++ doesn't
> guarantee binary compatibility in this particular case - so if your
> library was built with the old compiler version and the test program
> with the new, it may be an issue.
>
> > Partial core dump is shown below!!
> > -
> > Core was generated by `add_person'.
> > Program terminated with signal 11, Segmentation fault.
> > #0  0x0809dbc7 in std::_Rb_tree > const, std::pair >,
> > std::_Select1st
> >> >, std::less, std::allocator > std::pair > > >::insert_unique (this=0x4,
> > _...@0xbfbed9c8)
> > -
> >
> > This is fairly urgent for us and is a show-stopper - could you please
> > comment..
> >
> > thanks,
> >
> > Sushil.
> >
> > > >
>
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Re: Core Dump with c++ 4.1.1

[Monty Taylor]

Sushil Shelly wrote:
> Kenton and Team,
> 
> We recently moved to using c++ 4.1.1 and are getting a segmentation
> fault as shown below. We are simply building the tutorial code and then
> run 'add_person' (This same test runs fine when built with C++ 3.4.0).

Did you re-build protobuf after upgrading to 4.1.1? C++ doesn't
guarantee binary compatibility in this particular case - so if your
library was built with the old compiler version and the test program
with the new, it may be an issue.

> Partial core dump is shown below!!
> -
> Core was generated by `add_person'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x0809dbc7 in std::_Rb_tree const, std::pair >,
> std::_Select1st
>> >, std::less, std::allocator std::pair > > >::insert_unique (this=0x4,
> _...@0xbfbed9c8)
> -
> 
> This is fairly urgent for us and is a show-stopper - could you please
> comment..
> 
> thanks,
> 
> Sushil.
> 
> > 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Core Dump with c++ 4.1.1

[Sushil Shelly]

Kenton and Team,

We recently moved to using c++ 4.1.1 and are getting a segmentation fault as
shown below. We are simply building the tutorial code and then run
'add_person' (This same test runs fine when built with C++ 3.4.0).

Partial core dump is shown below!!
-
Core was generated by `add_person'.
Program terminated with signal 11, Segmentation fault.
#0  0x0809dbc7 in std::_Rb_tree >, std::_Select1st > >, std::less,
std::allocator > >
>::insert_unique (this=0x4,
_...@0xbfbed9c8)
-

This is fairly urgent for us and is a show-stopper - could you please
comment.

thanks,

Sushil.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---

Endian

[DaveP]

I quickly reviewed the documentation and the faq and found no
reference to the protocol demanding big endian or little endian or
associated byte order conversion.  Does the protocol require one or
the other?  Alternately, does the compiled have an "endian" switch/
conversion option in the resulting code?  Specifically, I'll be
generating binary data on a big endian and sending to a little endian.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---