Re: Repeated message field in c pacakge
Thank you very much Dave for your quick and detailed replies
I'll try that out soon.
Aviad
On 1 דצמבר, 04:50, daveb [EMAIL PROTECTED] wrote:
BTW: I just released protobuf-c 0.6 which include the item__init()
function (and it also supports default-values properly now).
- dave
On Nov 27, 7:48 pm, daveb [EMAIL PROTECTED] wrote:
I should mention too: I plan to eventually add a generated function
item__init(Item *) to initialize a message. i plan to add this when
i fix default values for non-fundamental types (which are currently
ignored).
(i haven't had a huge amount of time to deal with these few niggling
issues, but i've now got the design in my head, and the coding should
be pretty easy)
- dave
On Nov 27, 7:42 pm, daveb [EMAIL PROTECTED] wrote:
The reason why it is crashing is that all messages have a few builtin
members that are usually initialized via ITEM__INIT. If you look at
the generated Item it looks like:
struct _Item
{
ProtobufCMessage base_message;
uint32_t id;
};
The best way to construct a message is to use C's assignment
operator. Rewriting your code strictly:
LookupReply nlr = LOOKUP_REPLY__INIT;
Item item = ITEM__INIT;
(nlr)-replica_list = (Item**)malloc(1*sizeof(Item*));
(nlr)-n_replica_list = 1;
Item* item1 = (Item*)malloc(sizeof(Item));
item.id = 1;
*item1 = item;
(nlr)-replica_list[0] = item1;
int len = lookup_reply__get_packed_size(nlr);
But that's kind of overkill in a way, it's shorter just to write:
LookupReply nlr = LOOKUP_REPLY__INIT;
Item item1 = ITEM__INIT;
Item *items[1] = { item };
nlr.n_replica_list = 1;
nlr.replica_list = items;
item1.id = 1;
int len = lookup_reply__get_packed_size(nlr);
Furthermore, using the latter approach, you can avoid malloc(), and
therefore (most of) the possibility of memory leaks.
On Nov 25, 8:54 am, [EMAIL PROTECTED] wrote:
Hi,
I'm using the protocol buffers package for c. It's very useful and
handy. However I've encountered a problem when trying to use a message
with a nested repeated field of an other message type.
This is the simplified version -
message Item {
int32 id = 1;
}
message LookupReply {
repeated Item replica_list = 1;
}
Here is the code for constructing the message:
...
LookupReply nlr = LOOKUP_REPLY__INIT;
(nlr)-replica_list = (Item**)malloc(1*sizeof(Item*));
(nlr)-n_replica_list = 1;
Item* item1 = (Item*)malloc(sizeof(Item));
item1-id = 1;
(nlr)-replica_list[0] = item1;
int len = lookup_reply__get_packed_size(nlr);
...
The call to lookup_reply__get_packed_size causes a segmentation
fault.
I first tried using a string id field in the Item message, and that
failed too.
However when I used a string repeated field (with a few obvious
semantic changes) everything worked fine.
Anyone else has maybe used repeated message field in the c package and
got it to work (or if not, can explain why it doesn't work)?
Thank you,
Aviad-הסתר טקסט מצוטט-
-הראה טקסט מצוטט-
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Slicing support in Python
(Okay, back on track) On Tue, Dec 2, 2008 at 11:17 PM, Kenton Varda [EMAIL PROTECTED] wrote: On Tue, Dec 2, 2008 at 11:08 PM, Alek Storm [EMAIL PROTECTED] wrote: I would think encoding and decoding would be the main bottlenecks, so can't those be wrappers around C++, while let object handling (reflection.py and friends) be pure-python? It seems like the best of both worlds. Well, the generated serializing and parsing code in C++ is an order of magnitude faster than the dynamic (reflection-based) code. But to use generated code you need to be using C++ object handling. Not if you decouple them. Abstractly, the C++ parser receives a serialized message and descriptor and returns a tree of the form [(tag_num, value)] where tag_num is an integer and value is either a scalar or a subtree (for submessages). The Python reflection code takes the tree and fills the message object with its values. It's simple, fast, and the C++ parser can be easily swapped out for a pure-Python one on systems that don't support the C++ version. Run this backwards when serializing, and you get another advantage: you can easily swap out the function that converts the tree into serialized protobuf for one that outputs XML, JSON, etc. You're right. If it's a waste of time for them, most people won't use it. But if there's no point to it, why do normal Python lists have it? It's useful enough to be included there. And since repeated fields act just like lists, it should be included here too. I think Python object lists are probably used in a much wider variety of ways than protocol buffer repeated fields generally are. Let's include it - it gives us a more complete list interface, there's no downside, and the users can decide whether they want to use it. We can't predict all possible use cases. In fact, it doesn't even have to be useful for repeated composites. The fact that repeated scalars have it means it's automatically included for repeated composites, because they should have the exact same interface. Polymorphism is what we want here. But they already can't have the same interface because append() doesn't work. :) We don't have confirmation on that yet ;). Having the same interface is what we should be shooting for. Thanks, Alek Storm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protocol Buffers Compiler Maven Plug-In
The intertubes have led you to the right place. This has actually been done for a while, but I haven't written the tests (arg!). I'll definitely get it checked in by the end of the year. Sorry for the procrastination. On Wed, Dec 3, 2008 at 2:59 PM, Kenton Varda [EMAIL PROTECTED] wrote: [cc Gregory] On Wed, Dec 3, 2008 at 8:39 AM, Tom Nichols [EMAIL PROTECTED] wrote: Hi guys, Has anything become of this? I'm looking for a protobuf compiler Maven plugin and this is the only good lead I can find on the intertubes. Thanks. -Tom On Jul 24, 2:36 am, Tom [EMAIL PROTECTED] wrote: Greg, That's good news. I'm hoping to see your implementation soon. Can you ping me as soon as your code is available, will you? I'll also create separate project (open source) and check in my implementation so then we can easier compare and integrate our efforts if needed. Take care, Tom On Jul 23, 10:02 pm, Gregory Kick [EMAIL PROTECTED] wrote: Tom, As Kenton said, I'm working on apluginto do much of the same work and a bit more with regards to dependencies. I plan on having an alpha version by the end of the week and would love some feedback. If my implementation falls short, the next step would be to fill out the contributor agreement [1] and we'll integrate the two implementations. In any event, I'll send you an email as soon as something is publicly available. Sound good? [1]http://code.google.com/legal/individual-cla-v1.0.html On Wed, Jul 23, 2008 at 3:41 PM, Kenton Varda [EMAIL PROTECTED] wrote: Greg Kick (of Google) was also working on aMavenplugin. Greg, what should we do? On Wed, Jul 23, 2008 at 7:05 AM, Tom [EMAIL PROTECTED] wrote: Hi lads, I've implemented aMaven2 plug-in that invokes protoc (Protocol Buffers Compiler) to generate Java classes. It hooks into build life cycle at 'generate-sources' phrase. It scans specified directories to find .proto files and invokes protoc executable to generate Java classes into specified directory. Pretty useful stuff for allMaven users, I guess. I can contribute the source code to the main project if you are interested. Otherwise I'll put in on Mojo (Codehaus) or create separate project on Google Code. Please, lte me know if you are interested in my contribution. Cheers, Tom --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Slicing support in Python
On Dec 2, 10:49 pm, Kenton Varda [EMAIL PROTECTED] wrote:
C++ compatibility matters because eventually we want to be able to generate
Python code which just wraps C++ code for efficiency. C++ isn't garbage
collected, so append() can't easily be implemented in this case without
having ownership problems. Slice assignment has the same problem.
Also note that even pure-python protocol buffers have a sort of ownership
issue: Sub-messages all contain pointers back to their parents, so that
when a sub-message is modified, the parent's cached size can be marked
dirty. (Also, singular sub-messages have to inform their parents when the
first field within them is set, but that doesn't apply here.)
While you're on this topic, I ran into this ownership issue while
implementing
the Perl/XS wrapper around the generated C++ code. I think it is the
same
issue that would face the author of a Python or Ruby C++ extension of
the
generated C++. I ended up having to new() a copy of every message
that I
transferred from C++ to Perl or vice versa. So, for example, a
statement like
$team-member($i)-set_first_name('Dave');
won't have the same effect as (C++)
team.mutable_member(i)-set_first_name(Dave);
because $team-member($i) will generate a copy of the underlying C++
object, so that it can be managed by Perl's reference counting without
any
concern as to whether or not the underlying C++ object has been
deleted
because the containing message went out of scope.
Anyway, I thought it might be possible to allow for shared ownership
of a
message object if there were a reference counted variant of
RepeatedPtrFieldT
(something like RepeatedSharedPtrFieldT or whatever), which would
provide
incref() and decref() methods such that Perl and C++ could use the
same
underlying C++ objects in the generated code. This would really help
the
performance of the Perl/XS code if all of that copy construction could
be
avoided somehow. The C++ code generator would need an option that
would
instruct it to generate RepeatedSharedPtrFieldT members (and incref
and
decref calls, where appropriate) for repeated messages (instead of
using the
default RepeatedPtrFieldT).
What do you think? Is something like this possible, even though it
would
require a change to protobuf? It is an issue for all {Python, Perl,
Ruby, ...}/C++
extension wrappers for Protocol Buffers.
-dave
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Slicing support in Python
Ehhh... Reference counting is slow (assuming it needs to be thread-safe),
and I think even adding it as an option would add an excessive amount of
complication to the system.
On Wed, Dec 3, 2008 at 2:04 PM, Dave Bailey [EMAIL PROTECTED] wrote:
On Dec 3, 2:00 pm, Dave Bailey [EMAIL PROTECTED] wrote:
On Dec 2, 10:49 pm, Kenton Varda [EMAIL PROTECTED] wrote:
C++ compatibility matters because eventually we want to be able to
generate
Python code which just wraps C++ code for efficiency. C++ isn't
garbage
collected, so append() can't easily be implemented in this case without
having ownership problems. Slice assignment has the same problem.
Also note that even pure-python protocol buffers have a sort of
ownership
issue: Sub-messages all contain pointers back to their parents, so
that
when a sub-message is modified, the parent's cached size can be marked
dirty. (Also, singular sub-messages have to inform their parents when
the
first field within them is set, but that doesn't apply here.)
(Here is my post without all of the ridiculous formatting):
While you're on this topic, I ran into this ownership issue while
implementing the Perl/XS wrapper around the generated C++ code. I
think it is the same issue that would face the author of a Python or
Ruby C++ extension of the generated C++. I ended up having to new() a
copy of every message that I transferred from C++ to Perl or vice
versa. So, for example, a statement like
$team-member($i)-set_first_name('Dave');
won't have the same effect as (C++)
team.mutable_member(i)-set_first_name(Dave);
because $team-member($i) will generate a copy of the underlying C++
object, so that it can be managed by Perl's reference counting without
any concern as to whether or not the underlying C++ object has been
deleted because the containing message went out of scope.
Anyway, I thought it might be possible to allow for shared ownership
of a message object if there were a reference counted variant of
RepeatedPtrFieldT (something like RepeatedSharedPtrFieldT or
whatever), which would provide incref() and decref() methods such that
Perl and C++ could use the same underlying C++ objects in the
generated code. This would really help the performance of the Perl/XS
code if all of that copy construction could be avoided somehow. The C+
+ code generator would need an option that would instruct it to
generate RepeatedSharedPtrFieldT members (and incref and decref
calls, where appropriate) for repeated messages (instead of using the
default RepeatedPtrFieldT).
What do you think? Is something like this possible, even though it
would require a change to protobuf? It is an issue for all {Python,
Perl, Ruby, ...}/C++ extension wrappers for Protocol Buffers. I have
found that protobuf is a faster Perl data serialization mechanism that
the (generic) Storable module, but I think it can be even faster.
-dave
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Slicing support in Python
It still adds a lot of complication. And I think most cases where people
start out thinking thread-safety won't be an issue, particularly with
reference counting, they later find out otherwise.
On Wed, Dec 3, 2008 at 3:22 PM, Dave Bailey [EMAIL PROTECTED] wrote:
What if thread safety wasn't an issue?
-dave
On Dec 3, 2:41 pm, Kenton Varda [EMAIL PROTECTED] wrote:
Ehhh... Reference counting is slow (assuming it needs to be
thread-safe),
and I think even adding it as an option would add an excessive amount of
complication to the system.
On Wed, Dec 3, 2008 at 2:04 PM, Dave Bailey [EMAIL PROTECTED] wrote:
On Dec 3, 2:00 pm, Dave Bailey [EMAIL PROTECTED] wrote:
On Dec 2, 10:49 pm, Kenton Varda [EMAIL PROTECTED] wrote:
C++ compatibility matters because eventually we want to be able to
generate
Python code which just wraps C++ code for efficiency. C++ isn't
garbage
collected, so append() can't easily be implemented in this case
without
having ownership problems. Slice assignment has the same problem.
Also note that even pure-python protocol buffers have a sort of
ownership
issue: Sub-messages all contain pointers back to their parents, so
that
when a sub-message is modified, the parent's cached size can be
marked
dirty. (Also, singular sub-messages have to inform their parents
when
the
first field within them is set, but that doesn't apply here.)
(Here is my post without all of the ridiculous formatting):
While you're on this topic, I ran into this ownership issue while
implementing the Perl/XS wrapper around the generated C++ code. I
think it is the same issue that would face the author of a Python or
Ruby C++ extension of the generated C++. I ended up having to new() a
copy of every message that I transferred from C++ to Perl or vice
versa. So, for example, a statement like
$team-member($i)-set_first_name('Dave');
won't have the same effect as (C++)
team.mutable_member(i)-set_first_name(Dave);
because $team-member($i) will generate a copy of the underlying C++
object, so that it can be managed by Perl's reference counting without
any concern as to whether or not the underlying C++ object has been
deleted because the containing message went out of scope.
Anyway, I thought it might be possible to allow for shared ownership
of a message object if there were a reference counted variant of
RepeatedPtrFieldT (something like RepeatedSharedPtrFieldT or
whatever), which would provide incref() and decref() methods such that
Perl and C++ could use the same underlying C++ objects in the
generated code. This would really help the performance of the Perl/XS
code if all of that copy construction could be avoided somehow. The C+
+ code generator would need an option that would instruct it to
generate RepeatedSharedPtrFieldT members (and incref and decref
calls, where appropriate) for repeated messages (instead of using the
default RepeatedPtrFieldT).
What do you think? Is something like this possible, even though it
would require a change to protobuf? It is an issue for all {Python,
Perl, Ruby, ...}/C++ extension wrappers for Protocol Buffers. I have
found that protobuf is a faster Perl data serialization mechanism that
the (generic) Storable module, but I think it can be even faster.
-dave
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
