Why copy read stream and zero-copy read streams are not compatible?
Hi, The ZeroCopyInputStream and CopyingInputStream classes both define a Skip() function, but one returns a bool and the other returns an int. Because of this, they are not compatible, so I can't write a class that implements both interfaces for example. It is sometimes useful to be able to implement both interfaces. I wrote a zero copy socket input stream, and having the ability to use the copying interface is desirable for ease of use (and testing). So I wrote a Read() function that calls Next()/BackUp() then do a memcpy. This can even be a generic function in the "abstract" interface. This is what Java does for example: http://java.sun.com/javase/6/docs/api/java/io/InputStream.html It feels wrong to me that the two interfaces are incompatible by design, and it would be really cool if I can write something that implements both (so I can have a Skip() function for one). Making the ZeroCopy version of Skip() returns an int instead of bool would make this possible. What do you think? Frank --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Flush operation on ZeroCopyOutputStream
I recently ran into this problem as well. I wrote a socket stream
that has a flush operation. To use it I had to do
{
CodedOutputStream coded(&my_zero_copy_socket_stream);
message.SerializeToCodedStream(&coded);
}
// coded is now out of scope so it has back-up unused buffer
my_zero_copy_socket_stream.Flush();
I had to use coded stream temporarily to write, then call Flush() on
the underlying stream AFTER it goes out of scope. It feels a bit
ugly.
But then again, Flush() does not really make sense for many things,
and having its existence in the abstract interface clouds its meaning
(so you would have to say in ArrayOutputStream that Flush() does
nothing.
Frank
On Oct 20, 2:37 pm, ptab <[EMAIL PROTECTED]> wrote:
> Hello,
>
> Why there is no Flush() operation on ZeroCopyOutputStream and on
> CodedOutputStream? Is there any way to persist/send over network
> whole current content of buffers ?
> I think I would be able to provide implementation of the method for
> most stream's implementations, but I think that it was a design
> decision.
> What's the reason ?
>
> I would like to implemented some communication over network based on
> these
> streams and flush operation is crucial to avoid blocking on both ends
> of the protocol.
>
> Thanks,
> Piotr Tabor
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Patch for zero_copy_stream_impl.cc
Oh, and thanks for catching these!
On Mon, Oct 20, 2008 at 2:38 PM, Kenton Varda <[EMAIL PROTECTED]> wrote:
> Would you care to add some tests to the unit test to check for these bugs?
> Otherwise, I'll do that when I apply these patches.
>
>
> On Mon, Oct 20, 2008 at 2:13 PM, <[EMAIL PROTECTED]> wrote:
>
>>
>> Sorry, I did some reading, and this should the format I send patch in:
>>
>>
>> Index: src/google/protobuf/io/zero_copy_stream_impl.cc
>> ===
>> --- src/google/protobuf/io/zero_copy_stream_impl.cc (revision 67)
>> +++ src/google/protobuf/io/zero_copy_stream_impl.cc (working copy)
>> @@ -218,7 +218,8 @@
>> char junk[4096];
>> int skipped = 0;
>> while (skipped < count) {
>> -int bytes = Read(junk, min(count,
>> implicit_cast(sizeof(junk;
>> +int bytes = Read(junk, min(count - skipped,
>> + implicit_cast(sizeof(junk;
>> if (bytes <= 0) {
>> // EOF or read error.
>> return skipped;
>> @@ -757,7 +758,7 @@
>> }
>>
>> bool LimitingInputStream::Next(const void** data, int* size) {
>> - if (limit_ < 0) return false;
>> + if (limit_ <= 0) return false;
>>if (!input_->Next(data, size)) return false;
>>
>> limit_ -= *size;
>>
>>
>> Thanks,
>> Frank
>>
>> On Oct 17, 6:33 am, [EMAIL PROTECTED] wrote:
>> > Whoops, sorry, I meant the other way around
>> > 761c761
>> > < if (limit_ < 0) return false;
>> > ---
>> >
>> > > if (limit_ <= 0) return false;
>> >
>> > Frank
>> >
>> > On Oct 17, 6:32 am, [EMAIL PROTECTED] wrote:
>> >
>> > > Another patch:
>> > > 761c761
>> > > < if (limit_ <= 0) return false;
>> > > ---
>> >
>> > > > if (limit_ < 0) return false;
>> >
>> > > Frank
>> >
>> > > On Oct 17, 3:11 am, [EMAIL PROTECTED] wrote:
>> >
>> > > > I found a bug in zero_copy_stream_impl.cc. Here's the patch
>> > > > 221c221,222
>> > > > < int bytes = Read(junk, min(count,
>> > > > implicit_cast(sizeof(junk;
>> > > > ---
>> >
>> > > > > int bytes = Read(junk, min(count - skipped,
>> > > > >implicit_cast(sizeof(junk;
>> >
>> > > > Frank
>> >
>> > > > PS. Please let me know if this is the right way of sending in
>> patches.
>> >>
>>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Patch for zero_copy_stream_impl.cc
Would you care to add some tests to the unit test to check for these bugs?
Otherwise, I'll do that when I apply these patches.
On Mon, Oct 20, 2008 at 2:13 PM, <[EMAIL PROTECTED]> wrote:
>
> Sorry, I did some reading, and this should the format I send patch in:
>
>
> Index: src/google/protobuf/io/zero_copy_stream_impl.cc
> ===
> --- src/google/protobuf/io/zero_copy_stream_impl.cc (revision 67)
> +++ src/google/protobuf/io/zero_copy_stream_impl.cc (working copy)
> @@ -218,7 +218,8 @@
> char junk[4096];
> int skipped = 0;
> while (skipped < count) {
> -int bytes = Read(junk, min(count,
> implicit_cast(sizeof(junk;
> +int bytes = Read(junk, min(count - skipped,
> + implicit_cast(sizeof(junk;
> if (bytes <= 0) {
> // EOF or read error.
> return skipped;
> @@ -757,7 +758,7 @@
> }
>
> bool LimitingInputStream::Next(const void** data, int* size) {
> - if (limit_ < 0) return false;
> + if (limit_ <= 0) return false;
>if (!input_->Next(data, size)) return false;
>
> limit_ -= *size;
>
>
> Thanks,
> Frank
>
> On Oct 17, 6:33 am, [EMAIL PROTECTED] wrote:
> > Whoops, sorry, I meant the other way around
> > 761c761
> > < if (limit_ < 0) return false;
> > ---
> >
> > > if (limit_ <= 0) return false;
> >
> > Frank
> >
> > On Oct 17, 6:32 am, [EMAIL PROTECTED] wrote:
> >
> > > Another patch:
> > > 761c761
> > > < if (limit_ <= 0) return false;
> > > ---
> >
> > > > if (limit_ < 0) return false;
> >
> > > Frank
> >
> > > On Oct 17, 3:11 am, [EMAIL PROTECTED] wrote:
> >
> > > > I found a bug in zero_copy_stream_impl.cc. Here's the patch
> > > > 221c221,222
> > > > < int bytes = Read(junk, min(count,
> > > > implicit_cast(sizeof(junk;
> > > > ---
> >
> > > > > int bytes = Read(junk, min(count - skipped,
> > > > >implicit_cast(sizeof(junk;
> >
> > > > Frank
> >
> > > > PS. Please let me know if this is the right way of sending in
> patches.
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Flush operation on ZeroCopyOutputStream
Hello, Why there is no Flush() operation on ZeroCopyOutputStream and on CodedOutputStream? Is there any way to persist/send over network whole current content of buffers ? I think I would be able to provide implementation of the method for most stream's implementations, but I think that it was a design decision. What's the reason ? I would like to implemented some communication over network based on these streams and flush operation is crucial to avoid blocking on both ends of the protocol. Thanks, Piotr Tabor --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Patch for zero_copy_stream_impl.cc
Sorry, I did some reading, and this should the format I send patch in:
Index: src/google/protobuf/io/zero_copy_stream_impl.cc
===
--- src/google/protobuf/io/zero_copy_stream_impl.cc (revision 67)
+++ src/google/protobuf/io/zero_copy_stream_impl.cc (working copy)
@@ -218,7 +218,8 @@
char junk[4096];
int skipped = 0;
while (skipped < count) {
-int bytes = Read(junk, min(count,
implicit_cast(sizeof(junk;
+int bytes = Read(junk, min(count - skipped,
+ implicit_cast(sizeof(junk;
if (bytes <= 0) {
// EOF or read error.
return skipped;
@@ -757,7 +758,7 @@
}
bool LimitingInputStream::Next(const void** data, int* size) {
- if (limit_ < 0) return false;
+ if (limit_ <= 0) return false;
if (!input_->Next(data, size)) return false;
limit_ -= *size;
Thanks,
Frank
On Oct 17, 6:33 am, [EMAIL PROTECTED] wrote:
> Whoops, sorry, I meant the other way around
> 761c761
> < if (limit_ < 0) return false;
> ---
>
> > if (limit_ <= 0) return false;
>
> Frank
>
> On Oct 17, 6:32 am, [EMAIL PROTECTED] wrote:
>
> > Another patch:
> > 761c761
> > < if (limit_ <= 0) return false;
> > ---
>
> > > if (limit_ < 0) return false;
>
> > Frank
>
> > On Oct 17, 3:11 am, [EMAIL PROTECTED] wrote:
>
> > > I found a bug in zero_copy_stream_impl.cc. Here's the patch
> > > 221c221,222
> > > < int bytes = Read(junk, min(count,
> > > implicit_cast(sizeof(junk;
> > > ---
>
> > > > int bytes = Read(junk, min(count - skipped,
> > > > implicit_cast(sizeof(junk;
>
> > > Frank
>
> > > PS. Please let me know if this is the right way of sending in patches.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Cross compiling
The code generated by protoc is identical on all platforms, so you can use the same protoc that you compiled for Linux. You only have to cross-compile the libprotobuf library. Cross-compiling should work exactly like it does with any other autoconf-based package. On Mon, Oct 20, 2008 at 6:58 AM, <[EMAIL PROTECTED]> wrote: > > Hi, I am building and using protocol buffers 2.0.1 on a Linux host and > it works fine. > > Now I am moving to a target hardware with an ARM9 cpu, also running > Linux (and later to an ARM7 with no os) and would like to cross > compile to these platforms. I have tried to find information on how to > do this but have not found any information on the subject. > > Has anyone tested this? Can anybody help me out? > > Thanks for your help! > > /Magnus > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: protocol buffers not repeating items marked repeated
I don't see any obvious problems. Try printing the whole message just
before serializing and just after parsing to see if they are both as you
expect.
On Sat, Oct 18, 2008 at 11:24 PM, comsatcat <[EMAIL PROTECTED]> wrote:
>
> I have a simple client server in python... I have the following
> protobuf definition:
>
> message NodeListData {
>enum State {
>FALSE = 0;
>TRUE = 1;
>}
>required string node = 1;
>required string folder = 2;
>required State pendingcsr = 3;
> }
>
> message NodeList {
>repeated NodeListData items = 1;
> }
>
> message Packet {
>enum Type {
>ACK = 0;
>KEEP_ALIVE = 1;
>AUTHORIZE = 2;
>ADD_ENDPOINT = 3;
>GET_ENDPOINT_STATS = 4;
>ENDPOINT_STATS = 5;
>REMOVE_ENDPOINT = 6;
>NODE_LIST = 7;
>}
>required Type type = 1;
>optional Ack ack = 2;
>optional KeepAlive keepalive = 3;
>optional Authorize authorize = 4;
>optional AddEndpoint addendpoint = 5;
>optional EndpointStats endpointstats = 6;
>optional RemoveEndpoint removeendpoint = 7;
>optional NodeList nodelist = 8;
> }
>
> *note some fields have been removed for shortening this message*
>
> My problem is with the NodeList type...
>
> I've verified in my server that I am doing a
> packet.nodelist.items.add(), then proceeding to set the required
> values of the returned NodeItemList.
>
> I then send the packet over the wire.
>
> I receive the packet from the client and do a:
>
> for i in packet.nodelist.items:
>print str(i.node)
>
> and it only comes back with 1 result, no matter how many I build in
> the server (I've tried up to 3)...
>
> Any ideas?
>
> I'm doing this exactly the same way with a different packet type and
> it works fine... am I just missing something small?
>
> Thanks,
> Ben
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Cross compiling
Hi, I am building and using protocol buffers 2.0.1 on a Linux host and it works fine. Now I am moving to a target hardware with an ARM9 cpu, also running Linux (and later to an ARM7 with no os) and would like to cross compile to these platforms. I have tried to find information on how to do this but have not found any information on the subject. Has anyone tested this? Can anybody help me out? Thanks for your help! /Magnus --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
