Any way to dissect ProtBuf serialized data without knowing the structure in advance?

2008-10-22 Thread helium
I'm trying to consume data from an app that generates output serialized via Protocol Buffers but do not have the original spec for the specific structures that have been encoded. Is there a relatively straight-forward path to deserializing, or even just decoding, the serialized data stream without

Re: Serializing Large Collections: SerializePartial ParsePartial

2008-10-22 Thread bruce . weertman
OK, that makes sense. Thanks for the quick reply. I work at a seismic earthquake data center. We're looking at using protocol buffers as a means of internally moving around processed chunks of data. Seems to work pretty well, as long as the chunks aren't too large (which is a problem one way or a

Re: python setup.py test failing

2008-10-22 Thread jw
I'm also seeing this on an x86_64 machine. It seems like a problem with the test as stated above. Has it been confirmed? thanks. ~jason On Oct 6, 7:34 pm, "Kenton Varda" <[EMAIL PROTECTED]> wrote: > Petar, can you look into this? > > On Sun, Oct 5, 2008 at 2:47 AM, <[EMAIL PROTECTED]> wrote: >

Re: Serializing Large Collections: SerializePartial ParsePartial

2008-10-22 Thread Kenton Varda
The "Partial" serialize and parse routines actually do something completely unrelated: they allow the message to be missing required fields. So, that doesn't help you. I'm afraid protocol buffers are not designed for storing very large collections in a single message. Instead, you should be thin

Re: Standard for RPC proto

2008-10-22 Thread Kenton Varda
Google's RPC system is huge and bloated, and designed explicitly for our hardware and data center configurations. So, I don't think publishing our protocol would be very helpful, and it would take a lot of work just to document it properly. I'm not even sure if our protocol makes sense for use ov

Re: Make check reports error

2008-10-22 Thread Kenton Varda
On Wed, Oct 22, 2008 at 7:23 AM, Niclas Blomgren < [EMAIL PROTECTED]> wrote: > Solaris: > I'm guessing this was a sparc system, not x86? Was it 32-bit or 64-bit? I think someone else reported the same problem but we were not able to track it down. > Cygwin: > Can you include the text of the ac

Re: Failed Test

2008-10-22 Thread Kenton Varda
[+petar] I believe this is a known issue with 64-bit systems, and you can ignore it (it's only the test that is broken). On Tue, Oct 21, 2008 at 2:40 PM, <[EMAIL PROTECTED]> wrote: > > I just ran into some test errors before installing the python tools > and after installing the protocol buffer l

Re: Data structures using protocol buffers

2008-10-22 Thread Jeremy Leader
Protocol Buffers are a serialization format, rather than general-purpose data structures. To do computations, you'd probably want to build some auxiliary data structures, which you populate when you deserialize the protobuf data. You could have node objects that resemble your original .proto

Re: Data structures using protocol buffers

2008-10-22 Thread GDR
my bad. The code snippet should be as follows: for(UndirectedGraphNode node : UndirectedGraph.getNodesList() ) { double sum = 0; int count = 0; for(UndirectedGraphNodeReference neighbor : node.getNeighborsList() ) { sum += count++; } node.setWeight(sum/count); } --

Re: Data structures using protocol buffers

2008-10-22 Thread GDR
That does solve the duplicate information problem but it makes updates to node attributes (like weight) difficult. Let's say, I want to assign the weight of each node to the average of its neighbors. for(UndirectedGraphNode node : UndirectedGraph.getNodesList() ) { double sum = 0; int count

Re: Data structures using protocol buffers

2008-10-22 Thread Jeremy Leader
I was assuming all the properties of a node (weight, label, color, whatever) would be in UndirectedGraphNode; UndirectedGraphNodeReference would only have the id and nothing else. -- Jeremy Leader [EMAIL PROTECTED] GDR wrote: > Thanks Jeremy. That worked! > But we now have information about t

Re: Data structures using protocol buffers

2008-10-22 Thread GDR
Thanks Jeremy. That worked! But we now have information about the same node being replicated. For instance, let's say we have a field 'weight' attached to each node as shown below. This setup will replicate the weight information of a node as many times as its degree. If the weight of a node chang

Serializing Large Collections: SerializePartial ParsePartial

2008-10-22 Thread bruce . weertman
Is there a general strategy/methodology for dealing with very large collections so that they do not need to be completely held in memory before serializing and de-serializing? I see, for example, SerializePartialToOstream() and ParsePartialFromIstream() but no documentation of how to use it. --~--

Standard for RPC proto

2008-10-22 Thread gariev
To make different RPC implementations based on Protocol Buffers compatible with each other, they must use the same method of data serialization. However, it's not the case for RPC service fields like method name or message id: Java project Protorcp uses one specificaton (http:// protorpc.likbilen

Make check reports error

2008-10-22 Thread Niclas Blomgren
Hi! I am trying to make the protobuf (version 2.0.2) compiler on a Solaris 9 host and on a Windows Vista host using Cygwin. I get the error reports shown below. Do you know what the faults are? If not, how can I trouble-shoot them? Best Regards / Niclas Blomgren Solaris: --- make check m

C# port has new layout

2008-10-22 Thread Jon Skeet
My port of the Java code to C# (which I really must give a proper name some time) has just undergone a somewhat radical transformation. The github repository (http://github.com/jskeet/dotnet-protobufs/tree/ master) no longer contains any source code from the Google main project. There is no non-C