Re: GPB on non-Linux, non-Windows OS?

2009-02-26 Thread Kenton Varda
I've tested in on OSX, FreeBSD, and Solaris, but those are not terribly different from Linux. The Google implementations of protobufs definitely weren't designed for embedded systems, but there might be a case for taking the Google C++ implementation and ripping out all the "advanced" features to

Re: GPB on non-Linux, non-Windows OS?

2009-02-27 Thread Kenton Varda
or stripping protobuf of > file and stream support? > > Given our simple requirements and the perceived work involved with > stripping libprotobuf, however, my instinct tells me that protobuf- > c-0.7 would be a better starting point. It already is "stripped". I'm >

Re: GPB on non-Linux, non-Windows OS?

2009-03-01 Thread Kenton Varda
's only fair to compare your implementation against this case. Be careful about claiming that your code will be faster before actually running benchmarks. I've been burned many times doing that. :) > > > Josh > > On Feb 27, 1:55 pm, Tim wrote: > > That sounds li

Re: GZIPOutputStream (for C++)

2009-03-01 Thread Kenton Varda
On Sun, Mar 1, 2009 at 2:01 PM, bolson wrote: > I recently noticed that one of my protobuf-data-files still benefitted > from gzip at a ratio slightly better than 7 to 1. Just to note, this is not surprising. PB is a compact format but does not actual compression, so if you have a message cont

Re: GPB on non-Linux, non-Windows OS?

2009-03-01 Thread Kenton Varda
On Sun, Mar 1, 2009 at 3:42 PM, Joshua Haberman wrote: > What about strings and repeated elements? String assignment will at least > sometimes call malloc() to re-allocate the string, though I don't know > enough about STL to know how often this happens. And string assignment will > always copy

Re: GPB on non-Linux, non-Windows OS?

2009-03-01 Thread Kenton Varda
On Sun, Mar 1, 2009 at 6:56 PM, Kenton Varda wrote: > I believe GCC's std::string will only malloc if it needs to grow, so on > reuse there should be no new allocation. > I meant, reuse to store a string of the same or lesser s

Re: GPB on non-Linux, non-Windows OS?

2009-03-02 Thread Kenton Varda
On Mon, Mar 2, 2009 at 3:43 AM, Jon Skeet wrote: > I've ported the code to Java, but haven't sent it to Kenton yet. I don't > know exactly where we'd want to put it in the main distribution. Kenton, how > do you want to proceed with this? > A top-level "benchmark" directory seems like it would m

Re: Performance comparison of Thrift, JSON and Protocol Buffers

2009-03-02 Thread Kenton Varda
[+petar] Petar, want to share what you have so far on the C-extension stuff? Maybe someone with more time available would like to help with it. On Mon, Mar 2, 2009 at 10:04 AM, Stephan Richter wrote: > > On Monday 02 March 2009, Justin Azoff wrote: > > I actually posted a follow up: > > > http:/

Re: Using PB for object data storage in embedded c++ application

2009-03-02 Thread Kenton Varda
I don't think that's something we want to do in general, because storing the callbacks would make the message objects much bigger, and checking for their presence would add branches in all the setter methods, both of which would harm performance. That said, you could write a custom protoc which gen

Re: protocol buffers in .net

2009-03-02 Thread Kenton Varda
e short end of the stick in > terms of the number of implemented libraries, standards, oss projects. > Which is a non-trivial shortcoming all in all. I am not claiming that > the CLR ever had a technical, non-business reason to exist, but now > that it does exist... > > On Feb 16, 4:23 p

Re: build fix for protobuf-c

2009-03-03 Thread Kenton Varda
I agree, CXXFLAGS and LDFLAGS are the right way to specify the library and include dirs, but the configure script probably should check for the presence of libprotobuf, at least. :) On Tue, Mar 3, 2009 at 10:18 AM, wrote: > > It seems like you should be using > CXXFLAGS=-I/my/install/path/incl

Re: ZeroCopyInputStream behavior

2009-03-03 Thread Kenton Varda
On Tue, Mar 3, 2009 at 8:22 AM, aepensky wrote: > > I was reading through the code and I'm not sure I understand the > thought behing the ZeroCopyInputStream implementations. > > It seems to me that if the underlying data source actually *is* a > stream to begin with, "ZeroCopy" does not perform

Re: built libprotobuf for iphone but large footprint, any way to reduce?

2009-03-03 Thread Kenton Varda
Sorry, our implementation was designed for servers where binary footprint is not a big deal. Maybe try using protobuf-c? On Tue, Mar 3, 2009 at 7:09 AM, marc wrote: > > I built libprotobuf for the iphone (must be static for app store > apps). It's about 3mb. This results in a significant foot

Re: ZeroCopyInputStream behavior

2009-03-03 Thread Kenton Varda
On Tue, Mar 3, 2009 at 12:20 PM, aepensky wrote: > > Thanks for the quick response... > > So, to do an RPC implementation over a stream channel (e.g. TCP), > where the message length is being sent in advance of the message body, > the fastest way would still be > (1) allocate bytes[length] > (2)

Re: built libprotobuf for iphone but large footprint, any way to reduce?

2009-03-03 Thread Kenton Varda
[+dave] On Tue, Mar 3, 2009 at 11:33 AM, marc wrote: > > > > On Mar 3, 1:42 pm, Kenton Varda wrote: > > Sorry, our implementation was designed for servers where binary footprint > is > > not a big deal. Maybe try using protobuf-c? > > > > Ok thanks.

Re: protocol buffers in .net

2009-03-03 Thread Kenton Varda
On Mon, Mar 2, 2009 at 11:19 PM, Jon Skeet < sk...@pobox.com> wrote: > There's one major blocker at the moment though: all my copies of the > Google test .proto files are decorated with the C# options. *At the > moment* that means the Java and C++ code would have to build the C# > options as well

Re: protocol buffers in .net

2009-03-03 Thread Kenton Varda
n Mar 3, 11:06 pm, Kenton Varda wrote: > > sk...@pobox.com> wrote: > > > There's one major blocker at the moment though: all my copies of the > > > Google test .proto files are decorated with the C# options. *At the > > > moment* that means the Java and C++ c

Re: Memory leak?

2009-03-04 Thread Kenton Varda
http://code.google.com/p/protobuf/issues/detail?id=54 On Wed, Mar 4, 2009 at 3:50 AM, Adam wrote: > > Hi, > > I'm using protocol buffers in my project (C++), and while I was > hunting for memory leaks, I discovered that in the generated code, > static members are being allocated, but there's no

Re: Clear() function excessive CPU usage

2009-03-05 Thread Kenton Varda
Wow, that's interesting. I don't know why it would do that. Can you look deeper into your profiles and see what part of Clear() is taking so long? For example, is it spending the time clearing STL strings? On Thu, Mar 5, 2009 at 3:11 PM, Zachary Turner wrote: > > I have a fairly old version of

Re: Clear() function excessive CPU usage

2009-03-05 Thread Kenton Varda
able > to get it working. > > On Mar 5, 5:20 pm, Kenton Varda wrote: > > Wow, that's interesting. I don't know why it would do that. Can you > look > > deeper into your profiles and see what part of Clear() is taking so long? > > For example, is it spending the t

optimize_for option default

2009-03-05 Thread Kenton Varda
Hi all, As you know if you've read the docs carefully, when using C++ or Java protocol buffers, for best performance you need to add a line to your .proto files: option optimize_for = SPEED; Otherwise, by default, the compiler optimizes for code size. Optimizing for code size results in genera

Re: Clear() function excessive CPU usage

2009-03-05 Thread Kenton Varda
Technically it might work, but if it does I can't guarantee that it wouldn't break in the future. > > > On Thu, Mar 5, 2009 at 5:25 PM, Kenton Varda wrote: > >> Add this to your .proto file: >> option optimize_for = SPEED; >> >> Does it help? >&

Re: Clear() function excessive CPU usage

2009-03-05 Thread Kenton Varda
so that I can send it across the > wire. It would be nice if I could get rid of all this copying. Yeah, the implementation wasn't really designed for this sort of usage. :/ > > > > On Thu, Mar 5, 2009 at 6:24 PM, Kenton Varda wrote: > >> On Thu, Mar 5, 2009 at 4:0

Re: Initial benchmarking committed to svn (r100)

2009-03-06 Thread Kenton Varda
On Fri, Mar 6, 2009 at 5:28 AM, Jon Skeet wrote: > > On Mar 6, 1:24 pm, Justin Azoff wrote: > > On Mar 6, 1:13 am, Justin Azoff wrote:> I did > a quick port to python(pasted at the end, hopefully it wont be > > > garbled) > > > > well, that didn't work. > > I threw it up athttp://bouncybounc

Re: optimize_for option default

2009-03-06 Thread Kenton Varda
pressive syntax but don't want to go off the deep end in inventing a whole DSL for option munging. For now I think I'm going to go ahead and change the default for this option in the next release. On Fri, Mar 6, 2009 at 12:30 AM, Jon Skeet < sk...@pobox.com> wrote: > > On

Re: Memory leak?

2009-03-06 Thread Kenton Varda
On Fri, Mar 6, 2009 at 5:50 AM, marc wrote: > Isn't this a common scenario in C++ for which stdlib provides a simple > solution, std::auto_ptr? std::auto_ptr is a lightweight class designed > to "RAIIify" pointers. What am I missing? Deleting the objects on shutdown is easy. The problem is th

Re: parsing message with extensions in C++ implementation

2009-03-06 Thread Kenton Varda
On Fri, Mar 6, 2009 at 1:41 AM, anthonyp wrote: > > When receiving an encoded protocol buffer message containing > extensions it does not appear to parse correctly when using > > ParseFromArray(const void * data, int size) in C++ Can you describe what you mean by "does not appear to parse corre

Re: Memory leak?

2009-03-06 Thread Kenton Varda
e that the destructor is called with the background thread still running. On Fri, Mar 6, 2009 at 11:13 AM, Marc Vaillant wrote: > On Fri, Mar 06, 2009 at 09:52:07AM -0800, Kenton Varda wrote: > > On Fri, Mar 6, 2009 at 5:50 AM, marc wrote: > > > > Isn't this a com

Re: protobuf 2.0.3 *forces* Unicode

2009-03-06 Thread Kenton Varda
You need to use the "bytes" type instead of "string". "string" has always been defined in the docs as UTF-8, it just wasn't enforced in the Python implementation. On Fri, Mar 6, 2009 at 12:40 PM, Ross Vandegrift wrote: > > Hi everyone, > > I have a python app that passes objects using protocol

Re: Code Generator Modification Suggestion

2009-03-06 Thread Kenton Varda
This is pretty much what the "ctype=STRING_PIECE" option does... but it wasn't included in the public release, since it required some extra classes and we find we rarely use it internally. I've been meaning to port it at some point but haven't had time... I'll bump it up my todo list. On Fri, M

Re: Protorpc moved (and ported to Qt/c++)

2009-03-06 Thread Kenton Varda
Hmm, that's the same link I already had for you. I updated the wiki to list the languages you support. On Fri, Mar 6, 2009 at 1:26 PM, wrote: > > Protorpc has moved to http://protorpc.lekebilen.com I'll upload the > repositories for the different ports to github etc. to use a more > stable serv

Re: Protorpc moved (and ported to Qt/c++)

2009-03-06 Thread Kenton Varda
Sorry, fixed. On Fri, Mar 6, 2009 at 1:53 PM, wrote: > > On 6 Mar, 22:32, Kenton Varda wrote: > > Hmm, that's the same link I already had for you. I updated the wiki to > list > > the languages you support. > note lekebilen!=likbilen (and ple

Re: Memory leak?

2009-03-06 Thread Kenton Varda
On Fri, Mar 6, 2009 at 6:41 PM, Marc Vaillant wrote: > Ok, thanks. There are other problems though. If A::b was a B* instead > of a std::auto_ptr, then the data that A::b points to will be ok > because it won't be deleted. However, A::b would get deleted so access > to the data via A::b may be

Re: Memory leak?

2009-03-09 Thread Kenton Varda
On Mon, Mar 9, 2009 at 6:12 AM, Marc Vaillant wrote: > Except that you still have the shared lib load/unload problem :) > Yeah, well, I agree with you. I wish people wouldn't write programs that attempt to exit while background threads are still running, but a lot of people at Google (where we

Re: Swap() and RemoveLast() should be added to Reflection API

2009-03-09 Thread Kenton Varda
It wasn't added because the reflection interface is already huge, and there are a lot of different things that people could potentially want from it. That said, I'd be OK with adding these. Fortunately they don't need separate versions for every field type. On Mon, Mar 9, 2009 at 5:50 AM, Scott

Re: Inheriting from generated C++ classes for adding Qt QString accessors. Problems expected?

2009-03-09 Thread Kenton Varda
If you are OK with the possibility that future versions of protocol buffers may break your code (probably in minor ways), then you can do whatever you want. If you aren't OK with that, then you should not subclass protocol buffer types. There are tons of ways that future changes to the implementa

Re: Any protocol buffers MIME types?

2009-03-09 Thread Kenton Varda
We haven't defined a MIME type. Does it make sense to define a MIME type for protocol buffers in general, as opposed to MIME types for individual protocols? The latter makes more sense to me, since there's not much you can do with a protocol buffer without knowing its type. On Mon, Mar 9, 2009 at

Re: Two questions: uint16_t support and fixed length fields

2009-03-09 Thread Kenton Varda
On Mon, Mar 9, 2009 at 2:45 PM, anthrope wrote: > > Hello all, >I have only just begun looking at Protocol Buffers, and am > intrigued, to say the least. I have taken a quick look at the library, > and thought I would post a couple of questions before delvign further. > > First, is there supp

Re: Swap() and RemoveLast() should be added to Reflection API

2009-03-10 Thread Kenton Varda
s. On Tue, Mar 10, 2009 at 3:35 PM, Scott Stafford wrote: > > Are you interested in a patch from me to add them or would you just > add it yourselves at some point? > > On Mar 9, 2:53 pm, Kenton Varda wrote: > > It wasn't added because the reflection interface is alread

Re: parseFrom is blocking

2009-03-12 Thread Kenton Varda
parseFrom() reads and parses the entire stream, so it won't stop reading until it reaches EOF. If you don't want to close the socket on the sending end, then you should write the size of the message first, then the contents. Then, on the receiving end, you'll need to read that size first, then us

Re: Nullable ignores IsRequired

2009-03-12 Thread Kenton Varda
I'm guessing this question is about protobuf-net, specifically? (Not sure since I haven't used it myself.) On Thu, Mar 12, 2009 at 5:23 AM, Bryan wrote: > > I noticed in my test app, and then in the ProtoBuf source, that the > serialization of Nullable properties ignores the IsRequired > parame

Re: Avoiding object creation and data copying

2009-03-12 Thread Kenton Varda
It's not possible to set values by writing directly into a byte array since the size of a value depends on its contents. For example, integers are encoded using the varint encoding which uses fewer bytes for smaller numbers. So, if the value of a particular int32 field is 1, and then you set it t

Re: parseFrom is blocking

2009-03-12 Thread Kenton Varda
he end has been reached if you don't do this. The protocol buffer format is *not* self-delimiting -- it assumes that whatever container it is embedded in will keep track of the length. > > Is there any solution that is a bit more elegant? > > On Mar 12, 10:12 pm, Kenton Varda

Re: new actionscript port

2009-03-13 Thread Kenton Varda
Added. On Fri, Mar 13, 2009 at 10:11 AM, wrote: > > Hi, > > protocol-buffers-actionscript wasn't maintained and wasn't usable as- > is, so I spinned it off in > > http://code.google.com/p/protobuf-actionscript3/ > > It can be used to generate most common protofiles (including managing > packages

Re: same message, different serialization

2009-03-16 Thread Kenton Varda
On Mon, Mar 16, 2009 at 12:12 PM, Jeremy Leader wrote: > > I believe you can concatenate two (serialized) messages together, > yielding a message with all the fields from both messages. > > So if all your fields are optional, you could create one instance of > your message with just the fields co

Re: same message, different serialization

2009-03-16 Thread Kenton Varda
That is what I meant. Another option would be to define a custom option which identifies the fields of your message that should only be sent to one of the clients, and then write a reflection-based routine which recursively clears all such fields in a message. But personally I think having multipl

Re: Reading from a stream...

2009-03-18 Thread Kenton Varda
On Wed, Mar 18, 2009 at 8:31 AM, sauternes wrote: > I've read the various useful posts on this discussion group about > using popLimit() and pushLimit() to control how many bytes are read at > a time when using a CodedInputStream, but is there an elegant way of > detecting EOF/end of stream or is

Re: Serialization time in C++ using protobuf

2009-03-18 Thread Kenton Varda
First, are you using: option optimize_for = SPEED; ? If not, add that line to your .proto file. But even without that option, the speed shouldn't be that slow. Maybe you can run in a profiler to see what's taking so long? Are you writing to an in-memory buffer or some sort of output stream?

Re: Serialization time in C++ using protobuf

2009-03-19 Thread Kenton Varda
On Wed, Mar 18, 2009 at 11:23 PM, ShirishKul wrote: > Output in my case is a file to which i'm writing. > > But again, this doesn't seem to be matching with what your throughput > is. Do I still need to make changes to seek more optimum results? What happens if you serialize to an in-memory buf

Re: python performance

2009-03-20 Thread Kenton Varda
The Python implementation is pretty slow right now. I'm not familiar with the PHP implementation so couldn't say exactly why it is faster, but it's not terribly surprising. Petar is working on making the Python implementation use native C++ code for a lot of operations, which should make it much f

Re: What's the footprint on ARM7?

2009-03-20 Thread Kenton Varda
I haven't tried it, but I'm pretty sure the official C++ implementation will be way too large for you. Maybe protobuf-c would work? On Thu, Mar 19, 2009 at 7:59 PM, wrote: > > Hi all, > > Has anybody ported protocol buffers on ARM7 (hopefully STM32)? What's > the footprint? Is it suitable to us

Re: Protocol buffer unicode python vs. c++

2009-03-20 Thread Kenton Varda
If you changed all the "string" types to "bytes" instead, then you should not see that error. Are you sure you did that? If so, can you write a small demo program which produces this error, even when the protobuf type contains no "string" fields, and send it to me? On Fri, Mar 20, 2009 at 11:16

Re: Protocol buffer unicode python vs. c++

2009-03-20 Thread Kenton Varda
> ... > ... > > and then writing it back to my database > title_str_pb = t.SerializeToString() > > and now when I open it in c++, its not giving any error. > > So, I think when I was adding the original data, I should have > called .encode('utf-8') on all the

Re: About to google protocol buffer in j2me

2009-03-20 Thread Kenton Varda
The existing Java implementation is probably too heavyweight for J2ME. Sorry. The encoding is documented here: http://code.google.com/apis/protocolbuffers/docs/encoding.html Is this a Google service you want to talk to? If so, are you working with one of the Google mobile teams at all? They mig

Re: What's in 2.0.4 / Next Version?

2009-03-23 Thread Kenton Varda
You can see a list of the stuff that's already in SVN by looking at CHANGES.txt: http://code.google.com/p/protobuf/source/browse/trunk/CHANGES.txt It will probably just be another incremental update, nothing terribly significant. Pushing another release is high on my todo list, but somehow I've h

Re: Question about parsing Protocal buffers

2009-03-23 Thread Kenton Varda
On Mon, Mar 23, 2009 at 1:38 AM, ode wrote: > > There is api in c++, ParseFromIstream, but is there any similar api in > python? No, there's no Python equivalent right now. But, the parsed objects are bigger than the original serialized data, so if the original serialized data can't fit in mem

Re: Protocol Buffers License info

2009-03-23 Thread Kenton Varda
Protocol Buffers is licensed under the new BSD license. The license info is mentioned on the sidebar of the main project page: http://code.google.com/p/protobuf/ It is also found in the file COPYING.txt in the distribution: http://protobuf.googlecode.com/svn/trunk/COPYING.txt On Mon, Mar 23, 20

Re: Read proto files into Descriptor so it can be used with DynamicMessage

2009-03-23 Thread Kenton Varda
I'm a little confused by your message. You say you have the .proto files, but you don't know the types? But don't the .proto files define the types? Did you mean to say that you only have actual data encoded as these types, and you don't have the type definitions (the .proto files) at all? In t

Re: abort() in libprotobuf due to missing required field

2009-03-23 Thread Kenton Varda
Well, note that if you compile with -DNDEBUG, the abort won't happen. We have a no-exception policy in C++, so I don't think we can replace this with an exception. Is -DNDEBUG good enough for you or do you want to look for another solution? On Mon, Mar 23, 2009 at 2:32 PM, Dave Bailey wrote: >

Re: abort() in libprotobuf due to missing required field

2009-03-23 Thread Kenton Varda
BTW, you can always call IsInitialized() yourself before serializing. If it returns false, some required fields are missing. On Mon, Mar 23, 2009 at 2:50 PM, Kenton Varda wrote: > Well, note that if you compile with -DNDEBUG, the abort won't happen. We > have a no-exception policy

Re: abort() in libprotobuf due to missing required field

2009-03-23 Thread Kenton Varda
> > 1) never abort() > 2) never print anything to stderr > > -dave > > On Mar 23, 2:50 pm, Kenton Varda wrote: > > Well, note that if you compile with -DNDEBUG, the abort won't happen. We > > have a no-exception policy in C++, so I don't think we can

Re: Read proto files into Descriptor so it can be used with DynamicMessage

2009-03-23 Thread Kenton Varda
actually > needing the Java types generated from the proto files, but instead use > the DynamicMessage to get values out of a protobuf stream. > > Cheers, > Shay > > > On Mar 23, 11:47 pm, Kenton Varda wrote: > > I'm a little confused by your message. You s

Re: Can repeated structure like be mapped to java.util.Map

2009-03-24 Thread Kenton Varda
You have to construct the map yourself by iterating once through the entries. There have been some vague ideas thrown around regarding built-in map support but not much agreement on what it should look like or whether it is really needed... On Mon, Mar 23, 2009 at 11:23 PM, murthy wrote: > > Hi

Re: abort() in libprotobuf due to missing required field

2009-03-24 Thread Kenton Varda
On Tue, Mar 24, 2009 at 4:42 PM, David Anderson wrote: > At Google, 99% of assertions remain enabled at all times. There are > some nuances (as always, mindless dogma gets you nowhere), but in > general, if it should crash during testing, it should also crash in > production. The alternative is e

Re: abort() in libprotobuf due to missing required field

2009-03-25 Thread Kenton Varda
On Wed, Mar 25, 2009 at 2:01 AM, David Anderson wrote: > However, I'll note > that what you described for user-facing code, to log an error and try > to continue, is not the behavior you get with -DNDEBUG. Depends on the macro. Our LOG(DFATAL) macro aborts the process in debug mode (when NDEBU

Re: Storing complex graphs of objects?

2009-03-25 Thread Kenton Varda
To store a DAG (or a cyclic graph, for that matter) you would need to come up with some sort of identifier system for objects and then refer to objects by their identifier instead, so that multiple locations can refer to the same object without copying it. Example: message Graph { repeated N

Re: Visual Studio 200x fix

2009-03-26 Thread Kenton Varda
The existing format mimics GCC. Unfortunately I think if we changed it, it would break other tools. Maybe we could enable this format based on a command-line flag? On Wed, Mar 25, 2009 at 8:59 PM, os_ wrote: > > Hey there, below is a tiny patch that lets VS200x go to the correct > line in the

Re: Crash while initializing StaticDescriptorInitializer

2009-03-26 Thread Kenton Varda
Did you make sure to compile libprotobuf and your code using the same settings? (E.g. debug vs. release, same compiler version.) MSVC is pretty touchy about that when using STL in the interface to a library. On Thu, Mar 26, 2009 at 9:08 AM, wrote: > > I have libprotobuf.lib linked into a DLL,

Re: Any edition for as3?

2009-03-26 Thread Kenton Varda
There are a few AS ports listed here: http://code.google.com/p/protobuf/wiki/OtherLanguages I don't know how well any of them work. On Thu, Mar 26, 2009 at 2:03 AM, Ayr wrote: > > I want to use Protocol Buffers in my Flex project. > Is that possible? > > > --~--~-~--~~---

Re: Crash while initializing StaticDescriptorInitializer

2009-03-26 Thread Kenton Varda
On Thu, Mar 26, 2009 at 9:28 AM, Jesper wrote: > I think so, but I'll double-check. (Wouldn't that have showed up when > libprotobuf.lib was linked into the exe?) Unfortunately, probably not. Strange crashes at runtime are the usual symptom. :( --~--~-~--~~~---~--

Re: Crash while initializing StaticDescriptorInitializer

2009-03-27 Thread Kenton Varda
On Fri, Mar 27, 2009 at 3:33 AM, Jesper Eskilson wrote: > (BTW: I created a set of CMake (http://cmake.org) build files for > Protobufs, in case you're interested in not maintaining parallell > Makefiles/project files for Linux/Windows.) We considered using cmake originally, but decided it was

Re: Storing complex graphs of objects?

2009-03-27 Thread Kenton Varda
You could treat the protobuf classes as private implemnetation helpers. Your public interface would have methods like you describe, and internally they'd just copy the object's state into a protocol buffer and then serialize it. On Fri, Mar 27, 2009 at 2:32 AM, Paul Runyan wrote: > Hi Kenton -

Re: Streaming different types of messages

2009-03-27 Thread Kenton Varda
On Fri, Mar 27, 2009 at 9:08 AM, Dave Bailey wrote: > I don't suppose there'd ever be a way to mark a set of fields as > mutually exclusive? Nope. I've tossed around the idea before but no one has had time to work on it. --~--~-~--~~~---~--~~ You received this

Re: Memory leaks on VS2008/C++

2009-03-27 Thread Kenton Varda
t; is invoked. > I dunno. Did you make sure that DescriptorPool::generated_pool() is deleted? > > > Thanks, > Oleg. > > > On 3/27/2009 1:12 PM, Kenton Varda wrote: > > These aren't really "leaks". See this thread: > > http://groups.google.com

Re: Memory leaks on VS2008/C++

2009-03-27 Thread Kenton Varda
BTW, the only way libprotobuf allocates memory is via operator new, though it sometimes calls it directly like: void* buffer = operator new(size); So overriding operator new to collect stack traces should work. On Fri, Mar 27, 2009 at 2:34 PM, Kenton Varda wrote: > On Fri, Mar 27, 2009 a

Re: Memory leaks on VS2008/C++

2009-03-27 Thread Kenton Varda
On Fri, Mar 27, 2009 at 4:30 PM, Oleg Smolsky wrote: > So, the next question is - would it ever make its way into your SVN? If so, > do you see a need for a switch of some kind? > Yeah, reviewing and committing this is on my todo list for the next release. --~--~-~--~~~-

Re: Serialization performance comparison with Boost.serialization

2009-03-29 Thread Kenton Varda
What does your .proto file look like? And the code that uses it? On Sun, Mar 29, 2009 at 9:06 PM, Yingfeng wrote: > > Hi, > We are looking for a fast mechanism for serialization/deserialization. > Here is our comparison between pb and boost: > We hope to serialize/deserialize data in std contai

Re: Serialization performance comparison with Boost.serialization

2009-03-30 Thread Kenton Varda
files are attached > > Best > > > > On Mon, Mar 30, 2009 at 12:14 PM, Kenton Varda wrote: > >> What does your .proto file look like? And the code that uses it? >> >> On Sun, Mar 29, 2009 at 9:06 PM, Yingfeng wrote: >> >>> >>> Hi, >>&

Re: Serialization performance comparison with Boost.serialization

2009-03-30 Thread Kenton Varda
ol-buffer to deserilize vector of > size 1000 ! > > > > Best > > > > On Tue, Mar 31, 2009 at 2:07 AM, Kenton Varda wrote: > >> Several points: >> >> * Some of your test cases seem to be parsing from or serializing to files. >> This may be m

Re: Serialization performance comparison with Boost.serialization

2009-03-30 Thread Kenton Varda
s for boost to serialize vector of size 1000 ! > > It takes 0.05 seconds for boost to deserialize vector of size 1000 > ! > > > Best > > > > > On Tue, Mar 31, 2009 at 10:46 AM, Kenton Varda wrote: > >> That's more like it. :) >> >&

Re: Trouble initializing protobufs for Android.

2009-03-31 Thread Kenton Varda
I think you must have introduced a bug somewhere. It may be in descriptor.cc, but it could also be some other kind of parsing or serialization bug. Generated .pb.cc files contain encoded descriptors which are parsed at startup, and somehow the descriptor for descriptor.proto is coming up invalid.

Re: Serialization performance comparison with Boost.serialization

2009-03-31 Thread Kenton Varda
OK. But I believe Yingfeng's results were impossibly fast, unless the code has changed since I saw it. His data set is a vector of 10,000,000, each with a random size in the range [3,100]. That comes out to 515,000,000 bytes (491MB) of string data. If we totally ignore overhead of the vector, m

Re: string_as_array() returning null pointer when string is non-empty

2009-03-31 Thread Kenton Varda
MSVC9 is one of the platforms I test before releases, so it should work. MSVC is finicky about STL, though. You need to make sure that you compile libprotobuf and your project with the the same settings, for all settings that can affect the layout of STL objects. Unfortunately, with MSVC, there a

Re: identical messages?

2009-03-31 Thread Kenton Varda
No. Why would you want that? Is this supposed to be like a typedef? On Tue, Mar 31, 2009 at 5:21 PM, wrote: > > Is there a feature with which a message can be defined as identical to > another one, for instance, "message Y X" specifying that message "Y" > is identical to another previously def

Re: Trouble initializing protobufs for Android.

2009-03-31 Thread Kenton Varda
ore its first use. This leads me to think > maybe something similar has happened in that something else needs > to be initialized before the InternalBuildGeneratedFile is invoked. > > Is that a possibility, if so what would that be. > > -- Wink > > > On Tue, Mar 31, 2009 at 9

Re: Trouble initializing protobufs for Android.

2009-03-31 Thread Kenton Varda
not being constructed before its first use. This leads me to think >>> maybe something similar has happened in that something else needs >>> to be initialized before the InternalBuildGeneratedFile is invoked. >>> >>> Is that a possibility, if so what would that be.

Re: Visual Studio 200x fix

2009-03-31 Thread Kenton Varda
've attached a patch that adds a new command line option. > > Oleg. > > On Thu, Mar 26, 2009 at 9:14 AM, Kenton Varda wrote: > > The existing format mimics GCC. Unfortunately I think if we changed it, > it > > would break other tools. Maybe we could enable this form

Re: cat/combining serialized binary files a legit option?

2009-04-01 Thread Kenton Varda
Concatenating encoded protocol messages has the effect of merging them, with the same semantics as the MergeFrom() method. That is, repeated fields will be concatenated, singular primitive fields will take the *last* value, and singular embedded messages/groups will be merged. In your case, your c

Re: cat/combining serialized binary files a legit option?

2009-04-01 Thread Kenton Varda
On Wed, Apr 1, 2009 at 2:04 PM, le wrote: > > Thanks for the reply. I think I understand what is happening here but > have an additional question. > > You stated three rules. When you say "...singular embedded messages/ > groups will be merged.", does that imply that the other rules are > simpl

Re: Builder set* methods should be aware of null values

2009-04-02 Thread Kenton Varda
As Jerry said, see issue 57: http://code.google.com/p/protobuf/issues/detail?id=57 Summary: This is fixed in SVN by making setters themselves throw NPE if the input is null. Some people think this is the wrong behavior, and that setting a value to null should instead be equivalent to clearing it

Re: New line characters removal

2009-04-02 Thread Kenton Varda
No, the "newlines" are part of the binary data. (They aren't really "newlines" though since the data is not text. They are bytes with the value 10.) Or are you actually talking about the text format? On Thu, Apr 2, 2009 at 3:02 AM, ShirishKul wrote: > > In the binary file generated using Proto

Re: Platforms supported for C++ gen code

2009-04-02 Thread Kenton Varda
Most Unix-like platforms (Linux, Mac OSX, FreeBSD, Solaris, Cygwin, etc.) and Windows (MinGW, MSVC) are supported. libprotobuf does not use many OS-specific features so it's likely to be easy to port to other platforms. However, it is not well-designed for stripped-down embedded systems with limi

Re: Platforms supported for C++ gen code

2009-04-03 Thread Kenton Varda
gt; On Fri, Apr 3, 2009 at 11:38 AM, Shirish Kulkarni wrote: > >> Thanks Kenton. >> >> >> On Fri, Apr 3, 2009 at 12:18 AM, Kenton Varda wrote: >> >>> Most Unix-like platforms (Linux, Mac OSX, FreeBSD, Solaris, Cygwin, etc.) >>> and Windows (MinGW, MSVC) a

Re: More readable/compact text format for repeated.

2009-04-03 Thread Kenton Varda
You can easily implement your own text format by writing code based on protobuf reflection. In fact, the official TextFormat class is based on reflection. So take a look at the code for the TextFormat class and write something like that with whatever modifications you need. On Fri, Apr 3, 2009 at

Re: Protocol Buffers Vs. XML Fast Infoset

2009-04-03 Thread Kenton Varda
On Fri, Apr 3, 2009 at 2:40 AM, ShirishKul wrote: > I found that for a typical data to be transferred across the wire for > size of 500KB that a XML file would represent has corresponding file > size as 300KB for PB binary and around 130KB for XML Fast Infoset > binary file. What kind of data w

Re: Problem compiling after checking out from svn

2009-04-07 Thread Kenton Varda
You need a newer version of automake. 1.9 should be good enough. On Mon, Apr 6, 2009 at 11:05 PM, Wink Saville wrote: > I checked out the sources via svn and ran ./autogen.sh but it failed: > > w...@savu > :/usr/local/google/users/wink/svn-clients/protobuf/protobuf-read-only > $ ./autogen.sh >

Re: std::wstring serialization in protocol buffer?

2009-04-07 Thread Kenton Varda
Currently there is no support for wstring. Instead, strings are represented by STL string objects using UTF-8 encoding for Unicode characters. On Sat, Apr 4, 2009 at 4:52 AM, wrote: > > Hi, > > protocols buffers look very nice, taking into account that could be > used on different platforms and

Re: Protocol Buffers Vs. XML Fast Infoset

2009-04-07 Thread Kenton Varda
On Mon, Apr 6, 2009 at 1:53 AM, ShirishKul wrote: > I've *not* applied any kind of compression while using the FI. Then how did your FI data get so small? Protocol Buffers do not leave much room to further reduce the size without some sort of compression. Can you provide example files? --~--

Re: Tagged union types and "switch" statements

2009-04-07 Thread Kenton Varda
If you implemented this in ConsumeInteger(), it would not follow the usual symbol lookup semantics used everywhere else in the language. For example, you would not be able to refer to enums defined later in the file, or defined in a different file. You might be able to handle enums defined inside

Re: Compile error with imported Message

2009-04-07 Thread Kenton Varda
More specifically, you need to make sure that you use the same import path for all of the .proto files in your package. The easiest thing to do is to compile everything from the same top-level directory (and this is good style anyway -- recursive Makefiles can cause lots of problems). Another opt

<    1   2   3   4   5   6   7   8   9   10   >