Protoc and line endings
I'm using protoc as the raw .proto parser for protobuf-net (I then process the compiled binary for code-generation); at the moment, it is very sensitive about line endings - if it isn't LF, it won't work. This creates a bit of a nag for Windows users, as you have to go out of your way to get the right line endings. I could patch the input files, but that gets very tricky if the user is importing multiple other files (with the wrong line endings) into their proto - and it means that their CRLF files wouldn't work fully interoperably. I have no idea how complex it would be, but is there any chance that protoc could be less choosy about line endings? For now, I've added some code to protobuf-net's protogen, so that if protoc reports failure it checks the *known* files for CR (and advises the user) - but this is still tricky for the import scenario. Thoughts? Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
encode to binary stream example
Which class in the ProtoBuf Java api would be used to encode to a binary stream? Is there a short example or tutorial on how to encode (using Java) to a binary stream for RCP? a short code snippet on how to encode/decode would be super helpful. thanks jrv --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: output streams
You have to flush the CodedOutputStream. You should just do this instead: byte[] data = person.build().toByteArray(); On Thu, Jul 2, 2009 at 12:44 AM, J.V. jvsr...@gmail.com wrote: I'm trying to get the following to work but nothing is being written to the output stream. any ideas? private void testEncode() throws Exception { OutputStream out = new ByteArrayOutputStream(); CodedOutputStream cos = CodedOutputStream.newInstance(out); Person.person.Builder person = Person.person.newBuilder(); person.setFirstName(First); person.setLastName(Last); person.build().writeTo(cos); } When I try to decode cos it throws an exception Exception in thread main com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: encode to binary stream example
Just call writeTo(): void writeToStream(Message message, OutputStream out) { message.writeTo(out); } On Thu, Jul 2, 2009 at 7:47 AM, J.V. jvsr...@gmail.com wrote: Which class in the ProtoBuf Java api would be used to encode to a binary stream? Is there a short example or tutorial on how to encode (using Java) to a binary stream for RCP? a short code snippet on how to encode/decode would be super helpful. thanks jrv --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: output streams
Thanks, for some reason I'm' getting the same error. If you have an example of encoding to a stream and then decoding, it would be greatly appreciated. I will get it fully working and update the tutorial or post it somewhere if desired. thanks jrv Kenton Varda wrote: You have to flush the CodedOutputStream. You should just do this instead: byte[] data = person.build().toByteArray(); On Thu, Jul 2, 2009 at 12:44 AM, J.V. jvsr...@gmail.com mailto:jvsr...@gmail.com wrote: I'm trying to get the following to work but nothing is being written to the output stream. any ideas? private void testEncode() throws Exception { OutputStream out = new ByteArrayOutputStream(); CodedOutputStream cos = CodedOutputStream.newInstance(out); Person.person.Builder person = Person.person.newBuilder(); person.setFirstName(First); person.setLastName(Last); person.build().writeTo(cos); } When I try to decode cos it throws an exception Exception in thread main com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protoc and line endings
Protoc treats \r as plain whitespace, so it should have no problem with Windows line endings. I just tested this and sure enough, protoc works fine with .proto files that use Windows-style line endings. Mac pre-OSX line endings (\r with no \n) won't work if the file contains any comments. What kind of errors are you seeing? On Wed, Jul 1, 2009 at 11:48 PM, Marc Gravell marc.grav...@gmail.comwrote: I'm using protoc as the raw .proto parser for protobuf-net (I then process the compiled binary for code-generation); at the moment, it is very sensitive about line endings - if it isn't LF, it won't work. This creates a bit of a nag for Windows users, as you have to go out of your way to get the right line endings. I could patch the input files, but that gets very tricky if the user is importing multiple other files (with the wrong line endings) into their proto - and it means that their CRLF files wouldn't work fully interoperably. I have no idea how complex it would be, but is there any chance that protoc could be less choosy about line endings? For now, I've added some code to protobuf-net's protogen, so that if protoc reports failure it checks the *known* files for CR (and advises the user) - but this is still tricky for the import scenario. Thoughts? Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: output streams
Write to a stream: message.writeTo(stream); Parse from a stream: message = MyType.parseFrom(stream); If you need to read/write multiple messages on the same stream (or the stream does not end immediately after the first message), use writeDelimitedTo() and parseDelimitedFrom(). On Thu, Jul 2, 2009 at 10:56 AM, J.V. jvsr...@gmail.com wrote: Thanks, for some reason I'm' getting the same error. If you have an example of encoding to a stream and then decoding, it would be greatly appreciated. I will get it fully working and update the tutorial or post it somewhere if desired. thanks jrv Kenton Varda wrote: You have to flush the CodedOutputStream. You should just do this instead: byte[] data = person.build().toByteArray(); On Thu, Jul 2, 2009 at 12:44 AM, J.V. jvsr...@gmail.com mailto:jvsr...@gmail.com wrote: I'm trying to get the following to work but nothing is being written to the output stream. any ideas? private void testEncode() throws Exception { OutputStream out = new ByteArrayOutputStream(); CodedOutputStream cos = CodedOutputStream.newInstance(out); Person.person.Builder person = Person.person.newBuilder(); person.setFirstName(First); person.setLastName(Last); person.build().writeTo(cos); } When I try to decode cos it throws an exception Exception in thread main com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protoc and line endings
protoc actually expects its input to be UTF-8 (though non-ASCII characters are only allowed in default values for string fields). It just doesn't like the BOM. On Thu, Jul 2, 2009 at 12:44 PM, Marc Gravell marc.grav...@gmail.comwrote: My bad... it isn't the line endings - it is the UTF8 BOM; when I switched one it switched the other. (which is annoying; encoding is much trickier than just cr/lf!) Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protoc and line endings
OK... is there any way it /could/ silently ignore the BOM? ;-p I can try to advise the caller to use files without BOMs, but protoc reads UTF8 anyway it seems reasonable to accept a BOM? Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protoc and line endings
If you want to write up a patch to recognize a UTF8 BOM and ignore it, go ahead. You can just modify the Tokenizer class to recognize and discard a BOM appearing at the beginning of the input. On Thu, Jul 2, 2009 at 1:31 PM, Marc Gravell marc.grav...@gmail.com wrote: OK... is there any way it /could/ silently ignore the BOM? ;-p I can try to advise the caller to use files without BOMs, but protoc reads UTF8 anyway it seems reasonable to accept a BOM? Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
ThirdPartyAddOns
I notice the replacement ThirdPartyAddOns page; can you please also include protobuf-net in the RPC section? It provides 2 separate RPC implementations (WCF-based and bespoke). Also; I wonder if it might help to clarify (at the top) which languages are provided in the official codebase; then the entire list of implementations would be visible on a single page (just a thought). Thanks, Marc Gravell --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: ThirdPartyAddOns
Done and done. I didn't know protobuf-net had RPC or I would have listed it on the RPC implementations page some time ago. On Thu, Jul 2, 2009 at 1:47 PM, Marc Gravell marc.grav...@gmail.com wrote: I notice the replacement ThirdPartyAddOns page; can you please also include protobuf-net in the RPC section? It provides 2 separate RPC implementations (WCF-based and bespoke). Also; I wonder if it might help to clarify (at the top) which languages are provided in the official codebase; then the entire list of implementations would be visible on a single page (just a thought). Thanks, Marc Gravell --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: ThirdPartyAddOns
What's the difference between all these links? On Thu, Jul 2, 2009 at 4:08 PM, Chris Kuklewicz turingt...@gmail.comwrote: Kenton, Could you change the link on the http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns page for Haskell from http://darcs.haskell.org/packages/protocol-buffers/ to http://darcs.haskell.org/packages/protocol-buffers2/ And if there is room can you also add http://hackage.haskell.org/package/hprotoc ? -- Chris --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
error when using nested and packed in python
Hello protobuf gurus- I'm just getting started with the python version of protobuf, using v2.10. I want to store some lists of floats in a database's blob field. When using a nested Message structure with the packed=true option, I'm getting errors that look like this: Traceback (most recent call last): File stdin, line 1, in module File build/bdist.linux-i686/egg/google/protobuf/message.py, line 160, in ParseFromString File build/bdist.linux-i686/egg/google/protobuf/reflection.py, line 1215, in MergeFromString File build/bdist.linux-i686/egg/google/protobuf/reflection.py, line 1075, in _DeserializeOneEntity File build/bdist.linux-i686/egg/google/protobuf/reflection.py, line 899, in _RecursivelyMerge File build/bdist.linux-i686/egg/google/protobuf/internal/ decoder.py, line 181, in ReadMessageInto google.protobuf.message.DecodeError: Submessage told to deserialize from 14-byte encoding, but used only 12 bytes If I use a flat (not nested) message structure, I do not get these errors. If I use a nested, packed=false approach, I do not get these errors. The error happens only when both nested and packed. Here is some code to reproduce the problem. Note that the deserialization seems to succeed in putting the correct values into the list (at least in this test), in spite of the error message. my.proto file contains: message TestNest { optional ListData list_data = 1; message ListData { repeated float float_packed = 1 [packed=true]; repeated float float_notpacked = 2 [packed=false]; repeated int32 int_packed = 3 [packed=true]; repeated int32 int_notpacked = 4 [packed=false]; } } and then I run this set of commands for each of the fields [float_packed, float_notpacked, int_notpacked, int_packed] and see that the error happens only for the two fields that use packed=true. import my_pb2 a = my_pb2.TestNest() a.list_data.float_notpacked.extend([1,2,3]) a_string = a.SerializeToString() a_deserialized = my_pb2.TestNest() a_deserialized.ParseFromString(a_string) str(a_deserialized) I can think of three options to proceed, 2 of which I'm able to do myself: 1) don't use the packed=true option 2) catch the errors and ignore them 3) if this is actually a bug, ask you gurus to fix the protobuf code What do you recommend? Dan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
libprotobuf.lib, 29mb?
I'm on windows, using Visual Studio 2008 SP1, compiling a Release x64 build of libprotobuf. The lib file is 29,570KB big, is that expected? I'm using the provided visual studio projects, but I added the x64 configuration to them, perhaps I have some options set incorrectly or something. (The debug lib is 33,575KB) thanks! - Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: error when using nested and packed in python
Yeah, I think it's broken. In reflection.py there is a line: return decoder.Position() - content_start I think (untested) that it should actually be: return decoder.Position() - initial_position I have no idea how this got past testing and I have asked the people responsible to fix it. Surprisingly you are the first to report this, so maybe there's a mitigating factor that makes it unusual for the bug to harm anything. Sorry for the trouble. On Thu, Jul 2, 2009 at 5:14 PM, Dan danbr...@gmail.com wrote: Hello protobuf gurus- I'm just getting started with the python version of protobuf, using v2.10. I want to store some lists of floats in a database's blob field. When using a nested Message structure with the packed=true option, I'm getting errors that look like this: Traceback (most recent call last): File stdin, line 1, in module File build/bdist.linux-i686/egg/google/protobuf/message.py, line 160, in ParseFromString File build/bdist.linux-i686/egg/google/protobuf/reflection.py, line 1215, in MergeFromString File build/bdist.linux-i686/egg/google/protobuf/reflection.py, line 1075, in _DeserializeOneEntity File build/bdist.linux-i686/egg/google/protobuf/reflection.py, line 899, in _RecursivelyMerge File build/bdist.linux-i686/egg/google/protobuf/internal/ decoder.py, line 181, in ReadMessageInto google.protobuf.message.DecodeError: Submessage told to deserialize from 14-byte encoding, but used only 12 bytes If I use a flat (not nested) message structure, I do not get these errors. If I use a nested, packed=false approach, I do not get these errors. The error happens only when both nested and packed. Here is some code to reproduce the problem. Note that the deserialization seems to succeed in putting the correct values into the list (at least in this test), in spite of the error message. my.proto file contains: message TestNest { optional ListData list_data = 1; message ListData { repeated float float_packed = 1 [packed=true]; repeated float float_notpacked = 2 [packed=false]; repeated int32 int_packed = 3 [packed=true]; repeated int32 int_notpacked = 4 [packed=false]; } } and then I run this set of commands for each of the fields [float_packed, float_notpacked, int_notpacked, int_packed] and see that the error happens only for the two fields that use packed=true. import my_pb2 a = my_pb2.TestNest() a.list_data.float_notpacked.extend([1,2,3]) a_string = a.SerializeToString() a_deserialized = my_pb2.TestNest() a_deserialized.ParseFromString(a_string) str(a_deserialized) I can think of three options to proceed, 2 of which I'm able to do myself: 1) don't use the packed=true option 2) catch the errors and ignore them 3) if this is actually a bug, ask you gurus to fix the protobuf code What do you recommend? Dan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: libprotobuf.lib, 29mb?
On Thu, Jul 2, 2009 at 5:33 PM, Alex Black a...@alexblack.ca wrote: I'm on windows, using Visual Studio 2008 SP1, compiling a Release x64 build of libprotobuf. The lib file is 29,570KB big, is that expected? I was going to say no way, but sure enough, the 32-bit release build is 18,154k, so 64-bit could easily be double that. However, when you actually link a binary with the library, the size is much smaller. protoc.exe, for example, is 1MB, and it includes both libprotobuf.lib and libprotoc.lib, the latter of which is 10MB. I don't know much about MSVC, but perhaps it is including some sort of debug info in this library? Cygwin produces a 14MB libprotobuf.a, but after stripping it is down to 968k. Note that I consider 1MB to be too big for many users, which is why the next version will have a lite mode that sheds all the descriptors and reflection features and should be very small. I'm using the provided visual studio projects, but I added the x64 configuration to them, perhaps I have some options set incorrectly or something. (The debug lib is 33,575KB) thanks! - Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: how to use protobuf with Google App Engine?
You'd have to ask the AppEngine people. Someone from that team submitted the input_stream.py change. The problem is that both protocol buffers and AppEngine place themselves in the google package. Protocol Buffers has, from day one, declared the package as a namespace package using setuptools, which theoretically allows sharing of the package. For things to work, though, AppEngine has to cooperate by doing the same thing in their own code. On Thu, Jul 2, 2009 at 6:46 PM, Dan danbr...@gmail.com wrote: Hello- I'm working with GAE and ran into this error when trying to use protocol buffers: from google.protobuf import descriptor ImportError: No module named protobuf I saw some mention of people using hacks to do this in the past, but also saw that the release notes for protocol buffers v 2.10 said this: * Changes to input_stream.py should make protobuf compatible with appengine. I'm not sure what is meant by compatible in that statement, but I'd like to know if GAE still requires the hackery described in these threads referenced below. Is there an up-to-date howto or something like that somewhere that I missed? http://groups.google.com/group/protobuf/browse_thread/thread/6430dc5d7ceb145a/b8634dddbe4425f3?lnk=gstq=gae#b8634dddbe4425f3 http://groups.google.com/group/google-appengine/browse_thread/thread/db910ed40ae495fc/f41873a476db744e Dan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: how to use protobuf with Google App Engine?
OK, I'll ask over there. thanks On Jul 2, 6:55 pm, Kenton Varda ken...@google.com wrote: You'd have to ask the AppEngine people. Someone from that team submitted the input_stream.py change. The problem is that both protocol buffers and AppEngine place themselves in the google package. Protocol Buffers has, from day one, declared the package as a namespace package using setuptools, which theoretically allows sharing of the package. For things to work, though, AppEngine has to cooperate by doing the same thing in their own code. On Thu, Jul 2, 2009 at 6:46 PM, Dan danbr...@gmail.com wrote: Hello- I'm working with GAE and ran into this error when trying to use protocol buffers: from google.protobuf import descriptor ImportError: No module named protobuf I saw some mention of people using hacks to do this in the past, but also saw that the release notes for protocol buffers v 2.10 said this: * Changes to input_stream.py should make protobuf compatible with appengine. I'm not sure what is meant by compatible in that statement, but I'd like to know if GAE still requires the hackery described in these threads referenced below. Is there an up-to-date howto or something like that somewhere that I missed? http://groups.google.com/group/protobuf/browse_thread/thread/6430dc5d... http://groups.google.com/group/google-appengine/browse_thread/thread/... Dan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
RE: libprotobuf.lib, 29mb?
Thanks Kenton, thats reassuring. From: Kenton Varda [mailto:ken...@google.com] Sent: Thursday, July 02, 2009 8:56 PM To: Alex Black Cc: Protocol Buffers Subject: Re: libprotobuf.lib, 29mb? On Thu, Jul 2, 2009 at 5:33 PM, Alex Black a...@alexblack.ca wrote: I'm on windows, using Visual Studio 2008 SP1, compiling a Release x64 build of libprotobuf. The lib file is 29,570KB big, is that expected? I was going to say no way, but sure enough, the 32-bit release build is 18,154k, so 64-bit could easily be double that. However, when you actually link a binary with the library, the size is much smaller. protoc.exe, for example, is 1MB, and it includes both libprotobuf.lib and libprotoc.lib, the latter of which is 10MB. I don't know much about MSVC, but perhaps it is including some sort of debug info in this library? Cygwin produces a 14MB libprotobuf.a, but after stripping it is down to 968k. Note that I consider 1MB to be too big for many users, which is why the next version will have a lite mode that sheds all the descriptors and reflection features and should be very small. I'm using the provided visual studio projects, but I added the x64 configuration to them, perhaps I have some options set incorrectly or something. (The debug lib is 33,575KB) thanks! - Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---