Re: [protobuf] Re: WriteDelimited/parseDelimited in python
I was saying the user *could* do that, and that it's currently what I'm doing in my server-side code. The reason being, as you said, if you naively read from a stream and the message isn't all present then you need to block until it is with the way that the Java code works at present. If you are using it for client-side code then likely this is not an issue in the slightest, but a server that needs to be able to handle many clients at once just can not block on one of them... As to your other alternative, (a), I would suggest that this leaves too much of the underlying network protocol bare to the caller. This will make it very difficult to change the way that delimiting messages happens in the future should such a thing be required. If - for example - it is decided to go from having the length prefixed to having a special delimiting sequence after the message then it will cause all current calling code to need to be changed. It might be that this is considered a low enough level library that this is acceptable, but that would be a Google decision... One more alternative would be how the asn1c library works for parsing ASN.1 streams into objects, which is to be resumable. The decoder reads all the data it is given, and tries to build the object from this. If it doesn't have enough data yet then it does what it can, remembers where it got to and returns back to the user who can then supply more data when it becomes available. If the entire message does parse from the data provided then return back to the user the amount of data consumed so that they can discard this (reading from the stream directly makes this slightly cleaner still). At present, the Protobuf libraries (any of them) can not support this method of decoding an object, and it is not a trivial change to make it possible to do, but it does - IMO - give a much cleaner and easier to use method of use. -- Graham Cox On Tue, Jan 5, 2010 at 1:32 AM, Kenton Varda wrote: > Make sure to "reply all" so that the group is CC'd. > > So you are saying that the user should read whatever data is on the socket, > then attempt to parse it, and if it fails, assume that it's because there is > more data to read? Seems rather wasteful. I think what we ideally want is > either: > (a) Provide a way for the caller to read the size independently, so that > they can then make sure to read that many bytes from the input before > parsing. > (b) Provide a method that reads from a stream, so that the protobuf library > can automatically take care of reading all necessary bytes. > > Option (b) is obviously cleaner but has a few problems: > - We have to choose a particular stream interface to support. While the > Python "file-like" interface is pretty common I'm not sure if it's universal > for this kind of task. > - If not all bytes of the message are available yet, we'd have to block. > This might be fine most of the time, but would be unacceptable for some > uses. > > Thoughts? > > On Mon, Jan 4, 2010 at 3:09 PM, Graham Cox wrote: > >> I'm using it for reading/writing to sockets in my functional tests - works >> well enough there... >> In my Java-side server code, I read from the socket into a byte buffer, >> then deserialize the byte buffer into Protobuf objects, throwing away the >> data that has been deserialized. The python "MergeDelimitedFromString" >> function also returns the number of bytes that were processed to build up >> the Protobuf object, so the user could easily do the same - read the socket >> onto the end of a buffer, and then while the buffer is successfully >> deserializing into objects throw away the first x bytes as appropriate... >> >> Just a thought :) >> >> On Mon, Jan 4, 2010 at 9:57 PM, Kenton Varda wrote: >> >>> Hmm, it occurs to me that this currently is not useful for reading from a >>> socket or similar stream since the caller has to make sure to read an entire >>> message before trying to parse it, but the caller doesn't actually know how >>> long the message is (because the code that determines this is encapsulated). >>> Any thoughts on this? >>> >>> On Mon, Jan 4, 2010 at 12:11 PM, Kenton Varda wrote: >>> Mostly looks good. There are some style issues (e.g. lines over 80 chars) but I can clean those up myself. You'll need to sign the contributor license agreement: http://code.google.com/legal/individual-cla-v1.0.html -- If you own copyright on this change. http://code.google.com/legal/corporate-cla-v1.0.html -- If your employer does. Please let me know after you've done this and then I can submit these. On Fri, Jan 1, 2010 at 12:53 PM, Graham wrote: > On Jan 1, 7:32 am, Kenton Varda wrote: > > I don't think an equivalent has been added to the Python API. Want > to write > > up a patch? > > Well - if you insist... Here's a first run, which seems to work but > I'm a very long way from a competent python programmers so feel
[protobuf] How to generate File Descriptor from .proto file in Java code without generating .java files?
How to generate File Descriptor from .proto file in Java code without generating .java files? -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] How to generate File Descriptor from .proto file in Java code without generating .java files?
The parser is written in C++. You would need to either call it via JNI or execute protoc as a subprocess. protoc's --descriptor_set_out option will produce a FileDescriptorSet protocol buffer, as defined in descriptor.proto, which you can then parse in Java to produce FileDescriptors. On Tue, Jan 5, 2010 at 7:37 AM, Rosa wrote: > How to generate File Descriptor from .proto file in Java code without > generating .java files? > > -- > > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: building the protocol buffers with MinGW-4.4.0
OK, since this isn't the standard release, and the bug is clearly in MinGW itself, I'll leave it up to users to deal with it, unless you want to submit a patch (and it is simple and non-intrusive). On Mon, Jan 4, 2010 at 10:40 PM, lucas_ro wrote: > Ok, I try to make the stuffs more clear. > > In order to get the official MinGW 4.4.0 (aka gcc 4.4.0) you have to > follow the link http://sourceforge.net/projects/mingw/files/. Downward > that page please look for GCC Version 4 then expand it and find out > CurrentRelease_gcc-4.4.0. Under the latter branch you may find the > official release of gcc 4.4.0 for Windows. Please read the content of > the file gcc-4.4.0-mingw-notes.txt in order to have some hints on the > installation of this software package. > > If you find the above procedure too complicated (and indeed it is) > then the simplest way to have MinGW-4.4.0 is to download the following > file: ftp://ftp.qt.nokia.com/misc/MinGW-gcc440_1.zip . It is the > official release patched in order to allow the building of the > DirectShow under MinGW. > > The executable MinGW 5.1.6 installs the version 3.4.5 of gcc !!! > > On Jan 5, 4:38 am, Kenton Varda wrote: > > OK, I'm pretty confused now. Apparently there is a MinGW 5.1.6. > However, > > the MinGW version number does not appear to be related to the GCC version > > number -- it comes with gcc 3.4.5. Your link talks about building with > > MinGW and *GCC* 4.4.0. Last I knew, MinGW had not yet provided an > official > > release with GCC 4.x and AFAICT this is still the case. > > > > During the install I chose the "current" release. Maybe you chose the > > "candidate"? > > > > > > > > On Mon, Jan 4, 2010 at 12:02 PM, lucas_ro wrote: > > > It's the official stable release. As I observed by searching the web > > > for a solution the issue regarding 'libstdc++.dll.a' is well known. > > > Unfortunately it seems that they (mingw team) react very slowly in > > > fixing the signaled issues/bugs. > > > > > The link that helped me is > > >http://eschew.wordpress.com/2009/09/20/building-protobuf-with-mingw-g. > .. > > > . > > > (however it is not necessary to install the mozilla-build tree). > > > > > On Jan 4, 9:27 pm, Kenton Varda wrote: > > > > I see. Is mingw-4.4.0 a stable release or a beta? Unfortunately > their > > > web > > > > site is pretty terribly designed and I cannot figure this out. If > it's > > > an > > > > official release then we should probably work around the bug, but > > > otherwise > > > > I assume they'll fix it since it's a known issue. > > > > > > On Mon, Jan 4, 2010 at 11:07 AM, lucas_ro > wrote: > > > > > Eventually I succeeded by changing in the file libstdc++.la the > value > > > > > of the variable library_names from 'libstdc++.dll.a' to > 'libstdc++.a'. > > > > > The libtool was complaining about the missing of the file 'libstdc+ > > > > > +.dll.a'. It seems that this is a mingw-4.4.0 known issue. > > > > > > > Also I choose to build only the static library (passing the option > -- > > > > > disable-shared on to the command 'configure'). > > > > > > > On Jan 4, 8:07 pm, Kenton Varda wrote: > > > > > > Did you try and fail? If so, please post the error log. > > > > > > > > On Mon, Jan 4, 2010 at 5:52 AM, lucas_ro > > > wrote: > > > > > > > Hello, > > > > > > > > > Have anyone succeeded to build the protocol buffers using > > > MinGW-4.4.0 > > > > > > > under Windows ? > > > > > > > > > Thank you, > > > > > > > Florin > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the > Google > > > > > Groups > > > > > > > "Protocol Buffers" group. > > > > > > > To post to this group, send email to protobuf@googlegroups.com > . > > > > > > > To unsubscribe from this group, send email to > > > > > > > protobuf+unsubscr...@googlegroups.com > > > > > > > > > > > > > > > > . > > > > > > > For more options, visit this group at > > > > > > >http://groups.google.com/group/protobuf?hl=en. > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "Protocol Buffers" group. > > > > > To post to this group, send email to proto...@googlegroups.com. > > > > > To unsubscribe from this group, send email to > > > > > protobuf+unsubscr...@googlegroups.com > > > > > > > > > . > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/protobuf?hl=en. > > > > > -- > > > > > You received this message because you are subscribed to the Google > Groups > > > "Protocol Buffers" group. > > > To post to this group, send email to proto...@googlegroups.com. > > > To unsubscribe from this group, send email to > > > protobuf+unsubscr...@googlegroups.com > > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/protobuf?hl=en. > > -- > > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroup
