Re: Python implementation questions
On Wed, Nov 5, 2008 at 2:53 PM, Kenton Varda [EMAIL PROTECTED] wrote: [cc'ing Petar] On Tue, Nov 4, 2008 at 11:23 PM, DVusBoy [EMAIL PROTECTED] wrote: Hi, I have a couple of questions regarding the Python implementation. 1. There is a discrepancy between the function signatures of the prototype in google.protobuf.message.Message.ListFields(self, field_name) vs the actual implementation under google.protobuf.reflection._AddListFieldsMethod.ListFields(self). While it doesn't seem to hurt much as the concrete implementation trumps the prototype. It does look a bit weird that ListFields() should take a field_name argument. I'll fix this. 2. I also am wondering why doesn't a repeated field behaves more like a list in Python? Perhaps it is in keeping API consistency with the C+ + implementation, but why can I not do the following, for instance? message.repeated = [ 1, 2, 3 ] This (the slice assignment in general, like m.repeated[0:1] = [1, 2, 3]) was discussed some time ago. It isn't at all hard to add, the concern is that the it doesn't have a straight forward alternative in the C++ API (it can be done in the C++ API, but in a more complicated way). This may be a problem for the Python/C API that we are planing to have. It will be a Python C extension providing the same API as currently the pure Python API. If we allow slice assignment now, it will be very hard to forbid it if it turns to be a problem with the C extension. Thanks you for your insight, -kc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: speed - python implementation
On Sun, Nov 9, 2008 at 5:14 PM, codeazure [EMAIL PROTECTED] wrote: On Oct 31, 5:19 am, Petar Petrov [EMAIL PROTECTED] wrote: Yes, there are plans to improve performance. I have spent a little time on this without significant improvements. I think performance can hardly get a drastic improvement without a C++ extension module (which we are planning to have). Are you aware of anyone doing any work on a C++ Boost::Python interface for PB? No, we aren't aware of such. This would seem to be a relatively easy thing to write, implementing the __getattr__/__setattr__ Python methods in Boost::Python to interface to the reflection mechanism in PB. A few things. The current Python API has to remain pure-Python because some clients aren't able to use C/C++ extensions (like AppEngine). Boost is generally not accepted in Google, so a Boost::Pythonit interface will have to distribute separately. We are planning a Python C extension. It will likely consist of a separate python code generator to create Python code which wraps the C++ API and provides Python API similar to the current pure-Python protobuf API. If noone else is doing it, I might try this myself pass it on if it works. Regards, Jeff --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Serializing Large Collections: SerializePartial ParsePartial
I think the idea is to break up very large data sets into smaller packets so they can be 'streamed'. When I think of something like seismic data, stream based event handling makes the most sense. Can the data points be processed individually somehow, or do you need access to all of them (in memory) at once? -bmadigan On Oct 22, 6:54 pm, [EMAIL PROTECTED] wrote: OK, that makes sense. Thanks for the quick reply. I work at a seismic earthquake data center. We're looking at using protocol buffers as a means of internally moving around processed chunks of data. Seems to work pretty well, as long as the chunks aren't too large (which is a problem one way or another). But working with ~5 million data points doesn't seem to be any problem. some other container format. Not exactly sure what that would look like. Thanks again. -B On Oct 22, 3:26 pm, Kenton Varda [EMAIL PROTECTED] wrote: The Partial serialize and parse routines actually do something completely unrelated: they allow the message to be missing required fields. So, that doesn't help you. I'm afraid protocol buffers are not designed for storing very large collections in a single message. Instead, you should be thinking about representing each item in a collection using a protocol message, but then using some other container format. Using protocol buffers here makes the container format simpler because it only needs to deal with raw strings rather than having to worry about structured data. On Wed, Oct 22, 2008 at 10:19 AM, [EMAIL PROTECTED] wrote: Is there a general strategy/methodology for dealing with very large collections so that they do not need to be completely held in memory before serializing and de-serializing? I see, for example, SerializePartialToOstream() and ParsePartialFromIstream() but no documentation of how to use it. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Explanation of comment in descriptor.proto?
You don't have to worry about the uninterpreted_option field. It's an internal implementation detail of the parser. You should pretend it isn't there. The comment is saying that the uninterpreted_option field must have the name uninterpreted_option in all *Options messages because the parser uses reflection to access it. On Wed, Nov 12, 2008 at 2:47 AM, Jon Skeet [EMAIL PROTECTED] wrote: I was just looking over descriptor.proto for the option extensions, and I spotted this comment that I hadn't seen before: // Clients may define custom options as extensions of the *Options messages. // These extensions may not yet be known at parsing time, so the parser cannot // store the values in them. Instead it stores them in a field in the *Options // message called uninterpreted_option. This field must have the same name // across all *Options messages. We then use this field to populate the // extensions when we build a descriptor, at which point all protos have been // parsed and so all extensions are known. I don't understand the This field must have the same name across all *Options messages. Am I right in saying that's an implementation detail about the uninterpreted_option field? It's not saying that the extensions themselves need to have the same name across all *Options messages, right? Just thought it would be worth clarifying before I do the wrong thing :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Is it possible to download the documentation?
Hi - My company works on an intranet that doesn't have an internet connection. For 3rd-party libraries, we typically download the documentation and host it locally. I plan to use protobuf, but I don't see anywhere to download the docs. Any chance they'll be added to the download package or offered as a separate package? I can spider them but it's a pain and all the URLs and CSS are screwy... Thanks, Scott --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Is it possible to download the documentation?
Sorry, the docs are not actually stored in HTML form. They are EZT templates, so if we did make them downloadable you'd have to run an EZT web server to view them. The best we could do otherwise would be to spider them ourselves. So I don't think there's much worth doing here. On Wed, Nov 12, 2008 at 5:25 PM, Scott Stafford [EMAIL PROTECTED]wrote: Hi - My company works on an intranet that doesn't have an internet connection. For 3rd-party libraries, we typically download the documentation and host it locally. I plan to use protobuf, but I don't see anywhere to download the docs. Any chance they'll be added to the download package or offered as a separate package? I can spider them but it's a pain and all the URLs and CSS are screwy... Thanks, Scott --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
