[Zope-dev] Re: [Z3lab] Zope Corporation's Initial Reaction on the ZF Comments
Hadar Pedhazur wrote:
[clarifications and opinions]
Hey,
I just wanted to chip in that:
* I'm very happy a foundation has been announced. This is what I and
others advocated for, starting in earnest after the castle sprint last
year. This is a major step in the right direction. Thank you Zope
Corporation! I also wish to thank Nuxeo in their active role in taking
this further.
Some points I think we all agree on:
* We all agree that the Foundation should be vendor-neutral. Those of us
in business, we're competing and cooperating at the same time, and the
Foundation will need to be on the balance.
* We all agree that the Foundation should be about more than just
vendors. To those of us who are independent, the Foundation needs to
listen to their voice as well, not just the vendors. We have extremely
significant contributors in the community who are vendor neutral. I need
to mention only the authors of the two Zope 3 books currently published.
Now on a more personal note:
* I am confident that Zope Corporation will do the right thing. Time and
time again I've found that Zope Corporation does listen. Let's all not
forget this.
* From my perspective as a company co-owner: the Foundation should be
vendor-friendly. We should accept that one of the reasons it exists is
to help vendors sell Zope better. ("its future is assured in a
foundation with N members"). This self interest is fine, as long as we
can all cooperate in a fair way. (a state of self-interested cooperation
is ideal, I'd say, in fact)
* It's important for all parties involved to reach clarity about the
relationship of the Zope Foundation with existing foundations and
initiatives within the Zope community. We dont want uncertainty and
confusion and we want to *stimulate* initiatives by the community. The
community is after all one of our greatest resources.
* From my perspective as a developer: I hope politics won't get into the
way of development too much. We'll inevitably and necessarily have
politics in the period leading up to the creation of the Foundation, to
ensure everybody's interests are being looked after. We'll have some
more after the Foundation has been formed. Let's try to stay out of the
developers' hair. Many of them don't want to be involved in politics or
cannot be on a personal basis as they work for a company. We don't want
them worried or pressurized by all of this. Again, the community is one
of our greatest resources.
Regards,
Martijn
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.9 goals
Lennart Regebro wrote: OK, so becuase of the tomembased release schedule, let's not dicuss what goes in 2.9. let's discuss what features we found most urgent/desirable, so we can start working on that, like now. I think it's fine to discuss what we want to be in Zope 2.9. This way we can plan for it to actually be there. We just should realize that if nobody does the work in time, it won't actually be in there, but that should only get the pressure up. The hope is that a time-based schedule will actually speed up feature development, not slow it down. But you're right, work should start soon. This is also why I started the discussion now. Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.9 goals
As I mentioned before, I'd like to see Christian's blob work make it into 2.9. So that's one feature. ;-) On Mon, 2005-06-20 at 09:55 +0200, Martijn Faassen wrote: > Lennart Regebro wrote: > > OK, so becuase of the tomembased release schedule, let's not dicuss > > what goes in 2.9. let's discuss what features we found most > > urgent/desirable, so we can start working on that, like now. > > I think it's fine to discuss what we want to be in Zope 2.9. This way we > can plan for it to actually be there. > > We just should realize that if nobody does the work in time, it won't > actually be in there, but that should only get the pressure up. The hope > is that a time-based schedule will actually speed up feature > development, not slow it down. > > But you're right, work should start soon. This is also why I started the > discussion now. > > Regards, > > Martijn > ___ > Zope-Dev maillist - Zope-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Z3lab] Zope Corporation's Initial Reaction on the ZF Comments
Hi everybody, Martin Faassen wrote: > >* I'm very happy a foundation has been announced. Me too, and I am grateful to Zope Corp. for that. > >* We all agree that the Foundation should be about more than just >vendors [...] We have extremely significant contributors in the community [...] It makes sense to have also _users_ (people who use Zope but who doesn't contribute any code to the code base): we need help from the users to meet their needs. In my opinion Zope Corp. is company in which many people are very clever. >* From my perspective as a company co-owner: the Foundation should be >vendor-friendly. One of the reason that Zope Foundation exists is to support Zope vendors and other communities who use Zope as a technological basis (such as Plone, Nuxeo etc.) >* It's important for all parties involved to reach clarity about the >relationship of the Zope Foundation with existing foundations and >initiatives within the Zope community. It's also important to define how existing foundations (such as Plone Foundation, for instance), may cooperate with Zope Foundation. Regards Piero Giuseppe Goletto ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZOBD and pointers
Hi All, As my ZODB data files become larger and larger I am looking at ways to make the structure of my objects more efficient. To simplify my question, suppose I have two different classes and both contain a list of a objects from a third class: class x has the attribute x.elements = [objects of class z] class y has the attribute y.elements = [objects of class z] As far as I understand python the lists x.elements and y.elements contain pointers to the z objects previously defined. What I wanted to know is how ZODB handles that (or maybe I should say: how pickle handles that) when saving to a file. Will the pointers be converted to a copy of the z class objects or will one copy of the z class objects be saved and than the x.elements and y.elements will still be a list of pointers? Thanks for the help, Yair -- Yair Benita Utrecht University The Netherlands ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ZOBD and pointers
As far as I am aware, ZODB will store a list of pointers to the lists of z objects. What you should be careful of for efficient use of ZODB is that your list is stored in an efficient way, well if the list is updated often or long anyway. When you pack your ZODB does it take up a lot less space? If so it may be that a lot of space is being wasted storing the updated lists of object references. Unless you use a special PersistentList ZODB will have no choice but to store a new copy of the whole list when that list is modified. If you have long lists then this can be a big problem. The Persistent classes have special handling to make them more efficent. So instead of lists use PersistentLists and instead of dicts use BTrees, as these may be stored more efficiently in the ZODB. Also have a look at the analyze.py script to try and track down where the space is being used. My notes here may be helpful too http://zopelabs.com/cookbook/1114086617 Hope that helps, Laurence Yair Benita wrote: Hi All, As my ZODB data files become larger and larger I am looking at ways to make the structure of my objects more efficient. To simplify my question, suppose I have two different classes and both contain a list of a objects from a third class: class x has the attribute x.elements = [objects of class z] class y has the attribute y.elements = [objects of class z] As far as I understand python the lists x.elements and y.elements contain pointers to the z objects previously defined. What I wanted to know is how ZODB handles that (or maybe I should say: how pickle handles that) when saving to a file. Will the pointers be converted to a copy of the z class objects or will one copy of the z class objects be saved and than the x.elements and y.elements will still be a list of pointers? Thanks for the help, Yair ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZOBD and pointers
[Yair Benita] > ... suppose I have two different classes and both contain a list of a objects > from a third class: > > class x has the attribute x.elements = [objects of class z] > class y has the attribute y.elements = [objects of class z] > > As far as I understand python the lists x.elements and y.elements contain > pointers to the z objects previously defined. Yes, Python lists always contain pointers -- even if it's a list of integers, the list actually contains pointers to integer objects. But since that's always true, it's not much help in answering your real question. In general, pointers "make sense" only so long as an object resides in memory. > What I wanted to know is how ZODB handles that (or maybe I should say: > how pickle handles that) when saving to a file. Will the pointers be converted > to a copy of the z class objects or will one copy of the z class objects be > saved and than the x.elements and y.elements will still be a list of pointers? Persistence has its own rules: if an object is persistent (an instance of a subclass of Persistent|), then its current state is stored uniquely in the database, and all references to it just save away (in effect) its persistent object id (oid, usually a 64-bit identifier uniquely assigned to each persistent object, and which retains its value for as long as the database exists). There are no exceptions to this for persistent objects. Oids are effectively a mechanism for building "persistent pointers", and apply only to persistent objects. If an object is not persistent (is not an instance of a subclass of Persistent), it doesn't have an oid, and then there's very little possibility to share references to it on disk. Instead, on disk a copy of its state will usually get made everywhere it's referenced. So the answer to your specific question depends mostly on something you didn't reveal: does class z derive from Persistent? If it does, then _every_ reference on disk to an instance z1 is via z1's oid. If z doesn't derive from Perisistent, then almost all references on disk to an instance z1 will be via a physically distinct copy of z1's full state. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: ZOBD and pointers
[Laurence Rowe] > ... > Unless you use a special PersistentList ZODB will have no choice but > to store a new copy of the whole list when that list is modified. Caution: that's true of a PersistentList too. The purpose of PersistentList isn't realy to supply more-effecient storage (that's the purpose of the various BTree classes). The purpose of PersistentList is this: myobject.my_list_attibute[3] = 4 If my_list_attribute is a plain Python list, the persistence machinery has no way to know that my_list_attribute's state mutated, so the assignment above will not get stored to disk at the next commit unless you _also_ do myobject._p_changed = True # or 1 If my_list_attribute is a PersistentList, then the persistence machinery does know when its state mutates, and there's no need to manage _p_changed manually. But in either case, the entire state of my_list_attribute gets stored to disk whenever any part of it changes. The only difference in what gets stored in the example above is that myobject's state also gets stored to disk if my_list_attribute is a Python list (assuming myobject._p_changed gets set to a true value by hand), while myobject's state does not need to get written to disk again if my_list_attribute is a PersistentList (then myobject refers to my_list_attribute via the latter's oid, and that oid hasn't changed, so there's no need to store myobject's state again). The entire state of the list attribute gets written out in either case. > If you have long lists then this can be a big problem. Very true. > The Persistent classes have special handling to make them more efficent. Sometimes true, but not in the PersistentList case. > So instead of lists use PersistentLists If the goal is to save space, generally no, PersistentList won't help that; to the contrary, their state takes a little more space on disk than a plain list. > and instead of dicts use BTrees, That one's differenent: a BTree is really a graph of (potentially _very_) many distinct perisistent objects, and BTrees were designed to support space- and time- efficient mutation. > as these may be stored more efficiently in the ZODB. For BTrees, yes. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZOBD and pointers
As always. Clear, detailed and to the point. Thanks Tim. Actually, the z class isn't a subclass of persistent because it just holds data (has no methods) and never changes. Same goes to the lists of x and y, they tend to hold a few elements and also never change. The X and Y classes are more complex and are stored using BTrees. Reading this answer I understand that anything I store should be persistent, even if its a list I don't plan to edit. I was under the impression that a subclass of persistent will be larger in size for storage, so I avoided it in the cases mentioned. Is this true? Thanks again for the help, Yair On Jun 20, 2005, at 4:00 , Tim Peters wrote: [Yair Benita] ... suppose I have two different classes and both contain a list of a objects from a third class: class x has the attribute x.elements = [objects of class z] class y has the attribute y.elements = [objects of class z] As far as I understand python the lists x.elements and y.elements contain pointers to the z objects previously defined. Yes, Python lists always contain pointers -- even if it's a list of integers, the list actually contains pointers to integer objects. But since that's always true, it's not much help in answering your real question. In general, pointers "make sense" only so long as an object resides in memory. What I wanted to know is how ZODB handles that (or maybe I should say: how pickle handles that) when saving to a file. Will the pointers be converted to a copy of the z class objects or will one copy of the z class objects be saved and than the x.elements and y.elements will still be a list of pointers? Persistence has its own rules: if an object is persistent (an instance of a subclass of Persistent|), then its current state is stored uniquely in the database, and all references to it just save away (in effect) its persistent object id (oid, usually a 64-bit identifier uniquely assigned to each persistent object, and which retains its value for as long as the database exists). There are no exceptions to this for persistent objects. Oids are effectively a mechanism for building "persistent pointers", and apply only to persistent objects. If an object is not persistent (is not an instance of a subclass of Persistent), it doesn't have an oid, and then there's very little possibility to share references to it on disk. Instead, on disk a copy of its state will usually get made everywhere it's referenced. So the answer to your specific question depends mostly on something you didn't reveal: does class z derive from Persistent? If it does, then _every_ reference on disk to an instance z1 is via z1's oid. If z doesn't derive from Perisistent, then almost all references on disk to an instance z1 will be via a physically distinct copy of z1's full state. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZOBD and pointers
[Yair Benita] > ... > Reading this answer I understand that anything I store should be > persistent, even if its a list I don't plan to edit. I wouldn't say that. For example, for _most_ applications it would be foolish to create a subclass of Persistent to store an integer, as opposed to just storing an integer directly. I can conceive of (unlikely!) applications where there may be advantages to storing integers as perisistent objects, though. In the same vein, if there aren't multiple references to a single small list that doesn't change, there seems little (if any) point to making that a PersistentList. Note that there are other tradeoffs here too. For example, an attribute whose value is persistent is not loaded into RAM when its parent is loaded into RAM, but the full state of non-persistent attributes is loaded into RAM at the time their parent is loaded into RAM. That can have a major effect on time and memory demands, and in opposing directions. Or it may not -- it depends on details of the application's object access patterns. > I was under the impression that a subclass of persistent will be larger in > size > for storage, so I avoided it in the cases mentioned. Is this true? Create a specific class definition, and it's easy to measure. It depends on the class. Certainly it costs more space to create a persistent version of a builtin Python type, and for the same reason it costs more space too to create any user-defined subclass of a builtin Python type. But for an object of a user-defined class, a persistent version takes more RAM when it's in memory (because it has to store info like the oid, and _p_changed, that non-persistent objects don't have), but the on-disk size is at worst roughly the same (e.g., the values of persistent attributes like _p_changed and _p_state don't get stored to disk, they only exist while the persistent object is in RAM). If I were you, I'd spend some quality time with fsdump, and figure out where the bulk of your space is going. Details matter more than "general principles" here. If you use the fsdump.py from ZODB 3.4 (which can be used with .fs files created by ZODB 3.1 and 3.2 too), it displays the byte size of data records, which can be a real help in such analysis. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ZOBD and pointers
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tim Peters wrote: > [Yair Benita] > >>... >>Reading this answer I understand that anything I store should be >>persistent, even if its a list I don't plan to edit. > > > I wouldn't say that. For example, for _most_ applications it would be > foolish to create a subclass of Persistent to store an integer, as > opposed to just storing an integer directly. I can conceive of > (unlikely!) applications where there may be advantages to storing > integers as perisistent objects, though. As, for instance, where the integer changes much more frequently than the other attributes, which are large enough that re-storing them just because the integer attribute changed is painful. Making the attribute a persistent sub-object also eliminates the chance of a ConflictError based on changes to the other attributes. This is the use case which drives BTrees.Length, right? Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCtw/D+gerLs4ltQ4RAnEqAJ9PKCCRriJR3Qt4AWrGCUGk1V6RFQCgxTEl 9waizE6T/pk8Tz/Tkul/4TA= =Uief -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] FW: [Zope-Annce] Zope Foundation ideas
FYI, for the few of you who may not actually listen to the "bigger" lists ;-) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rob Page Sent: Monday, June 20, 2005 5:54 PM To: zope@zope.org; zope-announce@zope.org Subject: [Zope-Annce] Zope Foundation ideas In preparation for tomorrow's IRC session (reminder/details below) we have prepared some initial ideas about the Zope Foundation. These are available online at: o http://tinyurl.com/74pd3 Note -- the document is written with phrases like "the Foundation will", "Contributors shall", etc. This is NOT to be interpreted as though these terms/conditions are predetermined. It is written to close in on specific language that avoids misinterpretation. Zope Foundation IRC Session --- IRC Session Summary: - Who: Zope Corp and Zope Community - What: IRC session to discuss the Zope Foundation - When: Tue, 21 Jun 2005 10a - 12p (US EDT) - Where: irc.freenode.net #zope Please send specific questions to: mailto:[EMAIL PROTECTED] Hope to see you there. Regards, Rob -- Rob PageV: 540.361.1710 Zope CorporationF: 703.995.0412 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
