[Zope] Zope/Cache/Memory
I have a database that is just under 200K objects with plans that it will grow larger. The objects are created by using another object with methods that read files and create the objects from the data in the files. When creating the objects, I find I have to manually clear the cache or else zope eventually crashes with an out of memory error. There are 4GB of ram on the server. The same issue arises when I catalog the objects. I am trying maintain as much speed as possible, so I do not want to catalog until the end of the import. Is there a way to programmatically clear the cache during a large indexing to avoid out of memory errors? I am using zeo storages and setting the cache on those to a smaller size has no real effect in this case, because the cache seems to outgrow its defined size when it is very active. Is zope even good for this? Should I consider another database? It still seems that cataloging is an issue whether the objects are stored in the zodb or elsewhere. -Kevin ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Scalability
This is probably falls under the catagory of FAQ but I couldn't find a good answer through searching, so I'll ask. Are there any size limitations on file system storages? At what point can things become very slow? I'm wondering as much about the number of objects stored as well as file size for a Data.fs. And what about zeo storages. Is there on the number of zeo storages that can be connected to zope clients? As far as number of objects, I'm thinking in terms of 10,000,000 or more folderish and content objects. While I'm talking about it, are there any performance limitations on the zcatalog? Will it perform well even with this many objects? Thank you in advance, Kevin ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: accessing object from a list constructed in __init__.py
Thank you everybody for your input and help with this. What was explained to me and what I missed when reading about acquisition is that _getattr_ is used by acquisition to return attributes in context. If your objects are in a list, _getattr_ is not available, but _getitem_ is used to return the objects. So there are two solutions and two schools of thought on how to solve this. Some will over ride _getitem_ so that lists return object.__of__(self) , others will wrap the objects manually. The second solution is the one that I took. I wrote a method for my class that takes an integer and will return that element from the list as a wrapped object. simply def getObject(self,i): '''return wrapped object from list''' object = self.list.__of__(self) return object Thanks again everyone for your help. -Kevin ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] accessing object from a list constructed in __init__.py
I've changed my code a bit, adding the docstring, Persistent to all classes and __allow_access_to_unprotected_attributes__ = 1 to all classes. Still, I recieve the same error. I have changed the code so _attribute1a is a single instance of class1 instead of a list of containing multiple instances of class1 and I can access the attributes of the object without issue. Are the objects automatically unwrapped when added to a list? is there a way around this? I can send the code again if need be. so here is the code again, this time with only the single object in attribute1a. --myClass.py-- import Acquisition from persistent import Persistent class class1(Acquisition.Explicit,Persistent): ''' a simple class with some attributes and methods''' __allow_access_to_unprotected_attributes__ = 1 attribute1 = 'attribute 1' attribute2 = 'attribute 2' _number = '' def __init__(self,number2): '''simple init for the class''' self._number=number2*3 def method1(self): '''return a string methd1''' mthd1 = 'methd1' return mthd1 def method2(self): ''' return a string methd2''' mthd2 = 'method2' return mthd2 def _get_number(self): return self._number number = property(fget=_get_number) class class2(Acquisition.Explicit,Persistent): ''' a second simple class with some attributes and methods''' __allow_access_to_unprotected_attributes__ = 1 _attribute1a = [] attribute2a = [] def __init__(self): '''create a list of class1 objects in attribute1a''' #i=1 #while i 5: #obj = class1(i) #self._attribute1a.append(obj) #i=i+1 self._attribute1a = class1(5) def method1a(self): '''instantiate class1 as object and return object usable by zope''' obj = class1(5) return obj.__of__(self) def method2a(self): '''returns class1.method1()''' c1m1 = class1.method1() return c1m1 def get_attribute1a(self): #if self._attribute1a != []: #i = len(self._attribute1a) #j = 0 #while j != i-1: #self._attribute1a[j] = self._attribute1a[j].__of__(self) #j = j+1 #else: obj = self._attribute1a return obj.__of__(self) attribute1a = property(fget=get_attribute1a) attribute1b = property(fget=method1a) attribute2b = property(fget=method2a) Alexis Roda wrote: En/na kevin7kal ha escrit: Ok, thank you for that. I can create the list of objects from the __of__(self) attribute without the wrapper error now, but I still cannot access that attribute through zope. Here is my code again, along with my zope content. Another thing I've missed, class1 isn't persistent, but you store instances in Zclass2.attribute1a. This can explain the can't pickle error. HTH ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] accessing object from a list constructed in __init__.py
The traceback and 'offending' code. Traceback (innermost last): Module ZPublisher.Publish, line 115, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 41, in call_object Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.PythonScripts.PythonScript, line 323, in _exec Module None, line 7, in AMyZClass2 - PythonScript at /AMyZClass2 - Line 7 Unauthorized: You are not allowed to access 'number' in this context --Script(Python)-- MZ = context.Zclass2.myZclass() M2 = MZ.attribute1a print MZ print M2[0].number return printed kevin7kal wrote: I've changed my code a bit, adding the docstring, Persistent to all classes and __allow_access_to_unprotected_attributes__ = 1 to all classes. Still, I recieve the same error. I have changed the code so _attribute1a is a single instance of class1 instead of a list of containing multiple instances of class1 and I can access the attributes of the object without issue. Are the objects automatically unwrapped when added to a list? is there a way around this? I can send the code again if need be. so here is the code again, this time with only the single object in attribute1a. --myClass.py-- import Acquisition from persistent import Persistent class class1(Acquisition.Explicit,Persistent): ''' a simple class with some attributes and methods''' __allow_access_to_unprotected_attributes__ = 1 attribute1 = 'attribute 1' attribute2 = 'attribute 2' _number = '' def __init__(self,number2): '''simple init for the class''' self._number=number2*3 def method1(self): '''return a string methd1''' mthd1 = 'methd1' return mthd1 def method2(self): ''' return a string methd2''' mthd2 = 'method2' return mthd2 def _get_number(self): return self._number number = property(fget=_get_number) class class2(Acquisition.Explicit,Persistent): ''' a second simple class with some attributes and methods''' __allow_access_to_unprotected_attributes__ = 1 _attribute1a = [] attribute2a = [] def __init__(self): '''create a list of class1 objects in attribute1a''' #i=1 #while i 5: #obj = class1(i) #self._attribute1a.append(obj) #i=i+1 self._attribute1a = class1(5) def method1a(self): '''instantiate class1 as object and return object usable by zope''' obj = class1(5) return obj.__of__(self) def method2a(self): '''returns class1.method1()''' c1m1 = class1.method1() return c1m1 def get_attribute1a(self): #if self._attribute1a != []: #i = len(self._attribute1a) #j = 0 #while j != i-1: #self._attribute1a[j] = self._attribute1a[j].__of__(self) #j = j+1 #else: obj = self._attribute1a return obj.__of__(self) attribute1a = property(fget=get_attribute1a) attribute1b = property(fget=method1a) attribute2b = property(fget=method2a) Alexis Roda wrote: En/na kevin7kal ha escrit: Ok, thank you for that. I can create the list of objects from the __of__(self) attribute without the wrapper error now, but I still cannot access that attribute through zope. Here is my code again, along with my zope content. Another thing I've missed, class1 isn't persistent, but you store instances in Zclass2.attribute1a. This can explain the can't pickle error. HTH ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] accessing object from a list constructed in __init__.py
That did not work for me either. It was also suggested that I try inheriting from folder rather than re-inventing folder. Still no dice. I think there is something that I don't understand about acquisition and context. Why can an object be placed into context if it is stored as a simple attribute of another object, but not if it is an element in a list? Or what must be done differently when the objects are elements in a list? Alexis Roda wrote: En/na kevin7kal ha escrit: I've changed my code a bit, adding the docstring, Persistent to all classes and __allow_access_to_unprotected_attributes__ = 1 to all classes. Still, I recieve the same error. Sorry, my fault, should be __allow_access_to_unprotected_subobjects__. If it does work then add the security declarations. HTH ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] accessing object from a list constructed in __init__.py
I'm doing some work with Five and have an aquisition related issue. first, zope is zope 2.9.3 The trouble is with accessing the attributes of objects stored in the number property in class2. accessing class1.number via class2.method1a is no problem, the problem seems to stem from constructing a list of objects ala class2.__init__, if I try to append obj.__of__ in attribute1a I recieve an error that I can not pickle a wrapped class. Is there some other trick with acquisition to be able to access the attributes of an object which is stored in a list of object constructed in the __init__ function of a class? --myClass.py import Acquisition class class1(Acquisition.Explicit): ''' a simple class with some attributes and methods''' attribute1 = 'attribute 1' attribute2 = 'attribute 2' _number = '' def __init__(self,number2): '''simple init for the class''' self._number=number2*3 def method1(self): '''return a string methd1''' mthd1 = 'methd1' return mthd1 def method2(self): ''' return a string methd2''' mthd2 = 'method2' return mthd2 def _get_number(self): return self._number number = property(fget=_get_number) class class2(object): ''' a second simple class with some attributes and methods''' attribute1a = [] attribute2a = [] def __init__(self): '''create a list of class1 objects in attribute1a''' i=1 while i 5: obj = class1(i) self.attribute1a.append(obj) i=i+1 def method1a(self): '''instantiate class1 as object and return object usable by zope''' obj = class1(5) return obj.__of__(self) def method2a(self): '''returns class1.method1()''' c1m1 = class1.method1() return c1m1 attribute1b = property(fget=method1a) attribute2b = property(fget=method2a) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] accessing object from a list constructed in __init__.py
Ok, thank you for that. I can create the list of objects from the __of__(self) attribute without the wrapper error now, but I still cannot access that attribute through zope. Here is my code again, along with my zope content. ---myClass.py--- import Acquisition class class1(Acquisition.Explicit): ''' a simple class with some attributes and methods''' attribute1 = 'attribute 1' attribute2 = 'attribute 2' _number = '' def __init__(self,number2): '''simple init for the class''' self._number=number2*3 def method1(self): '''return a string methd1''' mthd1 = 'methd1' return mthd1 def method2(self): ''' return a string methd2''' mthd2 = 'method2' return mthd2 def _get_number(self): return self._number number = property(fget=_get_number) class class2(Acquisition.Explicit): ''' a second simple class with some attributes and methods''' attribute1a = [] attribute2a = [] def __init__(self): '''create a list of class1 objects in attribute1a''' i=1 while i 5: obj = class1(i) self.attribute1a.append(obj.__of__(self)) i=i+1 def method1a(self): '''instantiate class1 as object and return object usable by zope''' obj = class1(5) return obj.__of__(self) def method2a(self): '''returns class1.method1()''' c1m1 = class1.method1() return c1m1 attribute1b = property(fget=method1a) attribute2b = property(fget=method2a) myZclass.py- from zope.interface import implements from zope.schema.fieldproperty import FieldProperty from OFS.SimpleItem import Item, SimpleItem from persistent import Persistent import myClass from interfaces import * import Acquisition class Zclass2(SimpleItem,myClass.class2,Acquisition.Explicit): '''a simple zope class that inherits from my class''' implements(Izclass2) def __init__(self,tmpID='',tmpTITLE=''): '''non empty docstring''' cm = myClass.class2.__init__(self) return cm.__of__(self) def myZclass(self): cm = myClass.class2() return cm.__of__(self) Alexis Roda wrote: En/na kevin7kal ha escrit: I'm doing some work with Five and have an aquisition related issue. first, zope is zope 2.9.3 The trouble is with accessing the attributes of objects stored in the number property in class2. class2 must inherit from ExtensionClass.Base (or some subclass, like Acquisition.Explicit) in order to be acquisition aware. HTH ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
