Re: [Zope-dev] Do I really understanding caching?
Andy McKay wrote: What problem are you trying to solve -- response time, memory usage, disk usage? All of the above :) Describe some of the symptoms back on the list and let's talk about it there. For instance, you can trade RAM for performance by adjusting knobs on the catalog. --Paul ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Do I really understanding caching?
Andy McKay wrote: We have been looking at caching in Zope as a way of tweaking performance. Heres an example of what I think happens: - Supposing I have a 1,000 object catalog. If one person changes an catalog aware object, that instance of the catalog will be pulled out of the ZODB and changed. It will then be written to the ZODB. That last copy will stay in the cache for as long as the "Cache Parameters" are set to allow. - If somebody changes another catalog aware object, that will repeat the above process. - However simply accessing the catalog (no changes) will pull the object from the cache. - What would be really nice is if the object only got written to the cache when it is no longer used, that way every time the catalog changed it didnt write to ZODB instead it changed the cached version. Of course that does bring up the recovering from disaster problem. So do I understand it correctly? Not exactly. Have you seen the ZODB paper? http://www.zope.org/Members/jim/Info/IPC8/ZODB3 You can think of the normal ZODB cache as being a tool that manages the objects in memory. Whenever an object has been accessed, it is in the cache. The cache, together with the database are responsible for moving objects, and their state in and out of memory. An object can be in serveral in memory states. Two that are of interest, from a caching perspective, are the "ghost" state and the up-to-date state. The up-to-date state is a state in which the object and it's state are loaded. In the ghost state, the object is in memory, but it has now state. The cache manager tries to release an object's state and remove the object from memory when it is not used. An object's state is released when it hasn't been accessed in a long time. It can be removed when no other objects (not counting the cache itself) refer to it. Note that the cache parameters are only guides for an algorithm that is somewhat adaptive and a bit more complicated than the parameters suggest. For example, the target cache size is a target, not a limit. The cache sizes are usually mich higher than the target. Another thing to be aware of is that large objects are generally designed to spread their state over many subobjects so that they can make effective use of the cache. A catalog may have hundreds of sub-objects, only a small fraction of which are kept in memory at a time. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Technical Director (888) 344-4332 Python Powered! Digital Creationshttp://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Do I really understanding caching?
We have been looking at caching in Zope as a way of tweaking performance. Heres an example of what I think happens: - Supposing I have a 1,000 object catalog. If one person changes an catalog aware object, that instance of the catalog will be pulled out of the ZODB and changed. It will then be written to the ZODB. That last copy will stay in the cache for as long as the "Cache Parameters" are set to allow. - If somebody changes another catalog aware object, that will repeat the above process. - However simply accessing the catalog (no changes) will pull the object from the cache. - What would be really nice is if the object only got written to the cache when it is no longer used, that way every time the catalog changed it didnt write to ZODB instead it changed the cached version. Of course that does bring up the recovering from disaster problem. So do I understand it correctly? -- Andy McKay, Developer. ActiveState. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Do I really understanding caching?
Thanks (I'm only talking about text indices here.) Note that the Catalog is actually a tree of database/cache objects. There are a series of buckets and sub-buckets that end in a word object. The buckets, sub-buckets, and words are all database objects and can be cached/updated separately. Put another way, every branch and leaf on the tree in a Catalog is a separately stored and cached database object. The 1,027 words in the document (and the branches on the tree to get to them) that was updated would obey the cache parameters. Hmm, that is a significant optimization, and one I was not aware of. What problem are you trying to solve -- response time, memory usage, disk usage? All of the above :) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Do I really understanding caching?
Statement of possible BS: I am speaking from how I understand the catalog to work, and possibly not how it actually *does* work. :-) Andy McKay wrote: We have been looking at caching in Zope as a way of tweaking performance. Heres an example of what I think happens: - Supposing I have a 1,000 object catalog. If one person changes an catalog aware object, that instance of the catalog will be pulled out of the ZODB and changed. It will then be written to the ZODB. Actually, a subobject of the catalog which is stored in a BTree will change, and the Catalog instance will change, but it will not write the entire contents of the catalog back out to the ZODB. The current BTree implementation has some problems, so other unrelated objects in it may be changed on the write. This will be solved by updates to the catalog which use a new "broadtree" BTree implementation. That last copy will stay in the cache for as long as the "Cache Parameters" are set to allow. The changed subobjects will, yes. Unless they're manually ghosted for some other reason (rarely happens, if ever). - If somebody changes another catalog aware object, that will repeat the above process. Yes. - However simply accessing the catalog (no changes) will pull the object from the cache. Yes. - What would be really nice is if the object only got written to the cache when it is no longer used, that way every time the catalog changed it didnt write to ZODB instead it changed the cached version. Of course that does bring up the recovering from disaster problem. This would break transactional integrity. It's important that the catalog is in a self-consistent state as well as be consistent with other objects in the ZODB, and transactions help to maintain this. Is it essential that all your cataloged objects be catalog-aware? -- Chris McDonough Digital Creations, Publishers of Zope http://www.zope.org ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
