[Zope-dev] Re: test.py
On Wed, 2004-03-10 at 13:39, Evan Simpson wrote: Jeremy Hylton wrote: Make sure App.Product is imported first, so that test.py can work. This change does not affect the old utilities/testrunner.py. This checkin message caused me to notice 'test.py' for the first time. Can you point me to any discussion/docs on it? When did/will it replace testrunner? Shouldn't the python2.2 in the #! line be python2.3? There is minimal documentation in the test.py docstring. All the command line arguments are described. It's basic strategy for finding tests isn't documented, but I don't think it's too different from testrunner.py. test.py obsoleted testrunner.py as of Zope 2.7. test.py was being used by ZODB3 and Zope3 for a while before it was added to Zope 2.7. I'm not sure which #! line you mean, but I think you're right to say it should be python 2.3. All the software on the head requires python 2.3. Sorry for the flood of questions, but this and the recent doctest-style tests being checked in have made me wonder if I've been missing a significant shift in testing tech. test.py has all sorts of bells and whistle for running tests. It's a lot more flexible than testrunner.py. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Question about _v_ attributes and database connections
On Fri, 2004-03-05 at 16:57, Dieter Maurer wrote: 3) is it possible to disable caching of some object? No, but you can (in most cases) flush it again by calling its _p_deactivate method. You can always do it by assigning None to _p_changed, but this is highly unsafe. You should not do it (unless you really know what you are doing). Another was of saying this is that the cache performs an important role in maintaining object identity. It's not possible to use a persistent object without using a cached version of it. If you think you need to disable caching, you should probably say something more about what you are trying to do. 4) when an _v_ attribute is removed, what code of the object/attribute itself is called? The object's __del__ method (if defined). This means the object that was bound to the _v_ attribute not the object that had the _v_attribute. The setattr hook in cPersistence.c is called when an attribute is removed (del obj.attr). An __del__() method would only be called if the last reference to the object was also deleted. And __del__ is to be avoided whenever possible. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.7 breaks auto refresh
On Wed, 2004-03-03 at 08:42, Santi Camps wrote:
Hi again,
I've this traceback using Zope 2.7.0 final:
Module ZPublisher.Publish, line 163, in publish_module_standard
* Module ZPublisher.Publish, line 127, in publish
* Module Zope.App.startup, line 203, in zpublisher_exception_hook
* Module ZPublisher.Publish, line 104, in publish
* Module Zope.App.startup, line 221, in commit
* Module ZODB.Transaction, line 233, in commit
* Module ZODB.Transaction, line 348, in _commit_objects
* Module ZODB.Connection, line 435, in commit
__traceback_info__: (('Products.Transience.TransientObject',
'TransientObject'), '\x00\x00\x00\x00\x00\x00\x008', '')
I don't understand this traceback entirely. What is the exception that
is being raised? Also, the last line in the traceback doesn't match a
line on my copy of ZODB/Connection.py. (I've got revision 1.98.4.5.)
I've tryied to understand the problem. It seems that now Trasicience
uses a different ZODB connection (using DBTab), and I think this is the
cause of the problem.
In Zope/App/startup.py, there is the following lines:
# Initialize the app object
application = app()
OFS.Application.initialize(application)
if Globals.DevelopmentMode:
# Set up auto-refresh.
from App.RefreshFuncs import setupAutoRefresh
setupAutoRefresh(application._p_jar)
application._p_jar.close()
This handles the autoRefresh machinery to the main Connection, but I've
not found anywhere the same has been done with temporary Connection.
But in the ZODB/Connection.py there is a global_code_timestamp used to
compare all Connections _code_timestamp attribute.
Then, when the Connection tries to commit, as the _code_timestamp of
Trasicience connection hasn't been updated, the problem appears. Almost
is what I think.
I'm not very familiar with the App.RefreshFuncs code, but I think I can
offer some generic comments. setupAutoRefresh() calls
ZODB.Connection.updateCodeTimestamp() which will cause every Connection
to reset its cache the next time it is opened. It wouldn't be safe to
reset the cache at other times, because application code could have a
reference to existing objects.
I don't know how all this affects the Transience connection. When is it
opened and closed?
Jeremy
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Wed, 2004-03-03 at 04:55, Chris McDonough wrote: (boldly crossposting this to zodb-dev, please respond on one list or the other but not both) That error *appears* to be caused by reaching a state that is impossible to reach. The code in question is: for key in list(self._data.keys(None, max_ts)): assert(key = max_ts) STRICT and _assert(self._data.has_key(key)) for v in self._data[key].values(): to_notify.append(v) del self._data[key] I don't have much context for this question. It's definitely the case that a corrupt BTree there are keys you can reach using keys(), which follows the bucket next pointers, that can't reach using a lookup, which follows child pointers down through the interior nodes. If you could call the check functions on the BTrees in question. That's object._check() to check C internals and BTrees.check.check() to check value based consistency. So how is the BTree is question used? If the test is failing here, it seems most likely that the BTree was corrupted by a write somewhere else. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving big files thru zope?
On Mon, 2004-03-01 at 13:41, Paul Winkler wrote: However, ZEO is a whole other story. The time to load a 40 MB file from ZEO and serve it, is about another 10x slower than plain Zope without ZEO. This is painfully bad and readily apparent to users. But if your ZEO cache is large enough to hold all the data, subsequent download times are as fast as plain Zope. More investigation needed, I'm not sure about the accuracy or relevance of these results. But almost certainly, if using large files with Zope with ZEO, you will want to configure a much larger ZEO cache size than the default (20 MB). It really would be good to get to the bottom of the bad ZEO performance for loading an object the first time. I won't have time to do it until after the ZODB 3.3 release, but I think it's a worthwhile effort to understand what ZEO is doing. You're right, too, that a large ZEO cache makes a big difference. If you've got the disk space, setting it several hundred MB shouldn't be a problem. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] some breakage on the head
I was doing some branch merging and apparently got mislead by stray .pyc or .so files being left behind after .py and .c files were removed. So there's some broken code on the head; I'll fix ASAP. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] some breakage on the head
On Thu, 2004-02-19 at 13:48, Jeremy Hylton wrote: I was doing some branch merging and apparently got mislead by stray .pyc or .so files being left behind after .py and .c files were removed. So there's some broken code on the head; I'll fix ASAP. I think it's all fixed now, though the cause I suspected was wrong. Holler if you see any lingering problems. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Core Dump (Zope 2.7, Python 2.3.3, FreeBSD 4.5)
Dieter, Have you seen this behavior with a debug build of Python? Under a debug build, you're more likely to get complaints about dodgy C code and the garbage collection will complain about refcount problems it can detect. Jeffrey, Any luck with you're core dump? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Core Dump (Zope 2.7, Python 2.3.3, FreeBSD 4.5)
On Tue, 2004-02-17 at 11:45, Jeffrey P Shell wrote: What should I do next? Should I familiarize myself with gdb and inspect the core? What are some things I could look for if that's the next step? The code dump is an important clue, so it's definitely worth looking at. It's also pretty simple. Run gdb [python] [core] where python is the binary you use with Zope and core is the core file. A few commands will give the most benefit. bt will produce a backtrace, much like a Python traceback. It will tell us where the program actually failed and what was on the call stack. If you only do this, we'll probably be in good shape. If you put the following code in your ~/.gdbinit file, then you can also inspect the Python stack frames: # If you are in an eval_frame() function, calling pyframe with no # arguments will print the filename, function name, and line number. # It assumes that f is the name of the current frame. define pyframe x/s ((PyStringObject*)f-f_code-co_filename)-ob_sval x/s ((PyStringObject*)f-f_code-co_name)-ob_sval p f-f_lineno end That helps match the C stack and the Python stack and pinpoints what Python code was executing. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] TypeErrors during 'make test'
On Fri, 2004-02-13 at 14:38, Evan Simpson wrote: I'm getting several TypeErrors when I run 'make test' on the HEAD. Each of them involves a call to cAccessControl's 'validate' with 'roles' set to a PermissionRole instance, complaining that it's not iterable. Any ideas? These tests have been failing for weeks. I'm not sure what the actual cause of the failure is. The symption is that validate() is being passed a single PermissionRole object as its roles argument. validate() expects roles to be a sequence and fails with a TypeError iterable argument expected when it sees the PermissionRole. What isn't clear to me is if the argument should be a Python sequence that contains a PermissionRole or if it should be an imPermissionsRole, which is a sequence. I don't understand the code well enough to know which of these is more likely. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] change to zLOG.LOG() behavior
I noticed that Zope 2.7 and Zope 2.8 changed the default zLOG implementation to use the logging package. There's one subtle consequence of that change that I wonder if people are aware of. The zLOG.LOG() call takes an optional error argument that contains a 3-tuple of exception information as returned by sys.exc_info(). In the old zLOG, this exception was formatted and logged. In the new zLOG, the error argument is treated as a flag; if it is not None, the *current* exception is formatted and logged. Should I change my code to pass True for error instead of sys.exc_info()? There is at least one place in ZODB that is broken as a result of that change. The old API allowed code to capture an exception with sys.exc_info() and log it later. The common pattern seemed to be to capture the error, try to recover, and only log the original exception of recovery failed. I don't think there's a good solution to the problem, except to change all the code that tries to be clever about capturing exceptions. This affects some ZODB code that was a culprit in recent bug reports that locked up Zope. As a result my fix -- which was supposed to log a helpful traceback rather than prevent Zope from locking up -- probably didn't work. Should we give up on fixing this bug for Zope 2.7? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: backward compatibility of PythonScript bindings for 2.6.4 / 2.7.0
On Wed, 2004-01-21 at 10:42, Brian Lloyd wrote: What I don't like is that it is somewhat magical, and now the error you would get (probably 'None has no attribute xxx') if the user doesn't have access to the container doesn't tell you the real problem. What if you used a special object that would produce a useful error message if the user tries to access the container. Assuming an access involves an attribute access: class UnauthorizedContext: def __getattr__(self, attr): raise Unauthorized(user does not have access to context) I'm sure the details aren't right, but I think the idea is clear enough. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] CVS Head: Error Value: iterable argument required when adding objects
On Sat, 2004-01-17 at 17:22, Jamie Heilman wrote: Jeremy Hylton wrote: I committed a patch with the umask option a few days ago. I thought it only affected zdaemon and the tests all looked clean afterwards. I'm not sure how zopectl.py ends up being affected or why there aren't any tests of it. The various scripts to start and stop programs are usually hard to test, but they're usually the source of a lot of bugs, too. Speaking as a sysadmin I'd like to suggest that the zope daemons make no efforts to frob thier assigned umasks in any way. Thats generally something the sysadmin will take care of during the startup scripts, and to have daemons change it after the fact because they think they know better causes no end of frustration. You should take it up with the sysadmin on the zodb-dev list who wanted this feature :-). Daemons don't set the umask by themselves; they only do it when a sysadmin configures zdaemon to run with the --umask argument. All the advice I can find about writing daemon code suggests that setting the umask is desirable. The zdaemon code that we've been using is based on these guidelines http://www.hawklord.uklinux.net/system/daemons/d3.htm, which recommend resetting the umask in a daemon. They are basically the same as the coding rules for daemons in Steve's Advanced Programming in the Unix Environment. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] CVS Head: Error Value: iterable argument required when adding objects
On Sat, 2004-01-17 at 18:30, Jamie Heilman wrote: Its desirable in some circumstances, but not all. Part of the problem is people tend to blindly follow the traditional approach to daemon design without bothering to actually do any critical thinking. I expect you don't intend to sound rude, but this gives the impression you think I've failed to do some necessary critical thinking. Even if I you think that, it's hardly diplomatic to point it out. There are several very reasonable arguments for deviation from the historical approach. What are they? historical approach. Perhaps the most relevant argument is the same old one about the Unix design philosophy; many small programs working together is more a flexible and ultimately useful approach than a monolithic one-program-does-it-all design. I don't follow how this advice relates to the current discussion. We're talking about whether zdrun.py should have a --umask option. zdrun is a small program. It just allows some other program to run as a daemon. Anyway, if you want to question authority, consider reading: http://homepages.tesco.net/~J.deBoynePollard/FGA/unix-daemon-design-mistakes-to-avoid.html I don't see how this questions authority. It sounds entirely compatible with the design of zdaemon.(The TCP/IP stuff doesn't apply to zdaemon, and Zope works differently, but that's typical for app servers.) Are you familiar with zdaemon? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: CVS: Releases/Zope/lib/python/ZEO - simul.py:1.12.8.2.18.18
On Fri, 2004-01-16 at 13:58, Tres Seaver wrote: This is a result of some funny repository fiddling, which somehow got both 'ZEO' and 'BDBStorage' onto the 'Zope-2_6-branch'. I would like to remove the branch tag altogether, as neither module is part of the 2.6 release. The branch tag is used for making releases for both ZODB and Zope releases. We decided to use the same branch so that we didn't have to port bug fixes to so many different branches. One downside of that merge, which we didn't realized until after it was done, is that you now get a ZEO directory in a Zope checkout on the 2.6 branch. I think there was email about the discovery at the time, and I noticed that Brian did not include ZEO in the 2.6.3 releases; so I assume he's got some mechanism to suppress it from the actual release. I'd write this down somewhere more permanent if anyone can suggest a good place. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Zope] ANNOUNCE: Zope 2.6.3 Release and Security Update
On Thu, 2004-01-08 at 20:31, Dennis Allison wrote: Does this mean that Zope 2.6.3 is compatible with Python 2.3.3? I would be nice to retire 2.1.3. I'm not aware of any Zope Corp internal projects still using Python 2.1.3. I'm not aware of any serious incompatibilities. I suppose the only risk would be that fixing problems in Zope 2.6 specific to a Python version would have low priority because Zope 2.7 is on the horizon. Jeremy (not speaking for ZC) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] FileStorage instance has no attribute '_serial'
On Sun, 2004-01-04 at 23:28, Jeff Kowalczyk wrote: I'm not sure whether the collector issue 1164 I posted is simply a missing configuration step on my part. At any rate, I'm still unable to start the current checkout of Zope HEAD. The only edit I make to /var/zope/etc/zope.conf is 'effective-user zuser', and the problem below also occurs if I then bin/runzope as root. I haven't seen the collector issue, but the problem is that FileStorage.py was removed and replaced with a FileStorage package (directory). I would expect a cvs up -d to remove FileStorage.py. It may be that you've got a stale .pyc file there that needs to be removed by hand. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] FileStorage instance has no attribute '_serial'
Actually, this was a bug in CVS. I removed the old FileStorage.py on a branch, but didn't remove it on the trunk when I merged the branch. It should be fixed now. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] It's me again :(
Does the last report mean your problem is fixed? Or do you still need help tracking something down? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Head (2.8) breaks refresh
On Mon, 2003-12-15 at 07:34, Dieter Maurer wrote: I don't think it makes a lot of sense to put an __del__ method on a Persistent object. I just read in cPersistence.c:ghostify: /* We remove the reference to the just ghosted object that the ring * holds. Note that the dictionary of oids-objects has an uncounted * reference, so if the ring's reference was the only one, this frees * the ghost object. Note further that the object's dealloc knows to * inform the dictionary that it is going away. */ This means: *all* persistent objects have a special dealloc function. Hope, this dealloc function plays well with the cyclic garbage collector. All Python objects have a dealloc function -- whatever is in the tp_dealloc slot. The tp_dealloc function at the C level is completely different from an __del__ method at the Python level. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Head (2.8) breaks refresh
On Sun, 2003-12-14 at 16:53, Chris McDonough wrote: We still use ZODB 3.1 and at least there, the cache verification protocol seems quite stupid. We will soon switch to ZODB 3.2 and when cache validation still needs minutes, I will need to look into this... Apparently, the ZEO in 2.7 and the HEAD is better about doing as little work as possible for cache verification at startup than older versions were. Specifically, the server saves the list of invalidated objects for the last N transactions and the client keeps track of the last transaction id it received an invalidation for. One restart, if the last txn the client saw was within N of the current transaction, the server just sends the list of invalidated objects. For a large cache, this is much cheaper than having the client send the oid and serialno of every object in the cache to the server. N is configurable. I think the default value is 100. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] To the brave folks trying Zope HEAD...
I think there's a fix for this in CVS. Can you try it again? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] To the brave folks trying Zope HEAD...
On Thu, 2003-12-11 at 10:24, Sidnei da Silva wrote: On Thu, Dec 11, 2003 at 10:07:51AM -0500, Jeremy Hylton wrote: | I think there's a fix for this in CVS. Can you try it again? Just tried. Seems to have fixed it. Now, for the next step *wink*. If I try starting Zope with an old existing database I get the traceback below. Correct me if I'm wrong, but I think the error with 'UserDict' has nothing to do with the 'Could not import class 'BTree'' or am I wrong? I think the attribute error on data is just a symptom of the BTree failure. The object has no data attribute, because ZODB failed to unpickle the object that would be assigned to data. I don't know what happened with backwards compatibility for the old BTree classes. When Jim and I discussed it a while back, I thought we concluded that we would provide a script that would convert all the old BTree instances to new BTrees instance. But I don't know if that decision stuck or if the script was ever written. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] To the brave folks trying Zope HEAD...
On Thu, 2003-12-11 at 15:15, Sidnei da Silva wrote: Do'h, that's a bit difficult since Jim removed them for good :( Hmmm... I suppose you could write the script in Zope 2.7 and tell people they have to use that version of Zope to convert the database before upgrading. Now, back to the automagically-converting idea, would it be possible to provide an alias for the old BTree package that would feed the old pickle into a new BTrees constructor on unpickling *wink*? I just remembered why the problem is harder. Any object that references as BTree has the name of the BTree class embedded in its pickle. The conversion script needs to frob each of those database records to refer to the new class. I'm not sure how to do that. Unpickle and repickle each one? A simple binary rewrite of the pickle to past in the new name. Just not sure. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] To the brave folks trying Zope HEAD...
On Wed, 2003-12-10 at 14:01, Sidnei da Silva wrote: | Probably a good next step is to set a breakpoint in | ZODB/coptimizations.c where this exception is raised and find out what | object is causing the problem, which jar it's already in, and why it's | getting added to a different jar. Or you could add some print | statements. I figured out how to set a breakpoint in gdb, but I don't know how to get at the object and inspect it. Can I get a quick hint? I know there has to be some pointer cast probably, just need to know which one. :) I think you're looking for the helpers defined in Misc/gdbinit in the Python distribution. If you add them to your .gdbinit, you'll be golden. I've included that file's contents below so you don't have to go hunting. # -*- ksh -*- # # If you use the GNU debugger gdb to debug the Python C runtime, you # might find some of the following commands useful. Copy this to your # ~/.gdbinit file and it'll get loaded into gdb automatically when you # start it up. Then, at the gdb prompt you can do things like: # #(gdb) pyo apyobjectptr #module 'foobar' (built-in) #refcounts: 1 #address: 84a7a2c #$1 = void #(gdb) # Prints a representation of the object to stderr, along with the # number of reference counts it current has and the hex address the # object is allocated at. The argument must be a PyObject* define pyo print _PyObject_Dump($arg0) end # Prints a representation of the object to stderr, along with the # number of reference counts it current has and the hex address the # object is allocated at. The argument must be a PyGC_Head* define pyg print _PyGC_Dump($arg0) end # If you are in an eval_frame() function, calling pyframe with no # arguments will print the filename, function name, and line number. # It assumes that f is the name of the current frame. define pyframe x/s ((PyStringObject*)f-f_code-co_filename)-ob_sval x/s ((PyStringObject*)f-f_code-co_name)-ob_sval p f-f_lineno end ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] To the brave folks trying Zope HEAD...
On Wed, 2003-12-10 at 14:18, Sidnei da Silva wrote: 'jar' gives: object : attribute '_p_jar' of 'persistent.Persistent' objects type: getset_descriptor refcount: 2 address : 0x40621e6c Does that help? Yes. The code is getting an unexpected raw descriptor. Someone should be calling __get__() on this descriptor to get the value. I'll have to look into it to see how it's going wrong. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] To the brave folks trying Zope HEAD...
Actually, there are two equally valuable clues from the debugger session: object : class 'Products.Formulator.StandardFields.LabelField' type: ExtensionClass.ExtensionClass refcount: 7 address : 0x8503bec If I understand correctly: LabelField is a class that inherits from Persistence and some other object has a reference to it that isn't via __class__. persistent.Persistent._p_jar attribute '_p_jar' of 'persistent.Persistent' objects If you ask a class for an attribute, it seems you can get a descriptor back. I vaguely recall Guido saying that this was the intended behavior last time I asked about it. 'jar' gives: object : attribute '_p_jar' of 'persistent.Persistent' objects type: getset_descriptor refcount: 2 address : 0x40621e6c Thus, asking for the class's _p_jar gives the descriptor back. In ZODB4 I hacked around this by requiring that the value of _p_jar be None or a string. Would there be any objection to having the same requirement in ZODB3? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] To the brave folks trying Zope HEAD...
On Wed, 2003-12-10 at 15:36, Sidnei da Silva wrote: | 'jar' gives: | object : attribute '_p_jar' of 'persistent.Persistent' objects | type: getset_descriptor | refcount: 2 | address : 0x40621e6c | | Thus, asking for the class's _p_jar gives the descriptor back. In ZODB4 | I hacked around this by requiring that the value of _p_jar be None or a | string. Would there be any objection to having the same requirement in | ZODB3? I have no clue about the implications of this, but it seems like a reasonable constraint to me. Actually, that's just a brain cramp on my part. We can check that an oid is a string or None, but not a jar. I'll have to think about an efficient check. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Head (2.8) breaks refresh
On Mon, 2003-12-08 at 13:17, Dieter Maurer wrote: Jeremy Hylton wrote at 2003-12-7 23:01 -0500: On Sun, 2003-12-07 at 08:15, Dieter Maurer wrote: ATT: replacing the cache without clearing it can lead to huge memory leaks (everything in the old cache is leaked!). Without commenting on the rest of the bug report, I should mention that caches and persistent objects all participate in cyclic GC. They will not leak the way pre-2.8 caches did. Has the do not free when a cycle contains an object with destructor restriction been removed from the cyclic GC? It is not unlikely that the cache references an object with a __del__. This may keep the cache and all its content. I don't think it makes a lot of sense to put an __del__ method on a Persistent object. When would you expect it to be called? Every time the object is turned into a ghost? When the object is removed by pack? I can imagine some people would try to migrate non-persistent code to ZODB and have to deal with __del__ then. If I were doing that, I'd make sure to check gc.garbage when I tested and fix any code that caused leaks. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Zope Head (2.8) breaks refresh
On Mon, 2003-12-08 at 15:36, Dieter Maurer wrote: Dieter Maurer wrote at 2003-12-7 14:15 +0100: Playing with Zope Head (as of 2003-12-04) revealed problems with refresh. Turns out to be a bug in ZODB.Connection.Connection._setDB: ConnectionObjectReader used the old cache while refresh cause a new cache to be installed later in _resetCache. Patch attached. The patch doesn't make sense: --- ZODB/Connection.py~ 2003-11-28 17:44:49.0 +0100 +++ ZODB/Connection.py 2003-12-08 21:26:49.0 +0100 @@ -158,8 +158,6 @@ -self._reader = ConnectionObjectReader(self, self._cache, - self._db._classFactory) @@ -168,6 +166,8 @@ +self._reader = ConnectionObjectReader(self, self._cache, + self._db._classFactory) Is there any difference between these two execept for whitespace? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope Head (2.8) breaks refresh
On Mon, 2003-12-08 at 15:56, Yuppie wrote: Jeremy Hylton wrote: @@ -158,8 +158,6 @@ -self._reader = ConnectionObjectReader(self, self._cache, - self._db._classFactory) @@ -168,6 +166,8 @@ +self._reader = ConnectionObjectReader(self, self._cache, + self._db._classFactory) Is there any difference between these two execept for whitespace? Line numbers? HTH, Yuppie Yes. That helps :-). Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Head (2.8) breaks refresh
On Sun, 2003-12-07 at 08:15, Dieter Maurer wrote: ATT: replacing the cache without clearing it can lead to huge memory leaks (everything in the old cache is leaked!). Without commenting on the rest of the bug report, I should mention that caches and persistent objects all participate in cyclic GC. They will not leak the way pre-2.8 caches did. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: New-style ExtensionClasses (Zope 2.8) -- MRO issue
On Mon, 2003-11-24 at 13:22, Sidnei da Silva wrote: I think it was decided for the non-mro approach, though no one stated it clearly. If it was, they sure didn't. A summary of the opinions that lead to the decision would be nice. Even if it's decided, it wouldn't hurt to fix the inheritance problems in CMF. One of the key properties of C3 is local order is honored. If a class A has two base classes B and C, then a method defined in B and C (perhaps via their base classes) will be found in B first. This matches the way I think about inheritance, but not the way the old algorithm works. You may be able to change the inheritance hierarchy so that C3 and classic MROs both have the same results. Then programmers will be able to understand the code :-). Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] New-style ExtensionClasses (Zope 2.8) -- MRO issue
I am worried enough about breaking products that I'm inclined to go with option 3. Does anybody think we ought to use the new algorithm (option 2)? I think we should use the new algorithm. However, I don't have any products that I would be responsible for updating and maintaining. I'd be happy to help people work out problems, though. If we were starting from scratch, the C3 algorithm would be the obvious choice. The Dylan linearization paper makes a pretty good argument for it: The first reason for the change is that a monotonic linearization seems a better match for users' intuitions about how inheritance and linearizations work. With a monotonic linearization, it is easier to understand the behavior of classes when multiple inheritance is used, largely because behavior of instances of a class can be explained in terms of the direct superclasses. The depth-first resolution order can produce un-intuitive results because it doesn't honor local precedence order. It also leads to sloppy code where the same base class is mixed into the inheritance hierarchy at multiple levels. If we stick with a non-standard MRO, then Zope programmers will have to learn special rules for dealing with multiple inheritance with Zope code that don't apply to Python programming in general. I don't see much value in creating such a special case here. We'd also need to implement our own MRO, maintain it, make sure it's correct, etc. That's a long-term maintenance burden, as opposed to a short-term problem of updating existing code. We can write tools to help people update their code if it ends up being a real problem, and they get clearer code as an added bonus. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] New-style ExtensionClasses (Zope 2.8) -- MRO issue
On Fri, 2003-10-31 at 13:52, Sidnei da Silva wrote: | I personally don't like the new algorithm, but I don't really care | in the long run. One should avoid inheritence complex enough to show | a difference. I hearthly agree here. Home domestic of you wink. I do like the new algorithm, but we can all agree to avoid complex uses of inheritance. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope credits
Dean Goodmanson pointed out that the Zope credits file is a little out of date: http://www.pycs.net/sqr/categories/slicesOfPy/2003/10/30.html#a184 Wouldn't it be nice to get it in better shape for the 2.7 release? It seems like a task that is easy to do in parallel. If credits are too hard, how about an ACKS file like Python's? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: current best of breed ZEO for Zope 2.6.2 in production?
On Thu, 2003-10-02 at 13:34, Leonardo Rochael Almeida wrote: I thought that was easy to miss, so I changed the heading from Zope Enterprise Objects to Zope Enterprise Objects (OBSOLETE) And I still missed it! Amazing! it must've been the pre-lunch hunger or something... Maybe you should use a blink tag and a red font tag inside a one cell table with a thick border (/me runs from the HTML-standards mob). Maybe we should just delete it. On the other hand, Jim asked me today how someone would navigate from the zope.org home page to the current ZODB release. I don't have any idea. Anyone else know? I just use Google. It could probably use more text explaining what to do instead. I'd do that if I thought I knew wink. Yes, that's exactly what I'm looking for: what is the current blessed procedure for using ZEO with Zope? You can copy ZEO into lib/python or install ZODB into lib/python. The client will be using the binary Zope 2.6.2 package, BTW. I've (finally) read the ZODB page and it says Zope 2.6.2 doen't need a ZODB surgery to use ZEO. This is good since I don't want to scare the client with weird surgeries for a Production installation :-) (I'm mentoring them on the installation, not installing it for them). Since we'll be using python-2.1.3, ZODB 3.2 isn't an option. Does ZODB 3.1.4 (the last ZODB release according to the ZODB3.1 page) fix the one client changes an object but the others don't see the change bug reported here recently? It does. There are a few memory leaks and a potential deadlock bug that are only fixed in ZODB 3.2, but the majority of the critical bugs were fixed by 3.1.4. If yes, should I just copy over the ZEO package or do you guys still recomend that I replace all of Zope262 ZODB with the standalone one? Either one is fine with me. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZODB 3.2 feedback
Has anyone had a chance to test ZODB 3.2b3? We're hoping to do a release candidate tomorrow, and it would be good to know if anyone has tried to the beta release. If you've got a current CVS checkout of Zope 2.7, that's got the same code. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Zope-Dev Digest, Vol 2, Issue 20
Zope 2.6.2, Zeo 2.0.1b1 and python 2.2.3 I'm trying to get this combo working (I know its not officially supported). It works fine except for when a second client updates zeo the changes aren't seen from the first client (until its restarted). I ran the zeo tests with 2.2.3 and they all pass. Zope 2.6.2 seems to work fine even with zeo when only one client is doing updates. Where would I look to try and track down the problem (somewhere in the caching code?) Is this even worth pursuing? I have another library that I would like to use but it requires 2.2.3. You should download the latest ZODB 3.1 release and use the ZEO you find there. There have been many critical bugs fixed since 2.0.1b1; the current release is 2.0.4. See http://www.zope.org/Products/ZODB3.1. I'm not aware of any issues using ZEO with Python 2.2.3, but I can't comment on whether there are Zope issues. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Packing (still) hosed in 2.6.2?
I don't recall that any of the pack bugs that we fixed caused this sort of error. My first guess would be that the file is a bit damaged, perhaps in a way that the old pack did not check. Can you run fsdump.py and see if that output sheds any light. It will give you a detailed text summary of all the transactions and data records. You could use the dump to figure out if, for example, the two numbers in the error message are indeed transaction header locations. Jeremy -Original Message- From: Anthony Baxter [mailto:[EMAIL PROTECTED] Sent: Tuesday, August 12, 2003 4:18 AM To: [EMAIL PROTECTED] Subject: [Zope-dev] Packing (still) hosed in 2.6.2? I remember that 2.6 had issues with packing, but I was under the impression that they'd been fixed... However, attempting to pack a Data.fs in 2.6-current-cvs fails for me: File /export/01/zope/dev_csr_code/lib/python/ZODB/DB.py, line 526, in pack try: self._storage.pack(t,referencesf) File /export/01/zope/dev_csr_code/lib/python/ZODB/FileStorage.py, line 1503, in pack opos = p.pack() File /export/01/zope/dev_csr_code/lib/python/ZODB/fspack.py, line 680, in pack self.gc.findReachable() File /export/01/zope/dev_csr_code/lib/python/ZODB/fspack.py, line 464, in findReachable self.buildPackIndex() File /export/01/zope/dev_csr_code/lib/python/ZODB/fspack.py, line 486, in buildPackIndex self.checkData(th, tpos, dh, pos) File /export/01/zope/dev_csr_code/lib/python/ZODB/fspack.py, line 189, in checkData : %d != %d, dh.tloc, tpos) File /export/01/zope/dev_csr_code/lib/python/ZODB/fspack.py, line 170, in fail raise CorruptedError(s) ZODB.fspack.CorruptedError: /app/zope/dev_csr_server/var/Data.fs.packtest:9200838:data record does not point to transaction header: 17391715 != 9200760 This is reproducible. The particular Zope that uses this file was shut down cleanly before attempting to pack the DB. The code that fails to pack is: import ZODB from ZODB import FileStorage filename='/app/zope/dev_csr_server/var/Data.fs.packtest' store = FileStorage.FileStorage(filename) db = ZODB.DB(store) db.pack() # boom. Help? I'm using the ZODB that's in the current 2.6 branch of CVS - according to ZODB/__init__.py it's version 3.1.2. Is a newer version of ZODB going to help here? Anthony ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZODB 3.2b2 release
We have released ZODB 3.2b2 available from the usual place: http://www.zope.org/Products/ZODB3.2 This release includes several critical bug fixes for ZEO. The bugs are more likely to affect sites that see a lot of read conflicts (resolved or unresolved). The bugs can cause data loss or corruption. These are the same bug fixes that went into ZODB 3.1.2b2 last week. ZODB 3.2 has a number of new features and improvements over ZODB 3.1: - improve performance and stability of ZEO - new ZEO authentication protocol - new configuration language, ZConfig, for databases, storages, and ZEO servers - many bug fixes I have included the release notes for 3.2b2 below. Jeremy What's new in ZODB3 3.2 beta 2 == Release date: 16-Jun-2003 Fixed critical race conditions in ZEO's cache consistency code that could cause invalidations to be lost or stale data to be written to the cache. These bugs can lead to data loss or data corruption. These bugs are relatively unlikely to be provoked in sites with few conflicts, but the possibility of failure existed any time an object was loaded and stored concurrently. Fixed a bug in conflict resolution that failed to ghostify an object if it was involved in a conflict. (This code may be redundant, but it has been fixed regardless.) The ZEO server was fixed so that it does not perform any I/O until all of a transactions' invalidations are queued. If it performs I/O in the middle of sending invalidations, it would be possible to overlap a load from a client with the invalidation being sent to it. The ZEO cache now handles invalidations atomically. This is the same sort of bug that is described in the 3.1.2b1 section below, but it affects the ZEO cache. Fixed several serious bugs in fsrecover that caused it to fail catastrophically in certain cases because it thought it had found a checkpoint (status c) record when it was in the middle of the file. Two new features snuck into this beta release. The ZODB.transact module provides a helper function that converts a regular function or method into a transactional one. The ZEO client cache now supports Adaptable Persistence (APE). The cache used to expect that all OIDs were eight bytes long. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: deadlock patch ?
The change was too extensive to call it a patch. It was almost a rewrite. If you want to dig through the repository, look for all changes made to ZODB and ZEO between about October 31, 2002 and January 3, 2003. I imagine it would be less work to upgrade to a more recent version of Zope. Shane It was a fairly involved change. The chief changes were in Transaction.py, but it also affected all the storages. If it isn't possible to upgrade to a modern Zope, which has many benefits, it might be possible to integrate a modern ZODB with an old Zope. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZODB 3.1.2b2 released
We have released ZODB 3.1.2b2 available from the usual place: http://www.zope.org/Products/ZODB3.1 This release includes several critical bug fixes for ZEO. The bugs are more likely to affect sites that see a lot of read conflicts (resolved or unresolved). The bugs can cause data loss or corruption. I would strongly advise ZEO users to consider upgrading. This release has seen more internal stress testing that normal and should be fairly stable for production use. I hope to get a final release out as soon as possible. The release notes (below) describe the fixes in more detail. Jeremy What's new in ZODB3 3.1.2 beta 2? = Fixed critical race conditions in ZEO's cache consistency code that could cause invalidations to be lost or stale data to be written to the cache. These bugs can lead to data loss or data corruption. These bugs are relatively unlikely to be provoked in sites with few conflicts, but the possibility of failure existed any time an object was loaded and stored concurrently. Fixed a bug in conflict resolution that failed to ghostify an object if it was involved in a conflict. (This code may be redundant, but it has been fixed regardless.) The ZEO server was fixed so that it does not perform any I/O until all of a transactions' invalidations are queued. If it performs I/O in the middle of sending invalidations, it would be possible to overlap a load from a client with the invalidation being sent to it. The ZEO cache now handles invalidations atomically. This is the same sort of bug that is described in the 3.1.2b1 section below, but it affects the ZEO cache. Fixed several serious bugs in fsrecover that caused it to fail catastrophically in certain cases because it thought it had found a checkpoint (status c) record when it was in the middle of the file. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZEO install/runtime issues
[Please followup to zodb-dev.] Richard, You made some changes to the mkzeoinst.py script in April. I was busy then, and I've just had a chance to look at the changes now. I'd like to discuss some of the changes, and I'm including a wider discussion list to make sure we include anyone else who is interested. A number of the changes are Zope specific. (For example, you can't even run mkzeoinst.py without having a directory named Zope hanging off of sys.path.) ZEO and ZODB are intended for use separately from the rest of Zope, so we need to find a way to factor this out into a generic configuration and a Zope-specific configuration. One other requirement for ZEO is that it work with Zope 2.6. I expect there will be a ZODB 3.2 release long before there is a Zope 2.7 release. So we can't depend on any Zope 2.7 features in mkzeoinst.py. The other question I have is about the organization of software into a Zope home and an instance home. I'm not sure what the history of this arrangement is, but I recommend that people do not configure their ZEO servers to share software with their Zope app servers. It can cause fairly severe problems! The problem with sharing software is that the ZEO server can load arbitrary modules when it attempts to perform conflict resolution. If there is a conflict for an instance of class A.B.C, then ZEO will load A.B.C and see if it has an _p_resolveConflict() method. If the modules A or B have any side-effects at import time, then those side-effects will occur in the ZEO server. I've seen this method lookup cause all of CMF to get imported and try to initialize itself. This ended up brining down the ZEO server. A safer way to run a ZEO server is to have an isolated copy of the software that only contains software for objects that need to perform conflict resolution. In practice, many sites only need conflict resolution for BTrees. So none of the other Zope or product code needs to be accessible to the ZEO server. Do you have any ideas about how to support the features you need given the requirements I've suggested here? Do those requirements make sense? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] Re: [Zope-dev] Proposed installation changes for review
I'm not dismissing it, and I think you need to go back and read what I wrote again very very carefully without reading anything into it. I'm not trying to imply that using environment variables to configure the current codebase will reduce the code footprint. Even if it did, because of the distributed nature of the technique, its damnedly hard to maintain in a project as large as Zope. Also, I didn't say ZConfig was 374k of code, I said it was 374k of *work*. I chose that word very carefully, and obviously thats going to err on the side of conservatism as certainly the work was not isolated to that single directory tree. I don't know what work means in this context, but think the ZConfig project is thorough. In my checkout there are 180k of document, 180k of unit tests, and 136k of code. A measure of work that suggests that something with 0k of documentation and tests and 136k of code would be better makes no sense to me. The point I'm trying to make is that Zope has learned nothing from the UNIX philosophy. Yes, you can extend the config schema. You can grow I don't see where the UNIX philosophy has anything useful to say about configuration, but maybe I'm just a closet unix hater 0.5 wink. new, better config files, of extraordinary magnitude. The all-powerful server will grow from being all-powerful to being all-powerful + n. It will be able to read mail. Its heralds shall sit upon mountain high throwns hewn of the finest O'Reilly and New Riders scripture. But lo, still you won't be able to do something as mundane as limit the memory the FTP server is able to consume without affecting the HTTP server. Fracture the server infrastructure into small, seperate processes. The configuration of the individual pieces becomes trivial. The understanding of the overall data flow improves. When there's nothing left to remove from code, you've won. Some of the breaks have already been made, like the separation of the storage from its front-end. Thats good, we need more action along those lines. I don't see that configuration gets any easier if you have multiple processes. Even if Zope had N processes, there would still be the same amount of stuff to configure. You'd probably still want a single master config file for the whole thing, and a tool to check the configuration is valid separate from the process that uses the file to configure itself. As I watched everyone working on the ZConfig project, I was impressed with how many issues are involved in getting a decent configuration system. I don't think that break the server into multiple pieces would make it easier to configure, and I don't see what requirements could have been eliminated to make the project take less work. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Bare except dangerous to ZODB?
On Wed, 2003-02-12 at 04:23, Toby Dickenson wrote: On Tuesday 11 February 2003 5:21 pm, Jeremy Hylton wrote: On Tue, 2003-02-11 at 12:10, Shane Hathaway wrote: I'd like to do the transaction states, because it would keep the code in zodb3 and zodb4 similar. There are application-level reasons to mark a transaction as doomed, and I would like to keep *that* code looking similar ;-). The transaction states approach would work in that context too, right? Here's a late answer: If an application needs to mark a transaction as doomed, it is supposed to call get_transaction().abort(). If a transactional resource manager, like a database connection, needs to mark a transaction as doomed, it: - returns False from prepare() -- the ZODB4 spelling - raises an exception in tpc_vote() -- the ZODB3 spelling Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Bare except dangerous to ZODB?
On Thu, 2003-02-20 at 13:33, Toby Dickenson wrote: On Thursday 20 February 2003 5:17 pm, Jeremy Hylton wrote: There are application-level reasons to mark a transaction as doomed, and I would like to keep *that* code looking similar ;-). The transaction states approach would work in that context too, right? Here's a late answer: If an application needs to mark a transaction as doomed, it is supposed to call get_transaction().abort(). If a transactional resource manager, like a database connection, needs to mark a transaction as doomed, it: - returns False from prepare() -- the ZODB4 spelling - raises an exception in tpc_vote() -- the ZODB3 spelling Thanks for that idea. Normally in the Zope context, transaction and request boundaries are co-aligned. This assumption runs deep in zope. I can see problems because an application calling abort wont stop the publisher calling commit at the end of the request. Good point. One possibility is that the publisher should more explicitly manage the relationship between transaction and request. It could call get_transaction() at the start of a request to get a transaction object. Store that object explicitly and call its abort() or commit() method at the end of the request. Having application state revert mid-request to the pre-transaction state seems like a bad idea. Commiting application changes made in the second half of the request seems bad too. That does seem like a bad idea, and explicitly using a single transaction object would prevent it. On the other hand, maybe Zope app code that detects an error should not call abort(). Perhaps the publisher should be responsible for the transaction and the app code should information the publisher that something went wrong. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Bare except dangerous to ZODB?
On Tue, 2003-02-11 at 04:13, [EMAIL PROTECTED] wrote: Chris McDonough wrote: Could this be done by initializing a dictionary at startup keyed on thread-id that a ConflictError exception's __init__ could stick a marker into, then checking that dictionary at commit time and disallowing the commit if the marker still existed? Yes, but Transaction objects are already keyed on the thread ID, so I think you just have to set a doomed flag on the transaction. Attempts to commit doomed transactions result in either ConflictError or DoomedTransactionError. ;-) Aborting the transaction (or beginning a new transaction) resets the flag. Shane The doomed flag is the way zodb4 works. Each transaction has a status flag that can be one of: active, preparing, prepared, failed, committed, aborting, and aborted. Actions are only allowable in certain states. The only thing you can do to a failed transaction is abort it. And the only thing you can do to a prepared transaction is commit or abort it. I'd like to backport that to ZODB3, but I need to finish the transaction package for ZODB4 first. If the current problem is perceived to be a significant risk, we could increase the priority of the backport. One downside is that some people do strange things with transactions, like joining them during the commit; that doesn't work in ZODB4. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Bare except dangerous to ZODB?
On Tue, 2003-02-11 at 12:10, Shane Hathaway wrote: I added a test to testZODB.py on a new branch (shane-conflict-handling-branch) that exercises the conflict handling bug. The test currently fails. It might be simpler to go with Toby's implementation for now: add a veto object to the transaction that refuses any attempt to commit. But maybe your transaction states are better. Let me know what you want to do. I'd like to do the transaction states, because it would keep the code in zodb3 and zodb4 similar. Unless there's a reason to think there are problems with the transaction state approach. I didn't look carefully at the test, but if I remember the discussion last time around, the problem is with read conflicts caught outside of 2PC. In that case, we either need to mark the connection so that it votes no when it gets to prepare() or we need to veto() method. I'd prefer the vote-no-in-prepare because it keeps the API smaller, but veto() isn't so bad; maybe it's better to stop the transaction quickly. We have a similar need when things go wrong inside the persistent machinery. I can't recall the details, but I think there are comments in the code about what happens when loading an object fails -- some state transition fails at any rate. In that case, the transaction really needs to be marked as failed and further actions prevented. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [Bug] Zope's transaction behaviour flawed
Your basic point makes sense, although I'm not entirely clear on how transaction management is integrated into the Zope application. Speaking for ZODB alone, I believe we've recommended that people call get_transaction().abort() if they catch an exception. I can't recall getting into any nuances of error handling, but it certainly makes sense to call abort() after the error handling. Note that a new transaction is begin implicitly any time a persistence object is referenced. So the error handling code does execute in the context of a transaction. It may be that it's just not the transaction you'd like it to be in. I assume that some work done during error handling is getting committed with the next successful request. Yuck. How does user-level error handling get invoked in Zope? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] 2.6.1b2?
BL == Brian Lloyd [EMAIL PROTECTED] writes: Here's the status - an engagement that we're doing has been bringing up some issues regarding ZODB and ZEO in large-scale environments. I think that the fixes are useful enough that they should be in 2.6.1, but getting them finalized has taken longer than I expected. I'd love to know what kind of thing 'large-scale' implies here, and what kind of problems the fixes fixed. BL Large-scale meaning large numbers of ZEO clients, that mount BL multiple ZEO-served databases that are each replicated using ZRS BL (Zope Corp.'s replication / failover solution) :^) BL The changes have to do with coordination of transaction commit BL among multiple databases, manageability and BL fault-tolerance. I'll ask Jeremy to be sure to update the BL CHANGES.txt with the important changes. I've included the current list of changes from ZODB3/NEWS.txt. I believe the list is complete, but would want Barry and Guido to double-check. The first change is possibily controversial. I think the others are fairly conservative. Jeremy The Transaction hosed feature is disabled in this release. If a transaction fails during the tpc_finish() it is not possible, in general, to know whether the storage is in a consistent state. For example, a ZEO server may commit the data and then fail before sending confirmation of the commit to the client. If multiple storages are involved in a transaction, the problem is exacerbated: One storage may commit the data while another fails to commit. In previous versions of ZODB, the database would set a global variable hosed that prevented any other transaction from committing until an administrator could check the status of the various failed storages and ensure that the database is in a consistent state. This approach favors data consistency over availability. The new approach is to log a panic but continue. In practice, availability seems to be more important than consistency. The failure mode is exceedingly rare in either case. The BTrees-based fsIndex for FileStorage is enabled. This version of the index is faster to load and store via pickle and uses less memory to store keys. We had intended to enable this feature in an earlier release, but failed to actually do it; thus, it's getting enabled as a bug fix now. Two rare bugs were fixed in BTrees conflict resolution. The most probable symptom of the bug would have been a segfault. The bugs were found via synthetic stress tests rather than bug reports. A value-based consistency checker for BTrees was added. See the module BTrees.check for the checker and other utilities for working with BTrees. ZEO --- The ZEO version number was bumped to 2.0.2 on account of the below minor feature additions. The performance of full cache verification has improved dramatically. XXX Get measurements from Jim -- somewhere in 2x-5x recall. The implementation was fixed to use the very-fast getSerial() method on the storage instead of the comparatively slow load(). The ZEO server has an optional timeout feature that will abort a connection that does not commit within a certain amount of time. The timeout works by closing the socket the client is using, causing both client and server to abort the transaction and continue. This is a drastic step, but can be useful to prevent a hung client or other bug from blocking a server indefinitely. If a client was disconnected during a transaction, the tpc_abort() call did not properly reset the internal state about the transaction. The bug caused the next transaction to fail in its tpc_finish(). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Infuriating ZClass registry Heisenbug
I'll have to take a look, but may not get to it today. FWIW I thought I fixed the bug on Friday. I could certainly create ZClasses through the web without getting any obvious errors. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Infuriating ZClass registry Heisenbug
The nice thing about fixing this bug is that the code continues to be improved. The last round of beautification made the code substantially more readable. This was a direct result of trying to track down a reference count problem. The problem proved to be elsewhere, but it was nearly impossible for me to inspect the old coptimizations.c code to figure out if the problem was there. If we make the code easier to read, we reduce the burden we all share more keeping the software working. Inscrutable code hides subtle bugs and encourages work arounds that make the whole system harder to understand. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Infuriating ZClass registry Heisenbug
CD == Casey Duncan [EMAIL PROTECTED] writes: CD Yes, except I would argue that such work is best left on the CD HEAD rather than applied to a maintenance branch until CD necessitated by bug reports on that branch, or at least the CD knowledge that such a bug definitely exists and can be CD reproduced on that branch. The work was all done on the trunk, but no one noticed before Zope 2.6 was released. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Infuriating ZClass registry Heisenbug
SH == Shane Hathaway [EMAIL PROTECTED] writes: SH On Wed, 11 Dec 2002, Evan Simpson wrote: I think I've fixed my ZGlobals, but there's definitely a problem in coptimizations. I made Connections.py use the Python persistent_id, restarted twice, and ZGlobals was fine on the second restart. SH Ah-ha, you now have a reproducible test case. The SH coptimizations code has been undergoing beautification lately; SH apparently this particular ZClass-related functionality was not SH retained correctly. This ought to be a priority for Zope 2.6, I SH think. Evan, Can you give me a recipe from reproducing the problem? Ideally, we'd write a little unit test that verified that the persistent_id code handled ZClass properly. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] ZODB3 3.1.1b1
We have made a beta release of ZODB 3.1.1 and ZEO 2.0.1. The primary change is a fix that allows transactions with multiple databases to run without deadlock. These release will correspond exactly to Zope 2.6.1b1, which will also be released today. See http://www.zope.org/Products/StandaloneZODB http://www.zope.org/Products/ZEO Zope developers who have written customer data managers (aka _p_jars) should be sure to test this release. The deadlock prevention code changes the order in which data managers are invokved and prevents new data managers from registering after a transaction has begun. I've include the summary of changes below. Jeremy What's new in ZODB3 3.1.1? == Release date: 10-Dev-2002 It was possible for earlier versions of ZODB to deadlock when using multiple storages. If multiple transactions committed concurrently and both transactions involved two or more shared storages, deadlock was possible. This problem has been fixed by introducing a sortKey() method to the transaction and storage APIs that is used to define an ordering on transaction participants. This solution will prevent deadlocks provided that all transaction participants that use locks define a valid sortKey() method. A warning is raised if a participant does not define sortKey(). For backwards compatibility, BaseStorage provides a sortKey() that uses __name__. Added code to ThreadedAsync/LoopCallback.py to work around a bug in asyncore.py: a handled signal can cause unwanted reads to happen. A bug in FileStorage related to object uncreation was fixed. If an a transaction that created an object was undone, FileStorage could write a bogus data record header that could lead to strange errors if the object was loaded. An attempt to load an uncreated object now raises KeyError, as expected. The restore() implementation in FileStorage wrote incorrect backpointers for a few corner cases involving versions and undo. It also failed if the backpointer pointed to a record that was before the pack time. These specific bugs have been fixed and new test cases were added to cover them. A bug was fixed in conflict resolution that raised a NameError when a class involved in a conflict could not be loaded. The bug did not affect correctness, but prevent ZODB from caching the fact that the class was unloadable. A related bug prevented spurious AttributeErrors when a class could not be loaded. It was also fixed. The script Tools/zeopack.py was fixed to work with ZEO 2. It was untested and had two silly bugs. Some C extensions included standard header files before including Python.h, which is not allowed. They now include Python.h first, which eliminates compiler warnings in certain configurations. ZEO 2.0.1 (2002/12/10) The changes since the ZEO 2.0 release are: - The logging code for client connection was improved in several ways. In particular, client connection and disconnection messages include the server's address. - ClientCache.py: make some functions into methods; include the storage name in all log messages; append the storage name to the trace filename. - zrpc/trigger.py: in handle_read() (both versions!), catch socket errors from the recv() call and return early in that case. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Bugfix release?
TD == Toby Dickenson [EMAIL PROTECTED] writes: TD A bug in the transaction manager would be really bad news. I TD would have thought it appropriate to resolve these problems and TD leave at least a few weeks of beta test before release. I'm looking into the problem, as I reported on zodb-dev. Recall that we're talking about releasing Zope 2.6.1 beta 1 -- a beta maintenance release. I'm not sure how much testing Brian would like to see before a final release of 2.6.1, but a few weeks seems like the right order of magnitude. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex and PersistentList (Bug?)
JPS == Jeffrey P Shell Jeffrey writes: JPS What about ``if callable(aq_base(newKeywords)):`` to remove JPS potential acquisition wrappers? callable() returns True for any instance. Since PersistentList is an instance, you can't use callable() to determine whether it is callable. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] consistent naming in ZEO releases
I used distutils sdist to build the releases for ZEO and ZODB. I let distutils pick the file extension. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex and PersistentList (Bug?)
It must be any instance of an ExtensionClass. I think Python instances used to be true-for-callable as well, but that must have been fixed. Python 2.1.3 (#5, Aug 15 2002, 10:41:31) [GCC 2.95.3 19991030 (prerelease)] on linux2 Type copyright, credits or license for more information. from ZODB.PersistentList import * callable(PersistentList()) 1 Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] ZODB 3.2 release plan posted
TD == Toby Dickenson [EMAIL PROTECTED] writes: TD On Tuesday 05 November 2002 7:04 pm, Jeremy Hylton wrote: I have posted a tenative release plan for ZODB 3.2 in the ZODB Wiki at http://www.zope.org/Wikis/ZODB/ZODB%203.2%20Release There are a small set of proposed features centering around better configuration and management of ZEO and ZODB. TD Is 3.2 work happing on the trunk or a branch? It's happening where it's happening. Guido is doing the zdaemon stuff on the trunk. I've been working on the deadlock code on a branch, but I'll merge that to the trunk soon. This is the standard Zope development practice -- keep the trunk stable. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] deadlock prevention for ZODB3 / Zope 2.6
We recently discovered that ZODB3 applications, like Zope 2.6, can deadlock when run in a system that uses multiple storages. This was a fundamental design flaw in ZODB that, happily, has a simple fix. Brian and I are planning to commit these changes to the ZODB3 3.1 and Zope 2.6 release branches and make them part of an upcoming bug fix release. This is a slightly dodgy plan, because there is a lot of code that is being changed for this bug fix. But the problem is fairly serious and the solution is available now, we'd like to get it in the hands of users more quickly than ZODB 3.2 and Zope 2.7. In short, the solution is to sort storages before beginning the two-phase commit process used by the transaction manager. Since the two-phase commit occurs during a transaction commit, we know all the storages that will be used, and, thus, all the locks that will be acquired. By sorting the locks, we can guarantee that deadlock will not occur. The implementation of deadlock prevention requires changes to the storage and transaction data manager APIs. Each must now implement a sortKey() method that returns a string that can be used to sort the storages. The sortKey must be globally unique; two storages can't have the same key unless they are the same storage and one storage must have the same key regardless of when it is used. The sortKey must be a string, because other objects may not sort in the same order for all Python versions. We have provided default implementions of sortKey() for BaseStorage and Connection. These implementations aren't necessarily correct, but they will prevent outright breakage if you update to the maintenance release. BaseStorage uses the __name__ and Connection uses id(). All storages and data managers should be updated to the new API. The default implementations may not prevent deadlock in a multi-storage environment; they only prevent the update from breaking existing code. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] ZODB 3.2 release plan posted
I have posted a tenative release plan for ZODB 3.2 in the ZODB Wiki at http://www.zope.org/Wikis/ZODB/ZODB%203.2%20Release There are a small set of proposed features centering around better configuration and management of ZEO and ZODB. Feedback is welcome. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Debugging conflict errors? Hints please!
SHH == Stefan H Holek [EMAIL PROTECTED] writes:
SHH Hi All! I am experiencing a lot of 'ZODB conflict error at
SHH ...' that I can reliably reproduce by hitting my browser's
SHH Refresh button at a high rate. The bad news is that the
SHH conflicts happen on pages that are not supposed to change
SHH anything in the ZODB but only display results of some SQL
SHH queries. My question is now how to find out what actually is
SHH causing the conflicts. The log entry refers to the page being
SHH served ('/very/long/path/detail_view').
The debug log should show the oid that caused the conflict error.
From the oid, you ought to be able to figure out what the object is --
class, etc. -- and then what transactions have modified that object.
Can anyone recommend a simple strategy for either of these two steps?
There's got to be a good way to do it, but I'm not sure what the best
way is. I'd open the database in an interactive prompt:
db.open()[oid]
That would get me the object. Then I'd use fsdump to find all the
transactions that include that oid. That would be a very manual
process since I'd just grep the textual dump of the storage.
Jeremy
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZEO cache instrumentation -- any takers?
LRA == Leonardo Rochael Almeida [EMAIL PROTECTED] writes: LRA The last two installations belong to the same client, and are LRA both experiencing a problem that I believe is related to ZEO LRA latency. Both sites depend heavily on a single ZCatalog for LRA operation (one ZCatalog hit in almost every page, with some LRA pages with more than one ZCatalog hit). When site content is LRA being managed, we're hitting ReadConflict errors (They read LRA like this: ZODB.POSException.ReadConflictError on database read LRA conflict error (oid0004c0a4, class LRA BTrees.OOBTree.OOBucket)). The reference to BTrees makes me LRA think this is ZCatalog related, we don't use BTrees ourselves LRA anywhere else and we only use regular Folders as bases for our LRA folderish objects, and managing content causes the reindexing LRA of objects. Which I think is what is triggering the conflict LRA errors. Your analysis sounds correct. A read conflict occurs when on thread modifies an object while another thread is reading it (subject to certain other conditions that probably aren't relevant). I assume several users are managing content at the same time. If Zope gets a read conflict, it will retry the transaction. So you must get very unlikely in order to see the error. Each time the transaction is retried, something must go wrong. Are there a lot of concurrent transactions? Or do the updates affect a large number of objects at once? If either is true, it seems like it would make the problem more likely to occur. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope-Annce] ZEO 2.0 beta 1 released
TD == Toby Dickenson [EMAIL PROTECTED] writes: TD There is an O(n) option with easy logic... fragment big strings TD inside message_output. Sounds like a possibility. It would still be nice to have something in handle_write() to avoid extra slicing on partial send()s. But your patch is nice and simple. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope-Annce] ZEO 2.0 beta 1 released
TD == Toby Dickenson [EMAIL PROTECTED] writes: although we may do one more beta release to add some performance improvements. TD Do you have something specific in mind? I'd like to change this bit of code in handle_write(): if n len(v): # XXX It's unfortunate that we end up making many # slices of a large string. output.insert(0, v[n:]) break # we can't write any more If we send a very large message, which could be caused by a big pickle or just a transaction that touches a lot of objects, the message gets sent in little chunks. The current implementation uses string slicing to save the rest of the string, but that ends up making many copies of the large string -- an O(n^2) proposition. A possible solution is to store an index into the first string in __output and just increment the index. The logic will be a bit tricky to get right. After the alpha release, I suggested that there would be several more alpha releases to add new features. We didn't have the resources to pursue the new features. Instead, we plan to get a solid 2.0 final release out as soon as possible. New features will be postponed until 2.1. TD I would like to merge the branch 'toby-signal-branch' in before TD 2.0. This provides an easy way to use extensions to the storage TD API across ZEO. TD This feature is critical for DirectoryStorage. I am 100% happy TD with the current implementation (but not very happy with the TD name of the new method suggestions welcome) I'll look at the branch before the next beta. Anyone have ideas for a new name? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Corrupt Data.fs: Is there a way of fixing transaction ids?
JW == Joachim Werner [EMAIL PROTECTED] writes: JW Hi! I have a really large Data.fs file (1.3 GB) that has a JW number of errors. When I run the fstest.py script, I get this: JW Problem: 109963117 object serialno 0x034573c4b6a2cb6e does not JW match transaction id 0x034573c55c8c0dbb JW How can I fix the Data.fs file? It should be possible to scan JW the Data.fs and create correct serialnos or transaction ids for JW the broken entries. But how exactly would I do that? JW I'd also like to identify the corrupt entries (i.e. find out the JW Zope object they belong to) to be able to eliminate them. I'd start by running fsdump.py and identifying what transaction and objects are affected by this problem. fsdump will display transaction metadata (like time) and oid class for each object. JW My ultimate question is how these corrupt entries can exist at JW all. The Data.fs hit the 2 GB border once. So that could be a JW reason. But even then it would be really nice for the ZODB to JW not write corrupt entries ... I agree that ZODB should not write the corrupt entries. It's possible, of course, that it didn't. It's certainly possible for bit-rot to strike a Data.fs, although it seems odd that the only manifestation of a disk failure would be a mismatch between object serialno and transaction id. I might be able to help more by looking at the actual Data.fs. Are you in a position to share it? (I can be discrete ;-). Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Crash Burn with Zope 2.6.0a1 and ZEO 2.0a1
AS == Andrew Sydelko [EMAIL PROTECTED] writes: AS Trying Zope 2.6.0a1 and ZEO 2.0a1. The storage server seemed to AS come up fine, but when trying to start the ZEO client it pretty AS much sucked. It looks like you would have had the same problems without ZEO. It would be helpful if you could try running Zope without ZEO, just with FileStorage. AS I really should include the log file it produced but it's quite AS long. I put it up at AS http://eng.purdue.edu/people/sydelko/zope_2.6a1_log_1 AS It ended up dying at the end, but that I think I can fix. AS I'd really like someone to clue me into why everything else AS complained. The errors in the log are all about failures to import specific modules. (Let me know if I missed something else.) Failed to import class Splitter from module Products.PluginIndexes.TextIndex.Splitter.ZopeSplitter When I import Products...ZopeSplitter, I see a module with a Splitter attribute bound to a function. The other import problem I see is Zope Could not import Products.HTMLDocument Is HTMLDocument a standard product or a 3rd party product? I don't see one in my 2.6 checkout. If it's 3rd part, then I expect it needs to be upgrade to work with 2.6. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Crash Burn with Zope 2.6.0a1 and ZEO 2.0a1
AS == Andrew Sydelko [EMAIL PROTECTED] writes: AS On Tue, 18 Jun 2002 11:20:21 -0400 Jeremy Hylton AS [EMAIL PROTECTED] wrote: AS == Andrew Sydelko [EMAIL PROTECTED] writes: AS Trying Zope 2.6.0a1 and ZEO 2.0a1. The storage server seemed to AS come up fine, but when trying to start the ZEO client it pretty AS much sucked. It looks like you would have had the same problems without ZEO. It would be helpful if you could try running Zope without ZEO, just with FileStorage. AS Hmm... The first time I copied the 3gig Data.fs over and tried AS to start Zope a few times. After fiddling with the permissions AS of the var directory it finally started, but the Data.fs was AS only 2MB! AS So I copied it again. And once again, the 3gig Data.fs is blown AS away! What am I doing wrong??? What are the permissions of the AS var folder supposed to be? I'm starting Zope with: It wouldn't surprise me if this is a completely unrelated problem. Any chance that the python your using to test doesn't have large file support? Check by importing test.test_largefile. We recently discovered that there's a bad interaction between ZODB and Python without large file support running on a platform that supports large files. In this specific configuration, os.path.exists() returns false for files 2 GB and ZODB creates a new Data.fs. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Crash Burn with Zope 2.6.0a1 and ZEO 2.0a1
AS == Andrew Sydelko [EMAIL PROTECTED] writes: Failed to import class Splitter from module Products.PluginIndexes.TextIndex.Splitter.ZopeSplitter When I import Products...ZopeSplitter, I see a module with a Splitter attribute bound to a function. AS I don't understand what you're saying here. Do you mean it works AS without complaining for you? In the past I have deleted all the AS Products that were in the Products Control Panel to fix a AS similar bug. Is that the case here as well? The problem is that the alpha release had a bug that made it impossible to load Splitter objects from the database. Andreas fixed the bug on Monday, so you could copy the changed file Zope/lib/python/Products/PluginIndexes/TextIndex/Splitter/ZopeSplitter/__init__.py or you could delete whatever objects are related to the text index. The other import problem I see is Zope Could not import Products.HTMLDocument Is HTMLDocument a standard product or a 3rd party product? I don't see one in my 2.6 checkout. If it's 3rd part, then I expect it needs to be upgrade to work with 2.6. AS HTMLDocument is a 3rd party product. Is there a document that AS says what has changed enough since 2.5.1 that would break AS products? and how to fix them? The traceback in the log indicates that the problem is related to InterfaceBase. (IIRC, I don't have that browser window open any more.) I think that Jim changed the Interface package just before the alpha, backporting changes from the Zope3 package. Maybe Jim can summarize what people need to do to update their code. AS This is frustrating... I agree that it is frusterating, but I think it's hard to have an alpha release that works perfectly with all known third party extensions. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Crash Burn with Zope 2.6.0a1 and ZEO 2.0a1
AS == Andrew Sydelko [EMAIL PROTECTED] writes: AS Can someone tell me exactly how to compile python 2.1.3 with LFS AS support? I don't have the URL handy, but it's section 8.1.1 of the manual IIRC. Look for large file support in the library reference. Worked for me without problem on a RH 7.2 system. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] ZEO product installation Q
The documentation for ZEO contains the following note about product installation. The final sentence suggests that we intend to fix it, but I don't understand the problem exactly or whether it has been fixed. Can anyone tell me if this warning is still relevant? Jeremy Zope product installation Normally, Zope updates the Zope database during startup to reflect product changes or new products found. It makes no sense for multiple ZEO clients to do the same installation. Further, if different clients have different software installed, the correct state of the database is ambiguous. Zope will not modify the Zope database during product installation if the environment variable ZEO_CLIENT is set. Normally, Zope ZEO clients should be run with ZEO_CLIENT set so that product initialization is not performed. If you do install new Zope products, then you need to take a special step to cause the new products to be properly registered in the database. The easiest way to do this is to start Zope once with the environment variable FORCE_PRODUCT_LOAD set. The interaction between ZEO and Zope product installation is unfortunate. In the future, this interaction will be removed by ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] how bad are per-request-write-transactions
CM == Chris McDonough [EMAIL PROTECTED] writes: Completely agreed. My disagreement is portraying the counter problem as impossible with the zodb. I think some people, as evidenced by some of the responses, are willing to live with the tradeoffs. Other people will find managing a log file on disk to be a more manageable solution. CM It would be best to make make a dual-mode undoing and nonundoing CM storage on a per-object basis. I'd really like to do this for ZODB4, but it seems hard to get it into FileStorage, without adding automatic incremental packing to FileStorage. Example: Object A is marked as save enough revisions to do a single undo. When a transaction updates A and makes older revisions unnecessary, there's no obvious way to remove them without doing a pack. We could write a garbage collector that removed unneeded things (as opposed to packing everything to a particular date), but it doesn't seem very useful if it needs to be run manually. Also, how would you specifiy the object's packing policy? I'm thinking an _p_revision_control attribute or something like that. If the attribute exists on an object, it sets a particular policy for that object. Do individual transactions need to play in this game, too? I'm imagining a use case where an object is marked as no revisions but you want to be able to undo a particular transaction. I'm not sure if that means : - you can undo the transaction, but the no revisions object keeps its current state. - you can undo the transaction, and because the transaction is specially marked as undoable, there actually is a revision - you can't undo the transaction The first choice seems appropriate for a counter (I think), but I'm not sure if it makes sense for all possible revision-less objects. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] ZEO 1.0 final release
I am happy to announce that ZEO 1.0 final was released today. Zope Enterprise Objects (ZEO) provides the ability for multiple processes to use a single Zope Object Database. ZEO turns the Zope object system in a distributed architecture, allowing multiple processors, machines, and networks to act as one website. For more information and downloads, see the ZEO web page at http://www.zope.org/Products/ZEO. ZEO 1.0 will work with Zope 2.2 or higher and with StandaloneZODB 1.0. The 1.0 release comes after one year of beta testing, which results in several important bug fixes and increased stability. ZEO 1.0 has been used in production environment supporting hundreds of requests per second. See CHANGES.txt for details about bugs fixed. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] ZEO 1.0b6 released
I have just released ZEO 1.0 beta 6. It's been almost six months since the last beta release, so it was surely time. This release fixes two bugs that Zope Corp. has recently run into in the field. See http://www.zope.org/Products/ZEO/. Here are the changes: - Fixed a bug that could cause the server to stop committing transactions for a storage. The bug occurred under the following circumstances: - A synchronous client made a tpc_begin() request. - The request arrived while another transaction was in progress. - The other transaction finished. - The client closed the connection or an unexpected error occurred while the server was handling the client's tpc_begin(). The fix is to make the code to resume a waiting transaction robust in the face of disconnects and errors. - Fixed bugs in the storage server and the trigger module that caused the server to fail with uncaught exceptions when it received a signal. The start script installs several signal handlers, so the server should not fail when those signals are received. These bugs might have occurred in other circumstances, too. Note: The signal handlers are not particularly useful with the default zLOG backend (STUPID_LOG_FILE). - Upgrade to license to ZPL 2.0 I expect to release ZEO 1.0 final next week. I'll also take this opportunity to mention future ZEO development plans. I plan to make a ZEO 1.1 release sometime this month. This release will add some debugging support and a few minor features that have proved useful in high-volume ZEO clusters. I also plan to make an alpha release of ZEO 2.0. ZEO 2 is a partial re-implementation of ZEO that provides a new RPC protocol between the ZEO clients and server. The new RPC protocol will make it possible to extend ZEO with new features and to make changes that will improve performance. ZEO 2.0 alpha 1 will be a baseline that provides the new protocol, but probably none of the features. If there are any crucial feature requests for ZEO 1.1, now is the time to make them (patches preferred :-). Otherwise, new features will be postponed to ZEO 2.0. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Deleting versions from Control_Panel
On Mon, 4 Mar 2002 11:01:22 -0500 Andrew Sydelko [EMAIL PROTECTED] wrote: For some reason, when I try to view the Version Management screen under Control Panel, I get a bunch of gibberish for one of the versions... And... it won't let me click on the check box and Discard it... Is there a way to do this manually? Or even see which versions are available manually? We are running ZEO, so I should be able to import Zope and go from there... A low-level way to get a list of versions is to call the versions() method on the storage object. Example: from ZEO.ClientStorage import ClientStorage cs = ClientStorage(...) # your server address here cs.versions() Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Deleting versions from Control_Panel
On Mon, 4 Mar 2002 11:11:55 -0500 Andrew Sydelko [EMAIL PROTECTED] wrote: On Monday 04 March 2002 11:08, you wrote: On Mon, 4 Mar 2002 11:01:22 -0500 A low-level way to get a list of versions is to call the versions() method on the storage object. Example: from ZEO.ClientStorage import ClientStorage cs = ClientStorage(...) # your server address here cs.versions() If you wait a second or two, the client should connect. To actually, do something: import ZODB db = ZODB.DB(cs) cs.versions() # pick a version db.abortVersion(name) get_transaction().commit() Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.4.4b1 dumps core
AJ == Andreas Jung [EMAIL PROTECTED] writes: AJ Does this problem persist when you remove the 3rd-party products AJ ? Are you running Zope with enabled garbage collector ? Just a reminder that you *should* be running with the garbage collector enabled. We are aware of no current bugs in the garbage collector, which has been in use since Python 2.0. If you see a crash and the gdb stack trace points to the garbage collector, it is almost surely a sign of memory corruption caused by bugs elsewhere in Python or in a C extension. You can prevent those crashes by disabling the garbage collector, but you are more likely to get bizarre errors or storage corruptions since the interpreter memory is corrupted. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.4.4b1 dumps core
Python 2.1.2 has some extra safety checks that will immediately detect the stack overflow bugs that caused problems with PythonScripts before 2.4.4. The fact that you're seeing crashes in the garbage collector and not assertion failures in Python/ceval.c makes me think the problem isn't with the PythonScripts. I wouldn't rule it out completely, but it seems unlikely. Does the community have any Zen about how to narrow down bugs like this? It seems a daunting task, in general, because it's not obvious what particular request exercises the bug, and the error report doesn't come until long after the bug occurs. I wonder if you could crank up the garbage collection frequency -- either use gc.set_threshold() to see the threshold very low or add an explicit call to gc.collect() after each request. If you're lucky, this would cause the bug to be detect -- like right after the request that exercised the bug. Where exactly would you put the gc.collect() call to make this work? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Confused: Does PersistentList exist?
LR == Lennart Regebro [EMAIL PROTECTED] writes: LR I found seceral seemingly up to date docuemnts claiming that LR both PersistentMapping and PersistentList exists and is included LR with Zope. However, I couldn't find PersistentList, so I LR thought that it had been added after 2.3.3, which I am using. LR So I checked the CVS. Nope, nothing there. LR So whats the deal here, really? Andrew Kuchling wrote a PersistentList for his StandaloneZODB. I think it would be a good idea to include in ZODB eventually, but haven't gotten around to integrating it. IIRC, Jim wasn't sure he wanted it in ZODB. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Big problem.
Could you just undo enough transactions to get back to the old state? Or do you need to keep some of the changes you made? Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Big problem.
I don't know if a tool exists either. I think something quick could be hacked together, but I doubt I'd have time to do it anytime soon. I don't know if you up for some programming to accomplish this goal or not. There are a couple of ideas I have along these lines in case you or someone else can pursue them. There's a tool called fstest.py in two places in CVS: Zope/utilities/ZODBTools and StandaloneZODB/Tools If you run it with a -vv argument it will show you all the individual transactions an objects. It prints a transaction id, and could be modified to convert those transaction ids to timestamps. From the timestamps and oids, you could probably figure out the last good transaction. You can use transactionalUndo() on every transaction id following the last good one. You might be able to go into the storage and truncate it at the last good transaction. You'd need to blow away the index, too. This sounds risky, but might be the simplest solution. (Backup the file first :-). Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] disabling gc does not necessarily hide memory corruption
CM == Chris McDonough [EMAIL PROTECTED] writes: CM Ah ok... yeah, there were reportedly several leaks in the CM compiler stuff that have been fixed in CVS. I imagine that's CM what this is. Actually, I don't think the leaks haven't been fixed in CVS. It would be at best painful to try and get the compiler code to work without cycle GC. I really don't want to have to do it. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: EINTR ... was Re: [Zope-dev] browser closing connection
MTK == Matthew T Kromer [EMAIL PROTECTED] writes: MTK For what its worth, I tracked this down in the sources and MTK confirmed that in Zope 2.3, we shipped a modified asyncore.py MTK with Medusa that handled EINTR, but in Zope 2.4 we used stock MTK Python's asyncore which does NOT handle EINTR being returned MTK from select(). IMHO, the distributed Python 2.1 asyncore MTK behavior is incorrect. This is fixed in Python 2.2. A brief excerpt demonstrates the approach: def poll (timeout=0.0, map=None): if map is None: map = socket_map if map: r = []; w = []; e = [] for fd, obj in map.items(): if obj.readable(): r.append (fd) if obj.writable(): w.append (fd) try: r,w,e = select.select (r,w,e, timeout) except select.error, err: if err[0] != EINTR: raise In particular, I didn't use a while 1:. I believe an operator could send a signal to a process using asyncore and expect it to cause the app to fall out of a poll() call immediately, instead of waiting for the timeout to occur. (It might never occur.) I expect that the interrupted system call will be fairly uncommon, so it shouldn't matter than the poll() is returning without doing any work. In most cases, it will be called from loop() which already has a while loop. A similar fix was made in poll3(), which uses the select module's poll(2) interface. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: EINTR ... was Re: [Zope-dev] browser closing connection
JZ == John Ziniti [EMAIL PROTECTED] writes: JZ Replace your Python 2.1.1 asyncore.py with the one that is JZ attached. I've been using it for months now with no problems. JZ Notice, however, that it doesn't work on WinNT, b/c the author JZ didn't know what EINTR looked like on NT The asyncore.py in Python 2.2 has a number of bug fixes and improvements over 2.1.1 and over the patched version you attached. It does, for example, work correctly on win32. I'd recommend grabbing a copy of asyncore.py from a Python 2.2 beta and using it. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZEO on SMP
Since a single Python process has a single global interpreter lock, you'll never fully utilize a 2-processor machine. The lock is released whenever a socket call (send, receive, listen, etc.) is made, so that provides some opportunity for using two CPUs, but not much. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZEO client-server protocol endian-safe?
AB == Anthony Baxter [EMAIL PROTECTED] writes: AB Any known gotchas with running ZEO server and ZEO client on AB boxes of different endianness? e.g. the server running on sun AB solaris/sparc, while the clients are on x86 linux boxes. AB I'd expect it to be fine, but am feeling paranoid... All the data that ZEO passes between client and server is marshalled with cPickle, which uses a neutral format. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Greedy except clauses
"CW" == Chris Withers [EMAIL PROTECTED] writes: I am probably a bit idiosyncratic, but I prefer to avoid bare excepts at all costs. I often use "except Exception:", otherwise I add a CW Will that catch string exceptions? eg: raise 'Something bad CW happened' No. CW If not, then it's not much use in Zope, which is unfortunately CW riddled with String exceptions :-S That might be something worth fixing, too. :-) Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Greedy except clauses
"AB" == Anthony Baxter [EMAIL PROTECTED] writes: AB Now, it's obvious that not all of these are bad - some are AB probably fine - but there's also many that are _not_ good. The AB one in MailHost.py does eat exceptions. I'm sure there are AB others. I am probably a bit idiosyncratic, but I prefer to avoid bare excepts at all costs. I often use "except Exception:", otherwise I add a comment that specifically notes that I intend to catch everything. Otherwise, you can't tell the difference between intentional and accidental use. AB I've done a quickie hack script that runs over the source and AB produces a list of matches to lines in the file, and links to a AB lightly patched viewcvs so that you can jump straight to the AB lines. It seems like it would be reasonably easy to make this a AB bit more sophisticated to include things like the nesting 'def' AB and 'class' method. AB This, to me, seems like a pretty nice community type task - AB people could take one of them off the list of unchecked ones, AB and have a look through the code... no? It would also be interesting to see how many try/except blocks are including more code than they should. The larger the block, the more likely that a line of code raises an error that the author didn't consider; these errors can mean that real bugs get caught be accident and ignore. When the block is small -- preferably just a line or two -- there is litle chance for error. Of course, there are very good reasons to have longer except blocks. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
