[Zope-dev] Dare To get A m[o]rtgage
I am taking the liberty of writing you this letter instead of interrupting you by phone. We are glad to confirm that your application was accepted and you can get as low as a 3% fixed rate. Could we ask you to please fill out final details we need to complete you here: Quick Form Sincerely, Bill Hanks Account Manager - not interested ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss: ONLY IF YOUR APP CODE FORKS
[sathya] > tim thanks for confirming it. Wont loose sleep over it now. I did not > mean to sound like questioning anybodys track record. No, it didn't sound like you were. I mentioned that Dieter has an excellent track record because *I'm* giving him a hard time here . I'm sure he's seeing problems, but I'm frustrated by the lack of concrete information about how they were provoked and on which kind of system. > Since we have ZEO clusters in production it raised alarm bells thats all. Its good > to know the problem MAY occur only if u fork in your own app code A potential problem is that Unixish systems typically fork "under the covers" for things that look utterly harmless to an application writer -- like the os.system() and os.popen() Dieter mentioned. fork()ing can create horrible problems for threaded applications (which is essentially *why* POSIX only clones the thread calling fork() -- that creates problems too, but the POSIX spec I referenced before discusses all this, so I won't repeat it here). > and the core zeo/zope code by itself is not contributing to it. I don't think they are, but there's still not enough info here to say for sure. > I guess as a general rule of thumb its not a good idea to use forks in zope > application or product code anyway (unless you know what you are doing.) A more general rule of thumb is to avoid forking a threaded application of any kind. It's easy to follow that rule on OSes that don't have fork , but unfortunately tricky to avoid on those that do have fork. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss: ONLY IF YOUR APP CODE FORKS
tim thanks for confirming it. Wont loose sleep over it now. I did not mean to sound like questioning anybodys track record. Since we have ZEO clusters in production it raised alarm bells thats all. Its good to know the problem MAY occur only if u fork in your own app code and the core zeo/zope code by itself is not contributing to it. I guess as a general rule of thumb its not a good idea to use forks in zope application or product code anyway (unless you know what you are doing.) Regards sathya [sathya] ... The zeoclient causes threads to be created but there are no "forks" or "system" calls as far as I can tell (or strace for that matter) Can you please point out where in the zeo code does forking occur ? I will try and duplicate this condition. [tim] ZEO and ZEO never fork -- they wouldn't be portable if they did (only Unixish systems have fork()). The only forking you'll find here is in startup scripts specific to Unixish systems. Dieter is talking about forks (whether direct or indirect) in *application* code, not in the Zope/ZEO cores. If application code running in a Zope/ZEO process forks, and Zope/ZEO are running asyncore in a thread, and the fork() implementation does clone all threads (which does not happen under Windows or POSIX), then of course the fork() the application code does is going to create a clone of the asyncore thread too. That said, it would indeed be helpful if Dieter revealed enough details about what his app did, and on which kind of system, so it would be possible for others to try to duplicate his results. I have no doubt that he is seeing problems, BTW -- Dieter has an excellent track record. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss: would like to duplicate this
[sathya] > ... > The zeoclient causes threads to be created but there are no "forks" or > "system" calls as far as I can tell (or strace for that matter) > Can you please point out where in the zeo code does forking occur ? I > will try and duplicate this condition. ZEO and ZEO never fork -- they wouldn't be portable if they did (only Unixish systems have fork()). The only forking you'll find here is in startup scripts specific to Unixish systems. Dieter is talking about forks (whether direct or indirect) in *application* code, not in the Zope/ZEO cores. If application code running in a Zope/ZEO process forks, and Zope/ZEO are running asyncore in a thread, and the fork() implementation does clone all threads (which does not happen under Windows or POSIX), then of course the fork() the application code does is going to create a clone of the asyncore thread too. That said, it would indeed be helpful if Dieter revealed enough details about what his app did, and on which kind of system, so it would be possible for others to try to duplicate his results. I have no doubt that he is seeing problems, BTW -- Dieter has an excellent track record. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss:MAYBE NOT
[sathya] > thanks for the clarification below and also the pointers to the posix > behaviour of fork. The Warning about Zope/ZEO clients in the subject > line certainly caused some alarm bells to go off. > > I am assuming now that dieters description below of using forks does not > gel with the ZOPE/ZEO process model i.e there are no forks being called > within the code to cause the asyncore thread to be cloned I'm still not clear on either exactly what Dieter did, or on what kind of system. In any case, he wasn't talking about forks in Zope/ZEO directly, but about forks in *application* code. The presence (or absence) of those depends on what applications do, not on what Zope/ZEO do. > ( even if one were using a non posix compliant thread lib like native solaris it > would still be a non issue ). I don't think we know enough yet to say. I've never used native Solaris threads, and they're so different from POSIX threads in this respect that I'm not going to guess about how ugly life can be with them. I'm sure there's no problem here with native Windows threads. I don't *see* a way for there to be a problem under POSIX thread semantics, but then I'm still guessing too much about what Dieter actually did. If I were you, I'd ignore this thread until it reaches a conclusion -- if there were widespread problems here, we would have heard about them before in the multiple years Zope and ZEO have been in production (and even under Solaris, I believe most people chose to use Sun's POSIX threads implementation, not the native Solaris threads). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss: would like to duplicate this
Tim Peters wrote:
just to add my 2 cents
I have been looking at zserver code, the only time fork or system (which
i presume invokes execve ) calls are used are at startup to either
a) run a cmdline
b) daemonize
heres a snip from strace output
strace -o strace.txt -f -e trace=fork,execve ./runzope
on a zeo instance
15298 execve("./runzope", ["./runzope"], [/* 23 vars */]) = 0
15298 execve("/var/dev/vision/python233/bin/python",
["/var/dev/vision/python233/bin/py"..., "/var/dev/vision/Zope27/lib/python
which creates a child process with id 15299
at this point asyncore main loop thread has not even started so it is
safe to assume that the parent does not start the asyncore loop for any
servers created but happens in the forked child . which means we
probably cannot have multiple asyncore mainloops running
The zeoclient causes threads to be created but there are no "forks" or
"system" calls as far as I can tell (or strace for that matter)
Can you please point out where in the zeo code does forking occur ? I
will try and duplicate this condition.
-ty
sathya
[Dieter Maurer]
The problem occured in a ZEO client which called "asyncore.poll"
in the forked subprocess. This "poll" deterministically
stole ZEO server invalidation messages from the parent.
I'm sorry, but this is still too vague to guess what happened.
- Which operating system was in use?
- Which thread package?
- In the ZEO client that called fork(), did it call fork() directly, or
indirectly as the result of a system() or popen() call? Or what?
I'd like to understand a specific failure before rushing to
generalization.
- In the ZEO client that called fork() (whether directly or indirectly),
was fork called *from* the thread running ZEO's asyncore loop,
or from a different thread?
I read the Linux "fork" manual page and found:
fork creates a child process that differs from the parent process
only in its PID and PPID, and in the fact that resource utilizations
are set to 0. File locks and pending signals are not inherited.
...
The fork call conforms to SVr4, SVID, POSIX, X/OPEN, BSD 4.3
If it conforms to POSIX (as it says it does), then fork() also has to
satisfy the huge list of requirements I referenced before:
http://www.opengroup.org/onlinepubs/009695399/functions/fork.html
That page is the current POSIX spec for fork().
I concluded that if the only difference is in the PID/PPID
and resource utilizations, there is no difference in the threads between parent
and child.
Except that if you're running non-POSIX LinuxThreads, a thread *is* a
process (there's a one-to-one relationship under LinuxThreads, not the
many-to-one relationship in POSIX), in which case "no difference in
threads" is trivially true.
This would mean that the wide spread "asyncore.mainloop" threads could suffer
the same message loss and message duplication.
That's why all sane threading implementations do what POSIX
does on a fork(). fork() and threading don't really mix well under
POSIX either, but the "fork+exec" model for starting a new process is
an historical burden that bristles with subtle problems in a
multithreaded world; POSIX introduced posix_spawn() and posix_spawnp()
for sane(r) process creation, ironically moving closer to what most
non-Unix systems have always done to create a new process.
I did not observe a message loss/duplication in any
application with an "asyncore.mainloop" thread.
I don't understand. You said that you *have* seen message
loss/duplication in a ZEO client, and I assume the ZEO client was
running an asyncore thread. If so, then you have seen
loss/duplication in an application with an asyncore thread.
Or are you saying that you haven't seen loss/duplication under the
specific Linux flavor whose man page you quoted, but have seen it
under some other (so far unidentified) system?
Maybe, the Linux "fork" manual page is only not precise with respect
to threads and the problem does not occur in applications
with a standard "asyncore.mainloop" thread.
That "fork" manpage is clearly missing a mountain of crucial details
(or it's not telling the truth about being POSIX-compliant). fork()
is historically poorly documented, though.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
--
===
CEO
ZeOmega
Open minds' Open Solutions
Plano, Texas, USA
Bangalore, India
972-731-6750 (O)
214-733-3467 (M)
http://www.zeomega.com
Open source content management and workflow solutions
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman
Re: RE: [Zope-dev] RE: [ZODB-Dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss
[Dieter Maurer] > The problem occured in a ZEO client which called "asyncore.poll" > in the forked subprocess. This "poll" deterministically > stole ZEO server invalidation messages from the parent. I'm sorry, but this is still too vague to guess what happened. - Which operating system was in use? - Which thread package? - In the ZEO client that called fork(), did it call fork() directly, or indirectly as the result of a system() or popen() call? Or what? I'd like to understand a specific failure before rushing to generalization. - In the ZEO client that called fork() (whether directly or indirectly), was fork called *from* the thread running ZEO's asyncore loop, or from a different thread? > I read the Linux "fork" manual page and found: > > fork creates a child process that differs from the parent process > only in its PID and PPID, and in the fact that resource utilizations > are set to 0. File locks and pending signals are not inherited. > > ... > > The fork call conforms to SVr4, SVID, POSIX, X/OPEN, BSD 4.3 If it conforms to POSIX (as it says it does), then fork() also has to satisfy the huge list of requirements I referenced before: http://www.opengroup.org/onlinepubs/009695399/functions/fork.html That page is the current POSIX spec for fork(). > I concluded that if the only difference is in the PID/PPID > and resource utilizations, there is no difference in the threads between parent > and child. Except that if you're running non-POSIX LinuxThreads, a thread *is* a process (there's a one-to-one relationship under LinuxThreads, not the many-to-one relationship in POSIX), in which case "no difference in threads" is trivially true. > This would mean that the wide spread "asyncore.mainloop" threads could suffer > the same message loss and message duplication. That's why all sane threading implementations do what POSIX does on a fork(). fork() and threading don't really mix well under POSIX either, but the "fork+exec" model for starting a new process is an historical burden that bristles with subtle problems in a multithreaded world; POSIX introduced posix_spawn() and posix_spawnp() for sane(r) process creation, ironically moving closer to what most non-Unix systems have always done to create a new process. > I did not observe a message loss/duplication in any > application with an "asyncore.mainloop" thread. I don't understand. You said that you *have* seen message loss/duplication in a ZEO client, and I assume the ZEO client was running an asyncore thread. If so, then you have seen loss/duplication in an application with an asyncore thread. Or are you saying that you haven't seen loss/duplication under the specific Linux flavor whose man page you quoted, but have seen it under some other (so far unidentified) system? > Maybe, the Linux "fork" manual page is only not precise with respect > to threads and the problem does not occur in applications > with a standard "asyncore.mainloop" thread. That "fork" manpage is clearly missing a mountain of crucial details (or it's not telling the truth about being POSIX-compliant). fork() is historically poorly documented, though. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] RE: [ZODB-Dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss
Tim Peters wrote at 2004-6-27 04:46 -0400: > ... >[Dieter] >> When a process forks the complete state, including file descriptors, >> threads and memory state is copied and the new process >> executes in this copied state. >> We now have 2 "asyncore" threads waiting for the same events. > >A problem is that it's *not* the case that a POSIX fork() clones all >threads. Only the thread calling fork() exists in the child process. >There's a brief but clear discussion of that here: > >http://www.opengroup.org/onlinepubs/009695399/functions/fork.html > >POSIX doesn't even have a way to *ask* that all threads be duplicated, for >reasons explained there. > >Last I heard, Dieter was running LinuxThreads, which fail to meet the POSIX >thread spec in several respects. But, AFAICT, fork() under LinuxThreads is >the same as POSIX in this particular respect (since threads are distinct >processes under LinuxThreads, it would be bizarre if a fork() cloned >multiple processes!). I believe native Solaris threads act as Dieter >describes, though (fork() clones all native Solaris threads). > >Dieter, can you clarify which OS(es) and thread package(s) you're using >here? > Do the things you're doing that call fork() (directly or indirectly) >actually run from the thread running asyncore.loop()? The problem occured in a ZEO client which called "asyncore.poll" in the forked subprocess. This "poll" deterministically stole ZEO server invalidation messages from the parent. I read the Linux "fork" manual page and found: fork creates a child process that differs from the parent process only in its PID and PPID, and in the fact that resource utilizations are set to 0. File locks and pending signals are not inherited. ... The fork call conforms to SVr4, SVID, POSIX, X/OPEN, BSD 4.3 I concluded that if the only difference is in the PID/PPID and resource utilizations, there is no difference in the threads between parent and child. This would mean that the wide spread "asyncore.mainloop" threads could suffer the same message loss and message duplication. I did not observe a message loss/duplication in any application with an "asyncore.mainloop" thread. Maybe, the Linux "fork" manual page is only not precise with respect to threads and the problem does not occur in applications with a standard "asyncore.mainloop" thread. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Plone-users] Re: [ZODB-Dev] [Warning] Zope/ZEO clients: subprocesses can lead to non-deterministic message loss
Paul Winkler wrote at 2004-6-25 17:41 -0400: >On Fri, Jun 25, 2004 at 07:23:19PM +0200, Dieter Maurer wrote: >> ATTENTION: Crosspost -- Reply-To set to '[EMAIL PROTECTED]' >> >> Today, I hit a nasty error. >> >> The error affects applications under Unix (and maybe Windows) which >> >> * use an "asyncore" mainloop thread (and maybe other asyncore applications) >> >> Zope and many ZEO clients belong to this class >> >> and >> >> * create subprocesses (via "fork" and "system", "popen" or friends if >> they use "fork" internally (they do under Unix but I think not >> under Windows)). > >Hm. this applies to external methods and product code that makes >these calls? Discussion in "[EMAIL PROTECTED]" suggests that this problem cannot occur as described with a POSIX fork implementation. I observed the problem in a different setup and concluded from the Linux "fork" manual page only that Zope's "asyncore.mainloop" thread will suffer from the same problem. I did not observe the behaviour in this setup and it is well possible that it cannot occur. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Warning] Zope/ZEO clients: subprocesses can lead to non-deterministic message loss
ATTENTION: Crosspost -- Reply-To set to '[EMAIL PROTECTED]' On Friday, I reported a bug that can cause non-deterministic message loss and duplication of messages in forking applications with an "asyncore" mainloop thread. Unfortunately, the proposed workararound does not work for various reasons (as you may already have recognized and reported): * it modifies global variables without protection which is a receipe for desaster in a multi-threaded environment * it resets state that is already activated. Therefore, it is not effective in preventing the main problem. * when applied for "system", it blocks Zope until the the call returns which may be far too long. I am working at another work around. The main ideas is: * Inform "asyncore" about the actor for which its mainloop should execute. * When the actor is set, "asyncore" calls handlers only for this actor and does nothing otherwise. The problem: what is an "actor"? The most natural choice would be the process, identified via its process id. However, under Linux, the process id may not identify the process but the thread. I am still looking for an adequate, platform independent "actor" definition. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RE: [ZODB-Dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss:MAYBE NOT
hello tim, thanks for the clarification below and also the pointers to the posix behaviour of fork. The Warning about Zope/ZEO clients in the subject line certainly caused some alarm bells to go off. I am assuming now that dieters description below of using forks does not gel with the ZOPE/ZEO process model i.e there are no forks being called within the code to cause the asyncore thread to be cloned ( even if one were using a non posix compliant thread lib like native solaris it would still be a non issue ). At least thats how I see it so far ! please let me know if anybody thinks otherwise :) Regards Sathya [sathya] so can we safely assume that zeo does not mix the asyncore implementation with forks or threads and hence does not suffer from the "child concurrently operating on sockets along with parent" syndrome that dieter is experiencing ? appreciate any clarifications. It's normal for a ZEO application to run asyncore in its own thread. I don't really understand what Dieter is seeing, though: [Dieter] When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 "asyncore" threads waiting for the same events. A problem is that it's *not* the case that a POSIX fork() clones all threads. Only the thread calling fork() exists in the child process. There's a brief but clear discussion of that here: http://www.opengroup.org/onlinepubs/009695399/functions/fork.html POSIX doesn't even have a way to *ask* that all threads be duplicated, for reasons explained there. Last I heard, Dieter was running LinuxThreads, which fail to meet the POSIX thread spec in several respects. But, AFAICT, fork() under LinuxThreads is the same as POSIX in this particular respect (since threads are distinct processes under LinuxThreads, it would be bizarre if a fork() cloned multiple processes!). I believe native Solaris threads act as Dieter describes, though (fork() clones all native Solaris threads). Dieter, can you clarify which OS(es) and thread package(s) you're using here? Do the things you're doing that call fork() (directly or indirectly) actually run from the thread running asyncore.loop()? That's the only way a POSIX fork() should end up with a clone of the thread running the asyncore loop. But then the subsequent exec (if you're doing system() or popen()) should wipe out the cloned asyncore code before the child process returns to asyncore. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: CatalogBrains since Zope2.7.1b1
Optional arguments will still allow untrusted code to bypass security checks. Yes, that's true. Here are three solutions to this, two of which do not involve catalog changes: - Use a proxy role on the script that invokes getObject which grants the permissions needed. - Use self.unrestrictedTraverse(brain.getPath()) from trusted code - Add a private method: unrestrictedGetObject() to the catalog brain API which does no security checking, but is inaccessible to untrusted code. I think the last one is a good idea and I will implement it. The other two are available options for now. Ok, I think it will be useful. Until then, the second option is a good solution for me. Thanks a lot for the suggestion. Regards Santi Camps ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] RE: [ZODB-Dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss
[sathya] > so can we safely assume that zeo does not mix the asyncore implementation > with forks or threads and hence does not suffer from the "child > concurrently operating on sockets along with parent" syndrome that > dieter is experiencing ? appreciate any clarifications. It's normal for a ZEO application to run asyncore in its own thread. I don't really understand what Dieter is seeing, though: [Dieter] > When a process forks the complete state, including file descriptors, > threads and memory state is copied and the new process > executes in this copied state. > We now have 2 "asyncore" threads waiting for the same events. A problem is that it's *not* the case that a POSIX fork() clones all threads. Only the thread calling fork() exists in the child process. There's a brief but clear discussion of that here: http://www.opengroup.org/onlinepubs/009695399/functions/fork.html POSIX doesn't even have a way to *ask* that all threads be duplicated, for reasons explained there. Last I heard, Dieter was running LinuxThreads, which fail to meet the POSIX thread spec in several respects. But, AFAICT, fork() under LinuxThreads is the same as POSIX in this particular respect (since threads are distinct processes under LinuxThreads, it would be bizarre if a fork() cloned multiple processes!). I believe native Solaris threads act as Dieter describes, though (fork() clones all native Solaris threads). Dieter, can you clarify which OS(es) and thread package(s) you're using here? Do the things you're doing that call fork() (directly or indirectly) actually run from the thread running asyncore.loop()? That's the only way a POSIX fork() should end up with a clone of the thread running the asyncore loop. But then the subsequent exec (if you're doing system() or popen()) should wipe out the cloned asyncore code before the child process returns to asyncore. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
