[Zope-dev] commercial: Looking for CTO and multiple Zope and python developers
Dear friends in the Zope community, ZeOmega is growing. Given the aging populations in some parts of the world and health care reform in the US, we are seeing tremendous interest in our health care management solutions. We are almost doubling in size from 150 to 300 employees. We are hiring both in Dallas and Bangalore, India. We are looking for a CTO from the Zope community (who is available daylight time in the US ) as well as Zope and python developers who can be available during Indian standard time or US time zones during business hours. Please let us know at techjobs zeomega net with your interests and expectations. Best Regards -- Sathya Sam Rangaswamy Founder CEO ZeOmega 3010 Gaylord Parkway Suite 210 Frisco, TX 75034 Office:214-618-9880 ext 8002 Fax: 214-975-1258 Mobile:214-733-3467 s...@zeomega.com www.zeomega.com Proven. Progressive. Partner. *** CONFIDENTIALITY NOTICE: This message (including any attachments) may contain ZeOmega's confidential information, protected by law. Forwarding it to individuals, other than those with a need to know, without the permission of the sender, is prohibited. This message is intended for specific individual(s) or entity to whom it is intended even if addressed incorrectly. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure,copying, or distribution of this message or taking any action based upon it, is strictly prohibited. *** ___ ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] running out of available tcp ports on a zeo cluster
Hello folks, We are testing a zeo cluster in a vmware environment (zeo clients are running on multiple windows 2003 server guest vms). We are using Apache as a load balancer using mod_rewrite (on a separate windows 2003 guest vm ). We are using load runner to generate load After hitting about 40 concurrent users we are getting tcp errors on apache indicating that there are no available ports for accepting connections. It seems like the connections are hanging on for a longer period of time causing the available ports to get exhausted. The errors get worse on increasing the concurrent users. It seems as though there is a problem recycling ports for new connections. Has anybody seen this problem before ? Is there an issue with connections between apache and zope not closing down ? We have run similar tests on linux before and have not faced this problem ( I know, but the client wants windows :( ) Any feedback will be greatly appreciated. Regards sathya -- ZeOmega LLC Integrated Care Management software Frisco Tx 214-618-9880 www.zeomega.com -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope Foundation membership drive
Hello folks As some of you may know, The Zope foundation is set up and more details are available at www.zopefoundation.org Some of the membership rosters (especially solution providers,associate members, consumer members ) looks bare and not reflective of the community we have. Lets try to make the foundation stronger by growing the membership and show our support for Zope ! Best Regards Sathya ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] looking for Zope developers : Dallas Fort worth metroplex
Hello folks, ZeOmega (www.zeomega.com )a sponsor member of the python software foundation and a solutions provider member of the Zope foundation, a company built entirely on FOSS technology is looking for Zope/Plone programmers to work in our office in Frisco Tx. Our healthcare management product suite is very successful in the US care management market and is growing rapidly. It is used for example in managing Chronic heart failure, diabetes, asthma patients and chronic care improvement in several states. We also provide consulting services to companies such as Gartner and National Informatics Government of India . We use Python Zope and Plone on Linux and databases include MySQL PostgreSQL, Oracle and SQL server as mandated by our clients. We also work on open source projects and play an active role in the Python community by aiding advocacy efforts and sponsor international conferences such as Plone conference, EuroPython and PyCon. We are looking for Zope Plone developers who can add to to our product development efforts. We exclusively use python for all our SW development. Salary is based on experience. Please do send in your resumes to careers at zeomega dot com please include Zope Plone programmer in the subject. Best Regards Sathya sam Rangaswamy Founder and CEO ZeOmega ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] REQUEST: Looking for Zope Plone developers in the US
Hello Folks, ZeOmega is looking for Zope developers. CMF/Plone experience would be nice. We use Zope in our products for the health care management industry. The domain is medical informatics and we are doing some exciting work in the area of disease , chronic care management for public and commercial health populations and also injury management in workers compensation . We offer a good salary package that includes health benefits. The positions are in Dallas ,Texas. We will also consider telecommuting options for the right candidates. Please forward your resume to [EMAIL PROTECTED] You can find out all about us at www.zeomega.com Best Regards Sathya -- = CEO ZeOmega Open Minds' Open Solutions #2591 Dallas Parkway Suite 408 Frisco TX, 75034 214-618-9880 (O) 214-975-1258 (F) 214-733-3467 (M) http://www.zeomega.com == ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Commercial: looking for Zope/Plone developer
We are looking for a developer with atleast 3 years of development experience with knowledge of Zope and Plone to work in our Dallas Texas Office. We use Zope Plone for product development in managed care, workers comp and have an install base of 500,000 lives. We can consider telecommuters after atleast 6 months of onsite work. Please forward resumes to [EMAIL PROTECTED] Best Regards Sathya ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] INGRES R3 . If you plan to use Ingres please read this
http://opensource.ca.com/projects/ Chris Withers wrote: sathya wrote: Since Ingres Plone and APE have some things in common I thought I should post this on this forum. Your experience may be different and the above is just FYI . I hope things get better in the days ahead. Out of interest, where are APE and Ingres's python support downloadable from? cheers, Chris ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] INGRES R3 . If you plan to use Ingres please read this
Folks, Before putting Ingres R3 into production please make sure you can get support for it. We went live on a major implementation using zope 2.7.3 ZEO and Ingres R3 (int109) on RedHat AS 3.0 for a large customer (TPA with 200,000 lives) It has been a nightmare. The memory leaks in the ingres Python DBI are just the beginning. We fixed a few and submitted patches On the DB server we see the following in the log file Memory corruption errors Segvs iowait hits 100 % and system comes to a grinding stop. It could be our code (which worked well on postgres) but I am inclined to think its not the main cause We of course contacted CA for support and spent 4 days to learn that they are just getting the system in to take support calls and sell support on R3. The last time we had an issue with postgres a guy from command prompt inc logged in within 30 minutes and fixed our problem. Since Ingres Plone and APE have some things in common I thought I should post this on this forum. Your experience may be different and the above is just FYI . I hope things get better in the days ahead. Regards Sathya ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Making a ZSQL.DA fully multi-threaded?
I dont see a data loss problem either (unless there are hardware failures) If a connection is dropped due to inactivity it should not affect any transactions going to occur in the future as a reconnect is issued before submitted new transactions. However connections getting dropped due to transport layer problems is a different matter. if a connection gets dropped by either side after making the request then it should be detected at either end and the transaction aborted (assuming a well behaved transport layer). There cannot be any loss in data in this scenario. This is expected behaviour with any RDBMS connection infrastructure, i.e If the transport layer guarantees delivery and employs the customary process of error detection and informs the application there should not be a data inconsistency problem ( unless there is a hardware failure ) I beleive Whether or NOT the transaction is resubmitted should be a decision on part of the application and not the DA or Zpublisher. The application can make a decision based on the circumstance and then if it chooses to do so issue the conflict error to invoke the Zpublisher machinery. Brad Clements wrote: On 23 Jul 2004 at 20:08, Dieter Maurer wrote: The bad sequence can look as follows: * Zope starts a request (and thereby a transaction) * The request sends a modifying request to a relational database * The connection is lost; the former modification is discarded as the database performs an automatic abort on connection close The former modification cannot be lost because it was commited by the DA as part of the previous transaction. At least, gvibDA performs a database commit as part of the overall transaction machinery. So again, I don't see the loss.. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Making a ZSQL.DA fully multi-threaded?
sqlrelay seems to hold some promise for connection pooling although I am not sure if theres support for sapdb Brad Clements wrote: On 16 Jul 2004 at 8:57, Chris Withers wrote: This often means that many more database connections are opened than are actually necessary. And if you start timing out inactive database connectiosn on the database side, you're in for a world of pain and suffering. I branched ZOracleDA some time ago to move away from this model for that very reason... What's the actual problem you're experiencing? I have set the SAPDB timeout to 32400 seconds. However when running with more than 1 thread I still get mysterious segfaults in the sapdbapi component. The SAP folks say their adapter is multi-thread capable, but I've come to believe that only means that it's ok to open a connection in one thread and use it in another, but NOT to have 2 threads make requests on the same connection at the same time. Also, two threads can' t open a connection at the same time because their connection table management isn't thread safe. What I need, I suppose, is a DB connection pool mechanism of some kind. I also have Python extensions that need a connection as well. Getting the actual database connection from a DA stinks (ie, it's not really possible). I'd like to see an API extension to ZopeDA's to allow Pythonscripts, Products and external methods get an actual db connection. But that I suppose is a problem to solve later. I think I will take a look at ZOracleDA to see what you did. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Making a ZSQL.DA fully multi-threaded?
sathya wrote: we have been looking at psycopg for postgres. zope opens atleast one connection per zope thread. I have obeserved the same for mysql as well. Are you creating your own threads within zope ? in which case you have to do the locking yourself Brad Clements wrote: I am using ZsapdbDA with SAPDB (now MaxDB) and have run into problems when Zope has more than one thread. The folks at SAP suggest that there should be one database connection per thread, rather than sharing one connection across threads. I haven't yet looked at the Zsapdb source, but I think it has only one connection for all threads. Is there a document somewhere that explains how to make variables thread private in Zope? I vaguely recall something about one cache per thread, or that variables are read- only shared unless updated by a thread, and then they're copied.. This is probably quite wrong, but obviously I just don't understand how data and threading works in Zope. So, I'm looking for a document that explains this clearly. How to have data that is not shared between threads (perhaps a kind of connection pool thingy). Also, I have other projects where I really want to have just one object of it's kind no matter how many threads. For example, gvibDA uses Thunked_TM for this, but that doesn't really work with ZSQLDA because I've seen that multiple connect calls are made, even with the THUNKED_TM. Is there a Zen Of document about this somewhere? Thanks ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RE: [ZODB-Dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss:MAYBE NOT
hello tim, thanks for the clarification below and also the pointers to the posix behaviour of fork. The Warning about Zope/ZEO clients in the subject line certainly caused some alarm bells to go off. I am assuming now that dieters description below of using forks does not gel with the ZOPE/ZEO process model i.e there are no forks being called within the code to cause the asyncore thread to be cloned ( even if one were using a non posix compliant thread lib like native solaris it would still be a non issue ). At least thats how I see it so far ! please let me know if anybody thinks otherwise :) Regards Sathya [sathya] so can we safely assume that zeo does not mix the asyncore implementation with forks or threads and hence does not suffer from the child concurrently operating on sockets along with parent syndrome that dieter is experiencing ? appreciate any clarifications. It's normal for a ZEO application to run asyncore in its own thread. I don't really understand what Dieter is seeing, though: [Dieter] When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 asyncore threads waiting for the same events. A problem is that it's *not* the case that a POSIX fork() clones all threads. Only the thread calling fork() exists in the child process. There's a brief but clear discussion of that here: http://www.opengroup.org/onlinepubs/009695399/functions/fork.html POSIX doesn't even have a way to *ask* that all threads be duplicated, for reasons explained there. Last I heard, Dieter was running LinuxThreads, which fail to meet the POSIX thread spec in several respects. But, AFAICT, fork() under LinuxThreads is the same as POSIX in this particular respect (since threads are distinct processes under LinuxThreads, it would be bizarre if a fork() cloned multiple processes!). I believe native Solaris threads act as Dieter describes, though (fork() clones all native Solaris threads). Dieter, can you clarify which OS(es) and thread package(s) you're using here? Do the things you're doing that call fork() (directly or indirectly) actually run from the thread running asyncore.loop()? That's the only way a POSIX fork() should end up with a clone of the thread running the asyncore loop. But then the subsequent exec (if you're doing system() or popen()) should wipe out the cloned asyncore code before the child process returns to asyncore. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss: would like to duplicate this
Tim Peters wrote: just to add my 2 cents I have been looking at zserver code, the only time fork or system (which i presume invokes execve ) calls are used are at startup to either a) run a cmdline b) daemonize heres a snip from strace output strace -o strace.txt -f -e trace=fork,execve ./runzope on a zeo instance 15298 execve(./runzope, [./runzope], [/* 23 vars */]) = 0 15298 execve(/var/dev/vision/python233/bin/python, [/var/dev/vision/python233/bin/py..., /var/dev/vision/Zope27/lib/python which creates a child process with id 15299 at this point asyncore main loop thread has not even started so it is safe to assume that the parent does not start the asyncore loop for any servers created but happens in the forked child . which means we probably cannot have multiple asyncore mainloops running The zeoclient causes threads to be created but there are no forks or system calls as far as I can tell (or strace for that matter) Can you please point out where in the zeo code does forking occur ? I will try and duplicate this condition. -ty sathya [Dieter Maurer] The problem occured in a ZEO client which called asyncore.poll in the forked subprocess. This poll deterministically stole ZEO server invalidation messages from the parent. I'm sorry, but this is still too vague to guess what happened. - Which operating system was in use? - Which thread package? - In the ZEO client that called fork(), did it call fork() directly, or indirectly as the result of a system() or popen() call? Or what? I'd like to understand a specific failure before rushing to generalization. - In the ZEO client that called fork() (whether directly or indirectly), was fork called *from* the thread running ZEO's asyncore loop, or from a different thread? I read the Linux fork manual page and found: fork creates a child process that differs from the parent process only in its PID and PPID, and in the fact that resource utilizations are set to 0. File locks and pending signals are not inherited. ... The fork call conforms to SVr4, SVID, POSIX, X/OPEN, BSD 4.3 If it conforms to POSIX (as it says it does), then fork() also has to satisfy the huge list of requirements I referenced before: http://www.opengroup.org/onlinepubs/009695399/functions/fork.html That page is the current POSIX spec for fork(). I concluded that if the only difference is in the PID/PPID and resource utilizations, there is no difference in the threads between parent and child. Except that if you're running non-POSIX LinuxThreads, a thread *is* a process (there's a one-to-one relationship under LinuxThreads, not the many-to-one relationship in POSIX), in which case no difference in threads is trivially true. This would mean that the wide spread asyncore.mainloop threads could suffer the same message loss and message duplication. That's why all sane wink threading implementations do what POSIX does on a fork(). fork() and threading don't really mix well under POSIX either, but the fork+exec model for starting a new process is an historical burden that bristles with subtle problems in a multithreaded world; POSIX introduced posix_spawn() and posix_spawnp() for sane(r) process creation, ironically moving closer to what most non-Unix systems have always done to create a new process. I did not observe a message loss/duplication in any application with an asyncore.mainloop thread. I don't understand. You said that you *have* seen message loss/duplication in a ZEO client, and I assume the ZEO client was running an asyncore thread. If so, then you have seen loss/duplication in an application with an asyncore thread. Or are you saying that you haven't seen loss/duplication under the specific Linux flavor whose man page you quoted, but have seen it under some other (so far unidentified) system? Maybe, the Linux fork manual page is only not precise with respect to threads and the problem does not occur in applications with a standard asyncore.mainloop thread. That fork manpage is clearly missing a mountain of crucial details (or it's not telling the truth about being POSIX-compliant). fork() is historically poorly documented, though. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- === CEO ZeOmega Open minds' Open Solutions Plano, Texas, USA Bangalore, India 972-731-6750 (O) 214-733-3467 (M) http://www.zeomega.com Open source content management and workflow solutions ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http
Re: [Zope-dev] RE: [ZODB-Dev] [Warning] Zope/ZEO clients: subprocesses can lead tonon-deterministic message loss
Tim Peters wrote: hello tim, so can we safely assume that zeo does not mix the asyncore implementation with forks or threads and hence does not suffer from the child concurrently operating on sockets along with parent syndrome that dieter is experiencing ? appreciate any clarifications. Regards sathya [Dieter Maurer] ATTENTION: Crosspost -- Reply-To set to '[EMAIL PROTECTED]' Which I've honored. Today, I hit a nasty error. The error affects applications under Unix (and maybe Windows) which * use an asyncore mainloop thread (and maybe other asyncore applications) Zope and many ZEO clients belong to this class Note a possible complication: ZEO monkey-patches asyncore, replacing its loop() function with one of its own. This is done in ZODB's ThreadedAsync/LoopCallback.py. and * create subprocesses (via fork and system, popen or friends if they use fork internally (they do under Unix but I think not under Windows)). It may be an issue under Cygwin, but not under native Windows, which supports no way to clone a process; file descriptors may get inherited by child processes on Windows, but no code runs by magic. The error can cause non-deterministic loss of messages (HTTP requests, ZEO server responses, ...) destined for the parent process. It also can cause the same output to be send several times over sockets. The error is explained as follows: asyncore maintains a map from file descriptors to handlers. The asyncore main loop waits for any file descriptor to become active and then calls the corresponding handler. There's a key related point, though: asyncore.loop() terminates if it sees that the map has become empty. This appears to have consequences for the correctness of workarounds. For example, this is Python's current asyncore loop (the monkey-patched one ZEO installs is similar in this respect): def loop(timeout=30.0, use_poll=False, map=None): if map is None: map = socket_map if use_poll and hasattr(select, 'poll'): poll_fun = poll2 else: poll_fun = poll while map: poll_fun(timeout, map) If map becomes empty, loop() exits. When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 asyncore threads waiting for the same events. Sam Rushing created asyncore as an alternative to threaded approaches; mixing asyncore with threads is a nightmare; throwing forks into the pot too is a good working definition of hell wink. File descriptors are shared between parent and child. When the child reads from a file descriptor from its parent, it steals the corresponding message: the message will not reach the parent. While file descriptors are shared, memory state is separate. Therefore, pending writes can be performed by both parent and child -- leading to duplicate writes to the same file descriptor. A workaround it to deactivate asyncore before forking (or system, popen, ...) and reactivate it afterwards: as exemplified in the following code: from asyncore import socket_map saved_socket_map = socket_map.copy() socket_map.clear() # deactivate asyncore As noted above, this may (or may not) cause asyncore.loop() to plain stop, in parent and/or in child process. If there aren't multiple threads, it's safe, but presumably you have multiple threads in mind, in which case behavior seems unpredictable (will the parent process's thread running asyncore.loop() notice that the map has become empty before the code below populates the map again? asyncore.loop() will or won't stop in the parent depending on that timing accident). pid = None try: pid = fork() if (pid == 0): # child # ... finally: if pid != 0: socket_map.update(saved_socket_map) # reactivate asyncore Another approach I've seen is to skip mucking with socket_map directly, and call asyncore.close_all() first thing in the child process. Of course that's vulnerable to vagaries of thread scheduling too, if asyncore is running in a thread other than the one doing the fork() call. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] is threading in zope useful ?
[christian] That said , since ZEO ends up doing multiple python interpreters its possibly the more reliable approach to take advantage of SMP linux. please point out any flaws in my thought process . SMP is not doing any good for you when not running multiple Python processes. The GIL doesn't work per CPU but for the whole system. You very likely will end up locking your CPUs 2-n on a N-way system due to the GIL. I would think the GIL is relevant per process and not the whole system. If I ran multiple python processes on a N-way system they would not deadlock on the GIL as they dont contend for the same GIL. ZEO actually does the trick, but you should use CPU binding for the processes to make sure they won't affect other processors. What about threads on linux ? will binding a process to a CPU also mean all threads spawned by the process are also bound to the same processor ? I am not aware of linux support for binding threads to a CPU so I guess it does not matter . Further if extension modules such as RDBMS adapters do not cause the GIL to be freed up ,If I understand tim correctly any calls to only known thread safe APIS will cause the GIL to be unlocked. So python applications using thread safe implementations of MySQL db api will still suffer from the GIL bottleneck. bummer. time to deploy ZEO or hack the db adapter to be GIL aware ? I wonder if there is a way to inform python about thread safe api calls so that it can release the GIL thanks much for this discussion regards sathya Cheers, Christian ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] is threading in zope useful ?
Dario Lopez-Kästen wrote: thanks much for the info. turns out that the updated kernel rpms for RH8 and RH 9 2.4.20-28.8smp contain the patch for cpu affinity as well at this point seems like a good idea to deploy ZEO and bind each client to a processor and test it out ,hopefully with a 4way smp we should see 4X improvement in performance !!! (We hope ) one other point [dario] As I understand it. the GIL is not a problem *at all* unless you happen to run on smp machines. There are ofcourse optimisations to when and where to relase the GIL, but in general python takes care of that all by itself. How about when you use extension modules that python does not have apriori knowledge about (it only optimizes when it encounters known threadsafe api calls like the ones in clib or socket lib ). So, as I understand it if a python database module invokes a native library call python will not yeild the GIL. So if your zope app is making heavy use of an RDBMS we would still see zope threads waiting on the GIL until the long running thread fetched DB results and performed network IO to send back the response at which point the GIL would be released. Some GIL aware extesnion modules such as the mxodbc DA from egenix free the GIL for query API (http://www.egenix.com/mailman-archives/egenix-users/2003-February/000200.html). That said it is clear that using ZEO and CPU affinity guarantees the best utilization of SMP for your zope app Thanks tim ,dario and christian regards sathya sathya wrote: What about threads on linux ? will binding a process to a CPU also mean all threads spawned by the process are also bound to the same processor ? I am not aware of linux support for binding threads to a CPU so I guess it does not matter . For a full discussion of this and oter SMP issues, see this article on Zope org. http://www.zope.org/Members/glpb/solaris/multiproc It is possible to do CPU-affinity on Linux, and nowadyas there are even userspace tools that do this. http://www.hpl.hp.com/research/linux/kernel/o1-openmp.php http://www.tech9.net/rml/ http://kerneltrap.org/node/view/335 CPU.-affinity is avilable in RH 3.0 per default, and the same tools are avialable as a set of patches for various kernel versions. *In general, on an SMP box, Python processes MUST be bound to any one of the CPUs on a amchine* Not the same CPU, but each python process has to stay on the same CPU all the time. As I understand it. the GIL is not a problem *at all* unless you happen to run on smp machines. There are ofcourse optimisations to when and where to relase the GIL, but in general python takes care of that all by itself. ZEO won't save you from the GIL - ZEO will allow you to scale your Zope beyond the constraints of 1 singel zope process for serving the same content. ZEO implements storage of the data Zope serves in one centralised location (the ZEO storage server) and allows several Zope processes (acting as ZEO clients) to serve/update the same content. These Zope processes can ofcourse be placed in a kluster, each ZEO client in a different machine. /dario ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] is threading in zope really useful ?
greetings, I read somewhere that each zope thread acquires the global python interpreter lock before processing a request and until it releases it the other zope threads are essentially locked out. Does that mean zope threads are eventually serialized and there are no benefits to be gained from concurrent execution ? Any thoughts ? your ideas are much appreciated ty sathya ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] is threading within zope useful ?
greetings, I read somewhere that each zope thread acquires the global python interpreter lock before processing a request and until it releases it the other zope threads are essentially locked out. Does that mean zope threads are eventually serialized and there are no benefits to be gained from concurrent execution ? Any thoughts ? your ideas are much appreciated ty sathya ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] is threading in zope useful ?
greetings, I read somewhere that each zope thread acquires the global python interpreter lock before processing a request and until it releases it the other zope threads are essentially locked out. Does that mean zope threads are eventually serialized and there are no benefits to be gained from concurrent execution ? Any thoughts ? your ideas are much appreciated ty sathya ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] is threading in zope useful ?
Tim Peters wrote: tim thanks much for the explanation. some points below. [sathya] I read somewhere that each zope thread acquires the global python interpreter lock before processing a request and until it releases it the other zope threads are essentially locked out. [tim] No. The GIL effectively serializes threads doing pure computation in Python, but many pieces of the Python implementation release the GIL around calls to known-threadsafe C libraries. The most important examples of that are I/O, of many kinds. For example, if a Python thread opens a socket, the GIL is released around the C-level code that does the low-level socket open, which allows other Python threads to proceed in parallel (of course the thread that does the C-level socket open waits for the latter to finish; it reacquires the GIL after the socket operation is complete and it's ready to go back to interpreting Python bytecodes). Because Zope does a lot of network I/O, this form of parallelism can be very useful for it. [sathya] great ! If I understand correctly, if we had a zope process running on an smp linux machine and doing lots of RDBMS calls we would not be limited by the GIL. In essence threads running on multiple cpus would probably not have to wait on acquiring the GIL as most every page request results in a ZSQL method call which in turn would release the GIL. Therefore SMP may not be hindered by the GIL. That said , since ZEO ends up doing multiple python interpreters its possibly the more reliable approach to take advantage of SMP linux. please point out any flaws in my thought process . again thanks very much for putting it in persepective. it clears up some fog for us ! regards sathya ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
