Re: Semaphore Techniques
Carl Banks pavlovevide...@gmail.com (CB) wrote: CB On Jul 29, 7:14 am, Piet van Oostrum p...@cs.uu.nl wrote: Carl Banks pavlovevide...@gmail.com (CB) wrote: CB On Jul 28, 3:15 pm, John D Giotta jdgio...@gmail.com wrote: I'm looking to run a process with a limit of 3 instances, but each execution is over a crontab interval. I've been investigating the threading module and using daemons to limit active thread objects, but I'm not very successful at grasping the documentation. Is it possible to do what I'm trying to do and if so anyone know of a useful example to get started? CB It seems like you want to limit the number of processes to three; the CB threading module won't help you there because it deals with threads CB within a single process. CB What I'd do is to simply run the system ps to see how many processes CB are running (ps is pretty versatile on most systems and can find CB specifically targeted processes like you program), and exit if there CB are already three. That will surely run into some race conditions. CB What, the OS might not have gotten around to update the process table CB to include a process started minutes ago? (He said he was starting CB the processes over crontab intervals, not that he communicated what he CB wanted well.) No but between the time you get the ps output and decide not to start a new process one of the processes might have exited. As I said it probably is not a big deal, but you (he) should be aware of it I think. The other possible race condition: two processes starting at approximately the same time and both not detecting the other will probably not occur because of the time distance between starting the processes by cron. Unless the system is so busy that ps takes a lng time. The problem is similar to the sleeping barber problem (3 sleeping barbers actually). -- Piet van Oostrum p...@cs.uu.nl URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4] Private email: p...@vanoostrum.org -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
John D Giotta jdgio...@gmail.com wrote: I'm looking to run a process with a limit of 3 instances, but each execution is over a crontab interval. I've been investigating the threading module and using daemons to limit active thread objects, but I'm not very successful at grasping the documentation. Is it possible to do what I'm trying to do and if so anyone know of a useful example to get started? If you want a simple, cross platform way of doing it, then bind each process to a different local tcp port. Make a list of 3 ports, and try binding to each port in turn. If you can't find a port to bind to then there are already 3 instances running. Something like this import socket PORTS = range(1,10003) lock_sock = None def lock_process(_locks = []): for port in PORTS: sock = socket.socket() try: sock.bind((localhost, port)) except socket.error, e: sock = None else: _locks.append(sock) break else: raise Exception(Too many instances of me running) for i in range(5): print Trying,i+1 lock_process() Which prints Trying 1 Trying 2 Trying 3 Trying 4 Traceback (most recent call last): File stdin, line 20, in module File stdin, line 16, in lock_process Exception: Too many instances of me running You could do the same thing with lock files also very easily... -- Nick Craig-Wood n...@craig-wood.com -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
Carl Banks pavlovevide...@gmail.com (CB) wrote: CB On Jul 28, 3:15 pm, John D Giotta jdgio...@gmail.com wrote: I'm looking to run a process with a limit of 3 instances, but each execution is over a crontab interval. I've been investigating the threading module and using daemons to limit active thread objects, but I'm not very successful at grasping the documentation. Is it possible to do what I'm trying to do and if so anyone know of a useful example to get started? CB It seems like you want to limit the number of processes to three; the CB threading module won't help you there because it deals with threads CB within a single process. CB What I'd do is to simply run the system ps to see how many processes CB are running (ps is pretty versatile on most systems and can find CB specifically targeted processes like you program), and exit if there CB are already three. That will surely run into some race conditions. If the limit of 3 processes is soft then that wouldn't be a big deal, however. -- Piet van Oostrum p...@cs.uu.nl URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4] Private email: p...@vanoostrum.org -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
I'm working with up to 3 process session per server, each process running three threads. I was wishing to tie back the 3 session/server to a semaphore, but everything (and everyone) say semaphores are only good per process. -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
That was my original idea. Restricting each process by pid: #bash procs=`ps aux | grep script.pl | grep -v grep | wc -l` if [ $procs -lt 3 ]; then python2.4 script.py config.xml else exit 0 fi -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
John D Giotta schrieb: I'm working with up to 3 process session per server, each process running three threads. I was wishing to tie back the 3 session/server to a semaphore, but everything (and everyone) say semaphores are only good per process. That's not true. Named semaphores are the best solution for your problem and I said so yesterday. Christian -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
On Jul 29, 7:14 am, Piet van Oostrum p...@cs.uu.nl wrote: Carl Banks pavlovevide...@gmail.com (CB) wrote: CB On Jul 28, 3:15 pm, John D Giotta jdgio...@gmail.com wrote: I'm looking to run a process with a limit of 3 instances, but each execution is over a crontab interval. I've been investigating the threading module and using daemons to limit active thread objects, but I'm not very successful at grasping the documentation. Is it possible to do what I'm trying to do and if so anyone know of a useful example to get started? CB It seems like you want to limit the number of processes to three; the CB threading module won't help you there because it deals with threads CB within a single process. CB What I'd do is to simply run the system ps to see how many processes CB are running (ps is pretty versatile on most systems and can find CB specifically targeted processes like you program), and exit if there CB are already three. That will surely run into some race conditions. What, the OS might not have gotten around to update the process table to include a process started minutes ago? (He said he was starting the processes over crontab intervals, not that he communicated what he wanted well.) Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Semaphore Techniques
I'm looking to run a process with a limit of 3 instances, but each execution is over a crontab interval. I've been investigating the threading module and using daemons to limit active thread objects, but I'm not very successful at grasping the documentation. Is it possible to do what I'm trying to do and if so anyone know of a useful example to get started? -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
John D Giotta schrieb: I'm looking to run a process with a limit of 3 instances, but each execution is over a crontab interval. I've been investigating the threading module and using daemons to limit active thread objects, but I'm not very successful at grasping the documentation. Is it possible to do what I'm trying to do and if so anyone know of a useful example to get started? Since you are talking about crontab I assume that you are on an os that supports pthreads. You problem can easily be solved with a named semaphore (see sem_open(3) and sem_overview(7)). Unfortunately Python doesn't expose named semaphores. The multiprocessing library uses named semaphores but you can't set the name yourself. You have to write your own C wrapper or search on pypi and through Google. If you are going to write your own semaphore I highly recommend Cython. Christian -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
On Jul 28, 2009, at 7:19 PM, Christian Heimes wrote: John D Giotta schrieb: I'm looking to run a process with a limit of 3 instances, but each execution is over a crontab interval. I've been investigating the threading module and using daemons to limit active thread objects, but I'm not very successful at grasping the documentation. Is it possible to do what I'm trying to do and if so anyone know of a useful example to get started? Since you are talking about crontab I assume that you are on an os that supports pthreads. You problem can easily be solved with a named semaphore (see sem_open(3) and sem_overview(7)). Unfortunately Python doesn't expose named semaphores. The multiprocessing library uses named semaphores but you can't set the name yourself. You have to write your own C wrapper or search on pypi and through Google. If you are going to write your own semaphore I highly recommend Cython. My POSIX IPC extension permits manipulation of interprocess semaphores: http://semanchuk.com/philip/posix_ipc/ There's also one for SysV IPC: http://semanchuk.com/philip/sysv_ipc/ Enjoy P -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
John D Giotta jdgio...@gmail.com writes: I'm looking to run a process with a limit of 3 instances, but each execution is over a crontab interval. I've been investigating the threading module and using daemons to limit active thread objects, but I'm not very successful at grasping the documentation. Is it possible to do what I'm trying to do and if so anyone know of a useful example to get started? Does it have to be built into the tool, or are you open to handling the restriction right in the crontab entry? For example, a crontab entry like: * * * * * test `pidof -x script.py | wc -w` -ge 4 || path/script.py should attempt to run script.py every minute (adjust period as required) unless there are already four of them running. And if pidof isn't precise enough you can put anything in there that would accurately check your processes (grep a ps listing or whatever). This works because if the test expression is true it returns 0 which terminates the logical or (||) expression. There may be some variations based on cron implementation (the above was tested against Vixie cron), but some similar mechanism should be available. If you wanted to build it into the tool, it can be tricky in terms of managing shared state (the count) amongst purely sibling/cooperative processes. It's much easier to ensure no overlap (1 instance), but once you want 'n' instances you need an accurate process-wide counter. I'm not positive, but don't think Python's built-in semaphores or shared memory objects are cross-process. (Maybe something in multiprocessing in recent Python versions would work, though they may need the sharing processes to all have been executed from a parent script) I do believe there are some third party interfaces (posix_ipc, shm/shm_wrapper) that would provide access to posix shared-process objects. A semaphore may still not work as I'm not sure you can obtain the current count. But you could probably do something with a shared memory counter in conjunction with a mutex of some sort, as long as you were careful to clean it up on exit. Or, you could stick PIDs into the shared memory and count PIDs on a new startup (double checking against running processes to help protect against process failures without cleanup). You could also use the filesystem - have a shared directory where each process dumps its PID, after first counting how many other PIDs are in the directory and exiting if too many. Of course all of these (even with a PID check) are risky in the presence of unexpected failures. It would be worse with something like C code, but it should be reasonably easy to ensure that your script has cleanup code even on an unexpected termination, and it's not that likely the Python interpreter itself would crash. Then again, something external could kill the process. Ensuring accuracy and cleanup of shared state can be non-trivial. You don't mention if you can support a single master daemon, but if you could, then it can get a little easier as it can maintain and protect access to the state - you could have each worker process maintain a socket connection of some sort with the master daemon so it could detect when they terminate for the count, and it could just reject such connections from new processes if too many are running already. Of course, if the master daemon goes away then nobody would run, which may or may not be an acceptable failure mode. All in all, unless you need the scripts to enforce this behavior even in the presence of arbitrary use, I'd just use an appropriate crontab entry and move on to other problems :-) -- David -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
On Jul 28, 3:15 pm, John D Giotta jdgio...@gmail.com wrote: I'm looking to run a process with a limit of 3 instances, but each execution is over a crontab interval. I've been investigating the threading module and using daemons to limit active thread objects, but I'm not very successful at grasping the documentation. Is it possible to do what I'm trying to do and if so anyone know of a useful example to get started? It seems like you want to limit the number of processes to three; the threading module won't help you there because it deals with threads within a single process. What I'd do is to simply run the system ps to see how many processes are running (ps is pretty versatile on most systems and can find specifically targeted processes like you program), and exit if there are already three. If you really are talking about multiple threads on a single server process, then you want to use a thread pool (create three threads, and give them tasks as necessary). But you'll have to have a way for the process started by crontab to communicate with the server. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
