Re: How to safely maintain a status file

2012-07-09 Thread Plumo
and then on startup read from tmp_file if status_file does not exist. But this seems awkward.         It also violates your requirement -- since the crash could take place with a partial temp file. Can you explain why? My thinking was if crash took place when writing the temp file this

Re: How to safely maintain a status file

2012-07-09 Thread Plumo
Windows doesn't suppport atomic renames if the right side exists.  I suggest that you implement two code paths: if os.name == posix:     rename = os.rename else:     def rename(a, b):         try:             os.rename(a, b)         except OSError, e:             if e.errno != 183:    

Re: How to safely maintain a status file

2012-07-08 Thread Plumo
What are you keeping in this status file that needs to be saved several times per second?  Depending on what type of state you're storing and how persistent it needs to be, there may be a better way to store it. Michael This is for a threaded web crawler. I want to cache what URL's are

Re: asynchronous downloading

2012-02-23 Thread Plumo
My current implementation works fine below a few hundred threads. But each thread takes up a lot of memory so does not scale well. I have been looking at Erlang for that reason, but found it is missing useful libraries in other areas. -- http://mail.python.org/mailman/listinfo/python-list

Re: asynchronous downloading

2012-02-23 Thread Plumo
that example is excellent - best use of asynchat I have seen so far. I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat. -- http://mail.python.org/mailman/listinfo/python-list

Re: generate Windows exe on Linux

2012-02-22 Thread Plumo
thanks Jérôme. Closest I have found is pyinstaller added support for cross-compiling a year ago by mounting a Windows partition on Linux: https://groups.google.com/forum/?fromgroups#!topic/pyinstaller/KISZP5sHCWg But it was not stable so will be removed:

asynchronous downloading

2012-02-22 Thread Plumo
I want to download content asynchronously. This would be straightforward to do threaded or across processes, but difficult asynchronously so people seem to rely on external libraries (twisted / gevent / eventlet). (I would use gevent under different circumstances, but currently need to stick