Re: Advice on long running processes
Roy Smith wrote: > [EMAIL PROTECTED] wrote: > > >>Hello, >> >>I write a lot of CGI scripts, in Python of course. Now I need to >>convert some to long-running processes. I'm having trouble finding >>resources about the best practices to do that. >> >>I've found a lot of email discussions that say something like, "You >>need to educate yourself about the differences when you have long- >>running processes" but I've not had a lot of luck with finding things >>that explain the differences. > > > The biggest differences between run-and-exit vs. long running processes are > resource management and error recovery. Let's take them one at a time. > > Resource management. In a short-lived process, you really don't have to > worry about this at all. Snarf as much memory as you need, open as many > files as you want, and when you exit, the operating system cleans it all up > for you. With a long running process, you have to worry about stuff like > that. > > In Python, you're isolate from the low-level details of memory management, > but still need to think about it a bit. Imagine you had code that looked > like this in your main loop: > > for request in getNextRequest(): > requestList.append (request) > processRequest(request) > > requestList is going to keep growing without bounds and eventually will eat > up all available memory in the system and your process will crash. > Everything you store, you also need to delete when you're done with it. In particular, it is a good idea to call gc.collect() every now and then, specially if you are in such a loop. I don't know what is the gc policy in python, but an application of mine that seemed to eat as much memory as it was available was reduced to a constant small amount of memory after I started to call the gc directly. > The other big thing is error recovery. In a short lived process, if > something fails, you print an error message and exit. In a long running > process, you need to somehow recover from the error and keep going as best > you can. This can be tricky. You should have your main loop inside a try/except, to catch any exceptions that were not otherwise caught without exiting the application. Log the exceptions, of course, but in most long running applications work in a loop like Roy Smith's code above, so if one of them fails it won't disrupt the others to come. -- Bruno Barberi Gnecco The earth is like a tiny grain of sand, only much, much heavier. -- http://mail.python.org/mailman/listinfo/python-list
Re: RuntimeWarning: tp_compare
Chris Mellon wrote: > On 10/9/07, Bruno Barberi Gnecco > <[EMAIL PROTECTED]> wrote: > >>I'm getting the following exception when I call an external extension >>(pytst): >> >>/usr/lib/python2.5/threading.py:697: RuntimeWarning: tp_compare didn't return >>-1 or -2 for >>exception >> return _active[_get_ident()] >>Traceback (most recent call last): >> File "testDataMiner2.py", line 77, in >> testPlace() >> File "testDataMiner2.py", line 41, in testPlace >> data = db.getDescription(event['id']) >> File "testDataMiner2.py", line 28, in getDescription >> return self.getRow(query, (id,)) >> File "../database.py", line 73, in getRow >> self.readlock.acquire() >> File "/usr/lib/python2.5/threading.py", line 94, in acquire >> me = currentThread() >> File "/usr/lib/python2.5/threading.py", line 697, in currentThread >> return _active[_get_ident()] >>UnicodeEncodeError: 'ascii' codec can't encode character u'\xfa' in position >>52: ordinal >>not in range(128) >>awer >> > /usr/lib/python2.5/threading.py(700)currentThread() >>-> return _DummyThread() >> >> >>Note that the error occurs *after* the call that I isolated as >>affecting it (pytst.scan(), in the case). This doesn't happen for simple, >>isolated cases, but googling for "tp_compare threading" shows a lot of >>similar issues. Does anybody here know what this could be about? Any ideas >>to debug or work around it? >> > > > The various thread issues in the traceback aside, it looks like the > problem is that you passed a unicode object to pytst, which only > accepts plain (ascii) strings. That seems to have solved it. Thanks! -- Bruno Barberi Gnecco There is no time like the pleasant. -- http://mail.python.org/mailman/listinfo/python-list
RuntimeWarning: tp_compare
I'm getting the following exception when I call an external extension (pytst): /usr/lib/python2.5/threading.py:697: RuntimeWarning: tp_compare didn't return -1 or -2 for exception return _active[_get_ident()] Traceback (most recent call last): File "testDataMiner2.py", line 77, in testPlace() File "testDataMiner2.py", line 41, in testPlace data = db.getDescription(event['id']) File "testDataMiner2.py", line 28, in getDescription return self.getRow(query, (id,)) File "../database.py", line 73, in getRow self.readlock.acquire() File "/usr/lib/python2.5/threading.py", line 94, in acquire me = currentThread() File "/usr/lib/python2.5/threading.py", line 697, in currentThread return _active[_get_ident()] UnicodeEncodeError: 'ascii' codec can't encode character u'\xfa' in position 52: ordinal not in range(128) awer > /usr/lib/python2.5/threading.py(700)currentThread() -> return _DummyThread() Note that the error occurs *after* the call that I isolated as affecting it (pytst.scan(), in the case). This doesn't happen for simple, isolated cases, but googling for "tp_compare threading" shows a lot of similar issues. Does anybody here know what this could be about? Any ideas to debug or work around it? -- Bruno Barberi Gnecco I can read your mind, and you should be ashamed of yourself. -- http://mail.python.org/mailman/listinfo/python-list
IMAP library
It seems that this question has been asked a few times in this group before, but sometime has passed since the last post and maybe there's something new. I'm looking for a high level IMAP library--I know of imaplib, but it's awfully documented and I was hoping to avoid reading RFCs for a simple email reader routine. So, is there any example, routine, library or code in some software package that reads mails from IMAP, with attachments? I don't need anything fancy, just reading the emails and decoding them. If there isn't, I'm willing to write one and release it under LGPL. Anybody would like to help? -- Bruno Barberi Gnecco -- http://mail.python.org/mailman/listinfo/python-list
Which XML?
I've found a lot of XML libraries for Python. Any advices on which one to use (or *not* to use)? My requirements are: support for XPath, stability (a must, segfaults are not an option), with DOM API and good performance desirable. Thanks for any advice. -- Bruno Barberi Gnecco -- http://mail.python.org/mailman/listinfo/python-list
