On Monday 28 November, Guido van Rossum wrote: > On 11/24/05, Duncan Grisby <[EMAIL PROTECTED]> wrote:
> > I have encountered a problem with the re module. I have a > > multi-threaded program that does lots of regular expression searching, > > with some relatively complex regular expressions. Occasionally, events > > can conspire to mean that the re search takes minutes. That's bad > > enough in and of itself, but the real problem is that the re engine > > does not release the interpreter lock while it is running. All the > > other threads are therefore blocked for the entire time it takes to do > > the regular expression search. > > Rather than trying to fight the GIL, I suggest that you let a regex > expert look at your regex(es) and the input that causes the long > running times. As Fredrik suggested, certain patterns are just > inefficient but can be rewritten more efficiently. There are plenty of > regex experts on c.l.py. Part of the problem is certainly inefficient regexes, and we have improved things to some extent by changing some of them. Unfortunately, the regexes come from user input, so we can't be certain that our users aren't going to do stupid things. It's not too bad if a stupid regex slows things down for a bit, but it is bad if it causes the whole application to freeze for minutes at a time. > Unless you have a multi-CPU box, the performance of your app isn't > going to improve by releasing the GIL -- it only affects the > responsiveness of other threads. We do have a multi-CPU box. Even with good regexes, regex matching takes up a significant proportion of the time spent processing in our application, so being able to release the GIL will hopefully increase performance overall as well as increasing responsiveness. We are currently testing our application with the patch to sre that Eric posted. Once we get on to some performance tests, we'll post the results of whether releasing the GIL does make a measurable difference for us. Cheers, Duncan. -- -- Duncan Grisby -- -- [EMAIL PROTECTED] -- -- http://www.grisby.org -- _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com