Martin, On Wed, Apr 14, 2010 at 4:25 AM, Martin Holst Swende <mar...@swende.se> wrote: > Hi, > > I noticed that some greppers took extremely long time to run for certain > input, especially two of them almost appeared to halt when I ran them. > Those were ajax and svnusers. In ajax.py, the following regexp is used : > > regex_string = '< *?script.*?>.*?' > regex_string += > '(XMLHttpRequest|eval\(\)|ActiveXObject\("Msxml2.XMLHTTP"\)|' > regex_string += 'ActiveXObject\("Microsoft.XMLHTTP"\))' > regex_string += '.*?</ *?script *?>' > > This is a very 'loose' regexp, which has a lot of wildcards, therefore > it basically becomes ReDos:ed for certain pages. I suggest changing this > to just checking for the calls. Also, it looks like the construct > checking eval will check explicitly for "eval()", not "eval(foo)". > Something like this should work, if we want to check use of any eval :
Yes, I've been experiencing the same issue in my tests. One of the things I'm afraid of is that we'll never be able to make all the regexes work fast. Python's regular expression engine is not as fast as most people believe. w3af has one big problem right now: "We don't use the DOM in grep plugins", and we don't use it, because it's simply not there, the core is not providing a DOM for each HTTP response. If we would add that core feature, many plugins could stop using those regular expressions, which in some cases are *slow*, thus increasing the framework performance. But still, regular expressions are going to be needed... which needs some kind of enhancement also. Something that I started to research is the use of the Aho-Corasick algorithm [0] [1] that could enhance the speed a lot (I hope!) [0] http://sourceforge.net/apps/trac/w3af/ticket/160053 [1] http://pypi.python.org/pypi/acora/1.4 > regex_string = > '(XMLHttpRequest|eval\(|ActiveXObject\("Msxml2.XMLHTTP"\)|' > regex_string += 'ActiveXObject\("Microsoft.XMLHTTP"\))' > > svnusers.py contains the following > regex = '\$.*?: .*? .*? \d{4}[-/]\d{1,2}[-/]\d{1,2}' > regex += ' \d{1,2}:\d{1,2}:\d{1,2}.*? (.*?) (Exp )?\$' > > This can be enhanced by replacing wildcards with harder matches and > removing optional stuff at the end (Exp )?. > However, it seems to me that the following regexp would work and be much > quicker : > regex = "date:.*author:\W(\w+);" I will try these new regular expressions and see how they work out. > Additionally, both of them contains the construction ".*?" which is > strange. Unless I am not missing something special about python regexps, > this should be ".*", as * means zero or more times, and ? is optional, > which is one or zero times. As Tom said, the "?" indicates non-greedy match. > Regards, > Martin Holst Swende > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > W3af-develop mailing list > W3af-develop@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/w3af-develop > -- Andrés Riancho Founder, Bonsai - Information Security http://www.bonsai-sec.com/ http://w3af.sf.net/ ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop