Hi Christian,
Good morning Jens,On Sun, Aug 21, 2016 at 12:48:32PM +0200, Jens Schleusener wrote:One reason for the "silence" may be also the very laborious log file analysis. But to get just a first impression I found your script aliases collection at https://github.com/Apache-Labor/labor/blob/master/bin/.apache-modsec.alias very useful.Thanks. I should probably cover them in a blogpost. Right now they seem to be a bit of a secret to those who have read my (German) tutorials.On the server that I manage (httpd 2.4.23, modsecurity 2.9.1) CRS3-RC1 is now for nearly a month in use and it works very well (big thanks to the CRS3 team!).Glad it works!Although the server may be not "typical" since it uses itself neither PHP nor SQL but offers amongst others soure code browsing of FOSS software that contains for e.g. PHP, SQL and HTML files here an example output for 5 days (with roughly 200.000 page requests in total) using the script alias "melidmsg" (entries fom some special self-written rules are removed): cat modsec_audit.log.16081[5-9] | melidmsg | sort | uniq -c | sort -nrI guess you are aware of the alias 'sucs' and use the sort-uniq-c-sort combination here to make it more comprehensive for the other readers.
A "small" yes (but I wanted to reverse the sort order for this post) and there is for e.g. also a related alias "melmsgs" in your collection that does it all with a single command.
347 921140 HTTP Header Injection Attack via headers 308 910000 Request from Known Malicious Client (Based on previous traffic violations). 186 920350 Host header is a numeric IP address 105 932130 Remote Command Execution: Unix Shell Expression Found ...All in all, this is quite a big collection of alerts for 200K requests. Your site is special of course, but would this special content really affect the requests that much. Would not that be more a problem with the responses?
Sure, for e.g. if valued as meaningful the server tries to display a requested HTML member file of an available software package in rendered format but that leads sometimes to the mentioned errors like
105 932130 Remote Command Execution: Unix Shell Expression Found
if for e.g. that file includes stylesheets like
...common.css?_=${version}
But that was the case only for one member file of one package that caused
all that 105 matches.
There are a rules which point to a real positive. But with many others, we might face false positives. Did you investigate further, or do you think they point to malign requests?
As said that is very laborious if there hundred of entries. Often one had to analyze additionally the big Apache access log to find out the further behaviour of an suspicious IP and classify the according requests as "good" or "bad".
Helpful would be a general program or script that can grep blocks of lines given a main target pattern and two further patterns specifying the begin and the end of the block. I just write a very primitive one that allows for e.g. via
grep_blocks_by_sed -p 'id "921140"' \ -b '^--[0-9a-f]*-A--' -e '^--[0-9a-f]*-Z--' modsec_audit.logto extract all matching blocks related to a special rule ID (the number of matches can be restricted by an option "-n N") and filter them
appropriately further. So I found now that the 347 entries 347 921140 HTTP Header Injection Attack via headers belongs all to a single IP (probably a proxy) that adds to the end of the User-Agent header field a "\x0d". Also all the 59 entries 59 920430 HTTP protocol version is not allowed by policyare caused by a single client/bot (User-Agent: Java/1.6.0_04) that seems to have misinterpreted responses of previously requested URLs and requested now invalid URLs that the CRS3 rule seems to assign to the HTTP protocol version.
Found") in some cases seems to cause FPs (especially for the User-Agent "Go-http-client/1.1" that sends erroneously a "Connection: close, close" HTTP header).Would you please be so kind and open an issue on the github site for this false positive? Ideally including a full audit log entry for the request.
Ok (but a little bit later). Jens _______________________________________________ Owasp-modsecurity-core-rule-set mailing list [email protected] https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
