Hi Barry, FWIW, we've run into similar issues and re-implemented some rules to use Redis for storage instead. This had to be implemented via Lua scripting though, which takes a slight performance hit. Having said that, there is an open ticket to implement an alternative shared persistent storage mechanism:
https://github.com/SpiderLabs/ModSecurity/issues/378 -- - Josh On Wed, Mar 11, 2015 at 9:49 AM, Barry Pollard <[email protected]> wrote: > Thanks for those answers Chaim. Make sense to me. > > I do think the use of SDBM for storage for large scale tracking like for > IP addresses is a problem (with ModSecurity rather than with the core rule > set but not that that matters). Unfortunately it just doesn't work for even > relatively small amounts of traffic. > > Not sure how you could resolve it and I know nothing about ModSecurity is > written, or about writing really high performing systems like this, but > came up with below suggestions after a bit of a think. Not sure how > feasible any of these are. > > * Split out IP.pag into several files based on hashed IP address so bottle > neck is not on one file. Would need to be hashed IP address in case your > customer based is skewed towards certain IP ranges. Could also configure > number of files to split across, so busier sites have larger number of > IP.pag files, if the hashing algorithm could take that config number into > account. > > * Have a minimum update time e.g. If hit from IP in last 1 second then > don't bother saving another hit as record is reasonably up to date. Again > time could be configurable. Would obviously impact any DoS/Brute force > rules since we'd be filtering out hits here. > > * Make it easier to ignore the problem by ignoring certain alerts (e.g. > deleting stale collections fails... etc.) unless in high log level, and add > a safe way of ModSecurity clearing down the file and starting again at > certain times (e.g. daily/weekly) or after it reaches a certain size. > Massive fudge but could be short term solution. > > * Use some other persistent storage. Though I'd imagine, given the > potential rate of data updates for a busy site, this would also be a > problem in whatever we move to. > > * Use in-memory storage - though guess that would raise questions on > memory requirements and how to share between different threads/processes. > > * Have dedicated process/program to manage collections that all other > threads/processes talk via. Could address some of the issues with in using > in-memory storage raised above. Then again could bring in whole raft of > other problems :-) > > * Store less. Collections do seem to be quite large. Could you have a > minimal set for things like IP address with just last access time/num hits? > For rules that require more things tracked (e.g. Brute force attempts on a > specific URL) they could be tracked in a separate "full" collection which > would have a larger record set, but with less entries. > > * Reduce blocking time in code making the SDBM update. Imagine the code is > probably pretty tight here already but obviously any reduction here would > have benefit. Had a quick look myself (more out of nosiness) but the C code > scared me too much (been a while!). > > * Look into how other modules (e.g. mod_evasive) handle this problem to > see if can learn anything from them. > > Am sure you've probably already considered these and better solutions but > thought I'd suggest them anyway. > > Thanks, > Barry > > > On 11 Mar 2015, at 03:54, Chaim Sanders <[email protected]> wrote: > > > > we > _______________________________________________ > Owasp-modsecurity-core-rule-set mailing list > [email protected] > https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set >
_______________________________________________ Owasp-modsecurity-core-rule-set mailing list [email protected] https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
