Re: Calculating requests per second, per IP address
Hi all!! I appreciate everyone for the help. For now, I used the good old bash (awk, sort, uniq) to get the information I need, but I'll try later to analyze the other tools and tips you guys sent me! Thanks and a great 2024 to everyone. On Tue, Jan 2, 2024 at 12:24 AM jeremy ardley via nginx wrote: > > On 1/1/24 18:57, J Carter wrote: > > Other than log aggregators, writing a script (python, perl, bash) is > > likely the fastest approach. Consider sharing it if you do, I'm sure > > others will find it useful. > -- > An alternative option is to put haproxy as a front end to nginx. > haproxy has very configurable rate limiting. See > > https://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting > ___ > nginx mailing list > nginx@nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx > -- *Esta mensagem pode conter informações confidenciais ou privilegiadas, sendo seu sigilo protegido por lei. Se você não for o destinatário ou a pessoa autorizada a receber esta mensagem, não pode usar, copiar ou divulgar as informações nela contidas ou tomar qualquer ação baseada nessas informações. Se você recebeu esta mensagem por engano, por favor avise imediatamente ao remetente, respondendo o e-mail e em seguida apague-o. Agradecemos sua cooperação.* ___ nginx mailing list nginx@nginx.org https://mailman.nginx.org/mailman/listinfo/nginx
Re: Calculating requests per second, per IP address
On 1/1/24 18:57, J Carter wrote: Other than log aggregators, writing a script (python, perl, bash) is likely the fastest approach. Consider sharing it if you do, I'm sure others will find it useful. -- An alternative option is to put haproxy as a front end to nginx. haproxy has very configurable rate limiting. See https://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting ___ nginx mailing list nginx@nginx.org https://mailman.nginx.org/mailman/listinfo/nginx
Re: Calculating requests per second, per IP address
Hello, On Fri, 29 Dec 2023 09:54:30 -0300 Rejaine Monteiro wrote: > Hi all, > > I´m running Nginx community edition and need to implement rate limiting > > There's plenty of guides out there on how to do this, but no guides on how > to get real values/stats from the access logs > > > What I need to get from the NGINX access logs is: > > - Requests per minute, per IP address > > - Average requests per minute, derived from all IP addresses > > - Max requests per minute, per IP address > > We have a few endpoints with different functionalities and we simply cannot > have a common rule that works for everyone. > > Any tips on a tool or script that could help generate this type of > information (if possible in real time or collect this data for future > analysis)? > > > I appreciate any tips. > There isn't an existing bespoke tool for this (at least publicly available). Normally such metrics are generated by: A) Feeding access logs into a log aggregator platform (Splunk, Loki) B) Performing / creating custom queries on that platform, to generate such reports. Of note, Loki (which is AGPL/free) has a nice user experience for this. https://grafana.com/docs/loki/latest/query/metric_queries/ Other than log aggregators, writing a script (python, perl, bash) is likely the fastest approach. Consider sharing it if you do, I'm sure others will find it useful. If you're looking for existing scripts as a starting point, there may be similar tools for Apache that you could adapt for nginx. Both use the 'Common Log Format' for access logs by default. https://github.com/ajohnstone/apache-log-stats Something like this. ___ nginx mailing list nginx@nginx.org https://mailman.nginx.org/mailman/listinfo/nginx
Re: Calculating requests per second, per IP address
Thanks for the explanation I understand how Nginx works with limit_req, and what I'm looking for is precisely to find this "reasonable value" for each endpoint (testing the application side behavior is also important, but since there are multiple endpoints with hundreds of different functions, it might be a bit challenging to analyze in a short amount of time). We already use rate_limit in one of these applications, but I had to conduct a series of tests to arrive at a reasonable value. As we have several other services, I would like an analysis and monitoring tool to track requests over time and be able to assess requests per minute/second for each endpoint. This way, I can initially get a better idea and later make a more customized adjustment. For this specific endpoint, for example, I'm using: limit_req_zone $binary_remote_addr zone=api:10m rate=1r/s limit_req zone=api burst=10 nodelay On Fri, Dec 29, 2023 at 11:07 AM Maxim Dounin wrote: > Hello! > > On Fri, Dec 29, 2023 at 09:54:30AM -0300, Rejaine Monteiro wrote: > > > Hi all, > > > > I´m running Nginx community edition and need to implement rate limiting > > > > There's plenty of guides out there on how to do this, but no guides on > how > > to get real values/stats from the access logs > > > > > > What I need to get from the NGINX access logs is: > > > > - Requests per minute, per IP address > > > > - Average requests per minute, derived from all IP addresses > > > > - Max requests per minute, per IP address > > > > We have a few endpoints with different functionalities and we simply > cannot > > have a common rule that works for everyone. > > > > Any tips on a tool or script that could help generate this type of > > information (if possible in real time or collect this data for future > > analysis)? > > Note that nginx's limit_req module implements limiting based on > the "leaky bucket" algorithm, which permits traffic bursts when > properly configured, and therefore there is no need to calculate > exact rates of requests per minute and so on. > > Rather, I would recommend to focus on the number of requests a > typical legitimate user can generate to the protected resources > within a short period of time, and set large enough burst= value > to ensure legitimate users are not limited. Then use some > reasonable long-term rate - this does not need to be exact, but > rather somewhat reasonable, for example, to cover typical number > of requests per day from a legitimate user. > > Note well that using "nodelay" (or "delay=N") is recommended with > such approach, see http://nginx.org/r/limit_req for details. > > -- > Maxim Dounin > http://mdounin.ru/ > ___ > nginx mailing list > nginx@nginx.org > https://mailman.nginx.org/mailman/listinfo/nginx > -- *Esta mensagem pode conter informações confidenciais ou privilegiadas, sendo seu sigilo protegido por lei. Se você não for o destinatário ou a pessoa autorizada a receber esta mensagem, não pode usar, copiar ou divulgar as informações nela contidas ou tomar qualquer ação baseada nessas informações. Se você recebeu esta mensagem por engano, por favor avise imediatamente ao remetente, respondendo o e-mail e em seguida apague-o. Agradecemos sua cooperação.* ___ nginx mailing list nginx@nginx.org https://mailman.nginx.org/mailman/listinfo/nginx
Re: Calculating requests per second, per IP address
Hello! On Fri, Dec 29, 2023 at 09:54:30AM -0300, Rejaine Monteiro wrote: > Hi all, > > I´m running Nginx community edition and need to implement rate limiting > > There's plenty of guides out there on how to do this, but no guides on how > to get real values/stats from the access logs > > > What I need to get from the NGINX access logs is: > > - Requests per minute, per IP address > > - Average requests per minute, derived from all IP addresses > > - Max requests per minute, per IP address > > We have a few endpoints with different functionalities and we simply cannot > have a common rule that works for everyone. > > Any tips on a tool or script that could help generate this type of > information (if possible in real time or collect this data for future > analysis)? Note that nginx's limit_req module implements limiting based on the "leaky bucket" algorithm, which permits traffic bursts when properly configured, and therefore there is no need to calculate exact rates of requests per minute and so on. Rather, I would recommend to focus on the number of requests a typical legitimate user can generate to the protected resources within a short period of time, and set large enough burst= value to ensure legitimate users are not limited. Then use some reasonable long-term rate - this does not need to be exact, but rather somewhat reasonable, for example, to cover typical number of requests per day from a legitimate user. Note well that using "nodelay" (or "delay=N") is recommended with such approach, see http://nginx.org/r/limit_req for details. -- Maxim Dounin http://mdounin.ru/ ___ nginx mailing list nginx@nginx.org https://mailman.nginx.org/mailman/listinfo/nginx
Calculating requests per second, per IP address
Hi all, I´m running Nginx community edition and need to implement rate limiting There's plenty of guides out there on how to do this, but no guides on how to get real values/stats from the access logs What I need to get from the NGINX access logs is: - Requests per minute, per IP address - Average requests per minute, derived from all IP addresses - Max requests per minute, per IP address We have a few endpoints with different functionalities and we simply cannot have a common rule that works for everyone. Any tips on a tool or script that could help generate this type of information (if possible in real time or collect this data for future analysis)? I appreciate any tips. -- *Esta mensagem pode conter informações confidenciais ou privilegiadas, sendo seu sigilo protegido por lei. Se você não for o destinatário ou a pessoa autorizada a receber esta mensagem, não pode usar, copiar ou divulgar as informações nela contidas ou tomar qualquer ação baseada nessas informações. Se você recebeu esta mensagem por engano, por favor avise imediatamente ao remetente, respondendo o e-mail e em seguida apague-o. Agradecemos sua cooperação.* ___ nginx mailing list nginx@nginx.org https://mailman.nginx.org/mailman/listinfo/nginx
