cbaines pushed a commit to branch master
in repository maintenance.
commit 2ecf6bb33db59d84169811865f027c58fe978981
Author: Christopher Baines <m...@cbaines.net>
AuthorDate: Mon May 13 16:15:24 2024 +0100
hydra: bayfront: Block some common bots from scraping QA.
As they can make quite expensive requests.
* hydra/bayfront.scm (%qa.guix.gnu.org-nginx-servers): Block some
common bots.
---
hydra/bayfront.scm | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/hydra/bayfront.scm b/hydra/bayfront.scm
index 94ce73c..889f035 100644
--- a/hydra/bayfront.scm
+++ b/hydra/bayfront.scm
@@ -973,7 +973,12 @@ proxy_set_header Via $via;"
(body '("proxy_pass http://qa-frontpage;";
"proxy_http_version 1.1;"
"proxy_read_timeout 300;"
- "proxy_set_header Connection \"\";")))
+ "proxy_set_header Connection \"\";"
+
+ ;; Block bots that have taken a liking to QA
+ "if ($http_user_agent ~ (Bytespider|ClaudeBot) ) {
+ return 403;
+}")))
(nginx-location-configuration
(uri "/reproducible.json")
(body '("alias /var/lib/qa-frontpage/reproducible.json;"
