Hello Max, (Besides the fact that the this client seems to have a broken random URL generator)
Crawlers (like Nutch clients) may not always obey robot rules. If Nutch is not configured properly, it will not recognize your Nutch entry in your robots.txt file. If the requests come from a limited number of IP addresses (78.46.90.27, 188.40.65.130) , you could try to ban those. Or try to contact the people behind this address, they seem to live in Germany as well. Good luck, Mathijs On Nov 2, 2011, at 14:13 , Maximilian Laurenz wrote: > Hi there, > Because a Nutch client seems to cause errors on our web server, we changed > robots.txt for www.kinoundco.de to disallow Nutch. Unfortunately we still get > requests: > > 2011-11-01 05:50:35 W3SVC4 MB 62.128.28.16 GET > /Rango/+(Math.random()*100000)+ - 80 - 188.40.65.130 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 302 0 373 263 0 > 2011-11-01 05:52:23 W3SVC4 MB 62.128.28.16 GET /default.aspx > aspxerrorpath=/Atemlos-Gefaehrliche-Wahrheit/+(Math.random()*100000)+ 80 - > 188.40.65.130 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 200 0 32449 315 15 > 2011-11-01 05:59:15 W3SVC4 MB 62.128.28.16 GET > /Kleine-wahre-Luegen/+(Math.random()*100000)+ - 80 - 188.40.65.130 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 302 0 401 277 15 > 2011-11-01 05:59:31 W3SVC4 MB 62.128.28.16 GET > /Nachtasyl/+(Math.random()*100000)+ - 80 - 188.40.65.130 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 302 0 381 267 0 > 2011-11-01 06:35:30 W3SVC4 MB 62.128.28.16 GET > /Auf-der-anderen-Seite-der-Leinwand-100-Jahre-Moviemento/+(Math.random()*100000)+ > - 80 - 78.46.90.27 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 302 0 473 313 15 > 2011-11-01 06:35:33 W3SVC4 MB 62.128.28.16 GET > /Zoowaerter/+(Math.random()*100000)+ - 80 - 78.46.90.27 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 302 0 383 268 31 > 2011-11-01 06:35:40 W3SVC4 MB 62.128.28.16 GET /default.aspx > aspxerrorpath=/Sascha/+(Math.random()*100000)+ 80 - 78.46.90.27 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 200 0 32449 292 62 > 2011-11-01 06:36:35 W3SVC4 MB 62.128.28.16 GET > /Beschissenheit-der-Dinge/+(Math.random()*100000)+ - 80 - 78.46.90.27 > HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 302 0 411 282 31 > 2011-11-01 06:38:14 W3SVC4 MB 62.128.28.16 GET > /Auf-der-Suche/+(Math.random()*100000)+ - 80 - 78.46.90.27 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 302 0 389 271 15 > 2011-11-01 06:39:55 W3SVC4 MB 62.128.28.16 GET /Fall/+(Math.random()*100000)+ > - 80 - 78.46.90.27 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 302 0 371 262 15 > 2011-11-01 07:51:10 W3SVC4 MB 62.128.28.16 GET > /Midnight-in-Paris/+(Math.random()*100000)+ - 80 - 176.9.26.236 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 302 0 397 275 0 > 2011-11-01 07:51:40 W3SVC4 MB 62.128.28.16 GET > /Betty-Anne-Waters/+(Math.random()*100000)+ - 80 - 176.9.26.236 HTTP/1.0 > Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:2.0.1)+Gecko/20100101++++Firefox/4.0.1+++/Nutch-1.2 > - - www.kinoundco.de 302 0 397 275 15 > > Sincerely, > Max > Maximilian Laurenz > S&L Medien Gruppe GmbH > Aidenbachstraße 54 > 81379 München > Tel. +49 89 790862-49 > Fax +49 89 790862-55 > [email protected] > http://www.slmedien.de > S&L Medien Gruppe GmbH | Geschäftsführung: Maria-Theresia von Seidlein, > Torsten Weihrich, Olaf Wiehler | Sitz der Gesellschaft: München | Amtsgericht > München | HRB 99977 >
