Dear Berry, as you can see in robots.txt file og myhotelcar, I have blocked all the bots except the google, yahoo slurp etc, so Ahref bot if it obeys it it should not hit site.
I have also searched so many thing and I found the Ahref bot doesn't obey robots principal. Many people has suggested that I can prohibit them via htaccess file, I don't want to use that way as in google app engine hosting I didn't find htaccess file. So please provide me any way to filter out these spam bots. On Monday, April 20, 2015 at 5:54:42 PM UTC+5:30, barryhunter wrote: > > > > On 20 April 2015 at 12:28, Ashutosh Mishra <ashutosh.n...@gmail.com > <javascript:>> wrote: > >> y I am annoyed by the hitting of Spam Bot mainly Ahref bot. >> > > The Ahref bot (if its the legitimate one of course!) definitly obays the > robots.txt > https://ahrefs.com/robot > > Looking at > http://www.myhotelcar.com/robots.txt > > there is nothing blocking that particular bot. > > But a number of other oddities. The crawl delay will only apply to the * > group, which is disallowed from any crawling, meaning it has no effect. > > The 'directories' rules, will only apply the group they placed in (so only > affect MJ12bot/v1.4.5 - which is blocked completely by the first rule) > > > > > >> >> I don't find any way to stop them as I didn't see .HTaccess file in >> Google App Engine >> >> > Not as such. You would need to handle any such directives directly in > code. ie your javas handlers, could check the User-Agent and do 'stuff' > selectively. > > > > Tehre is also > https://cloud.google.com/appengine/docs/java/config/dos > but its utility for this is limited. (unless you can identify specific > IP/ranges to block) > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/77e74589-aa22-47e1-bb0c-7fa6c2e92364%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
