Hi berry, No you can see that I have updated the robots.txt to block specifically ahref bot file for myhotelcar.com but still issue remains un-resolved bots hitting the site regularly badly. I need your expert advice to stop any kind of spam bots or bots which doesn't obey robots.txt file rules. Any help will be appreciated.
On Monday, April 20, 2015 at 5:54:42 PM UTC+5:30, barryhunter wrote: > > > > On 20 April 2015 at 12:28, Ashutosh Mishra <ashutosh.n...@gmail.com > <javascript:>> wrote: > >> y I am annoyed by the hitting of Spam Bot mainly Ahref bot. >> > > The Ahref bot (if its the legitimate one of course!) definitly obays the > robots.txt > https://ahrefs.com/robot > > Looking at > http://www.myhotelcar.com/robots.txt > > there is nothing blocking that particular bot. > > But a number of other oddities. The crawl delay will only apply to the * > group, which is disallowed from any crawling, meaning it has no effect. > > The 'directories' rules, will only apply the group they placed in (so only > affect MJ12bot/v1.4.5 - which is blocked completely by the first rule) > > > > > >> >> I don't find any way to stop them as I didn't see .HTaccess file in >> Google App Engine >> >> > Not as such. You would need to handle any such directives directly in > code. ie your javas handlers, could check the User-Agent and do 'stuff' > selectively. > > > > Tehre is also > https://cloud.google.com/appengine/docs/java/config/dos > but its utility for this is limited. (unless you can identify specific > IP/ranges to block) > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/152e34c2-c272-4007-a2e8-f9a00fa94998%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
