On Thu, Nov 7, 2013 at 10:38 AM, Barry Hunter
wrote:
> Well yes, as a webmaster you do have to decide if you willing to 'pay' for
> crewlers wanting to access your site.
> Its a tradeoff, the 'cost' verses the potential benefit of having your
> site included in their search engine.
> You could o
Oh! And wait until you have to pay for the SQL injection attacks! :-)
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegro
You could also try detecting the crawlers via UserAgent and return a response
(html) with keywords but without links, so the crawling stops there.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop rece
Well yes, as a webmaster you do have to decide if you willing to 'pay' for
crewlers wanting to access your site.
Its a tradeoff, the 'cost' verses the potential benefit of having your site
included in their search engine.
You could opt to proactivly manage crawler activity, in the first case with
Hello,
yesterday my app was visited 35 times by HUMANS. It seems however that a
machine was crawling the website.
I was overquota in a few hours. Today i pay max 5USD per day. for 35 real
people it seems way too much.
I dont feel really good paying for crawlers that block the access to my
