Ah, finally some information. Ok, so now we know that this is a "not so good" Google project as this spider definitely doesn't behave itself - what can be done?
As it stands right now, 75.125 is completely blocked because this thing is hitting from IP's all across the range. The problem is, the block is affecting other things like Google Web Preview, etc. Also, is there a reason that a webmaster would want Google to crawl pages for the web preview (or not want) as it appears that the bot also doesn't necessarily honor robots.txt (as reported in other posts) What else comes from 75.125.X.X Why would projects such as Google DOTS that have potential to cause these sort off issues be permitted to run from IP addresses where Google App Engine traffic and other Google Spider traffic originate? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/zQ0_X02n0BEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
