I wanted to post a followup for those following this thread Well as I mentioned before, we moved away from blocking entire IP blocks to just blocking user agents like DOTS which was effective until today. Now it seems that from the IP's that DOTs is originating, this project (or another) has resorted to completely stripping the user agent in an attempt to bypass these blocks. I fail to understand why this is occurring and why does Google tolerate activity such as what is reported in the links above and as I have described here? We have seen this activity thus far on
74.125.52.* and 74.125.126.* Now due to this, we have to resort back to IP blocking....... Taking this along with the other issues, I guess it just makes things a bit frustrating due to the fact there is a bit of a lack of transparency as to what exactly is going on and why the cloak and dagger with these bots/projects/etc from Google IP addresses. On Aug 8, 9:13 pm, "Brandon Wirtz" <drak...@digerat.com> wrote: > I don't have solution other than to block requests by agent. > > And I really only provided this information so that someone at GOOG could go > track him down and beat him (again). > > From: google-appengine@googlegroups.com > [mailto:google-appengine@googlegroups.com] On Behalf Of Viper > Sent: Monday, August 08, 2011 2:28 PM > To: google-appengine@googlegroups.com > Cc: carolinasviperc...@gmail.com > Subject: Re: RE: [google-appengine] Is an APP ENGINE Application Attacking > websites? - DOTS user agent / browser > > Ah, finally some information. Ok, so now we know that this is a "not so > good" Google project as this spider definitely doesn't behave itself - what > can be done? > > As it stands right now, 75.125 is completely blocked because this thing is > hitting from IP's all across the range. The problem is, the block is > affecting other things like Google Web Preview, etc. Also, is there a > reason that a webmaster would want Google to crawl pages for the web preview > (or not want) as it appears that the bot also doesn't necessarily honor > robots.txt (as reported in other posts) > > What else comes from 75.125.X.X Why would projects such as Google DOTS that > have potential to cause these sort off issues be permitted to run from IP > addresses where Google App Engine traffic and other Google Spider traffic > originate? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web > visithttps://groups.google.com/d/msg/google-appengine/-/zQ0_X02n0BEJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
