You can also try things like randomizing your sleep time in-between requests and changing the user-agent string you send—basically make your robot seem more like an office full of regular users. Be nice to their servers and you may be able to get search results undetected.
For the record, the Google Web Search API<https://developers.google.com/web-search/terms>, which is deprecated (but still worked last time I checked it), states, in part, on the TOS that *you will not*: - *use any robot*, spider, site search/*retrieval application*, or other device *to retrieve or index any portion of Google Search Results* or to collect information about users for any unauthorized purpose; I have to believe that since they've deprecated the web search API, they want you to scrape their results now even less than they used to. On Sun, Nov 18, 2012 at 7:33 PM, PHPBABY3 <[email protected]> wrote: > > On Nov 18, 9:29 pm, Geoffrey Hoffman <[email protected]> > wrote: > > Throttle your requests throug multiple IPs. > > > > Is that all it is - frequency of request? I lately started getting it > when I manually copy a complete set of results (all pages.) What > would the threshold frequency be? > > Charlie > > > > > On Sun, Nov 18, 2012 at 7:11 PM, PHPBABY3 <[email protected]> wrote: > > > I have an application in which I need to know the results of a > > > variable Google (or other search engine) results i.e. the first page > > > with the number of results and the first 10, and a link to the next > > > page if there are more. > > > > > When I get it through a PHP function at some point Google only sends > > > a > > > message that it thinks I'm a robot (which I am.) > > > > > Charlie > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "Google AJAX APIs" group. > > > To post to this group, send email to > > > [email protected] > > > To unsubscribe from this group, send email to > > > [email protected] > > > To view this message on the web, visit > > >http://groups.google.com/group/google-ajax-search-api?hl=en_US > > > For more options, visit this group at > > >http://groups.google.com/group/google-ajax-search-api?hl=en?hl=en > > -- > You received this message because you are subscribed to the Google > Groups "Google AJAX APIs" group. > To post to this group, send email to > [email protected] > To unsubscribe from this group, send email to > [email protected] > To view this message on the web, visit > http://groups.google.com/group/google-ajax-search-api?hl=en_US > For more options, visit this group at > http://groups.google.com/group/google-ajax-search-api?hl=en?hl=en > -- You received this message because you are subscribed to the Google Groups "Google AJAX APIs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] To view this message on the web, visit http://groups.google.com/group/google-ajax-search-api?hl=en_US For more options, visit this group at http://groups.google.com/group/google-ajax-search-api?hl=en?hl=en
