On Sat, Mar 7, 2020 at 5:35 AM Kurtis Rader <kra...@skepticism.us> wrote:
> The web site you are accessing thinks you are executing a DDOS attack or > are otherwise violating their terms of service. This has nothing to do with > the Go language. You need to rate limit your requests of that site. > > To add: Look up the concept of a circuit breaker and run it in reverse. Rate limit your requests, and if the site takes too long to respond, or if it returns a 429 or the like, you should trip the circuit breaker for a while to cool down. Scraping can put a tremendous load on a web site and it is your duty as the scraper to be careful. It is especially important in a language like go, where you can easily make ten thousand requests to a website from a single program. -- J. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAGrdgiWDCNtPgfMsWJOeU%3DbYy0930FmuN9dhX8-4qnBcvQmJuA%40mail.gmail.com.
