tl;dr Do you of any libraries for parallelizing HTTP requests with per-server concurrency control and handling of retries?
I'm writing a service that fetches many independent small binary blobs (map tiles) over HTTP from a several upstream servers and package them together in to a single archive. I want to parallelize the fetching of the small binary blobs. Currently there are O(10) upstream servers and O(1000) small binary blobs fetched from each. Making parallel HTTP requests in Go is trivially easy and is demonstrated in many Go tutorials and blog posts. However, I'm looking for a "production ready" library that supports: * Per upstream server concurrency limits. * Overall (across all upstream servers) concurrency limits. * Controllable retries with exponential backoff in the case of upstream server errors. * Timeouts for upstream requests. * context.Context support. This would seem to be a common enough task that I would expect to find an existing library that does all of the above. Existing Go web scrapers, e.g. colly <http://go-colly.org>, likely have this functionality internally but do not expose it in their API and are instead focused on crawling web pages. Do you know of any such library? Many thanks, Tom -- Confidentiality Notice: This electronic message and any attached documents contain confidential and privileged information and is for the sole use of the individual or entity to whom it is addressed. If you are not the addressee of this email, or the employee or agent responsible for delivering it to the addressee, you are hereby notified that any dissemination, distribution or copying of this transmission is strictly prohibited. If you receive this message in error, please notify the sender immediately by return e-mail or telephone and destroy the attached message (and all attached documents) immediately. Thank you for your cooperation. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cfb138b2-88d4-46ee-9315-996389718bad%40googlegroups.com.
