I am also in the same boat as tom, there is certainly a demand for this
type of library.
-- Sam Fourman
On Tue, Aug 20, 2019 at 3:33 PM 'Thomas Bushnell, BSG' via golang-nuts <
golang-nuts@googlegroups.com> wrote:
> I am of the opinion that a case like this is best handled by simply
> writing the thing you want.
>
> Concurrency limits are easily managed by using tokens to gate fetches. One
> simple technique is to make a channel of struct{} with capacity equal to
> the maximum number of concurrent connections you are allowed. You can
> either fill it with things at startup, and then read from it before a
> request and send back to the channel when done, or start with it empty and
> send to it before a request and read back after. Either is equivalent.
>
> I'm not sure the point of overall concurrency limits in general, but the
> same thing works. It's unlikely to be a problem IMO for the size of job you
> describe.
>
> Retries are best done inside each fetch; wrap http.Get with the logic you
> want. There is no one-size-fits all here. There is a public backoff library
> available, but it's a bit complex and the code could easily be simpler if
> you address exactly what you want directly.
>
> For contexts, just use the http package's (*Request).WithContext method.
> That accomplishes timeouts too.
>
> On Mon, Aug 19, 2019 at 12:20 PM tom via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
>> tl;dr Do you of any libraries for parallelizing HTTP requests with
>> per-server concurrency control and handling of retries?
>>
>> I'm writing a service that fetches many independent small binary blobs
>> (map tiles) over HTTP from a several upstream servers and package them
>> together in to a single archive. I want to parallelize the fetching of the
>> small binary blobs. Currently there are O(10) upstream servers and O(1000)
>> small binary blobs fetched from each.
>>
>> Making parallel HTTP requests in Go is trivially easy and is demonstrated
>> in many Go tutorials and blog posts. However, I'm looking for a "production
>> ready" library that supports:
>> * Per upstream server concurrency limits.
>> * Overall (across all upstream servers) concurrency limits.
>> * Controllable retries with exponential backoff in the case of upstream
>> server errors.
>> * Timeouts for upstream requests.
>> * context.Context support.
>>
>> This would seem to be a common enough task that I would expect to find an
>> existing library that does all of the above. Existing Go web scrapers, e.g.
>> colly <http://go-colly.org>, likely have this functionality internally
>> but do not expose it in their API and are instead focused on crawling web
>> pages.
>>
>> Do you know of any such library?
>>
>> Many thanks,
>> Tom
>>
>> Confidentiality Notice:
>> This electronic message and any attached documents contain confidential
>> and privileged information and is for the sole use of the individual or
>> entity to whom it is addressed. If you are not the addressee of this email,
>> or the employee or agent responsible for delivering it to the addressee,
>> you are hereby notified that any dissemination, distribution or copying of
>> this transmission is strictly prohibited. If you receive this message in
>> error, please notify the sender immediately by return e-mail or telephone
>> and destroy the attached message (and all attached documents) immediately.
>> Thank you for your cooperation.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/cfb138b2-88d4-46ee-9315-996389718bad%40googlegroups.com
>> <https://groups.google.com/d/msgid/golang-nuts/cfb138b2-88d4-46ee-9315-996389718bad%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CA%2BYjuxvjvAQU-R7PAeocEJ%3Dp9-k0MSqRL%2BtcL4XhXaXjV%3DUepw%40mail.gmail.com
> <https://groups.google.com/d/msgid/golang-nuts/CA%2BYjuxvjvAQU-R7PAeocEJ%3Dp9-k0MSqRL%2BtcL4XhXaXjV%3DUepw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
--
Sam Fourman Jr.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAOFF%2BZ0hpyrJ8aHV0QWVC4j8BiYQKZ2RFwbkBhrBobMLESYgxg%40mail.gmail.com.
