Hello Parallelizers! I'm using GNU Parallel to run the AWS CLI utility, and I want to use Parallel's --limit option to avoid being throttled.
Before I sit down and write something, has anybody already invented this wheel? Essentially, AWS has a token-bucket rate-limiting system: * Every API has two numbers associated with it: a bucket size (integer) and a refill rate (integer). * Every second, the bucket's value is incremented by the second number, until it reaches the first number. * Every unthrottled request decrements the bucket's value by one. * If the bucket's value is zero, requests are denied with a throttling error. The script would emulate this logic, and given the two numbers (bucket size and rate) return a '1' if a throttling error would occur and a '0' otherwise. It would be nice if the script could auto-detect the two numbers, but it's unclear how that could be done without being throttled at least twice. Maybe it could analyze the job log of a previous run? Thanks! -- CH ([email protected])
