mustafaiman commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable URL: https://github.com/apache/hadoop/pull/1795#issuecomment-577902804 @steveloughran ``` we have the InconsistentS3AClient so simulate failures to S3; used for S3Guard. I've been wondering what it would take to actually simulate throttling there as well; some random probability of the client considering itself overloaded, and then having a window where it blocks. Or even better -could we actually let you configure a throttle load and have it trigger when the request rate exceeded it. Simulating request timeouts would be simpler -but as or fault injecting client goes above the AWS SDK, it won't be testing their internals. ``` Simulating throttling looks achievable. AWS SDK passes throttle errors up to hadoop-aws and that is where we catch throttling error and implement retry mechanism. So you can simulate throttling in InconsistentAmazonS3Client. However, `fs.s3a.connection.request.timeout` affects the retry mechanism below AWS SDK. So as you already pointed out, it is not useful in testing this particular configuration.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
