[
https://issues.apache.org/jira/browse/HADOOP-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013285#comment-17013285
]
Mustafa Iman commented on HADOOP-16792:
---------------------------------------
[~ste...@apache.org] [~fabbri] thanks for directions.
I added a short documentation.
I tested the patch with "mvn -Dparallel-tests -DtestsThreadCount=16
-Daws.region=us-west-2 -Dscale clean verify" with AWS Oregon region. The
following is the "auth-keys.xml" with sensitive information removed
{{<property>}}
{{ <name>fs.s3a.scale.test.huge.filesize</name>}}
{{ <value>2G</value>}}
{{ </property>}}
{{ <property>}}
{{ <name>test.fs.s3a.name</name>}}
{{ <value>s3a://hadoop-patch-test</value>}}
{{ </property>}}{{<property>}}
{{ <name>fs.contract.test.fs.s3a</name>}}
{{ <value>${test.fs.s3a.name}</value>}}
{{ </property>}}{{<property>}}
{{ <name>fs.s3a.access.key</name>}}
{{ <description>AWS access key ID. Omit for IAM role-based
authentication.</description>}}
{{ <value>accesskey</value>}}
{{ </property>}}{{<property>}}
{{ <name>fs.s3a.secret.key</name>}}
{{ <description>AWS secret key. Omit for IAM role-based
authentication.</description>}}
{{ <value>secretkey</value>}}
{{ </property>}}{{<property>}}
{{ <name>test.sts.endpoint</name>}}
{{ <description>Specific endpoint to use for STS requests.</description>}}
{{ <value>sts.amazonaws.com</value>}}
{{ </property>}}{{<property>}}
{{ <name>test.fs.s3a.endpoint</name>}}
{{ <value>${oregon.endpoint}</value>}}
{{ </property>}}{{<property>}}
{{ <name>fs.s3a.server-side-encryption.key</name>}}
{{ <value>key-arn</value>}}
{{ </property>}}{{<property>}}
{{ <name>fs.s3a.assumed.role.arn</name>}}
{{ <value>test-role-arn</value>}}
{{ </property>}}{{<property>}}
{{ <name>fs.s3a.delegation.token.endpoint</name>}}
{{ <value>fs.s3a.assumed.role.sts.endpoint</value>}}
{{ </property>}}{{<property>}}
{{ <name>fs.s3a.scale.test.enabled</name>}}
{{ <value>true</value>}}
{{</property>}}
> Let s3 clients configure request timeout
> ----------------------------------------
>
> Key: HADOOP-16792
> URL: https://issues.apache.org/jira/browse/HADOOP-16792
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Mustafa Iman
> Assignee: Mustafa Iman
> Priority: Major
>
> S3 does not guarantee latency. Every once in a while a request may straggle
> and drive latency up for the greater procedure. In these cases, simply
> timing-out the individual request is beneficial so that the client
> application can retry. The retry tends to complete faster than the original
> straggling request most of the time. Others experienced this issue too:
> [https://arxiv.org/pdf/1911.11727.pdf] .
> S3 configuration already provides timeout facility via
> `ClientConfiguration#setTimeout`. Exposing this configuration is beneficial
> for latency sensitive applications. S3 client configuration is shared with
> DynamoDB client which is also affected from unreliable worst case latency.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
