steveloughran commented on a change in pull request #1826: HADOOP-16823. Manage
S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#discussion_r377181603
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -174,10 +174,34 @@ private Constants() {
public static final String PROXY_DOMAIN = "fs.s3a.proxy.domain";
public static final String PROXY_WORKSTATION = "fs.s3a.proxy.workstation";
- // number of times we should retry errors
+ /**
+ * Number of times the AWS client library should retry errors before
+ * escalating to the S3A code: {@value}.
+ */
public static final String MAX_ERROR_RETRIES = "fs.s3a.attempts.maximum";
+
+ /**
+ * Default number of times the AWS client library should retry errors before
+ * escalating to the S3A code: {@value}.
+ */
public static final int DEFAULT_MAX_ERROR_RETRIES = 10;
+ /**
+ * Experimental/Unstable feature: should the AWS client library retry
+ * throttle responses before escalating to the S3A code: {@value}.
+ *
+ * When set to false, the S3A connector sees all S3 throttle events,
+ * And so can update it counters and the metrics, and use its own retry
+ * policy.
+ * However, this may have adverse effects on some operations where the S3A
+ * code cannot retry as efficiently as the AWS client library.
+ *
+ * This only applies to S3 operations, not to DynamoDB or other services.
+ */
+ @InterfaceStability.Unstable
+ public static final String EXPERIMENTAL_AWS_INTERNAL_THROTTLING =
+ "fs.s3a.experimental.aws.internal.throttling";
Review comment:
Updating the value, also changing the name to
""fs.s3a.experimental.aws.s3.throttling" to make clear its s3 only
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
