[
https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18915:
--------------------------------
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Description:
Increases existing pool sizes, as with server scale and vector
IO, larger pools are needed
fs.s3a.connection.maximum 200
fs.s3a.threads.max 96
Adds new configuration options for v2 sdk internal timeouts,
both with default of 60s:
fs.s3a.connection.acquisition.timeout
fs.s3a.connection.idle.time
All the pool/timoeut options are covered in performance.md
Moves all timeout/duration options in the s3a FS to taking
temporal units (h, m, s, ms,...); retaining the previous default
unit (normally millisecond)
Adds a minimum duration for most of these, in order to recover from
deployments where a timeout has been set on the assumption the unit
was seconds, not millis.
Uses java.time.Duration throughout the codebase;
retaining the older numeric constants in
org.apache.hadoop.fs.s3a.Constants for backwards compatibility;
these are now deprecated.
Adds new class AWSApiCallTimeoutException to be raised on
sdk-related methods and also gateway timeouts. This is a subclass
of org.apache.hadoop.net.ConnectTimeoutException to support
existing retry logic.
+ reverted default value of fs.s3a.create.performance to false;
inadvertently set to true during testing.
was:
Increases existing pool sizes, as with server scale and vector
IO, larger pools are needed
fs.s3a.connection.maximum 200
fs.s3a.threads.max 96
Adds new configuration options for v2 sdk internal timeouts,
both with default of 60s:
fs.s3a.connection.acquisition.timeout
fs.s3a.connection.idle.time
All the pool/timoeut options are covered in performance.md
Moves all timeout/duration options in the s3a FS to taking
temporal units (h, m, s, ms,...); retaining the previous default
unit (normally millisecond)
Adds a minimum duration for most of these, in order to recover from
deployments where a timeout has been set on the assumption the unit
was seconds, not millis.
Uses java.time.Duration throughout the codebase;
retaining the older numeric constants in
org.apache.hadoop.fs.s3a.Constants for backwards compatibility;
these are now deprecated.
Adds new class AWSApiCallTimeoutException to be raised on
sdk-related methods and also gateway timeouts. This is a subclass
of org.apache.hadoop.net.ConnectTimeoutException to support
existing retry logic.
+ reverted default value of fs.s3a.create.performance to false;
inadvertently set to true during testing.
> Tune/extend S3A http connection and thread pool settings
> --------------------------------------------------------
>
> Key: HADOOP-18915
> URL: https://issues.apache.org/jira/browse/HADOOP-18915
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Ahmar Suhail
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.7-aws
>
>
> Increases existing pool sizes, as with server scale and vector
> IO, larger pools are needed
> fs.s3a.connection.maximum 200
> fs.s3a.threads.max 96
> Adds new configuration options for v2 sdk internal timeouts,
> both with default of 60s:
> fs.s3a.connection.acquisition.timeout
> fs.s3a.connection.idle.time
> All the pool/timoeut options are covered in performance.md
> Moves all timeout/duration options in the s3a FS to taking
> temporal units (h, m, s, ms,...); retaining the previous default
> unit (normally millisecond)
> Adds a minimum duration for most of these, in order to recover from
> deployments where a timeout has been set on the assumption the unit
> was seconds, not millis.
> Uses java.time.Duration throughout the codebase;
> retaining the older numeric constants in
> org.apache.hadoop.fs.s3a.Constants for backwards compatibility;
> these are now deprecated.
> Adds new class AWSApiCallTimeoutException to be raised on
> sdk-related methods and also gateway timeouts. This is a subclass
> of org.apache.hadoop.net.ConnectTimeoutException to support
> existing retry logic.
> + reverted default value of fs.s3a.create.performance to false;
> inadvertently set to true during testing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
