[ https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shilun Fan updated HADOOP-18915: -------------------------------- Hadoop Flags: Reviewed Target Version/s: 3.4.0 Description: Increases existing pool sizes, as with server scale and vector IO, larger pools are needed fs.s3a.connection.maximum 200 fs.s3a.threads.max 96 Adds new configuration options for v2 sdk internal timeouts, both with default of 60s: fs.s3a.connection.acquisition.timeout fs.s3a.connection.idle.time All the pool/timoeut options are covered in performance.md Moves all timeout/duration options in the s3a FS to taking temporal units (h, m, s, ms,...); retaining the previous default unit (normally millisecond) Adds a minimum duration for most of these, in order to recover from deployments where a timeout has been set on the assumption the unit was seconds, not millis. Uses java.time.Duration throughout the codebase; retaining the older numeric constants in org.apache.hadoop.fs.s3a.Constants for backwards compatibility; these are now deprecated. Adds new class AWSApiCallTimeoutException to be raised on sdk-related methods and also gateway timeouts. This is a subclass of org.apache.hadoop.net.ConnectTimeoutException to support existing retry logic. + reverted default value of fs.s3a.create.performance to false; inadvertently set to true during testing. was: Increases existing pool sizes, as with server scale and vector IO, larger pools are needed fs.s3a.connection.maximum 200 fs.s3a.threads.max 96 Adds new configuration options for v2 sdk internal timeouts, both with default of 60s: fs.s3a.connection.acquisition.timeout fs.s3a.connection.idle.time All the pool/timoeut options are covered in performance.md Moves all timeout/duration options in the s3a FS to taking temporal units (h, m, s, ms,...); retaining the previous default unit (normally millisecond) Adds a minimum duration for most of these, in order to recover from deployments where a timeout has been set on the assumption the unit was seconds, not millis. Uses java.time.Duration throughout the codebase; retaining the older numeric constants in org.apache.hadoop.fs.s3a.Constants for backwards compatibility; these are now deprecated. Adds new class AWSApiCallTimeoutException to be raised on sdk-related methods and also gateway timeouts. This is a subclass of org.apache.hadoop.net.ConnectTimeoutException to support existing retry logic. + reverted default value of fs.s3a.create.performance to false; inadvertently set to true during testing. > Tune/extend S3A http connection and thread pool settings > -------------------------------------------------------- > > Key: HADOOP-18915 > URL: https://issues.apache.org/jira/browse/HADOOP-18915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.4.0 > Reporter: Ahmar Suhail > Assignee: Steve Loughran > Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.7-aws > > > Increases existing pool sizes, as with server scale and vector > IO, larger pools are needed > fs.s3a.connection.maximum 200 > fs.s3a.threads.max 96 > Adds new configuration options for v2 sdk internal timeouts, > both with default of 60s: > fs.s3a.connection.acquisition.timeout > fs.s3a.connection.idle.time > All the pool/timoeut options are covered in performance.md > Moves all timeout/duration options in the s3a FS to taking > temporal units (h, m, s, ms,...); retaining the previous default > unit (normally millisecond) > Adds a minimum duration for most of these, in order to recover from > deployments where a timeout has been set on the assumption the unit > was seconds, not millis. > Uses java.time.Duration throughout the codebase; > retaining the older numeric constants in > org.apache.hadoop.fs.s3a.Constants for backwards compatibility; > these are now deprecated. > Adds new class AWSApiCallTimeoutException to be raised on > sdk-related methods and also gateway timeouts. This is a subclass > of org.apache.hadoop.net.ConnectTimeoutException to support > existing retry logic. > + reverted default value of fs.s3a.create.performance to false; > inadvertently set to true during testing. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org