[jira] [Updated] (HADOOP-13637) improve setting of max connections in AWS client
[ https://issues.apache.org/jira/browse/HADOOP-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13637: Parent Issue: HADOOP-15620 (was: HADOOP-15220) > improve setting of max connections in AWS client > > > Key: HADOOP-13637 > URL: https://issues.apache.org/jira/browse/HADOOP-13637 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > things can go badly wrong if the S3A FS creates a thread pool for IO > than > the number of pooled AWS http connections (set by property > MAXIMUM_CONNECTIONS); you also need some for any other IO requests coming in. > The max connections property is currently independent of thread pool size, > and has a default value of 1. > this is why there is a troubleshooting section in the docs showing the stack > trace and instructions to fix". > Better: have a dynamic minimum like thread pool size + n, for a value of n to > be chosen. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13637) improve setting of max connections in AWS client
[ https://issues.apache.org/jira/browse/HADOOP-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13637: Target Version/s: (was: 3.1.0) > improve setting of max connections in AWS client > > > Key: HADOOP-13637 > URL: https://issues.apache.org/jira/browse/HADOOP-13637 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > things can go badly wrong if the S3A FS creates a thread pool for IO > than > the number of pooled AWS http connections (set by property > MAXIMUM_CONNECTIONS); you also need some for any other IO requests coming in. > The max connections property is currently independent of thread pool size, > and has a default value of 1. > this is why there is a troubleshooting section in the docs showing the stack > trace and instructions to fix". > Better: have a dynamic minimum like thread pool size + n, for a value of n to > be chosen. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13637) improve setting of max connections in AWS client
[ https://issues.apache.org/jira/browse/HADOOP-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13637: - Target Version/s: 3.1.0 (was: 2.9.0) > improve setting of max connections in AWS client > > > Key: HADOOP-13637 > URL: https://issues.apache.org/jira/browse/HADOOP-13637 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > things can go badly wrong if the S3A FS creates a thread pool for IO > than > the number of pooled AWS http connections (set by property > MAXIMUM_CONNECTIONS); you also need some for any other IO requests coming in. > The max connections property is currently independent of thread pool size, > and has a default value of 1. > this is why there is a troubleshooting section in the docs showing the stack > trace and instructions to fix". > Better: have a dynamic minimum like thread pool size + n, for a value of n to > be chosen. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13637) improve setting of max connections in AWS client
[ https://issues.apache.org/jira/browse/HADOOP-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13637: Parent Issue: HADOOP-14831 (was: HADOOP-13204) > improve setting of max connections in AWS client > > > Key: HADOOP-13637 > URL: https://issues.apache.org/jira/browse/HADOOP-13637 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > things can go badly wrong if the S3A FS creates a thread pool for IO > than > the number of pooled AWS http connections (set by property > MAXIMUM_CONNECTIONS); you also need some for any other IO requests coming in. > The max connections property is currently independent of thread pool size, > and has a default value of 1. > this is why there is a troubleshooting section in the docs showing the stack > trace and instructions to fix". > Better: have a dynamic minimum like thread pool size + n, for a value of n to > be chosen. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
