Antoni Reus created HADOOP-19640:
------------------------------------
Summary: Resource leak in AssumedRoleCredentialProvider
Key: HADOOP-19640
URL: https://issues.apache.org/jira/browse/HADOOP-19640
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.4.1
Reporter: Antoni Reus
When `org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider` is used in a
Hadoop Configuration object, it will attempt to `resolveCredentials()` inside
the constructor.
(lines 165-167)
```
// and force in a fail-fast check just to keep the stack traces less
// convoluted
resolveCredentials();
```
If this method fails, because the current identity is not able to assume role,
the constructor will throw an exception, and fail to close the `stsClient`,
`stsProvider` and any other resources that are created in the constructor,
leaking threads and other resources.
In a long running application, that handles Hadoop S3 file systems, where the
user can dynamically change to configured role to assume, and external id, this
will lead to eventually the system running out of resources due to the leaked
threads created by the AWS SDK clients that are not closed when a wrong role or
external id is used.
There are two potential fixes for this problem:
- Don't attempt to `resolveCredentials()` inside the constructor
- Wrap the `resolveCredentials()` in the constructor in a try/catch block,
that cleans the resources and rethrows the exception in the catch block.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
