Ryan Despain created FLINK-37683:
------------------------------------
Summary: [Connector][AWS] SSO credentials fail because of shaded
dependencies missing
Key: FLINK-37683
URL: https://issues.apache.org/jira/browse/FLINK-37683
Project: Flink
Issue Type: Bug
Components: Connectors / AWS
Affects Versions: aws-connector-5.0.0, aws-connector-4.3.0,
aws-connector-4.2.0, aws-connector-4.1.0, aws-connector-4.0.0,
aws-connector-3.0.0, aws-connector-3.1.0, aws-connector-4.4.0,
aws-connector-5.1.0
Environment: * Apache Flink (any version using {{flink-connector-aws}}
based on AWS SDK v2)
* {{flink-connector-aws}} (using {{{}KinesisStreamsSource{}}})
* Java application
* AWS credentials configured via {{~/.aws/config}} using SSO profiles (e.g.,
{{{}sso-session{}}}, {{{}sso_account_id{}}}, {{{}sso_role_name{}}}).
* Running locally via {{flink run}} or deploying a job using the shaded
connector.
Reporter: Ryan Despain
*Problem:* Users attempting to authenticate to AWS services (specifically
Kinesis via {{{}KinesisStreamsSource{}}}) using AWS SSO profiles configured in
{{~/.aws/config}} encounter a runtime error when using the standard shaded
{{{}flink-connector-aws{}}}. The application fails to load credentials. This
issue prevents users who rely on AWS SSO from using the connector.
*Steps to Reproduce:*
# Configure local AWS credentials using an SSO profile in
{{{}~/.aws/config{}}}. Ensure the profile works with the AWS CLI.
# Create a simple Flink Java application that uses {{KinesisStreamsSource}} to
connect to an AWS Kinesis stream.
# Include the {{org.apache.flink:flink-connector-aws:<version>}} dependency
(which brings in the shaded AWS SDK).
# Attempt to run the job using {{flink run}} or by building an application jar
and running it.
# _Expected Result:_ The job connects successfully using the SSO credentials.
# _Actual Result:_ The job fails with an error related to loading credentials.
_(Optional: Link to reproduction repository provided by the user:
[soisyourface/flink-kinesis-blah|https://www.google.com/search?q=https://github.com/soisyourface/flink-kinesis-blah])_
*Error Message:*
Unable to load credentials from any of the providers in the chain ...{{}}
And/or specifically:
{noformat}
To use Sso related properties in the 'default' profile, the 'sso' servicee
module must be on the class path.{noformat}
*Root Cause Analysis:* The AWS SDK for Java v2 uses
{{software.amazon.awssdk.auth.credentials.internal.ProfileCredentialsUtils}}
which dynamically loads the {{SSO_PROFILE_CREDENTIALS_PROVIDER_FACTORY}} using
{{{}ClassLoaderHelper.loadClass{}}}.
The {{flink-connector-aws}} project shades AWS SDK dependencies into
{{flink-connector-aws-base}} to avoid classpath conflicts. However, the
necessary AWS SDK modules for SSO functionality
({{{}software.amazon.awssdk:sso{}}} and {{{}software.amazon.awssdk:ssooidc{}}})
are not included as dependencies in the
{{{}flink-connector-aws-base/pom.xml{}}}.
Consequently, when Flink shades the dependencies, these modules are omitted
from the final shaded jar. At runtime, the {{ProfileCredentialsUtils}} attempts
to load the SSO factory class, but it cannot be found on the classpath because
it wasn't included in the shaded artifact, leading to the observed error.
*Proposed Solution:* Add {{software.amazon.awssdk:sso}} and
{{software.amazon.awssdk:ssooidc}} as {{<dependency>}} entries within the
{{<dependencies>}} section of the {{flink-connector-aws-base/pom.xml}} file in
the {{apache/flink-connector-aws}} repository. This will ensure these modules
are included when the connector is built and shaded.
*Verification:* The user who reported this issue verified locally that building
{{flink-connector-aws}} with these added dependencies resolved the problem in
their Flink application.
*Related PR:* A Pull Request with the proposed fix has been submitted:
[apache/flink-connector-aws#197|https://www.google.com/search?q=https://github.com/apache/flink-connector-aws/pull/197]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
