[jira] [Created] (FLINK-32964) KinesisStreamsSink cant renew credentials with WebIdentityTokenFileCredentialsProvider
PhilippeB created FLINK-32964: - Summary: KinesisStreamsSink cant renew credentials with WebIdentityTokenFileCredentialsProvider Key: FLINK-32964 URL: https://issues.apache.org/jira/browse/FLINK-32964 Project: Flink Issue Type: Bug Components: Connectors / Kinesis Affects Versions: 1.17.1, 1.16.2, 1.15.4 Reporter: PhilippeB (First time filling a ticket in Flink community, please let me know if there are any guidelinges I need to follow) I noticed a very strange behavior with the Kinesis Sink. I actually using Flink in containerized and Application (reactive) mode on EKS with high availability on S3. Kinesis is configured with IAM role and appropried policies. {code:java} //Here a part of my flink-config.yaml: parallelism.default: 2 scheduler-mode: reactive execution.checkpointing.interval: 10s env.java.opts.jobmanager: -Dkubernetes.max.concurrent.requests=200 containerized.master.env.KUBERNETES_MAX_CONCURRENT_REQUESTS: 200 aws.credentials.provider: WEB_IDENTITY_TOKEN aws.credentials.role.arn: role aws.credentials.role.sessionName: session aws.credentials.webIdentityToken.file: /var/run/secrets/eks.amazonaws.com/serviceaccount/token {code} When my project is deployed the application and cluster are working well but when the project has been started for about an hour, I suppose the IAM roles session need to be renew, then the job become to crashing continuously. {code:java} 2023-08-24 10:35:55 java.lang.IllegalStateException: Connection pool shut down at org.apache.flink.kinesis.shaded.org.apache.http.util.Asserts.check(Asserts.java:34) at org.apache.flink.kinesis.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57) at org.apache.flink.kinesis.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) at org.apache.flink.kinesis.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.flink.kinesis.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:63) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:77) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:56) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:39) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.in
[jira] [Commented] (FLINK-32964) KinesisStreamsSink cant renew credentials with WebIdentityTokenFileCredentialsProvider
[ https://issues.apache.org/jira/browse/FLINK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760762#comment-17760762 ] PhilippeB commented on FLINK-32964: --- Hello [~mapohl] for the response. I can help for adding or completing some information on the issue. The connector Kinesis is actually shade some package from apache client and sdk v2. The probleme is actually hitting who using WebIdentityToken on EKS. I figure out some issue from aws-sdk-v2 project and can be the root cause. [https://github.com/aws/aws-sdk-java/issues/1282] [https://github.com/aws/aws-sdk-java-v2/issues/4221] > KinesisStreamsSink cant renew credentials with > WebIdentityTokenFileCredentialsProvider > -- > > Key: FLINK-32964 > URL: https://issues.apache.org/jira/browse/FLINK-32964 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.15.4, 1.16.2, 1.17.1 >Reporter: PhilippeB >Priority: Major > > (First time filling a ticket in Flink community, please let me know if there > are any guidelinges I need to follow) > I noticed a very strange behavior with the Kinesis Sink. I actually using > Flink in containerized and Application (reactive) mode on EKS with high > availability on S3. > Kinesis is configured with IAM role and appropried policies. > {code:java} > //Here a part of my flink-config.yaml: > parallelism.default: 2 > scheduler-mode: reactive > execution.checkpointing.interval: 10s > env.java.opts.jobmanager: -Dkubernetes.max.concurrent.requests=200 > containerized.master.env.KUBERNETES_MAX_CONCURRENT_REQUESTS: 200 > aws.credentials.provider: WEB_IDENTITY_TOKEN > aws.credentials.role.arn: role > aws.credentials.role.sessionName: session > aws.credentials.webIdentityToken.file: > /var/run/secrets/eks.amazonaws.com/serviceaccount/token {code} > When my project is deployed the application and cluster are working well but > when the project has been started for about an hour, I suppose the IAM roles > session need to be renew, then the job become to crashing continuously. > {code:java} > 2023-08-24 10:35:55 > java.lang.IllegalStateException: Connection pool shut down > at > org.apache.flink.kinesis.shaded.org.apache.http.util.Asserts.check(Asserts.java:34) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:63) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:77) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:56) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestS
[jira] [Comment Edited] (FLINK-32964) KinesisStreamsSink cant renew credentials with WebIdentityTokenFileCredentialsProvider
[ https://issues.apache.org/jira/browse/FLINK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760762#comment-17760762 ] PhilippeB edited comment on FLINK-32964 at 8/31/23 8:42 AM: Hello [~mapohl] thanks for the response. I can help for adding or completing some information on the issue. The connector Kinesis is actually shade some package from apache client and sdk v2. The probleme is actually hitting who using WebIdentityToken on EKS. I figure out some issue from aws-sdk-v2 project and can be the root cause. [https://github.com/aws/aws-sdk-java/issues/1282] [https://github.com/aws/aws-sdk-java-v2/issues/4221] was (Author: JIRAUSER301979): Hello [~mapohl] for the response. I can help for adding or completing some information on the issue. The connector Kinesis is actually shade some package from apache client and sdk v2. The probleme is actually hitting who using WebIdentityToken on EKS. I figure out some issue from aws-sdk-v2 project and can be the root cause. [https://github.com/aws/aws-sdk-java/issues/1282] [https://github.com/aws/aws-sdk-java-v2/issues/4221] > KinesisStreamsSink cant renew credentials with > WebIdentityTokenFileCredentialsProvider > -- > > Key: FLINK-32964 > URL: https://issues.apache.org/jira/browse/FLINK-32964 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.15.4, 1.16.2, 1.17.1 >Reporter: PhilippeB >Priority: Major > > (First time filling a ticket in Flink community, please let me know if there > are any guidelinges I need to follow) > I noticed a very strange behavior with the Kinesis Sink. I actually using > Flink in containerized and Application (reactive) mode on EKS with high > availability on S3. > Kinesis is configured with IAM role and appropried policies. > {code:java} > //Here a part of my flink-config.yaml: > parallelism.default: 2 > scheduler-mode: reactive > execution.checkpointing.interval: 10s > env.java.opts.jobmanager: -Dkubernetes.max.concurrent.requests=200 > containerized.master.env.KUBERNETES_MAX_CONCURRENT_REQUESTS: 200 > aws.credentials.provider: WEB_IDENTITY_TOKEN > aws.credentials.role.arn: role > aws.credentials.role.sessionName: session > aws.credentials.webIdentityToken.file: > /var/run/secrets/eks.amazonaws.com/serviceaccount/token {code} > When my project is deployed the application and cluster are working well but > when the project has been started for about an hour, I suppose the IAM roles > session need to be renew, then the job become to crashing continuously. > {code:java} > 2023-08-24 10:35:55 > java.lang.IllegalStateException: Connection pool shut down > at > org.apache.flink.kinesis.shaded.org.apache.http.util.Asserts.check(Asserts.java:34) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > at > org.apache.flink.kinesis.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231) > at > org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228) > at > org.apache.flink.kinesis.shaded.software.amazon.awss