[
https://issues.apache.org/jira/browse/FLINK-25949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hamdy updated FLINK-25949:
--------------------------------
Description:
h2. Description
As captured by {{KinesisFirehoseITCase}} merged in
[PR#18314|http://https://github.com/apache/flink/pull/18314] , Firehose client
created using default values in {{KinesisFirehoseSinkBuilder}} is unstable.
h2.Proposed changes
- Default protocol overriden in {{KinesisFirehoseSinkBuilder}} is instance of
{{software.amazon.awssdk.http.Protocol}} while required type is {{String}}.
Protocol inserted should be of type {{String}}.
- Unit tests should be added to capture the issue if reoccured.
was:
Default protocol overriden in {{KinesisFirehoseSinkBuilder}} is instance of
{{software.amazon.awssdk.http.Protocol}} while required type is {{String}}.
Construction of firehose client using default protocol in
{{KinesisFirehoseITCase}} is unstable
Example of error log trace due in a test failure is:
{code:java}
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2022-02-01T21:47:41.9320469Z Feb 01 21:47:41 at
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2022-02-01T21:47:41.9321158Z Feb 01 21:47:41 at
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:141)
2022-02-01T21:47:41.9321834Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2022-02-01T21:47:41.9322409Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2022-02-01T21:47:41.9370284Z Feb 01 21:47:41 ... 4 more
2022-02-01T21:47:41.9371234Z Feb 01 21:47:41 Caused by:
org.apache.flink.connector.aws.util.AWSAuthenticationException: Encountered
non-recoverable exception relating to mis-configured client
2022-02-01T21:47:41.9371989Z Feb 01 21:47:41 at
org.apache.flink.connector.aws.util.AWSCredentialExceptionWrappers.lambda$static$1(AWSCredentialExceptionWrappers.java:43)
2022-02-01T21:47:41.9372744Z Feb 01 21:47:41 at
org.apache.flink.util.ThrowableWrapper.shouldSuppress(ThrowableWrapper.java:43)
2022-02-01T21:47:41.9373321Z Feb 01 21:47:41 at
org.apache.flink.util.ThrowableWrapper.shouldSuppress(ThrowableWrapper.java:48)
2022-02-01T21:47:41.9373870Z Feb 01 21:47:41 at
org.apache.flink.util.ThrowableWrapper.shouldSuppress(ThrowableWrapper.java:48)
2022-02-01T21:47:41.9374504Z Feb 01 21:47:41 at
org.apache.flink.util.ThrowableWrapper.shouldSuppress(ThrowableWrapper.java:48)
2022-02-01T21:47:41.9375075Z Feb 01 21:47:41 at
org.apache.flink.util.ThrowableWrapper.shouldSuppress(ThrowableWrapper.java:48)
2022-02-01T21:47:41.9375755Z Feb 01 21:47:41 at
org.apache.flink.connector.firehose.sink.KinesisFirehoseSinkWriter.isRetryable(KinesisFirehoseSinkWriter.java:221)
2022-02-01T21:47:41.9376482Z Feb 01 21:47:41 at
org.apache.flink.connector.firehose.sink.KinesisFirehoseSinkWriter.handleFullyFailedRequest(KinesisFirehoseSinkWriter.java:188)
2022-02-01T21:47:41.9377235Z Feb 01 21:47:41 at
org.apache.flink.connector.firehose.sink.KinesisFirehoseSinkWriter.lambda$submitRequestEntries$2(KinesisFirehoseSinkWriter.java:165)
2022-02-01T21:47:41.9377906Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2022-02-01T21:47:41.9378479Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2022-02-01T21:47:41.9379065Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2022-02-01T21:47:41.9379649Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2022-02-01T21:47:41.9380273Z Feb 01 21:47:41 at
software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76)
2022-02-01T21:47:41.9380901Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2022-02-01T21:47:41.9381496Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2022-02-01T21:47:41.9382084Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2022-02-01T21:47:41.9382726Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2022-02-01T21:47:41.9383442Z Feb 01 21:47:41 at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:54)
2022-02-01T21:47:41.9384146Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2022-02-01T21:47:41.9384807Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2022-02-01T21:47:41.9385376Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2022-02-01T21:47:41.9386085Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2022-02-01T21:47:41.9386808Z Feb 01 21:47:41 at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda$execute$2(AsyncApiCallTimeoutTrackingStage.java:67)
2022-02-01T21:47:41.9387514Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2022-02-01T21:47:41.9388089Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2022-02-01T21:47:41.9388672Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2022-02-01T21:47:41.9389251Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2022-02-01T21:47:41.9389890Z Feb 01 21:47:41 at
software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76)
2022-02-01T21:47:41.9390504Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2022-02-01T21:47:41.9391206Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2022-02-01T21:47:41.9391793Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2022-02-01T21:47:41.9392357Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2022-02-01T21:47:41.9393065Z Feb 01 21:47:41 at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
2022-02-01T21:47:41.9393867Z Feb 01 21:47:41 at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:181)
2022-02-01T21:47:41.9394725Z Feb 01 21:47:41 at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
2022-02-01T21:47:41.9395412Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2022-02-01T21:47:41.9396079Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$WrappedErrorForwardingResponseHandler.onError(MakeAsyncHttpRequestStage.java:158)
2022-02-01T21:47:41.9405001Z Feb 01 21:47:41 at
software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.lambda$notifyIfResponseNotCompleted$2(ResponseHandler.java:402)
2022-02-01T21:47:41.9405783Z Feb 01 21:47:41 at
software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:181)
2022-02-01T21:47:41.9406460Z Feb 01 21:47:41 at
software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.notifyIfResponseNotCompleted(ResponseHandler.java:402)
2022-02-01T21:47:41.9407122Z Feb 01 21:47:41 at
software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.channelInactive(ResponseHandler.java:153)
2022-02-01T21:47:41.9407785Z Feb 01 21:47:41 at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
2022-02-01T21:47:41.9421085Z Feb 01 21:47:41 Caused by:
java.util.concurrent.CompletionException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: Server failed to send complete response. The channel was closed.
This may have been done by the client (e.g. because the request was aborted),
by the service (e.g. because there was a handshake error, the request took too
long, or the client tried to write on a read-only socket), or by an
intermediary party (e.g. because the channel was idle for too long).
2022-02-01T21:47:41.9422365Z Feb 01 21:47:41 at
software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:62)
2022-02-01T21:47:41.9423177Z Feb 01 21:47:41 at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
2022-02-01T21:47:41.9423916Z Feb 01 21:47:41 at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
2022-02-01T21:47:41.9426537Z Feb 01 21:47:41 Caused by:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: Server failed to send complete response. The channel was closed.
This may have been done by the client (e.g. because the request was aborted),
by the service (e.g. because there was a handshake error, the request took too
long, or the client tried to write on a read-only socket), or by an
intermediary party (e.g. because the channel was idle for too long).
2022-02-01T21:47:41.9427818Z Feb 01 21:47:41 at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
2022-02-01T21:47:41.9432517Z Feb 01 21:47:41 Caused by: java.io.IOException:
Server failed to send complete response. The channel was closed. This may have
been done by the client (e.g. because the request was aborted), by the service
(e.g. because there was a handshake error, the request took too long, or the
client tried to write on a read-only socket), or by an intermediary party (e.g.
because the channel was idle for too long).
'''
> [FLIP-171] Kinesis Firehose sink builder falls back to wrong http protocol.
> ---------------------------------------------------------------------------
>
> Key: FLINK-25949
> URL: https://issues.apache.org/jira/browse/FLINK-25949
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
> Reporter: Ahmed Hamdy
> Assignee: Ahmed Hamdy
> Priority: Major
> Labels: pull-request-available
>
> h2. Description
> As captured by {{KinesisFirehoseITCase}} merged in
> [PR#18314|http://https://github.com/apache/flink/pull/18314] , Firehose
> client created using default values in {{KinesisFirehoseSinkBuilder}} is
> unstable.
> h2.Proposed changes
> - Default protocol overriden in {{KinesisFirehoseSinkBuilder}} is instance of
> {{software.amazon.awssdk.http.Protocol}} while required type is {{String}}.
> Protocol inserted should be of type {{String}}.
> - Unit tests should be added to capture the issue if reoccured.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
