Hey Priya,
Having local dockerized integration testing would help devs to run the
integration tests when they don't have AWS service credentials.
You could have a look to the DynamoDB [1] or Kinesis [2] connector
integration tests. DynamoDBContainer extends interface from
Testcontainers [3].
Best,
Muhammet
[1]:
https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/sink/DynamoDbSinkITCase.java#L74-L78
[2]:
https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisITCase.java#L87-L89
[3]:
https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/testutils/DynamoDbContainer.java
On 2024-04-15 20:31, Dhingra, Priya wrote:
Thanks Danny/ Muhammet for the feedback. Updated the FLIP accordingly.
I am not familiar with docker based integration testing but if that is
what we have used for other AWS services, I can follow the same here.
If you can share any existing sample code link that would be very
helpful.
Thanks
Priya
On 4/15/24, 5:22 AM, "Danny Cranmer" <dannycran...@apache.org
<mailto:dannycran...@apache.org>> wrote:
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you can confirm the sender and
know the content is safe.
Thanks for the FLIP Priya. I have seen many custom implementations of
SQS
sinks, so having an Apache supported connector will be great.
I agree on the points about implementation detail in the FLIP. There is
still too much code really, we will review that in the PR. The FLIP is
mainly to agree we should add this connector, we can leave the
implementation details for later.
nit: Wrt to versioning, it is actually Flink 1.18 and 1.19 now, since
1.17
is out of support. But since we are using the flink-connector-aws repo
we
will use the supported version there.
Would it make sense to use here docker based integration testing using
Testcontainer
+1, we typically use localstack for the Integration tests and hit AWS
in
the e2e test packages. Having coverage of both will be great.
Thanks,
Danny
On Mon, Apr 15, 2024 at 3:23 AM Muhammet Orazov
<mor+fl...@morazow.com.inva <mailto:mor+fl...@morazow.com.inva>lid>
wrote:
Hey Priya,
Thanks for the FLIP and driving it!
One question from my side on test plan:
> End to end integration tests that hit the real Amazon SQS service.
> These tests will be enabled when credentials are defined.
Would it make sense to use here docker based integration testing using
Testcontainer[1], for example, with Localstack[2]? Seems like the SQS
service is provided on the free tier.
[1]: https://java.testcontainers.org/modules/localstack/
<https://java.testcontainers.org/modules/localstack/>
[2]: https://docs.localstack.cloud/user-guide/aws/sqs/
<https://docs.localstack.cloud/user-guide/aws/sqs/>
On 2024-04-05 18:16, Dhingra, Priya wrote:
> Hi Dev,
>
> I would like to start a discussion about FLIP-438: Amazon SQS Sink
> Connector<
https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector>
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector>>.
> This FLIP is proposing to add support for AWS SQS sink in
> flink-connector-aws repo.
>
> For more details, see FLIP-438. Looking forward to your feedback.
>
https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector>
>
> Thanks,
> Priya
