Thanks Danny/ Muhammet for the feedback. Updated the FLIP accordingly. I am not familiar with docker based integration testing but if that is what we have used for other AWS services, I can follow the same here. If you can share any existing sample code link that would be very helpful.
Thanks Priya On 4/15/24, 5:22 AM, "Danny Cranmer" <dannycran...@apache.org <mailto:dannycran...@apache.org>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Thanks for the FLIP Priya. I have seen many custom implementations of SQS sinks, so having an Apache supported connector will be great. I agree on the points about implementation detail in the FLIP. There is still too much code really, we will review that in the PR. The FLIP is mainly to agree we should add this connector, we can leave the implementation details for later. nit: Wrt to versioning, it is actually Flink 1.18 and 1.19 now, since 1.17 is out of support. But since we are using the flink-connector-aws repo we will use the supported version there. > Would it make sense to use here docker based integration testing using Testcontainer +1, we typically use localstack for the Integration tests and hit AWS in the e2e test packages. Having coverage of both will be great. Thanks, Danny On Mon, Apr 15, 2024 at 3:23 AM Muhammet Orazov <mor+fl...@morazow.com.inva <mailto:mor+fl...@morazow.com.inva>lid> wrote: > Hey Priya, > > Thanks for the FLIP and driving it! > > One question from my side on test plan: > > > End to end integration tests that hit the real Amazon SQS service. > > These tests will be enabled when credentials are defined. > > Would it make sense to use here docker based integration testing using > Testcontainer[1], for example, with Localstack[2]? Seems like the SQS > service is provided on the free tier. > > [1]: https://java.testcontainers.org/modules/localstack/ > <https://java.testcontainers.org/modules/localstack/> > [2]: https://docs.localstack.cloud/user-guide/aws/sqs/ > <https://docs.localstack.cloud/user-guide/aws/sqs/> > > On 2024-04-05 18:16, Dhingra, Priya wrote: > > Hi Dev, > > > > I would like to start a discussion about FLIP-438: Amazon SQS Sink > > Connector< > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector> > > <https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector>>. > > > This FLIP is proposing to add support for AWS SQS sink in > > flink-connector-aws repo. > > > > For more details, see FLIP-438. Looking forward to your feedback. > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector > > <https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector> > > > > Thanks, > > Priya >