Hi Saurabh, Thank you for sharing the FLIP for the SQS source connector. An SQS source connector will be a great addition to the Flink ecosystem, as there is a growing demand for SQS source/sink integration.
I have a few queries: 1. AWS has announced JSON protocol support in SQS [1]. Can you shed some light on how different protocols will be supported? 2. AWS SQS has two types of queues [2]. What are the implementation detail differences for the source connector? 3. Will the SQS source connector implement any kind of callbacks [3] on success to offer any kind of guarantee? [1] https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-sqs-support-json-protocol/ [2] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-types.html [3] https://docs.aws.amazon.com/step-functions/latest/dg/callback-task-sample-sqs.html Bests, Samrat On Fri, 19 Jul 2024 at 9:53 PM, Saurabh Singh <saurabhsingh9...@gmail.com> wrote: > Hi Fink Devs, > > Our team has been working on migrating various data pipelines to Flink to > leverage the benefits of exactly-once processing, checkpointing, and > stateful computing. We have several use cases built around the AWS SQS > Service. For this migration, we have developed an SQS Source Connector, > which enables us to run both stateless and stateful Flink-based jobs. > > We believe that this SQS Source Connector would be a valuable addition to > the existing connector set. Therefore, we propose a FLIP to include it. > > For more information, please refer to the FLIP document. > > > https://docs.google.com/document/d/1lreo27jNh0LkRs1Mj9B3wj3itrzMa38D4_XGryOIFks/edit?usp=sharing > > Thanks > Saurabh & Abhi >
