Re: Using S3 as stream source in Flink

2023-01-26 Thread Sriram Ganesh
I saw in aws-samples https://github.com/aws-samples/flink-stream-processing-refarch/blob/master/kinesis-taxi-stream-producer/src/main/java/com/amazonaws/flink/refarch/utils/TaxiEventReader.java they are not using FileSource. Now I got it. Thanks, Martijn. On Wed, Jan 25, 2023 at 9:07 PM Martijn

Re: Using S3 as stream source in Flink

2023-01-25 Thread Martijn Visser
Hi Sriram G, Both the DataStream and Table API support filesystem as a source in unbounded (streaming mode) with exactly once guarantees. This is documented at https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/filesystem/ and

Using S3 as stream source in Flink

2023-01-24 Thread Sriram Ganesh
Hi Everyone, I am thinking of switching my input source from Kafka to S3. First, I couldn't find any streaming source connector for S3. I have some basic questions about 1. How S3 will work as a streaming source with proper checkpointing.? 2. How Flink will manage the last offset processed from