Re: Suggestions for aggregating records to a Kinesis Sink, (or generic Async Sink)?

2024-04-29 Thread Michael Marino
Hi Ahmed, hi Hong, Thanks for your responses. It sounds like the most promising would be to initially focus on the Global Window with the custom trigger. We don't need to be compatible with the aggregation used by the KPL (actually we would likely combine records in protobuf, and my impression i

Re: Suggestions for aggregating records to a Kinesis Sink, (or generic Async Sink)?

2024-04-29 Thread Ahmed Hamdy
Hi Michael, Unfortunately the new `KinesisDataStreamsSink` doesn't support aggregation yet. My suggestion if you want to use native kinesis aggregation is to use the latest connector version that supports KPL as sink for Table API, that would be 1.14.x. you could package the connector of that versi

Suggestions for aggregating records to a Kinesis Sink, (or generic Async Sink)?

2024-04-29 Thread Michael Marino
Hi all, We are currently using Flink 1.18.1 (AWS Managed Flink) and are writing to Kinesis streams in several of our applications using the Table API. In our use case, we would like to be able to aggregate multiple records (rows) together and emit them in a single Kinesis record. As far as I und