Re: [DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Hello Thomas, Thank-you for your vote and feedback on the FLIP. Q: Do you see the polling (non-EFO) mode as a permanent option going forward? A: I will follow up on this, I have forwarded the question on. But generally speaking AWS do not usually deprecate APIs. KDS (Kinesis Data Streams) will

Re: [DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Thanks for the excellent proposal! Big +1 for introducing EFO as an incremental feature while retaining backward compatibility! This will make it easier for users to adopt. Thanks for mentioning the reasons why one might not want to use EFO. Regarding "Streams with a single consumer would not

Re: [DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Wow, that is one thorough FLIP! I didn't fully go into all the technical details but I think the general direction of this is good. If no one objects I'd say we can proceed to voting and figure out the technical details during implementation/review (if any remain unclear). Best, Aljoscha On

Re: [DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Thanks for updating the FLIP Danny. Changes look good to me. Please feel free to proceed with a vote soon. On Tue, Jun 30, 2020 at 11:19 PM Cranmer, Danny wrote: > Hey Gordon, > > I have updated the FLIP [1] to include support for configurable > registration strategies: > - Added 2 additional

Re: [DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Hey Gordon, I have updated the FLIP [1] to include support for configurable registration strategies: - Added 2 additional configuration keys - Added Registration/De-registration Configuration section - Updated Stream Consumer Registration/Tear Down section - Remove rejected alternative (since we

Re: [DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Hey Gordon, Thank-you for you review and feedback. I agree with your suggestion for the contribution plan. I have updated the FLIP to include the additional step with the FanOutKinesisProxy/AWS SDK 2.x dependency. I have also added another precursor step to generally improve test coverage. I

Re: [DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Also, if it wasn't clear, I'll be happy to provide committer support on reviewing and merging this FLIP, if it gets approved :) On Mon, Jun 29, 2020 at 12:12 PM Tzu-Li (Gordon) Tai wrote: > Hi Cranmer, > > Thank you for proposing the feature and starting the discussion thread. > This is really

Re: [DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Hi Cranmer, Thank you for proposing the feature and starting the discussion thread. This is really great work! Overall, +1 to adding EFO support to the Kinesis connector. I can see that having a dedicated throughput quota for each consuming Flink application is definitely a requirement for AWS

Re: [DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Hi, Cranmer. I'm Roland Wang. I've read the FLIP you wrote, and agree with your design. Recently, I'm working on this feature too, and have made some progress: 1. I add two methods: getOrRegisterConsumer & subscribeToShard on KinesisProxyInterface. 2. I re-implement the

[DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Hello everyone, This is a discussion thread for the FLIP I have recently created: * FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers *

[DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

Hello everyone, This is a discussion thread for the FLIP [1] regarding Enhanced Fan Out for AWS Kinesis Consumers. Enhanced Fan Out (EFO) allows AWS Kinesis Data Stream (KDS) consumers to utilise a dedicated read throughput, rather than a shared quota. HTTP/2 reduces latency and typically