Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was working on, so I ended up submitting that as https://github.com/apache/beam/pull/9899.
But since that still needs work to be fully functional (right now it does not handle resharding), I figured I could at least contribute the updates to make KinesisIO compatible with the AWS SDK v2, so I split that out and submitted it as: https://github.com/apache/beam/pull/11318. Now some totally reasonable concerns have been raised about maintaining two separate codebases, and having to dual-commit every bugfix or feature enhancement. That seems to already be the case for all of the other AWS IOs in amazon-web-services2 (dynamodb, sns, sqs), but do we want to continue that trend? What do members of the community think? One solution is obviously to rewrite the AWS IOs with an added abstraction layer so that the SDK-specific code can be factored out. However, the KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm not aware of a way that this could be done so that it's backwards-compatible. This means that everyone currently using KinesisIO would most likely have to update their code to use the new interface whether they wanted to use the new v2 SDK or continue to use v1. I think a rewrite would also be a non-trivial task with more than a few details to be worked out first. >From my perspective, the options are: - Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO and I can continue working on enhanced fanout while a rewrite is being worked on, but the community now has to dual-commit any changes to KinesisIO. - Close #11318 and continue working on #9899 and I alone have to dual-commit the changes until it can be merged, at which point everyone can use enhanced fanout while the rewrite is being worked on. Then the community will also have to dual-commit but only after enhanced fanout is available. - Close both PRs and just wait for a rewrite, and no one can use the v2 AWS SDK or enhanced fanout in KinesisIO until after it's done. (Not to imply that I wouldn't be interested in contributing to that effort, just for the sake of these 2 PRs.) - Don't do the rewrite at all and just live with dual-committing until the KinesisIO based on the v1 SDK reaches its end of life. No one has to update their code to continue using the v1 SDK, only if they want to move to v2. There may be other options or considerations I'm missing here. What do others think? Thanks, Jonothan
