I’d like to get back to this question, fairly raised by Jonothan a while ago, since actually it affects not only KinesisIO but also all other AWS IOs in general, that uses AWS SDK of two different versions.
My personal opinion - I’d strongly like to avoid a copy-pasted code for two different versions of the same IO (and double support of this), that is SDK-independent. In this case, if we wish too provide IO for two AWS SDK versions, we have to extract this core logic into one common and SDK-independent framework that could used by any of SDK versions then. In the same time, if it will require a lot of efforts to achieve that, then probably we need to decide to leave only one IO based on single AWS SDK version (I guess it will be V2 as more modern) and deprecate old version (and then remove it to avoid a confusion for users). What others think about that? > On 18 Apr 2020, at 01:48, Jonothan Farr <jonothan.f...@gmail.com> wrote: > > Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was > working on, so I ended up submitting that as > https://github.com/apache/beam/pull/9899 > <https://github.com/apache/beam/pull/9899>. > > But since that still needs work to be fully functional (right now it does not > handle resharding), I figured I could at least contribute the updates to make > KinesisIO compatible with the AWS SDK v2, so I split that out and submitted > it as: https://github.com/apache/beam/pull/11318 > <https://github.com/apache/beam/pull/11318>. > > Now some totally reasonable concerns have been raised about maintaining two > separate codebases, and having to dual-commit every bugfix or feature > enhancement. That seems to already be the case for all of the other AWS IOs > in amazon-web-services2 (dynamodb, sns, sqs), but do we want to continue that > trend? What do members of the community think? > > One solution is obviously to rewrite the AWS IOs with an added abstraction > layer so that the SDK-specific code can be factored out. However, the > KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm not > aware of a way that this could be done so that it's backwards-compatible. > This means that everyone currently using KinesisIO would most likely have to > update their code to use the new interface whether they wanted to use the new > v2 SDK or continue to use v1. I think a rewrite would also be a non-trivial > task with more than a few details to be worked out first. > > From my perspective, the options are: > > - Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO > and I can continue working on enhanced fanout while a rewrite is being worked > on, but the community now has to dual-commit any changes to KinesisIO. > - Close #11318 and continue working on #9899 and I alone have to dual-commit > the changes until it can be merged, at which point everyone can use enhanced > fanout while the rewrite is being worked on. Then the community will also > have to dual-commit but only after enhanced fanout is available. > - Close both PRs and just wait for a rewrite, and no one can use the v2 AWS > SDK or enhanced fanout in KinesisIO until after it's done. (Not to imply that > I wouldn't be interested in contributing to that effort, just for the sake of > these 2 PRs.) > - Don't do the rewrite at all and just live with dual-committing until the > KinesisIO based on the v1 SDK reaches its end of life. No one has to update > their code to continue using the v1 SDK, only if they want to move to v2. > > There may be other options or considerations I'm missing here. What do others > think? > > Thanks, > Jonothan
