Oups I meant 'would you like to start rolling this plan?' sorry.
On Fri, May 8, 2020 at 12:03 AM Ismaël Mejía <[email protected]> wrote: > > Achieving good abstractions will prove elusive since the APIs differ > and we will end up with a ton of extra maintenance work that should be > not Beam's responsibility. I know that similar code (almost copy > pasteable) is not nice to have but we should consider this as a > temporary measure and probably try to make deprecation (and possible > removal) of older versions as soon as possible. > Please remember that this was already discussed in the past [1] and > the soft consensus was around rapid deprecation AWS SDK v1 IOs already > available in their v2 version and only do improvements on v1 IOs for > security related issues and dependency updates. This has not happened > yet but the only missing thing is someone to take action. > > [email protected] would you have to me to start rolling this plan? > otherwise someone else can jump and do it too. > > [1] > https://lists.apache.org/thread.html/130cb60e6bcdd58c5afdd0c375663eaf05e705aab9ee0196535cd17f%40%3Cdev.beam.apache.org%3E > > On Thu, May 7, 2020 at 11:05 PM Luke Cwik <[email protected]> wrote: > > > > I think you should try and share as much as is reasonable. Using what is > > shared between AWS V1 and V2 SDKs would be a good signal as to what should > > be shared in Beam. There might be some places where a trivial wrapper could > > help but I wouldn't try to create a bunch of grand abstractions that fit > > both versions of the AWS SDKs. > > > > On Wed, May 6, 2020 at 9:59 AM Alexey Romanenko <[email protected]> > > wrote: > >> > >> I’d like to get back to this question, fairly raised by Jonothan a while > >> ago, since actually it affects not only KinesisIO but also all other AWS > >> IOs in general, that uses AWS SDK of two different versions. > >> > >> My personal opinion - I’d strongly like to avoid a copy-pasted code for > >> two different versions of the same IO (and double support of this), that > >> is SDK-independent. In this case, if we wish too provide IO for two AWS > >> SDK versions, we have to extract this core logic into one common and > >> SDK-independent framework that could used by any of SDK versions then. > >> > >> In the same time, if it will require a lot of efforts to achieve that, > >> then probably we need to decide to leave only one IO based on single AWS > >> SDK version (I guess it will be V2 as more modern) and deprecate old > >> version (and then remove it to avoid a confusion for users). > >> > >> What others think about that? > >> > >> On 18 Apr 2020, at 01:48, Jonothan Farr <[email protected]> wrote: > >> > >> Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was > >> working on, so I ended up submitting that as > >> https://github.com/apache/beam/pull/9899. > >> > >> But since that still needs work to be fully functional (right now it does > >> not handle resharding), I figured I could at least contribute the updates > >> to make KinesisIO compatible with the AWS SDK v2, so I split that out and > >> submitted it as: https://github.com/apache/beam/pull/11318. > >> > >> Now some totally reasonable concerns have been raised about maintaining > >> two separate codebases, and having to dual-commit every bugfix or feature > >> enhancement. That seems to already be the case for all of the other AWS > >> IOs in amazon-web-services2 (dynamodb, sns, sqs), but do we want to > >> continue that trend? What do members of the community think? > >> > >> One solution is obviously to rewrite the AWS IOs with an added abstraction > >> layer so that the SDK-specific code can be factored out. However, the > >> KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm > >> not aware of a way that this could be done so that it's > >> backwards-compatible. This means that everyone currently using KinesisIO > >> would most likely have to update their code to use the new interface > >> whether they wanted to use the new v2 SDK or continue to use v1. I think a > >> rewrite would also be a non-trivial task with more than a few details to > >> be worked out first. > >> > >> From my perspective, the options are: > >> > >> - Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO > >> and I can continue working on enhanced fanout while a rewrite is being > >> worked on, but the community now has to dual-commit any changes to > >> KinesisIO. > >> - Close #11318 and continue working on #9899 and I alone have to > >> dual-commit the changes until it can be merged, at which point everyone > >> can use enhanced fanout while the rewrite is being worked on. Then the > >> community will also have to dual-commit but only after enhanced fanout is > >> available. > >> - Close both PRs and just wait for a rewrite, and no one can use the v2 > >> AWS SDK or enhanced fanout in KinesisIO until after it's done. (Not to > >> imply that I wouldn't be interested in contributing to that effort, just > >> for the sake of these 2 PRs.) > >> - Don't do the rewrite at all and just live with dual-committing until the > >> KinesisIO based on the v1 SDK reaches its end of life. No one has to > >> update their code to continue using the v1 SDK, only if they want to move > >> to v2. > >> > >> There may be other options or considerations I'm missing here. What do > >> others think? > >> > >> Thanks, > >> Jonothan > >> > >>
