On Tue, Nov 26, 2019 at 6:17 PM Reza Rokni <r...@google.com> wrote: > Hi Alexey, > > With regards to @Experimental there are a couple of discussions around its > usage ( or rather over usage! ) on dev@. It is something that we need to > clean up ( some of those IO are now being used on production env for > years!). >
I agree that we should move some IO connectors out of experimental state and probably this should be a separate discussion. Also, this issue is probably more than for IO connectors since there are other parts of the code that is marked as experimental as well, sometimes for a good reason (for example, SDF). > > Cheers > > Reza > > On Wed, 27 Nov 2019 at 04:50, Luke Cwik <lc...@google.com> wrote: > >> I suggested the wrapper because sometimes the intent of the APIs can be >> translated easily but this is not always the case. >> >> Good to know that it is all marked @Experimental. >> >> On Tue, Nov 26, 2019 at 12:30 PM Cam Mach <cammac...@gmail.com> wrote: >> >>> Thank you, Alex for sharing the information, and Luke for the questions. >>> I like the idea that just depreciate the V1 IOs, and just maintain V2 >>> IOs, so we can support whoever want continue with V1. >>> Just as Alex said, a lot of users, including my teams :-) , use the V1 >>> IOs in production for real workload. So it'll be hard to remove V1 IOs or >>> force them migrate to V2. But let hear if there are any other ideas? >>> >>> Btw, making V1 a wrapper around V2 is not very positive, code will get >>> more complicated since V2 API is very different from V1's. >>> >>> Thanks, >>> >>> >>> >>> On Tue, Nov 26, 2019 at 8:21 AM Alexey Romanenko < >>> aromanenko....@gmail.com> wrote: >>> >>>> AFAICT, all AWS SDK V1 IOs (SnsIO, SqsIO, DynamoDBIO, KinesisIO) are >>>> marked as "Experimental". So, it should not be a problem to gracefully >>>> deprecate and finally remove them. We already did the similar procedure for >>>> “HadoopInputFormatIO”, which was renamed to just “HadoopFormatIO” (since it >>>> started to support HadoopOutputFormatI as well). Old “HadoopInputFormatIO” >>>> was deprecated and removed after *3 consecutive* Beam releases (as we >>>> agreed on mailing list). >>>> >>>> In the same time, some users for some reasons would not be able or to >>>> want to move on AWS SDK V2. So, I’d prefer to just deprecate AWS SDK V1 IOs >>>> and accept new features/fixes *only* for V2 IOs. >>>> >>> +1 for deprecating AWS V1 IO connectors as opposed to removing as well unless we can confirm that usage is extremely limited. > >>>> Talking about “Experimental” annotation. Sorry in advance If I missed >>>> that and switch a subject a bit, but do we have clear rules or an agreement >>>> when IO becomes stable and should not be marked as experimental anymore? >>>> *Most* of our Java IOs are marked as Experimental but many of them >>>> were using in production by real users under real load. Does it mean that >>>> they are ready to be stable in terms of API? Perhaps, this topic deserves a >>>> new discussion if there are several opinions on that. >>>> >>> Probably, decision to move component APIs (for example, an IO connector) out of experimental state should be done on a case-by-case basis. Thanks, Cham > >>>> On 26 Nov 2019, at 00:39, Luke Cwik <lc...@google.com> wrote: >>>> >>>> Phase I sounds fine. >>>> >>>> Apache Beam follows semantic versioning and I believe removing the IOs >>>> will be a backwards incompatible change unless they were marked >>>> experimental which will be a problem for Phase 2. >>>> >>>> What is the feasibility of making the V1 transforms wrappers around V2? >>>> >>>> On Mon, Nov 25, 2019 at 1:46 PM Cam Mach <cammac...@gmail.com> wrote: >>>> >>>>> Hello Beam Devs, >>>>> >>>>> I have been working on the migration of Amazon Web Services IO >>>>> connectors into the new AWS SDK for Java V2. The goal is to have an >>>>> updated >>>>> implementation aligned with the most recent AWS improvements. So far we >>>>> have already migrated the connectors for AWS SNS, SQS and DynamoDB. >>>>> >>>>> In the meantime some contributions are still going on V1 IOs. So far >>>>> we have dealt with those by porting (or asking contributors) to port the >>>>> changes into V2 IOs too because we don’t want features of both versions to >>>>> be unaligned but this may quickly become a maintenance issue, so we want >>>>> to >>>>> discuss a plan to stop supporting (deprecate) V1 IOs and encourage users >>>>> to >>>>> move to V2. >>>>> >>>>> Phase I (ASAP): >>>>> >>>>> - Mark migrated AWS V1 IOs as deprecated >>>>> - Document migration path to V2 >>>>> >>>>> Phase II (end of 2020): >>>>> >>>>> - Decide a date or Beam release to remove the V1 IOs >>>>> - Send a notification to the community 3 months before we remove >>>>> them >>>>> - Completely get rid of V1 IOs >>>>> >>>>> >>>>> Please let me know what you think or if you see any potential issues? >>>>> >>>>> Thanks, >>>>> Cam Mach >>>>> >>>>> >>>> > > -- > > This email may be confidential and privileged. If you received this > communication by mistake, please don't forward it to anyone else, please > erase all copies and attachments, and please let me know that it has gone > to the wrong person. > > The above terms reflect a potential business arrangement, are provided > solely as a basis for further discussion, and are not intended to be and do > not constitute a legally binding obligation. No legally binding obligations > will be created, implied, or inferred until an agreement in final form is > executed in writing by all parties involved. >
