Thanks Alex for the summary, and all for your contributes. I have been waiting for a couple of days, and so far we don't have any objections. So I guess we can move forward with this approach. We can, of course, wait until next week to see if anyone else has any ideas, and then make a final decision to close this discussion :-)
Thanks, Cam On Thu, Nov 28, 2019 at 7:47 AM Alexey Romanenko <aromanenko....@gmail.com> wrote: > Going back to main subject of this thread, I just wanted to make things > clear for all. > > Seems like that everybody is agree that we will *just deprecate* AWS SDK > V1 connectors once the alternative will be available, *don’t remove* them > and still *distribute artifacts* [1] with new releases along with > artifacts with IOs based on AWS SDK V2 [2]. > > Do we need to vote for this or we can accept it without voting if there > are no objections? > > [1] > https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services > [2] > https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services2 > > > On 27 Nov 2019, at 18:44, Kenneth Knowles <k...@google.com> wrote: > > > On Tue, Nov 26, 2019 at 7:00 PM Chamikara Jayalath <chamik...@google.com> > wrote: > >> >> >> On Tue, Nov 26, 2019 at 6:17 PM Reza Rokni <r...@google.com> wrote: >> >>> With regards to @Experimental there are a couple of discussions around >>> its usage ( or rather over usage! ) on dev@. It is something that we >>> need to clean up ( some of those IO are now being used on production env >>> for years!). >>> >> >> I agree that we should move some IO connectors out of experimental state >> and probably this should be a separate discussion. Also, this issue is >> probably more than for IO connectors since there are other parts of the >> code that is marked as experimental as well, sometimes for a good reason >> (for example, SDF). >> > > Yes, let's have a separate thread on @Experimental. There are a ton of > threads that start talking about it, and they all seem to agree it isn't > working. Only one direct thread* that was about something a bit more > specific > https://lists.apache.org/thread.html/302bd51c77feb5c9ce39882316d391535a0fc92e7608a623d9139160@%3Cdev.beam.apache.org%3E > > > > >> On Tue, Nov 26, 2019 at 8:21 AM Alexey Romanenko < >>>>> aromanenko....@gmail.com> wrote: >>>>> >>>>>> AFAICT, all AWS SDK V1 IOs (SnsIO, SqsIO, DynamoDBIO, KinesisIO) are >>>>>> marked as "Experimental". So, it should not be a problem to gracefully >>>>>> deprecate and finally remove them. We already did the similar procedure >>>>>> for >>>>>> “HadoopInputFormatIO”, which was renamed to just “HadoopFormatIO” (since >>>>>> it >>>>>> started to support HadoopOutputFormatI as well). Old >>>>>> “HadoopInputFormatIO” >>>>>> was deprecated and removed after *3 consecutive* Beam releases (as >>>>>> we agreed on mailing list). >>>>>> >>>>>> In the same time, some users for some reasons would not be able or to >>>>>> want to move on AWS SDK V2. So, I’d prefer to just deprecate AWS SDK V1 >>>>>> IOs >>>>>> and accept new features/fixes *only* for V2 IOs. >>>>>> >>>>> >> +1 for deprecating AWS V1 IO connectors as opposed to removing as well >> unless we can confirm that usage is extremely limited. >> > > +1 to deprecate as soon as there is an alternative. > > Trying to measure usage is a good idea, but hard. If the maven coordinates > were different we could look at download numbers and dependencies. > > > Talking about “Experimental” annotation. Sorry in advance If I missed that >>>>>> and switch a subject a bit, but do we have clear rules or an agreement >>>>>> when >>>>>> IO becomes stable and should not be marked as experimental anymore? >>>>>> *Most* of our Java IOs are marked as Experimental but many of them >>>>>> were using in production by real users under real load. Does it mean that >>>>>> they are ready to be stable in terms of API? Perhaps, this topic >>>>>> deserves a >>>>>> new discussion if there are several opinions on that. >>>>>> >>>>> >> Probably, decision to move component APIs (for example, an IO connector) >> out of experimental state should be done on a case-by-case basis. >> > > Let's repeat these good points on a dedicated thread. > > Kenn > > > >> >> Thanks, >> Cham >> >> >>> >>>>>> On 26 Nov 2019, at 00:39, Luke Cwik <lc...@google.com> wrote: >>>>>> >>>>>> Phase I sounds fine. >>>>>> >>>>>> Apache Beam follows semantic versioning and I believe removing the >>>>>> IOs will be a backwards incompatible change unless they were marked >>>>>> experimental which will be a problem for Phase 2. >>>>>> >>>>>> What is the feasibility of making the V1 transforms wrappers around >>>>>> V2? >>>>>> >>>>>> On Mon, Nov 25, 2019 at 1:46 PM Cam Mach <cammac...@gmail.com> wrote: >>>>>> >>>>>>> Hello Beam Devs, >>>>>>> >>>>>>> I have been working on the migration of Amazon Web Services IO >>>>>>> connectors into the new AWS SDK for Java V2. The goal is to have an >>>>>>> updated >>>>>>> implementation aligned with the most recent AWS improvements. So far we >>>>>>> have already migrated the connectors for AWS SNS, SQS and DynamoDB. >>>>>>> >>>>>>> In the meantime some contributions are still going on V1 IOs. So far >>>>>>> we have dealt with those by porting (or asking contributors) to port the >>>>>>> changes into V2 IOs too because we don’t want features of both versions >>>>>>> to >>>>>>> be unaligned but this may quickly become a maintenance issue, so we >>>>>>> want to >>>>>>> discuss a plan to stop supporting (deprecate) V1 IOs and encourage >>>>>>> users to >>>>>>> move to V2. >>>>>>> >>>>>>> Phase I (ASAP): >>>>>>> >>>>>>> - Mark migrated AWS V1 IOs as deprecated >>>>>>> - Document migration path to V2 >>>>>>> >>>>>>> Phase II (end of 2020): >>>>>>> >>>>>>> - Decide a date or Beam release to remove the V1 IOs >>>>>>> - Send a notification to the community 3 months before we remove >>>>>>> them >>>>>>> - Completely get rid of V1 IOs >>>>>>> >>>>>>> >>>>>>> Please let me know what you think or if you see any potential issues? >>>>>>> >>>>>>> Thanks, >>>>>>> Cam Mach >>>>>>> >>>>>>> >>>>>> >>> >>> -- >>> This email may be confidential and privileged. If you received this >>> communication by mistake, please don't forward it to anyone else, please >>> erase all copies and attachments, and please let me know that it has gone >>> to the wrong person. >>> The above terms reflect a potential business arrangement, are provided >>> solely as a basis for further discussion, and are not intended to be and do >>> not constitute a legally binding obligation. No legally binding obligations >>> will be created, implied, or inferred until an agreement in final form is >>> executed in writing by all parties involved. >>> >> >
