It's been for a week now, so it's time for us to conclude this discussion. A summary: 1) We have discussed about removing or keeping the existing V1 IOs. And, we decided to keep them, but not support and maintain, except for security reasons 2) We are going to mark the V1 IOs as deprecated and document to encourage forks to use V2.
Thanks again for your contributions to this thread Cam Mach On Fri, Nov 29, 2019 at 4:41 AM Cam Mach <cammac...@gmail.com> wrote: > Thanks Alex for the summary, and all for your contributes. > I have been waiting for a couple of days, and so far we don't have any > objections. So I guess we can move forward with this approach. > We can, of course, wait until next week to see if anyone else has any > ideas, and then make a final decision to close this discussion :-) > > Thanks, > Cam > > > > > On Thu, Nov 28, 2019 at 7:47 AM Alexey Romanenko <aromanenko....@gmail.com> > wrote: > >> Going back to main subject of this thread, I just wanted to make things >> clear for all. >> >> Seems like that everybody is agree that we will *just deprecate* AWS SDK >> V1 connectors once the alternative will be available, *don’t remove* >> them and still *distribute artifacts* [1] with new releases along with >> artifacts with IOs based on AWS SDK V2 [2]. >> >> Do we need to vote for this or we can accept it without voting if there >> are no objections? >> >> [1] >> https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services >> [2] >> https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services2 >> >> >> On 27 Nov 2019, at 18:44, Kenneth Knowles <k...@google.com> wrote: >> >> >> On Tue, Nov 26, 2019 at 7:00 PM Chamikara Jayalath <chamik...@google.com> >> wrote: >> >>> >>> >>> On Tue, Nov 26, 2019 at 6:17 PM Reza Rokni <r...@google.com> wrote: >>> >>>> With regards to @Experimental there are a couple of discussions around >>>> its usage ( or rather over usage! ) on dev@. It is something that we >>>> need to clean up ( some of those IO are now being used on production env >>>> for years!). >>>> >>> >>> I agree that we should move some IO connectors out of experimental state >>> and probably this should be a separate discussion. Also, this issue is >>> probably more than for IO connectors since there are other parts of the >>> code that is marked as experimental as well, sometimes for a good reason >>> (for example, SDF). >>> >> >> Yes, let's have a separate thread on @Experimental. There are a ton of >> threads that start talking about it, and they all seem to agree it isn't >> working. Only one direct thread* that was about something a bit more >> specific >> https://lists.apache.org/thread.html/302bd51c77feb5c9ce39882316d391535a0fc92e7608a623d9139160@%3Cdev.beam.apache.org%3E >> >> >> >> >>> On Tue, Nov 26, 2019 at 8:21 AM Alexey Romanenko < >>>>>> aromanenko....@gmail.com> wrote: >>>>>> >>>>>>> AFAICT, all AWS SDK V1 IOs (SnsIO, SqsIO, DynamoDBIO, KinesisIO) are >>>>>>> marked as "Experimental". So, it should not be a problem to gracefully >>>>>>> deprecate and finally remove them. We already did the similar procedure >>>>>>> for >>>>>>> “HadoopInputFormatIO”, which was renamed to just “HadoopFormatIO” >>>>>>> (since it >>>>>>> started to support HadoopOutputFormatI as well). Old >>>>>>> “HadoopInputFormatIO” >>>>>>> was deprecated and removed after *3 consecutive* Beam releases (as >>>>>>> we agreed on mailing list). >>>>>>> >>>>>>> In the same time, some users for some reasons would not be able or >>>>>>> to want to move on AWS SDK V2. So, I’d prefer to just deprecate AWS SDK >>>>>>> V1 >>>>>>> IOs and accept new features/fixes *only* for V2 IOs. >>>>>>> >>>>>> >>> +1 for deprecating AWS V1 IO connectors as opposed to removing as well >>> unless we can confirm that usage is extremely limited. >>> >> >> +1 to deprecate as soon as there is an alternative. >> >> Trying to measure usage is a good idea, but hard. If the maven >> coordinates were different we could look at download numbers and >> dependencies. >> >> >> Talking about “Experimental” annotation. Sorry in advance If I missed >>>>>>> that and switch a subject a bit, but do we have clear rules or an >>>>>>> agreement >>>>>>> when IO becomes stable and should not be marked as experimental anymore? >>>>>>> *Most* of our Java IOs are marked as Experimental but many of them >>>>>>> were using in production by real users under real load. Does it mean >>>>>>> that >>>>>>> they are ready to be stable in terms of API? Perhaps, this topic >>>>>>> deserves a >>>>>>> new discussion if there are several opinions on that. >>>>>>> >>>>>> >>> Probably, decision to move component APIs (for example, an IO connector) >>> out of experimental state should be done on a case-by-case basis. >>> >> >> Let's repeat these good points on a dedicated thread. >> >> Kenn >> >> >> >>> >>> Thanks, >>> Cham >>> >>> >>>> >>>>>>> On 26 Nov 2019, at 00:39, Luke Cwik <lc...@google.com> wrote: >>>>>>> >>>>>>> Phase I sounds fine. >>>>>>> >>>>>>> Apache Beam follows semantic versioning and I believe removing the >>>>>>> IOs will be a backwards incompatible change unless they were marked >>>>>>> experimental which will be a problem for Phase 2. >>>>>>> >>>>>>> What is the feasibility of making the V1 transforms wrappers around >>>>>>> V2? >>>>>>> >>>>>>> On Mon, Nov 25, 2019 at 1:46 PM Cam Mach <cammac...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hello Beam Devs, >>>>>>>> >>>>>>>> I have been working on the migration of Amazon Web Services IO >>>>>>>> connectors into the new AWS SDK for Java V2. The goal is to have an >>>>>>>> updated >>>>>>>> implementation aligned with the most recent AWS improvements. So far we >>>>>>>> have already migrated the connectors for AWS SNS, SQS and DynamoDB. >>>>>>>> >>>>>>>> In the meantime some contributions are still going on V1 IOs. So >>>>>>>> far we have dealt with those by porting (or asking contributors) to >>>>>>>> port >>>>>>>> the changes into V2 IOs too because we don’t want features of both >>>>>>>> versions >>>>>>>> to be unaligned but this may quickly become a maintenance issue, so we >>>>>>>> want >>>>>>>> to discuss a plan to stop supporting (deprecate) V1 IOs and encourage >>>>>>>> users >>>>>>>> to move to V2. >>>>>>>> >>>>>>>> Phase I (ASAP): >>>>>>>> >>>>>>>> - Mark migrated AWS V1 IOs as deprecated >>>>>>>> - Document migration path to V2 >>>>>>>> >>>>>>>> Phase II (end of 2020): >>>>>>>> >>>>>>>> - Decide a date or Beam release to remove the V1 IOs >>>>>>>> - Send a notification to the community 3 months before we >>>>>>>> remove them >>>>>>>> - Completely get rid of V1 IOs >>>>>>>> >>>>>>>> >>>>>>>> Please let me know what you think or if you see any potential >>>>>>>> issues? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Cam Mach >>>>>>>> >>>>>>>> >>>>>>> >>>> >>>> -- >>>> This email may be confidential and privileged. If you received this >>>> communication by mistake, please don't forward it to anyone else, please >>>> erase all copies and attachments, and please let me know that it has gone >>>> to the wrong person. >>>> The above terms reflect a potential business arrangement, are provided >>>> solely as a basis for further discussion, and are not intended to be and do >>>> not constitute a legally binding obligation. No legally binding obligations >>>> will be created, implied, or inferred until an agreement in final form is >>>> executed in writing by all parties involved. >>>> >>> >>
