Hi Jarek, thank you for your support on this effort!
For sure. One comment here. One of the ways we discussed with Kamil > about the approach we want to take is to engage more people in > implementing it - so Kamil would create a basic framework and we might > have many people implementing and testing each endpoint separately. > With Kaxil we are co-mentors in the Outreachyprogram and Airflow will > have an intern paid by Outreachy helping with API development. > Few Outreachy candidates recently started to contribute small tasks and > we are going to select the intern at the beginning of April. I think that's a fantastic idea. The more we can delegate and isolate tasks the quicker we can move on this. Once we finish the migration to github issues I'll be glad to help write up smaller tickets for each endpoint We should provide some tools to migrate automatically (if possible) and > asses the effort required to migrate and base our decision on tha > I think automatic migration tools are absolutely necessary and we can even write system tests around those tools/use them for simultaneously testing 1.10 and 2.0 Of course, we should make our decision on our own based > on what we hear, but I think 2 months for some of the Airflow users is not > even > enough time to start planning the migration. I'd say 6-12 months feels much > more appropriate, but that's just feeling for now. > I agree especially for larger-scale users migrations are a difficult process. Perhaps we can adopt something similar to a blockchain fork (e.g. determine X known airflow using companies, and start the countdown as soon as Y% of them migrate). I really just want to make sure we don't end up with a python2/3 situation. Even if we continue support it should only be for bugfixes and we should not add any new features into 1.10. Absolutely. I think the system tests approach i kicked-off today might be > the > best way to ensure that the 2.0 "providers" part is battle-tested way > before > we release 2.0. This might make the whole migration process much > smoother and faster. > I couldn't agree more :). If we can start people writing (close to) 2.0 compliant DAGs before the release of 2.0 that will make the migration process so much easier :). If there aren't any extra steps or features that we need to add (beyond the ones discussed here), I think a good next step would be to create an official checklist just so we can see all of these features in one place (and hopefully start breaking them down into as small of changes as possible). Does that sound ok? On Fri, Mar 20, 2020 at 9:57 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > > Hello, fellow Airflowers! > > > > I wanted to start this discussion in hopes of "kicking the tires" and > > agreeing on a timeline for the Airflow 2.0 release. > > Cool! > > > > > But I want to add _____ feature which would be so awesome in a 2.0 press > > release! > > > > We should also > > consider that the 2.0 release is ALREADY going to take a monumental > testing > > effort, and adding more features will only make that worse. > > Very true. > > > So what should we do now? > > > > Spring Cleaning of Necessary 2.0 tasks > > > > A while back, we created a spring cleaning list of 2.0 necessary fixes. > > Let's review, triage the tickets no longer relevant, and give the highest > > priority to the remaining tickets. > > Happy to help here. > > > Determine Breaking Changes > > > > Of all of the significant efforts in play right now, let's determine > which > > of those efforts are breaking. A major version bump is the only time we > can > > reasonably expect users to accept modifying DAGs, so let's categorize > those > > issues and prioritize them. > > I think a lot of that is already in UPDATING.md. But w will likely find > more. > > > Release API and Scheduler HA > > > > Of the remaining efforts that could potentially cause breaking changes, I > > believe that the new API, which replaces the “Experimental API”, and the > > Scheduler HA have the most potential upside for users. Let's make sure to > > include these in our planning for releasing 2.0. > > For sure. One comment here. One of the ways we discussed with Kamil > about the approach we want to take is to engage more people in > implementing it - so Kamil would create a basic framework and we might > have many people implementing and testing each endpoint separately. > With Kaxil we are co-mentors in the Outreachyprogram and Airflow will > have an intern paid by Outreachy helping with API development. > Few Outreachy candidates recently started to contribute small tasks and > we are going to select the intern at the beginning of April. > > > Set a Timeline for 1.10 releases > > > > One question users will ask when this release comes out is, "how long > will > > the 1.10.x release stream continue to be enhanced?" In creating a > timeline, > > we want to ensure that customers are not left high and dry while also > > preventing any form of Python 2/3 issue where users feel no urgency to > > update. I propose a 2 month window where the 1.10.x release stream will > > continue getting critical bug fixes, but not any additional features. > > I think that the decision on that should depend on the difficulty of > migration. > We should provide some tools to migrate automatically (if possible) and > asses the effort required to migrate and base our decision on that (maybe > we should also query our biggest users what their plans are for migration > after > we provide that. Many of our users are corporate users and we know it takes > time to migrate. Of course, we should make our decision on our own based > on what we hear, but I think 2 months for some of the Airflow users is not > even > enough time to start planning the migration. I'd say 6-12 months feels much > more appropriate, but that's just feeling for now. Some survey might be a > good way to check at least what are the expectations of our users. > > > > > Enhance Tests to Avoid Regressions > > > > Let’s make sure that any new commits have tests suitable for both streams > > while we get ready for 2.0. Let's continue our work on writing system > > tests. Let’s all contribute scenarios so that we can ensure that these > are > > covered before we launch 2.0. > > Absolutely. I think the system tests approach i kicked-off today might be > the > best way to ensure that the 2.0 "providers" part is battle-tested way > before > we release 2.0. This might make the whole migration process much > smoother and faster. I am looking forward to help from the community there > :) > > > I'm ecstatic for the next year in Apache Airflow. Everything is speeding > > up, and by cutting a 2.0 release, I think we can ensure that we keep up > > with the pace of our growing community and demand. > > Yeah! > > J > > -- > > Jarek Potiuk > Polidea | Principal Software Engineer > > M: +48 660 796 129 >
