Hello all, I've been reviewing This wiki page <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0> and I wanted to discuss which of these features are still relevant/are blockers for 2.0
I figured we should start with the status of the high priority fixes 1. *Knative Executor (handled by KEDA) *The KEDA Autoscaler has accomplished everything that we would've hoped for from a KnativeExecutor. I think we can consider this work done. 2. *Improve Webserver performance* @Kaxil Naik <kaxiln...@gmail.com> It appears that DAG serialization is already merged? Are there other steps needed for webserver performance? 3. *Enhanced real-time UI *While this would make for a great "wow!" feature, I don't think this is necessary for a 2.0. Unless anyone wants to work on this in the short term :). 4. *Improve Scheduler performance and reliability @Ash Berlin-Taylor <a...@apache.org> *has been hard at work on scheduler HA, and we've already 5. *Extend/finish the API @Kamil Breguła <kamil.breg...@polidea.com> *I see that there is an active PR here <https://github.com/apache/airflow/pull/7549>, are we getting close on this? 6. *Production Docker image *The PR here should be merged hopefully today! Now there's a second section of "needs more info" tickets and I wanted to see which of these we could triage for after 2.0 Rework Subdags to be less "bolted-on" and more native to the scheduler > > There are all sorts of edge cases around subdags that result from the > tasks in the subdag being run by another executor, instead of being handled > and scheduled by the core Scheduler. We should make the scheduler "see in" > to the Subdags and make it responsible for scheduling tasks. This should > make subdags less error prone and more predictable. It may involve > replacing/significantly changing the SubDagOperator I think it's pretty damn crucial we fix subdags and backfills. I'm on the fence about this one. On the one hand it could possibly wait. On the other hand it would be embarrasing to release a 2.0 and still have this feature broken Move (tested) components out of contrib folder > > https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E > I think this has been something we've actively worked on. Is this task complete? Filter passwords/sensitive info from logs. > > Jenkins does this if the password comes from a connection - it would be > good if we could do this too While I agree that this is an important issue, I think if no one has picked this up yet we should triage for a later release (I also don't think this should break anything) Allow Backfill runs to be handled by the scheduler/triggered from UI > > It would be nice to not need console access to run airflow backfill, and > to have not it not stop if the SSH session is closed. > > Lots of details to work out here though around how this would work, where > would it show up in UI, priority of tasks, ways of reducing > concurrency/load to allow normal tasks to run etc. Once again I think this is a feature that's a bit embarrasing NOT to fix for a 2.0 release, but also understand that there's only so much manpower. I also imagine this would be a HUGE undertaking so I think this would push back 2.0 significantly. Rationalize HA in Connections > > Right now it is possible to create multiple connections with the same ID > and *some* Connections/hooks will support this and pick a random one > from the list. This feature isn't well documented or understood (and the > CLI doesn't support it as well as the UI for instance) so we should examine > if this makes sense, or if we should support it individually in certain > connection types instead. I honestly don't know enough about this to have a strong opinion on it Make setting up HTTPS connections easier/more expected > > It appears that @Kamil Breguła <kamil.breg...@polidea.com> has an open PR <https://github.com/apache/airflow/pull/5239/files> for this one. @Kamil Breguła <kamil.breg...@polidea.com> do you think this would be hard to merge? Front end/"browser" testing > > The Airflow UI is non trivial and there have been a number of JS/html bugs > that could have been caught by better front-end testing. > > It has been suggested to look at Cypress for this over Selenium. What ever > we choose we need to pay careful attention to avoid slow or flakey UI tests. > This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've been taking time to learn cypress recently, and once the airflow helm chart is merged I think merging a set of integration/behavior/UI tests is crucial What does everyone think? Open to suggestions on this assessment! On Tue, Mar 31, 2020 at 11:46 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > Got it. Thanks Daniel for leading this > > On Tue, Mar 31, 2020 at 7:40 PM Daniel Imberman <daniel.imber...@gmail.com> > wrote: > >> I think including both is fine as long as the old one contains >> deprecation warnings/force a feature flag to allow it >> (e.g. —allow-deprecated) >> >> via Newton Mail >> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2> >> >> On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła < >> kamil.breg...@polidea.com> wrote: >> >> On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <r...@bidnamic.com> wrote: >> >> > Also does the new API need to be feature complete or just enough >> > functionality to warrant removing the existing experimental one. >> > >> >> I think we should release at least one version that will contain the >> new and old REST APIs simultaneously. It is not easy to upgrade two >> complex systems at the same time. However, if we do this, some users >> will have to do it. Older versions can be hidden behind the feature >> gate. We can also add deprecation warnings. >> >> > >> > R >> > >> > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <daniel.imber...@gmail.com> >> > wrote: >> > >> > > Great! Hope to get a few more folx to give +1's but I think we have a >> good >> > > path forward here :) >> > > >> > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk < >> jarek.pot...@polidea.com> >> > > wrote: >> > > >> > > > > >> > > > > >> > > > > I agree especially for larger-scale users migrations are a >> difficult >> > > > > process. Perhaps we can adopt something similar to a blockchain >> fork >> > > > (e.g. >> > > > > determine X known airflow using companies, and start the >> countdown as >> > > > soon >> > > > > as Y% of them migrate). I really just want to make sure we don't >> end up >> > > > > with a python2/3 situation. Even if we continue support it should >> only >> > > be >> > > > > for bugfixes and we should not add any new features into 1.10. >> > > > > >> > > > >> > > > I think we are in perfect sync - I think feature-migration should >> end >> > > > almost immediately after we release 2.0. But bug-fixing should >> continue >> > > > for quite some time. On that front - having backport packages will >> help >> > > > with releasing "integrations" quite independently from 1.10/2.0 >> version >> > > > (which I think is good for those who are - for this or another >> reason - >> > > > stuck on 2.0). On the other hand we should make sure that the >> important >> > > > stuff for 2.0 that is not "feature" is also backported to 1.10. For >> > > example >> > > > a lot of recent performance improvements that we have now in 2.0 >> will >> > > > be possible (and not that complex) to backport to 1.10. Some of this >> > > effort >> > > > is actually easier to do in 2.0 and then apply to 1.10 in similar >> fashion >> > > > as it is easier to understand and reason about the 2.0 code now when >> > > > we have some refactoring/pylints etc in place. So we should make >> sure >> > > > we get the latest 1.10 to a "good" state - before we freeze it for >> bugfix >> > > > only. >> > > > I know it might mean that some people will stay with 1.10 for >> longer, but >> > > > that's also OK for them. The reason to migrate to 2.0 should be not >> > > > performance but some important features (like API or HA) that come >> > > > with it. >> > > > >> > > > I couldn't agree more :). If we can start people writing (close to) >> 2.0 >> > > > > compliant DAGs before the release of 2.0 that will make the >> migration >> > > > > process so much easier :). >> > > > > >> > > > >> > > > Yeah. I even thought that we should write a >> > > > "How good your DAGs are for 2.0" assessment tool. >> > > > >> > > > >> > > > > If there aren't any extra steps or features that we need to add >> (beyond >> > > > the >> > > > > ones discussed here), I think a good next step would be to create >> an >> > > > > official checklist just so we can see all of these features in one >> > > place >> > > > > (and hopefully start breaking them down into as small of changes >> as >> > > > > possible). >> > > > > >> > > > > Does that sound ok? >> > > > > >> > > > >> > > > Perfectly OK for me! >> > > > >> > > > >> > > > > >> > > > > -- >> > > > >> > > > Jarek Potiuk >> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > > > >> > > > M: +48 660 796 129 <+48660796129> >> > > > [image: Polidea] <https://www.polidea.com/> >> > > > >> > > >> >>
