I've made a _small_ start on updating the road map for 2.0: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
There is still a lot of work to do, but I wanted to get _something_ in place before I'm off for 3 weeks! I have probably missed many things off the list. It is not complete, and things that we plan to get in 2.0 may be pushed back to a 2.1 (or 2.x) or even 3.0. Being on the list doesn't promise any timescale. Feedback and ideas greatly appreciated! > On 15 Apr 2019, at 17:53, James Meickle <jmeic...@quantopian.com.INVALID> > wrote: > > Now that this thread has been open for a minute, I'll loop back to this > with my own thoughts... > > I'm not complaining about the overall pace of development. Instead I'd just > like us to discuss how and why that pace is concentrated in bugfixes, UI > tweaks, new operators, and so on. Check out the changelog: > https://airflow.apache.org/changelog.html - there comparatively isn't as > much work going into redesigns of platform features that are painful (or > missing) in Airflow currently like backfill, parametrized on-demand DAGs, > subdags, xcoms, historical DAG definitions, operating/monitoring airflow, > metadata features, etc. > > My guess is: working in these areas requires more consensus on what > architectural direction is correct; it's really hard to test "all of > Airflow" (and all its use cases!) for compatibility changes; and it's hard > to merge big PRs that affect a lot of rapidly-moving code. So I'm not > saying that anyone just needs to "work harder", but that we might all > benefit from process changes that incentivize or enable work on deeper > features of Airflow as a platform. > > For a suggested process change, I really like some of what Jarek is > proposing. I would certainly find it useful to have a clear contributor > guide. Expecting more from contributors is fine, as long as those > expectations are laid out. If we have firm process expectations, we can > start automating around them. > > Another example process change: I think that Airflow development would > benefit from a clearer statement of audience and supported use cases. Is > the goal to focus on being the "best ETL tool", or do we want to support > more diverse scheduling workflows? Are there any workflows that we never > want to support? I see a lot of discussions where the response to changing > a feature is, "It's that way for historical reasons" (often, because it was > for ETL at Airbnb that way). That's not an argument either for or against > keeping it that way, though, so it's really hard to respond to that in > discussion. If we have some statements about user workflows, we can at > least start discussion with a set of principles to point to. > > Another example process change: I think that at least Airflow major > versions should have roadmaps, which should match our target audience. The > most recent I can find is two years old! > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0 A good > roadmap should be based on evidence of user and developer interest; it > should be updated on a regular cadence with a clear process; and it should > make a convincing case for platform improvements that, by themselves, > wouldn't be worth implementing. It's not just "when will this feature > happen", but "why are features happening in this order" to facilitate > designing something to be more than the sum of its parts. > > So... what would it look like to develop features influenced by a roadmap? > > An example roadmap goal: what if we wanted Airflow 3.0 to be > container-native, with most deployments occurring on containers? That's not > really a "thing", as there is no one feature that makes something > container-native. But if we called ourselves that it might refer to a lot > of smaller features that, by themselves, aren't as exciting: > > - Documentation on a "day one" Airflow installation backed by popular cloud > services like ECS, GKE, etc. > - Documentation of how to deploy an Airflow-compatible, locked-down Docker, > so that people can port existing envs to a local container executor > - A rock-solid Docker Compose workflow for local development of Airflow > - An API for creating DAG-run based ephemeral volumes, which can be > propagated downstream, to support easy migration of "requires local > filesystem" workflows > - Integrating Airflow with container security features (like disallowing > root or capabilities) > - Improved Kubernetes pod logs > - Getting operators runnable in containers without a full Airflow worker > process (possibly even without a DAG sync process) > - Help building containers: at minimum a good official base image and > documentation, but possibly build scripts, or even DAG code to help you > build containers > > Here's another goal that's more cross-cutting: what if we wanted Airflow > 3.0 to be better for scientific reproducibility? That's pretty broad, but > might mean coming up with a set of features like: > > - All executions stored immutably, either built in or as an integration > with a metadata service > - All DAGs and tasks execute in containers, and track container metadata > - Richer metadata/provenance about each task instance > - Each DAGRun saves the DAG version that it was based on / accurate > historical execution info > - Backfills default to using the historical version of the DAG, and have > first-class operations for "recalculate the same DAG with new data" and > "recalculate the same data with a new DAG" > > This isn't to say that we will never support these usecases or features, > but that without community decision and guidance, they'll arrive slower or > only in parts. I'm glad that we're a "big tent" software development > project that accepts most PRs, but it's still worthwhile to revise > contribution guidelines and set project goals, because they also shape > community behavior and interest. > > On Sat, Apr 13, 2019 at 7:58 AM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > >> Hello Everyone, >> >> Few thoughts which I had after proposing a few changes/PRs/AIPs. I think we >> should not only look at it from the system point of view but also from >> human psychology and emotions point of view :) >> >> * I personally felt pretty demotivated (that's the emotion part) by seeing >> 200 opened PRs and 20 AIPS that you do not know which are actively being >> worked on, which are abandoned which are ideas only etc. >> >> * It becomes much better (I feel enthusiastic emotions for that) with AIPs >> starting getting voted and adopted recently (big +) and the discussions we >> had about mentoring/piloting AIPs but I think we have still too many AIPs >> per committer/PMC member. Hopefully more committers will be added and some >> of the big proposals will get implemented and we will learn all how to get >> from AIP to implementation quickly and maybe it will help with speeding up >> AIP implementation process. >> >> * I feels pretty sad (emotions again) with PRs though. I think I never gone >> past the second page (and some of my PRs are on page 3 or 4 now even if I >> actively work on it now). There is simply no way to classify the PRs now >> easily. >> >> * However I don't think this should be PMC or committer role to classify >> and manage PRs properly. This should be the task of contributor who "owns" >> the PR to make sure that his or her PR is properly "promoted". All the >> nagging and asking people for review and making sure PR is in a good shape >> should be on the side of the person who "owns" it. Otherwise PMC/Committers >> will be swamped by purely administrative tasks which is not the point. The >> system should be designed in the way that it self-manages with the help of >> contributors who should have incentive in managing their PRs properly. I >> sympathise (emotions again) with committers/PMC members that they should >> actually do an interesting work and contribute where their expertise is >> most needed and not loose time for things that are only distractions. So I >> think it should be clear for everyone who is the "owner" of the PR to do >> all the work necessary to drag attention of PMC/committers to take a look >> at their PRs. But it should be clearly stated in Contributors document that >> this is the process and expectation from the contributors. And it should be >> easy for contributors to know who is the best person to contact (that part >> ain't easy now). This is already partially solved by "Recommended" reviewer >> in Github but maybe some guidelines on who from the committers is an expert >> in which areas - this might be super-helpful. >> >> Similarly we should have a clear guidance on how to label the PRs (by the >> owner!!!) and have a rule that requires proper label before reviewer takes >> a look at it. This can even be automated by checks on Travis. >> >> *Proposal 1: have a short overview in CONTRIBUTORS with the committers/PMC >> members and their area of involvement/expertise* >> >> *Proposal 2: have a rule that PR has to be properly labeled in order to get >> committer/PMC member even starts taking a look at it.* >> >> * I think there is a psychological effect that I really like about the >> current way we "do not handle and let rot" some of the PRs. I think it's >> quite in the nature of such open-source project that people will have many >> ideas and some of them will rush in discussing them, proposing sometimes >> even opening draft PR but many of those PRs turn out to be >> not-that-important and the original author abandons them. And that's fine. >> Let those pr to "rot". I think this is how human brain works, that >> sometimes you think about a new idea and you start working on it but when >> it turns out that it's not that important we abandon it and then it is >> fairly difficult to force the author to "clean-up". And that's fine as well >> - we should not expect more from the authors. People just work like that >> and we won't change it in scale. However it's all that stale, inactive PRs >> that are the reason for the "mess" we are experiencing. Why don't we close >> all the stale PRs automatically? There are no drawbacks to that, there is >> the bot that we can easily employ to do that for us: >> https://github.com/probot/stale . There is even explanation in there why >> closing PRs automatically is actually good for contributors (otherwise they >> get false expectations). And as contributor you can always re-open such PR >> easily and effortlessly - marking that the contributor is still interested >> in pushing it forward. >> >> *Proposal 3: employ probot to close stale PRs automatically* >> >> J. >> >> >> >> On Thu, Apr 11, 2019 at 8:12 AM Deng Xiaodong <xd.den...@gmail.com> wrote: >> >>> Some personal thoughts about the PR processing speed specifically. >>> >>> I'm trying to benchmark Airflow with other Apache projects (like Spark, >>> Kafka), in terms of PR reviewing/merging speed: as at this moment, there >>> are 400+ open PRs in Spark and 500+ open PRs in Kafka. On the other hand, >>> there are 26 committers of Kafka and 68 committers of Spark. For Airflow, >>> we have less than 20 committers, and recently the # of open PRs remain at >>> about 200. >>> >>> (highlight: this benchmarking is not 100% precise, as I didn't consider >> the >>> total # of PRs processed per day. But seems the # of commits per day of >>> Kafka is roughly close to Airflow) >>> >>> Don't get me wrong: I never think we have done well enough, and I do >> agree >>> that there is big room for improvement. But to be fair, the situation of >>> Airflow here is not that bad. >>> >>> I was just nominated as a committer about 1 month ago. Earlier as a PR >>> submitter, I also had the feeling "why my PRs are processed so slowly"; >> but >>> now when I start to consider more about reviewing/approving/merging, I >>> realize the current pace is fairly good (big thanks to the other >>> committers). >>> >>> Another thing I would like to suggest. Currently we committers almost >> never >>> give "-1" for PRs. Even when committers disagree on a change proposal, >> they >>> don’t close it. I would like to suggest PMC to have this discussion: >>> whether we can close a PR is we have a few "-1"s from committers (say 3 >> or >>> 4). I believe this would somehow help. >>> >>> >>> Best regards, >>> >>> XD >>> >>> On Thu, Apr 11, 2019 at 13:54 airflowuser >>> <airflowu...@protonmail.com.invalid> wrote: >>> >>>> 1. Getting more contributes is important but it's also important to >> give >>>> attention to the current contributes. >>>> I noticed that if PR had no reviews and it reached page 3 and above it >> is >>>> likely to be forgotten. >>>> take this one for example: >>>> https://github.com/apache/airflow/pull/4473 >>>> The author is required to rebase again. It's not very "welcomey" to new >>>> contributes. There are more open PRs like this. One suggestion might >> be a >>>> monthly status check of all open PRs to see if something was missed? >>>> >>>> >>>> 2. The attention of committers doesn't always pointer to what the >>>> community needs. Check this one >>>> https://github.com/apache/airflow/pull/1936 a problem that bugs many >>>> people but there is no discussion how to solve this. There has been >> more >>>> than 4 releases after this PR was introduced and the problem it tries >> to >>>> fix wasn't addressed nor discussed. The author commented that he can >>> update >>>> the branch but he needs committers to be involved. >>>> >>>> Again, since everything is volunteer base it make sense and >>> understandable >>>> however if the project wishes to get more contributors it might be >> easier >>>> to start with the PRs that we already have rather than putting effort >> on >>>> trying to invite new contributors. >>>> >>>> >>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >>>> On Thursday, April 11, 2019 3:46 AM, Aizhamal Nurmamat kyzy >>>> <aizha...@google.com.INVALID> wrote: >>>> >>>>> Hello all, >>>>> >>>>> The Beam project has had problems similar to these also. One of the >>>> things >>>>> they did is formalize how contributions are tracked. I understand >> that >>>>> tracking this sort of information is difficult for the PMC, so if >>> there's >>>>> interest, I'd be happy to work with the PMC to make tools to track >>>>> contributions (e.g. a simple spreadsheet tracking contributions on >> PRs, >>>>> StackOverflow answering, public speaking, documentation, etc). So >> that >>> we >>>>> can streamline the "promotion" of new committers. This may also help >>>>> incentivize "housekeeping" work, such as triaging of JIRA issues, >>>> testing, >>>>> release management, etc. >>>>> >>>>> This may also help provide early feedback to people on track to >> being a >>>>> committer. (e.g. private emails of the kind "hi X. The Airflow PMC >> has >>>>> noticed and appreciates your contributions. We think you could >> improve >>> by >>>>> doing Y or Z" >>>>> >>>>> Let me know what you all think. >>>>> >>>>> Best, >>>>> Aizhamal >>>>> >>>>> On Wed, Apr 10, 2019 at 5:24 PM Gabriel Silk >> gs...@dropbox.com.invalid >>>>> wrote: >>>>> >>>>>>> A lot of the problems that Quantopian experiences with Airflow >>> can't >>>> be >>>>>>> tackled without either "hacks" on top of Airflow; or deep >>> reworkings >>>> of >>>>>>> Airflow components. But that kind of rework is very challenging >> to >>>>>>> implement with the current Airflow contribution process. >>>>>> >>>>>> Can you elaborate on what some of the problems are that Quantopian >>> has >>>>>> encountered, which would require significant re-work to Airflow to >>>> address? >>>>>> On Wed, Apr 10, 2019 at 8:19 AM Driesprong, Fokko >>> fo...@driesprong.frl >>>>>> wrote: >>>>>> >>>>>>> Hi James, >>>>>>> Adressing your concerns one by one: >>>>>>> >>>>>>> - There are a lot of users of Airflow, but their use cases and >>>> feature >>>>>>> usage are not well described. Something that seems trivial or >>>> unnecessary >>>>>>> to one user turns out to be what someone else's entire >> workflow >>>> depends >>>>>>> on. >>>>>>> >>>>>>> >>>>>>> I think in general it is all about scheduling stuff. For me, this >>> is >>>> also >>>>>>> true for many software packages. 80% of the users only use 20% of >>> the >>>>>>> functionality. I think it is up to the committers to make sure >> that >>>> we >>>>>>> don't remove any functionality too easily, and break the workflow >>> for >>>>>>> others. However, sometimes this is what you want, for example >>>> dropping >>>>>>> Python 2 support. I strongly believe that the flexibility offered >>> by >>>>>>> Airflow is both a strength and a weakness, it allows you to do >>>> virtually >>>>>>> everything, on the other hand, maybe you should not do that :-) >>>>>>> >>>>>>> - The Airflow JIRA feels completely unmaintained. Most of the >>>> issues I've >>>>>>> reported have never even been acknowledged, and it's hard to >>>> know what >>>>>>> versions an issue applies to. This makes it hard to know what >>> to >>>> work on >>>>>>> or >>>>>>> what would be most impactful to other users. >>>>>>> >>>>>>> >>>>>>> Keeping track of Jira is a full-time job. Periodically I go >> through >>>> all >>>>>>> the >>>>>>> tickets, but it is also (mis)used for dumping stack traces, or >> any >>>> other >>>>>>> error. We should be more strict on this. As a community. If >> you're >>>>>>> interested in doing this, let me know so I can grand you editor >>>>>>> permissions. >>>>>>> >>>>>>> - Hacking on Airflow is challenging, especially if you need to >>> run >>>> a real >>>>>>> workload to examine your changes. (I saw the work for an >>>> improved local >>>>>>> dev >>>>>>> process - great stuff!) >>>>>>> >>>>>>> >>>>>>> This is a known problem. I think the community is doing an >> awesome >>>> job >>>>>>> here. For example, Breeze by Polidea ( >>>>>>> https://www.youtube.com/watch?v=ffKFHV6f3PQ) and Whirl by >>>>>>> ING/GoDataDriven ( >>>>>>> >>> https://blog.godatadriven.com/open-source-airflow-local-development >>>> ). >>>>>>> >>>>>>> - Keeping track of what's on master vs. what's in a release is >>>>>>> challenging, >>>>>>> particularly since so many commits are for operators we'll >>> never >>>> use. (I >>>>>>> know there's some discussion about breaking operators into >>> their >>>> own >>>>>>> repos, >>>>>>> and I hope that goes through.) >>>>>>> >>>>>>> >>>>>>> The main job of the committers is to keep compatibility on the >>>>>>> interfaces. >>>>>>> The versions are clearly set in Jira when a ticket is being >> worked >>>> on. >>>>>>> Based on if the change is compatible with the new minor version, >> it >>>> will >>>>>>> be >>>>>>> included, otherwise, it will be set to the next major version. >>>>>>> >>>>>>> - The PMCs are too busy to guarantee timely reviews, and >> rebasing >>>> is >>>>>>> extremely costly with how much code reorganization is >>> happening. >>>> This >>>>>>> strongly discourages putting in time to develop anything >> other >>>> than >>>>>>> relatively isolated features, often new features. >>>>>>> >>>>>>> >>>>>>> The code grew rapidly over time. This required to reorganize a >> lot >>> of >>>>>>> code. >>>>>>> This is required to keep development possible and make the code >>> more >>>>>>> accessible to newcomers. For example the splitting up of the >>> infamous >>>>>>> models.py (a file with well over 6k lines), was quite a pain with >>>>>>> circular >>>>>>> imports. This is periodically necessary to keep the code >> organized. >>>>>>> Please >>>>>>> note that it isn't a task for only the PMC to do reviewing. But >>> this >>>> is >>>>>>> also for the committers and contributors. If there any >>>> functionalities >>>>>>> that >>>>>>> you use a lot, please also provide reviews on that topic. >>>>>>> For me, being committer and PMC on the project is just something >>>> that I >>>>>>> do >>>>>>> out of passion for Airflow. It isn't my job and I don't get paid >>> for >>>> it. >>>>>>> That being said, I do agree with getting more committers on board >>> to >>>>>>> strengthen the workforce. >>>>>>> We're now preparing for Airflow 2.0, including a couple of AIP's. >>> The >>>>>>> question if there will be a true container-native, or >> cloud-native >>>>>>> version >>>>>>> of Airflow, is completely up to you and the community. I'm in >> favor >>>> of >>>>>>> jumping on the container train, but this requires to rework on >> the >>>>>>> codebase >>>>>>> of Airflow. >>>>>>> Cheers, Fokko >>>>>>> Op wo 10 apr. 2019 om 16:56 schreef Szymon Przedwojski < >>>>>>> szymon.przedwoj...@polidea.com>: >>>>>>> >>>>>>>> I think it is quite clear that Airflow needs more committers. >>>>>>>> Looking at AIPs, PRs and this devlist there are quite a few >>> active >>>>>>>> people >>>>>>> >>>>>>>> who might be a good fit to become them. >>>>>>>> With the community and the project growing I think this should >> be >>>>>>>> natural >>>>>>> >>>>>>>> to increase the number of committers as well. I know there >> comes >>> a >>>> new >>>>>>>> committer every now and then, but maybe it’s still not enough >> and >>>> maybe >>>>>>>> Airflow should recruit them more “aggressively”? >>>>>>>> Szymon Przedwojski >>>>>>>> Polidea | Software Engineer >>>>>>>> M: +48 500 330 790 >>>>>>>> E: szymon.przedwoj...@polidea.com >>>>>>>> >>>>>>>>> On 10 Apr 2019, at 16:47, airflowuser < >>>> airflowu...@protonmail.com >>>>>>>>> .INVALID> >>>>>>>>> wrote: >>>>>>>> >>>>>>>>> The Jira is a mess and it require committers time to organize >>> it. >>>>>>>>> Ideally users should report issues and committers should tag >>> them >>>>>>>>> with >>>>>>> >>>>>>>> priority, milestone / fix version, labels (This is how for >>> example >>>>>>>> it's >>>>>>> >>>>>>>> done with https://github.com/pandas-dev/pandas ) >>>>>>>> >>>>>>>>> When I have time I try to stack list of Jira issues that >>> require >>>>>>>>> committers attention and ashb fix them but it's progressing >>>> slowly. >>>>>>>>> I think that at least it would be great if the version field >> in >>>> the >>>>>>>>> Jira >>>>>>>>> will be mandatory when user submit ticket. >>>>>>>> >>>>>>>>> At the end... committers simply don't have time for this. >> They >>>> don't >>>>>>>>> have enough time for reviewing PRs as well so I doubt >> something >>>> will >>>>>>>>> change >>>>>>>>> in the near future. >>>>>>>> >>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >>>>>>>>> On Wednesday, April 10, 2019 5:18 PM, James Meickle < >>>>>>>>> jmeic...@quantopian.com.INVALID> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> I've been following Airflow development fairly actively for >>>> over a >>>>>>>>>> year. In >>>>>>>>> >>>>>>>>>> that time, the company I work at (Quantopian) has gone >> all-in >>>> on >>>>>>>>>> Airflow. >>>>>>>>> >>>>>>>>>> It's a core part of our business and required for daily >>>> operations. >>>>>>>>>> However, I've had some concerns over the future of the >>>> project. Part >>>>>>>>>> of >>>>>>>> >>>>>>>>>> these concerns are because it's difficult to contribute to >>>> Airflow: >>>>>>>>>> >>>>>>>>>> - There are a lot of users of Airflow, but their use >> cases >>>> and >>>>>>>>>> feature >>>>>>>>>> >>>>>>>> >>>>>>>>>> usage are not well described. Something that seems trivial >> or >>>>>>>>>> unnecessary >>>>>>>>> >>>>>>>>>> to one user turns out to be what someone else's entire >>> workflow >>>>>>>>>> depends on. >>>>>>>>> >>>>>>>>>> - The Airflow JIRA feels completely unmaintained. Most of >>> the >>>>>>>>>> issues >>>>>>>>>> >>>>>>> >>>>>>>> I've >>>>>>>> >>>>>>>>>> reported have never even been acknowledged, and it's hard >> to >>>> know >>>>>>>>>> what >>>>>>>>> >>>>>>>>>> versions an issue applies to. This makes it hard to know >> what >>>> to >>>>>>>>>> work on or >>>>>>>>> >>>>>>>>>> what would be most impactful to other users. >>>>>>>>>> >>>>>>>>>> - Hacking on Airflow is challenging, especially if you >> need >>>> to >>>>>>>>>> run a >>>>>>>>>> >>>>>>> >>>>>>>> real >>>>>>>> >>>>>>>>>> workload to examine your changes. (I saw the work for an >>>> improved >>>>>>>>>> local dev >>>>>>>>> >>>>>>>>>> process - great stuff!) >>>>>>>>>> >>>>>>>>>> - Keeping track of what's on master vs. what's in a >> release >>>> is >>>>>>>>>> challenging, >>>>>>>>>> >>>>>>>>> >>>>>>>>>> particularly since so many commits are for operators we'll >>>> never >>>>>>>>>> use. (I >>>>>>>>> >>>>>>>>>> know there's some discussion about breaking operators into >>>> their >>>>>>>>>> own >>>>>>>>>> repos, >>>>>>>> >>>>>>>>>> and I hope that goes through.) >>>>>>>>>> >>>>>>>>>> - The PMCs are too busy to guarantee timely reviews, and >>>> rebasing >>>>>>>>>> is >>>>>>>>>> >>>>>>> >>>>>>>>>> extremely costly with how much code reorganization is >>>> happening. >>>>>>>>>> This >>>>>>>> >>>>>>>>>> strongly discourages putting in time to develop anything >>> other >>>>>>>>>> than >>>>>>> >>>>>>>>>> relatively isolated features, often new features. >>>>>>>>>> A lot of the problems that Quantopian experiences with >>> Airflow >>>>>>>>>> can't >>>>>>>>>> be >>>>>>>> >>>>>>>>>> tackled without either "hacks" on top of Airflow; or deep >>>>>>>>>> reworkings >>>>>>>>>> of >>>>>>>> >>>>>>>>>> Airflow components. But that kind of rework is very >>> challenging >>>>>>>>>> to >>>>>>> >>>>>>>>>> implement with the current Airflow contribution process. >>>>>>>>>> I'm glad that we've recently adopted AIPs, but the way >> we're >>>>>>>>>> using >>>>>>> >>>>>>>> them >>>>>>>> >>>>>>>>>> seems better suited to planning isolated features. The >>> Airflow >>>>>>>>>> project does >>>>>>>>> >>>>>>>>>> not have a well-maintained roadmap, nor any mechanism to >>>> produce >>>>>>>>>> one >>>>>>>>>> by >>>>>>>> >>>>>>>>>> weighing AIPs based on synergy vs. developer interest vs. >>> user >>>>>>>>>> interest. >>>>>>>>> >>>>>>>>>> I think that this lack of long-term planning makes it even >>> more >>>>>>>>>> challenging >>>>>>>>> >>>>>>>>>> to propose larger reworks that might require multiple AIPs >> to >>>>>>>>>> implement, >>>>>>>>> >>>>>>>>>> each of which individually might yield little benefit. I >>> worry >>>>>>>>>> that >>>>>>> >>>>>>>> we may >>>>>>>> >>>>>>>>>> approve a series of "promising" AIPs that, taken together, >>>> don't >>>>>>>>>> amount to >>>>>>>>> >>>>>>>>>> anything greater than a "pile of new features"; instead of >>>>>>>>>> balancing >>>>>>>> >>>>>>>>>> feature improvements with platform improvements that will >>>> unlock >>>>>>>>>> more >>>>>>>> >>>>>>>>>> fundamental changes to how Airflow can work. >>>>>>>>>> I'd like to see some discussion of what it would look like >> to >>>> set >>>>>>>>>> long term >>>>>>>>> >>>>>>>>>> goals for Airflow. What is Airflow 2 going to look like? >> How >>>> much >>>>>>>>>> backwards >>>>>>>>> >>>>>>>>>> compat will it break? When should we expect Airflow 3? Are >>> they >>>>>>>>>> going to be >>>>>>>>> >>>>>>>>>> "business as usual" releases, or will they embrace any new >>>>>>>>>> concepts >>>>>>> >>>>>>>> or >>>>>>>> >>>>>>>>>> idioms? Will there be a true container-native, or >>> cloud-native >>>>>>>>>> version of >>>>>>>>> >>>>>>>>>> Airflow? Will we work to be better for current users, or to >>>>>>>>>> embrace >>>>>>> >>>>>>>> new >>>>>>>> >>>>>>>>>> classes of users? >>>>>>>>>> I have some thoughts of my own, of course, but I'd like to >>> hear >>>>>>>>>> what >>>>>>>>>> other >>>>>>>> >>>>>>>>>> people have to say on this topic first! >>>> >>>> >>>> >>> >> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> E: jarek.pot...@polidea.com >>
