OK. Got to it. I personally think we should only release any "old" branch, when there is an absolute critical problem that cannot be easily circumvented by the users who already have the latest version from that line released.
I strongly believe that the problems of our users with upgrades is that they do not upgrade frequently enough. Yeah. You read it right. If our users would upgrade rather soon after the release, they would have avoided a lot of "migration pains". There are few reasons: 1. Because they would be used to migration , this would be a "small event" in their product life cycle rather than a huge event, they would have ready procedures and people who know how airflow is configured etc. 2. With the small tenure of people in the industry (at most 2 years staying at one position), you are all but guaranteed that if you upgrade every few years, the people who were there when you migrated last time are not there and you have no chance to learn how to do it. If you do it every few months there will be "overlapping" people in the team who were there the last time almost for sure 3. Airflow is a "batch" scheduler. It will pick up where left, its availability is almost never "critical". You can usually afford short-to-medium downtime if you plan it accordingly There are also few reasons why we (contributors) do not NEED to release usually 4. There are very rarely critical enough bug-fixes that cannot wait - usually there are some workarounds you can apply while waiting 6. We (contributors) want to encourage people to install ONLY from the latest branch and migrate quickly. In order to induce a faster (and more secure, less burden) migration cycle we invested quite a lot into making airflow easy to upgrade: * SemVER as a communication tool - you should be safe to migrate) * Splitting airflow to "core" and providers. - the "sensitive" non-core providers that are changing much more often and are bound to be needed more often. Those have already a 12 months compatibility period - way longer than 4-5 months where "latest branch" is actively developed * In Airlflow 2.3 we've added a lot of missing capabilities about managing version of the Metadata DB - there is a CLI that allows you to freely upgrade/downgrade and generate manual migration scripts that you can update yourself if you need some fixes - this makes our migration capabilities a "two-way" route. If you migrate and you will find some unrecoverable problems later, there is a path to downgrade now. 5. This is OSS - you can always take that single problem that you really care about fixing and patch it - whether you have your own installation, or manage it for others - you can always take a patch from the upstream version and apply it. We have a very strong culture about implementing cherry-pickable changes, so this is usually easy. 6. Maintaining and especially testing any previous version of the software is a cost. Mostly cost of time of people who maintain it, cost of infrastructure to run the tests, cost of organising manual and release activities around it. It should not come for free if there is another viable path that costs the community less. There are companies out there that can even patch such an "oldish" system for you if you cannot do it. Eventually patching old systems costs money. There is no reason the "OSS community" should effectively pay the money (either from free time of volunteers or from paid time by stakeholders who contribute their people's time. If you NEED to upgrade an old system and community is very straightforward and upfront about release cadency, schedule and expectations - this is what you get "for free". You also get "for free" capability to upgrade when we release a new version. If you as a user want to follow another path and stick to the old version - this is perfectly fine. OSS licensing allows you to do it - but you need to pay for diverging from the community recommended approach. Either by paying your own people or hiring an external company to patch such an older version for you. Actually when you think of it, this is a very good and naturally behaving incentive to migrate quickly. Because either you have to pay the migration cost, or maintenance cost of an old version. If you do not have to pay the cost of maintaining the old version, then you will delay the migration as much as possible. Such "maintenance" cost has a natural tendency to be higher, the more you delay - then there you hit the moment where the maintenance cost is bigger than migration cost. This will be a different moment for different users, as some users have higher cost of migration and lower cost of maintenance as their people can maintain Airflow on its own - and that's fine. And this is exactly what we (contributors) want. Honestly, when you maintain old versions for free and there is no "growing" maintenance cost for the users - there is absolutely no incentive for the users to migrate (apart from new features but this is a different story). If your goal is to get people to migrate as soon as they can, supporting older versions is likely the worst thing that you can do to achieve the goal. If you do it, you are working against your own goals, which is, I don't know, stupid ? So in summary - I think over the last 2 years we invested a lot as a community so that we actually can safely say "we do not have to maintain older branches" . We developed tools and use processes that allow our user to choose whichever path they want - either follow the community "upgrade fast" or pay to maintain old versions if they want to. J. On Wed, May 11, 2022 at 2:17 PM Jarek Potiuk <[email protected]> wrote: > Or a busy time :) . I will respond - no worries :). > > On Wed, May 11, 2022 at 1:39 PM Robin Edwards <[email protected]> wrote: > >> Perhaps not a hot topic :-) >> >> Never the less i'd be interested on hearing your thoughts Jarek. >> >> R >> >> On Wed, 4 May 2022 at 17:57, Jarek Potiuk <[email protected]> wrote: >> > >> > This is a very good point. I'd love to hear what others think about it. >> I have my thoughts there but will keep my mouth shut for a while this time >> to hear from others first :) >> > >> > On Wed, May 4, 2022 at 6:32 PM Robin Edwards <[email protected]> wrote: >> >> >> >> This is probably slightly touching on the issues Jarek and Kevin were >> >> discussing in the release announcement however i think it warrants its >> >> own thread. >> >> >> >> Firstly i'd like to thank everyone for their hard work in 2.3, I >> >> haven't had time to try it out yet but i do look forward giving it a >> >> spin. >> >> >> >> We run a fairly large Airflow installation that has been running from >> >> early in the 1. series. >> >> >> >> One thing i've observed since the start of the 2 series is that the >> >> minor releases 2.1, 2.2, 2.3 contain quite ambitious feature changes. >> >> These series often don't mature until 2 or so patch releases. I am not >> >> pointing fingers here it's just the nature of shipping software. >> >> >> >> When a new minor release comes out any outstanding fixes for the >> >> previous series (2.2) now get moved and applied to the new series >> >> (2.3). This can be quite problematic for a user, either bite the >> >> bullet and do a risky upgrade to a .0 release or run our own build >> >> with the given patches applied. The obvious issue with the latter is >> >> your potentially running different code paths to everyone else which >> >> makes getting support hard. >> >> >> >> As far as i am aware the larger vendors maintain their own builds with >> >> extra patches applied. For smaller teams (or new users) doing this is >> >> prohibitive. I guess this is one of the selling points of paying for a >> >> managed service. >> >> >> >> Would it be possible to continue support for the previous minor series >> >> with patch releases whilst the new minor release matures? I know such >> >> a thing isn't uncommon in other projects such as Postgres (all be it >> >> with major releases). >> >> >> >> Obviously I am aware a lot of time and effort goes into cutting a >> >> release, for which I am eternally greatful :-) >> >> >> >> R >> >
