I beg to differ too and I do think that Alexander is right in what he wants to accomplish. Large installations do want to do rolling upgrades and not bring a cluster down for an upgrade. It should be possible to run Airflow Core 2.4 with for example 2.3 workers. It should however not happen through reliance on the DB schema but through a (non-public) API specification. This requires significant architectural improvements. It means decoupling workers from the DB, but also Tasks. It requires backoff periods and strict versioning on the API. It would really improve security (availability, integrity). It is partially addressed in one of the AIPs but imho not sufficiently.
Now Airflow is just a monolith packaged in PODs or microservices. Something for the future (hopefully :-) ). B. On 22 November 2022 at 12:40:07, Abhishek Bhakat ( abhishek.bha...@astronomer.io.invalid) wrote: Hi, I Beg to differ with Alexander and agree with Jarek. There are multiple ways to deploy Airflow. Mostly commonly used is docker images, in that case using one image for all components is standard practice. If using native pip installations, airflow components are launched by a single pip module. So, to have different versions of components (as you mentioned) is adding extra work just to keep them out of sync. A basic common sense would be not to take extra steps to self sabotage. Thanks, Abhishek On 22-Nov-2022 at 4:35:09 PM, Alexander Shorin <kxe...@gmail.com> wrote: > On Tue, Nov 22, 2022 at 1:37 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> BTW. "Workers from 2.2" used with "Airflow 2.4" is not even a thing. >> This is something that you should never, ever, try to do. >> This is even more common sense, and there are of course limits of what >> you can describe in the docs (whatever you come up with, someone might >> have a super crazy idea that you have not thought about and - for >> example - run Airflow 1.10 worker With Airflow 2 (why not? We have not >> written it should not happen). > > > At scale, you cannot upgrade all the versions and keep them in sync all > the time. For minor versions compatibility is expected. Obviously, it > doesn't for major one. It is common sense and practice in the real world, > sorry. > > -- > ,,,^..^,,, > > >
