potiuk commented on issue #26834: URL: https://github.com/apache/airflow/issues/26834#issuecomment-1304806529
Thanks @HadarSha. > I apologize for not being able to provide more detailed logs/explanations, perhaps if we tried again at some point (maybe with the current version or thereabout) we could reproduce it - though in its essence it's very similar, if not identical, to what the author of this issue had already attached. It would be great if you try with 2.4.2 - particularly https://github.com/apache/airflow/pull/27030 from 2.4.2 improved dag collection time and overhead and while unlikely to solve the problem. it will speed up the upgrade time. > Update Airflow package from 2.3.4 to 2.4.0 (we also updated provider packages, I believe) from one of the pods (e.g. Airflow Webserver), using pip > Run airflow db upgrade so as to upgrade the metadata DB tables, schemas etc. before actually updating the rest of the Airflow components (schedulers, workers, etc.) to 2.4.0 Do I understand correctly that you did not shutdown the schedulers and they were running in parallell to the upgrade command? If so then THIS is the reason of your problem. You should never, ever run any component of airlfow while running `airflow db upgrade` - same as trying to run several `airflow db upgrade` commands. This WILL cause problems, this is also why we have `airflow db check-migrations` command in init containers for all our components in our Helm Chart. The right sequence of doing migration is: 1) Stop all airflow components 2) Upgrade ALL airflow components 3) Run (one and only one instance of) `airflow db upgrade` 4) Start all components Step 3 can be achieved easily in fully-automated upgrade process where you would not have to do any manual sequence of operations. If you add `airflow db check-migrations` as init containers (or just command to run before the component starts). If you upgraded all components before, this command will wait until `airflow db upgrade` completes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
