potiuk commented on issue #31972: URL: https://github.com/apache/airflow/issues/31972#issuecomment-1595559666
The way it looks - it's a problem of your Database. You likely try to use some active-active DB (not standard Postgres, but something that builds on top of Postgres). Likely this is some non-standard active-active setup with some load balancing to route your DB connection between those two. - i.e. multiple DB instances, each serving the traffic, each trying to insert its own keys. Airflow (and generally any application that is not build to run in such setup, and specifically any application that is build using SQLAlchemy without specifically accounting for such active-active setup) will fail because different replicas of the DB will attempt to insert the same unique keys when attempting to create a record in a table with automated unique key generated by SqlAlchemy. Airflow will only work with Active-Passive high availability DB - i.e one that you have only one node handling traffic at a time and the second is a replica, which can be used as a failover. The solution to your problem is to have a database that does not have active-active setup. BTW. Using Airflow 2.5.0 is a bad idea, there were probably few dosens bugfixes in 2.5 line alone (2.5.3 was release quite a few months ago) so using 2.5.0 when those fixes are out there makes very little sense, because you do not give yourself a chance to make use of those bug-fixes that were implemented (even if you do not want to switch to latest 2.6.* which I woudl heartily recommend). Converting it into discussion to continue discussion there because it is unlikely to be an airflow issue. @ginwakeup -> I am curious to see later what kind of database solution you use, almost for certain this is the reason of your problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
