potiuk commented on issue #31972:
URL: https://github.com/apache/airflow/issues/31972#issuecomment-1595559666

   The way it looks - it's a problem of your Database.  You likely try to use 
some active-active DB (not standard Postgres, but something that builds on top 
of Postgres).
   
   Likely this is some non-standard active-active setup with some load 
balancing to route your DB connection between those two. - i.e. multiple DB 
instances, each serving the traffic, each trying to insert its own keys. 
   
   Airflow (and generally any application that is not build to run in such 
setup, and specifically any application that is build using SQLAlchemy without 
specifically accounting for such active-active setup) will fail because 
different replicas of the DB will attempt to insert the same unique keys when 
attempting to create a record in a table with automated unique key generated by 
SqlAlchemy.
   
   Airflow will only work with Active-Passive high availability DB - i.e one 
that you have only one node handling traffic at a time and the second is a 
replica, which can be used as a failover. 
   
   The solution to your problem is to have a database that does not have 
active-active setup.
   
   BTW. Using Airflow 2.5.0 is a bad idea, there were probably few dosens 
bugfixes in 2.5 line alone (2.5.3 was release quite a few months ago) so using 
2.5.0 when those fixes are out there makes very little sense, because you do 
not give yourself a chance to make use of those bug-fixes that were implemented 
(even if you do not want to switch to latest 2.6.* which I woudl heartily 
recommend). 
   
   Converting it into discussion to continue discussion there because it is 
unlikely to be an airflow issue.
   
   @ginwakeup -> I am curious to see later what kind of database solution you 
use, almost for certain this is the reason of your problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to