Thanks Jorrick - very interesting perspective.. I wonder if others have other experiences/thoughts they can share about it :)
On Wed, Oct 22, 2025 at 11:50 PM Jorrick Sleijster <[email protected]> wrote: > Hey all, > > My two cents; > > Most likely, this is very specific to our setup, though it was good to > share nonetheless. > We use an internally managed HA Postgres cluster. Primary fail-overs for > this cluster are handled through DNS changes. > Furthermore, during maintenance on networking or data centers which could > impact the primary, we switch over the primary to another data center as an > extra precaution. > There is a slight difference in how the tools react: > - SQLAlchemy is reactive to DNS changes, waiting for a failure of a > connection before doing another DNS lookup. Furthermore, the SQLAlchemy DNS > lookup could potentially hit the DNS Cache of the machine if the DNS record > still has an active TTL. > - PGBouncer does (configurable) pro-active DNS lookups, thereby upon > fail-overs it is considerably faster to pick-up the new DNS name and > AIrflow never notices. > > Given we are not on Airflow 3 yet, I can not judge on the connection > pooling with a full picture and whether SQLAlchemy or PGBouncer would be > better. > What I can say is having connection pooling on in Airflow and PGBouncer > gave us quite some headachages with scaling & limits while not having any > benefits. > And although I'm stating the obvious; going from no connection pooling to > enabling connection pooling gave some db-heavy processes a 10x performance > boost on machines with a local Postgres database. > > Therefore, my opinion on the matter is that it depends on the database > setup, but the preferred setup should be *enable either of them, not both*. > > Kind regards, > Jorrick Sleijster > > > Op ma 20 okt 2025 om 20:30 schreef Jarek Potiuk <[email protected]>: > > > Hello Here, > > > > I wonder if people have some thoughts (experiences?) about using > PGBouncer > > for Airflow 3 ? People are asking questions - for example here > > https://github.com/apache/airflow/discussions/56890 > > > > While in Airflow 2 PGBouncer was pretty much mandatory (due to workers > > using direct connections) - this is (likely) different story for Airflow > 3. > > I guess we should be able to (finally) - properly configure SQLAlchemy > > connection pools (how big? what's proper?) and PGBouncer might be simply > > unnecessary (leading to likely a bit less complex and faster deployment). > > > > Do people have thoughts / experience about it, I wonder? > > >
