Would love to hear more experiences here - Maybe some from Astronomer, Amazon, Google ? I know all of them use Postgres a lot, and have some Airflow 3 experiences already :) ?
On Thu, Oct 23, 2025 at 11:08 AM Blain David <[email protected]> wrote: > We still use PGBouncer in Airflow 3, as we already used it for Airflow 2, > but also because we use YugabyteDB, there we would need to install the > Yugabyte specific Postgres driver if we would want to ditch PGBouncer, > which we don't know what the effects would be as Yugabyte isn't > (officially) supported on Airflow, but we are planning to test that in the > future. > > -----Original Message----- > From: Jarek Potiuk <[email protected]> > Sent: 23 October 2025 09:59 > To: [email protected] > Subject: Re: [DISCUSS] Using PGBouncer vs, SQL Alchemy pooling in Airflow > 3 ? > > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze > niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, > stuur deze e-mail als bijlage naar [email protected]<mailto: > [email protected]>. > > Thanks Jorrick - very interesting perspective.. I wonder if others have > other experiences/thoughts they can share about it :) > > On Wed, Oct 22, 2025 at 11:50 PM Jorrick Sleijster <[email protected]> > wrote: > > > Hey all, > > > > My two cents; > > > > Most likely, this is very specific to our setup, though it was good to > > share nonetheless. > > We use an internally managed HA Postgres cluster. Primary fail-overs > > for this cluster are handled through DNS changes. > > Furthermore, during maintenance on networking or data centers which > > could impact the primary, we switch over the primary to another data > > center as an extra precaution. > > There is a slight difference in how the tools react: > > - SQLAlchemy is reactive to DNS changes, waiting for a failure of a > > connection before doing another DNS lookup. Furthermore, the > > SQLAlchemy DNS lookup could potentially hit the DNS Cache of the > > machine if the DNS record still has an active TTL. > > - PGBouncer does (configurable) pro-active DNS lookups, thereby upon > > fail-overs it is considerably faster to pick-up the new DNS name and > > AIrflow never notices. > > > > Given we are not on Airflow 3 yet, I can not judge on the connection > > pooling with a full picture and whether SQLAlchemy or PGBouncer would > > be better. > > What I can say is having connection pooling on in Airflow and > > PGBouncer gave us quite some headachages with scaling & limits while > > not having any benefits. > > And although I'm stating the obvious; going from no connection pooling > > to enabling connection pooling gave some db-heavy processes a 10x > > performance boost on machines with a local Postgres database. > > > > Therefore, my opinion on the matter is that it depends on the database > > setup, but the preferred setup should be *enable either of them, not > both*. > > > > Kind regards, > > Jorrick Sleijster > > > > > > Op ma 20 okt 2025 om 20:30 schreef Jarek Potiuk <[email protected]>: > > > > > Hello Here, > > > > > > I wonder if people have some thoughts (experiences?) about using > > PGBouncer > > > for Airflow 3 ? People are asking questions - for example here > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi > > > thub.com%2Fapache%2Fairflow%2Fdiscussions%2F56890&data=05%7C02%7Cdav > > > id.blain%40infrabel.be%7C2cb1dfd0cfbb47e8ad3a08de120a1006%7Cb82bc314 > > > ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638968031591893364%7CUnknown%7CTW > > > FpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMi > > > IsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fVqKPrQCopG5%2 > > > BTSlaergm2N%2BiYssbKe%2BcK%2BlNqPHywY%3D&reserved=0 > > > > > > While in Airflow 2 PGBouncer was pretty much mandatory (due to > > > workers using direct connections) - this is (likely) different story > > > for Airflow > > 3. > > > I guess we should be able to (finally) - properly configure > > > SQLAlchemy connection pools (how big? what's proper?) and PGBouncer > > > might be simply unnecessary (leading to likely a bit less complex and > faster deployment). > > > > > > Do people have thoughts / experience about it, I wonder? > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] >
