Hi all, Sorry for the long email, but here’s an update on bumping to SQLAlchemy 2.0.
TL;DR (consensus needed): it’ll cost us compatibility with a few smaller databases, and I want an OK on that. Here’s the full story: We've been chipping away at a SQLAlchemy 2.0 migration for a while now (see Discussion #40273 [1], started by @hy144328). Steps 1-5 of that plan are done and merged, hooray! The current PR [2] is to prep the actual core `sqlalchemy` version bump is open, which brings every SQLAlchemy-dependent driver package we ship up to the newest version that still works under both 1.4 and 2.0, so each driver bump can be reviewed and tested independently before we flip the core version. That prep work surfaced something we haven't had a real conversation about yet, thus this email. A handful of databases don't have a SQLAlchemy 2.0-compatible driver available at all, and a few more can only move in lockstep with the core bump itself, with zero room for a staged rollout. Before we commit to step 6, I want this list in front of the list, not just buried in a PR description. Why SQLA 2.0 now: This isn't purely a "should modernize" push. Two pains are driving it: * Dremio's SQLAlchemy dialect (and probably others now, if not soon) now requires 2.0. * We're effectively pinned to Python 3.11 (nearing EOL) as a downstream consequence of the SQLAlchemy 1.4 pin so we seemingly must bump it to keep up with Python itself. SQLAlchemy 1.4 itself has no official EOL date but is "virtually there," per upstream, and 2.1 is coming, which will make 1.4 fully obsolete. So… what I want consensus on: This would break databases with no SQLAlchemy 2.0-compatible driver anywhere upstream. These would need to be dropped when we move to 2.0-only drivers, unless someone (us?) does the upstream work first: - Azure Data Explorer / Kusto (sqlalchemy-kusto) — upstream is actively maintained (releases into this year) but hard-pinned to `sqlalchemy==1.4.*` even in its latest release. - Aurora, Data-API mode specifically (sqlalchemy-aurora-data-api). Our own fork (`preset-io/sqlalchemy-aurora-data-api`) has been dormant since 2021. A more active community fork exists but has an open, unresolved SQLAlchemy 2.0 issue. Note this is *only* the serverless HTTP Data API access mode, and regular Aurora MySQL/Postgres-protocol access uses our standard mysql/postgres engine specs and is unaffected. - Apache Solr (sqlalchemy-solr) — effectively dormant upstream, dependabot-only bumps since 2024. - Cloudflare D1 (sqlalchemy-d1) — very young project (first real commits Nov 2025), no signal either way yet. I don't have usage telemetry on how many real deployments run these four. If anyone does, that data point would help a lot here — my working guess is these are low-traffic engines, but I'd rather not assume. Less risky: Databases that aren't lost, but require a synchronized cutover. These drivers jumped straight from 1.4-only to 2.0-only with no dual-compatible bridge release, so the driver bump has to land in the *same* release as the core bump, not before and not after: - RisingWave. The first 2.0-only release only landed this week, so this is freshly resolved, not long-settled. - Exasol. Its Dec 2025 release requires 2.0+. - Firebird. Worth noting this driver has “required" SQLAlchemy 2.0 on any Python 3.8+ for three years already so Firebird is probably broken now unless anyone can validate otherwise. - Redshift. Clean cutover, ~3-year gap with zero intermediate releases between the 1.4-only and 2.0-only versions. - Dremio. This is messier: the currently-pinned version has an open-ended version constraint, so pip metadata alone wouldn't block installing it under 2.0, but whether it actually *works* under 2.0 isn't documented anywhere I could find, so it needs an independent smoke test, but id does have a 2.0 targeted release available. Unrelated to the above but worth fixing regardless: Teradata's recommended SQLAlchemy dialect package (teradatasqlalchemy) isn't pinned anywhere in our dependency files at all, but there's already a SQLAlchemy 2.0-safe release on PyPI, so Teradata itself isn't at risk. We’ve Just been shipping an unmanaged dependency here and should pin it. Heads up: Flask-SQLAlchemy bump needed We’ll need to rekindle something like https://github.com/apache/superset/pull/35117 Version 3.0+ changes session scoping from per-thread to per-app-context, and that broke our Celery task-boundary handling in real CI runs when tested (a reproducible `AttributeError`s across dashboards/security/ tasks tests, plus MySQL lock-wait timeouts). Flask-SQLAlchemy 3.1+ requires SQLAlchemy 2.0, so we can't dodge this forever if we do the core bump, and it needs its own investigation. I think this is a bigger open risk to how long remaining steps actually take than any single database driver above, and didn't want to bury it under “driver compatibility." What I'd like to hear on this thread Does anyone have usage data or objections on the four "would be dropped” engines (Kusto, Aurora Data API, Solr, D1)? That's the single biggest input into how controversial this actually is. Any volunteers to smoke-test Dremio and sanity-check current Firebird support? Given the Python-EOL pressure described above, does the timeline urgency outweigh the disruption to these ~9 engines? And getting this into the “breaking change window” 7.0 offers us? Per lazy consensus, if there are no objections raised in 3 days, I’ll take that as license to move forward with attempting the upgrade, adjust docs and UPDATING.md with compatibility notes, and starting to try to contribute upstream to restore compatibility. Thanks, Evan - [1] Battleplan discussion: https://github.com/apache/superset/discussions/40273 - [2] Driver-compat prep PR: https://github.com/apache/superset/pull/42542 - [3] Original migration attempt (bumping w/flask-sqlalchemy): https://github.com/apache/superset/pull/35117 Evan Rusackas Preset | preset.io
