Looking forward to it. We are going to have a bit of hard time until FAB 5
is out to be able to migrate - but since we already have FAB5 integration
(alpha) and keep iterating on it, this is a perfect opportunity to rebase
on top of it and make incremental progress so that we can merge it quickly
after FAB is released. thanks Dev-IL for leading the efforts!
I think what really helps is to (and we've been doing that successfully for
a number of changes like that):
1) create inventory (in semi-automated way) of things to do -> ideally
small tasks that people can tackle individually and incrementally
2) create an issue to track progress and invite people to help with several
examples and clear instructions how to work on it (having breeze helps as
the environment and setup is highly reproducible and you can provide
usually several exact commands to run to get the "errors" to fix
3) In your case - you will likely need people to branch off from your PR
and submit changes to your PR so that you can merge it in and continue
rebasing. It's not a "standard" approach so having (linked in the issue)
instructions on how to do it would be great aid even for newcomers to help
4) overlooking, rebasing, encouraging and informing people of the progress
is crucial to keep the engagement - keeping people updated on the devlist
is cool
5) Once ready - continue rebasing until we can merge it.
I think this has all the benefits of building collaboration, gives people
opportunities to contribute and learn and to further grow the community
(i.e. "community over code" motto from the ASF).
J.
On Fri, Jun 27, 2025 at 11:52 AM Dev-iL <gid....@gmail.com> wrote:
> Hi Jens,
>
> Thank you for your input!
>
> * If your code suggestion is an acceptable approach, I don't mind
> doing it that way.
> * I haven't followed the migration guide yet. My effort so far was
> very preliminary (trying to understand how much work is even
> needed), and it involved applying the fixes suggested in the
> exception messages themselves. I got around fixing 2-3 of them
> before posting on the devlist. I will, however, go over it before
> making further attempts.
>
> Best,
> Dev-iL
>
>
> On 2025/06/26 19:48:25 Jens Scheffler wrote:
> > Hi,
> >
> > thanks for taking the lead on this!
> >
> > Regarding (1) I fear we need to make a dual version support for a moment
> > because some providers also use SQLA (fab, edge3 at least I know of,
> > standard also soonish if human operators are implemented). If we
> > directyl move to 2.0 then this would be sort of a breaking change and
> > all providers need to move as well - and then newer providers would not
> > be working on older Airflow versions. So I fear this is a coupling
> > problem... and also means we need to consider how long we actually keep
> > 1.4 support.
> >
> > I am not an expert which implications and "messy" workarounds we need to
> > consider for a dual version compatibility. Would it be possible also to
> > use the same model and just make the field definitions dynamically
> > depending on the library version? Like...
> >
> > classDagRun(Base, LoggingMixin):
> > __tablename__="dag_run"
> > if SQLA3:
> > id: Mapped[int] = mapped_column(Integer, primary_key=True) dag_id:
> > Mapped[str] = mapped_column(String, ForeignKey("dag.id"))
> > else:
> > id=Column(Integer, primary_key=True)
> > dag_id=Column(StringID(), nullable=False)
> >
> > But there is also a larger migration guide. I assume you have digested
> > and considered this already?
> >
> > Jens
> >
> > On 26.06.25 14:49, Dev iL wrote:
> > > Hello everyone,
> > >
> > > With the release of Airflow 3 and the refactoring that moved
> > > flask-appbuilder (FAB) into a separate provider, as well as recent
> work by
> > > the FAB developers to support SQLAlchemy v2.0 (SQLA2), it is finally
> > > possible to work on supporting SQLA2 in Airflow as well.
> > >
> > > To get this process going, I added two CI tasks that install SQLA2 (PR
> > > #52233) and am now slowly adding various workarounds to overcome
> repeated
> > > test failures. After doing this for a while, there's a couple of
> discussion
> > > points I want to raise:
> > >
> > > 1. What is the plan regarding dual support of 1.4 & 2.x? If we want to
> > > have that, it might lead to pretty messy code (see details below).
> > > 2. Should we work on supporting SQLA 2.0 or jump straight to 2.1
> (which
> > > is currently in beta)?
> > >
> > > I think we should decide on the refactoring principles and start
> moving in
> > > that direction. With some luck, it might be possible to get this
> done in
> > > time for 3.1.
> > >
> > > I'd love your inputs on how to best approach this task. More
> importantly,
> > > if anyone wants to participate - please leave your comment on my
> PR, and
> > > let's coordinate. I suggest you use my branch as a starting point,
> since it
> > > has the tests in place.
> > >
> > > Best,
> > > Dev-iL
> > >
> > > P.S.
> > > At the time of writing, the status of running the tests (via `breeze
> > > testing core-tests --upgrade-sqlalchemy --maxfail=1000`) is: 134
> failed,
> > > 6469 passed, 47 skipped, 8 xfailed, 1 warning
> > > -------------------
> > >
> > > SQLA2 introduced stricter type annotation requirements for ORM mapped
> > > attributes. All mapped columns need to use the `Mapped[]` generic type
> > > annotation:
> > >
> > > <pre>
> > > class TaskInstance(Base):
> > > __tablename__ = "task_instance"
> > >
> > > # Before (SQLA 1.4)
> > > # id = Column(Integer, primary_key=True)
> > > # dag_id = Column(String, ForeignKey("dag.id"))
> > > # dag_model = relationship("DagModel")
> > >
> > > # After (SQLA 2.0)
> > > id: Mapped[int] = mapped_column(Integer, primary_key=True)
> > > dag_id: Mapped[str] = mapped_column(String, ForeignKey("dag.id"))
> > > dag_model: Mapped["DagModel"] = relationship()
> > > </pre>
> > >
> > > To address differences of similar nature, and assuming we're
> interested in
> > > dual SQLA version support, I can think of two extremes:
> > >
> > > 1. We implement two copies of each model, one for each version of
> SQLA,
> > > and use the relevant one based on a runtime check.
> > > 2. We use workarounds that disable the breaking features of SQLA2
> (e.g.
> > >
>
> https://github.com/apache/airflow/pull/52233/commits/90d738880553707fa01ee2a6b4acd40609e8cc2f
> )
> > > where possible, and resort to runtime branching where impossible, thus
> > > postponing the rewrite to whenever SQLA1.4 support will be dropped.
> > >
> > > ----------------------
> > >
> > > References:
> > >
> > > 1. Main SQLA2 support issue:
> > > https://github.com/apache/airflow/issues/28723
> > > 2. My PR adding SQLA2 to the CI:
> > > https://github.com/apache/airflow/pull/52233
> > > 3. FAB's SQLA2 support PR:
> > > https://github.com/dpgaspar/Flask-AppBuilder/pull/2241
> > >
> >
