What's the current thinking on a 2.11.1?
Totally understandable if this was too much work and has been dropped,
but just trying to gauge what advice I should giving to cautious upgraded
on a path to Airflow 3.x.
Damian
-----Original Message-----
From: Jarek Potiuk <[email protected]>
Sent: Sunday, October 5, 2025 3:44 AM
To: [email protected]
Subject: Upcoming Airflow 2.11.1 release [was: [DISCUSS] Possible
Werkzeug vulnerabilities fix for Airflow 2]
Hello here,
*TL;DR; I wanted to start a process of preparing to 2.11.1 release and
I would like the community to be aware of it as I am taking the role of
release manager for it. *
I will need help with reviewing PRs from the committers (I will try to
move it forward even during the Summit, but realistically speaking, I think
I will start release process some time after the Summit as likely a lot of
us won't have the usual attention/time.
*First: good news.* We are unblocked with long overdue Werkzeug upgrade
- with a serious vulnerabiity (via Connexion 2.15.0) - there are also few
small security-related patches that we want to implement alongside.
*Then: not so good news* (well, depends for whom): while we are going
to release 2.11.1, this is is going to be **critical bugfixes only +
security** release. There will be absolutely no new features, or fixes
to - even annoying - issues in 2.11 if they are not critical.
You can skip the rest of the message if you are not interested in more
details or do not want to be involved in the 2.11.1 release testing.
*MORE DETAILS:*
*Again - what is going to be included?*
Only absolutely critical issues and security related changes.
If you think there is an absolutely critical fix that should be
included - please let me know and explain why - here in this discussion.
But the approach I am going to take is that only absolutely critical/
security related fixes should be included in this release - and there has
to be a really good justification to fix anything in 2.11.
I will also absolutely expect, that whoever wants to get any fix there
and we will agree here that it's a good idea, it's **on the one who proposes
it** to make a green PR to v2-11-test with the fix and that they 100%
commit to testing and verifying it when the release candidate is out.
If you think that something should be included in 2.11.1 because of
security reasons - please do not write about it in public. Send an email to
[email protected] explaining the issue and ideally solution
/ PR to backport. Generally follow our Security Policy
https://github.com/apache/airflow/security/policy
*Help needed*
Eventually - I will need community help in testing it - especially for
authentication/FAB integration because this part will be changed a bit. I
will ask for a bit longer time of testing likely and will need community
support from people who are already at 2.11.0 to test it.
*A little more details on wha triggered it*
It took a LOONG time, but finally - with help of some friends of mine
who did a little nudging and conveniently just before coming back from my
vacations - which will happen on Monday BTW - we finally have Connexion
2.15.0 released. This was a bit of a blocker that we waited for - this
**should** help us to solve one of the longest standing issue with
Werkzeug dependency version of ours having a critical vulnerability.
I think (that was few months ago) I fixed all the compatibility issues
for Airflow 2.11.
It was done some time ago on a version of Connexion built from a branch
and it required a few changes (the way how percent encoding of urls are
handled by Werkzeug 2.3.0 and few internal things + i had to implement a
bit of a "hack" on Serialization in flask-session, this PR
https://github.com/apache/airflow/pull/51681 - should likely
eventually lead to a green build.
*A little more details on what is going to happen*
I will need to do a few more steps to get there:
1) I need to release Fab provider 1.5.4 (initially beta, but when I get
it
tested) from providers/fab/v1-5 (working on it). This is needed to
"unblock" some of the depenendency limits in 1.5.3 and adapt provider to a
new flask-session that is needed for the upgrade..
2) I will continue with the "connexion-2.15" PR
https://github.com/apache/airflow/pull/51681 to use this new provider
version, get constraints generated - and **hopefullly** get v2-11-test
branch green (might require some tweaks to the old branches - they are a
bit rusty I am afraid)
3) then I will apply remaining critical changes, That will be the time
when anyone who thinks a change should be included, should work on
backporting critical/implementing security related PRs.
What this will allow (fingers crossed it will not be too difficult) -
is to release 2.11.1 version of Airflow with bumped Werkzeug and few other
dependencies, and critical changes that we plan for 2.11.1 - following the
regular release process.
J.
On Sun, Jun 22, 2025 at 8:55 AM Jarek Potiuk <[email protected]> wrote:
Good news. As a result of our request, Connection 2.15.0rc2 was
released in PyPI this morning with Flask>3. I am running now tests
with it
https://github.com/apache/airflow/pull/51681 and we **finally** have
non-conflicting dependencies in Airflow 2.11 with it.
It still fails - i.e. we will have to fix things with session handling
(we knew we will have to do it because of flask-session upgrade) but
this is something we are now unblocked with :).
Hopefully soon we will get rid of the Werkzeug drama.
root@a20ed58d4f59:/opt/airflow# pip freeze | grep lask
Flask==2.3.3
Flask-AppBuilder==4.5.2
Flask-Babel==2.0.0
Flask-Bcrypt==1.0.1
Flask-Caching==2.3.1
Flask-JWT-Extended==4.7.1
Flask-Limiter==3.11.0
Flask-Login==0.6.3
Flask-Session==0.8.0
Flask-SQLAlchemy==2.5.1
Flask-WTF==1.2.2
root@a20ed58d4f59:/opt/airflow# pip freeze | grep erkzeug
*Werkzeug==3.1.3*
root@a20ed58d4f59:/opt/airflow#
J.
On Thu, Jun 19, 2025 at 7:44 AM Jarek Potiuk <[email protected]>
wrote:
Dear Airflow community,
Thank you. You are amazing. With all the upvotes and comments we had
the contributor of connexion working on bringing Flask 2.3.3+ back to
the upcoming Connexion release
https://github.com/spec-first/connexion/pull/2058/
Particularly Kamil - thanks for the thoughtful comments and the
diligent check on what Flask version we need. We are currently at 2.2
in Airflow 2.11 but I checked that if Connexion sets their limit to
=2.3.3, we should be able update to that version in 2.11 (and it's
good in general as 2.3+ is now the only recommended branch still
being "supported" for Flask 2 for security issues it seems. So we get
additional benefit there that we will be less likely to hit similar
issues until Airflow 2 EOL.
J.
On Wed, Jun 18, 2025 at 8:07 PM Jarek Potiuk <[email protected]>
wrote:
Thank you Kamil - that's very thoughtful and nice to see your
message back on the devlist :D
On Wed, Jun 18, 2025 at 7:38 PM Kamil Breguła <[email protected]>
wrote:
I proposed to split the new connexion release into two versions.
First release one release that supports the new Werkzereg release,
and then release a new Connexion release that supports Flask 3
only. This is not ideal, because Airflow 2 will still be on an
unsupported version of Connexion, but we will have at least one
release that has the new Werkzeug version and has a fix for the CVE
bug. This might be easier to do, as I understand that connexion
might not want to support Flask 2 if there is no specific end date
for when other dependencies will support Flask 3, but it may still
turn out to be enough for us.
śr., 18 cze 2025 o 08:54 Jarek Potiuk <[email protected]>
napisał(a):
I WOULD LIKE TO TAP INTO POWER OF OUR COMMUNITY... PLEASE HELP.
We again had another issue with FAB where the root cause was our
old Werkzeug version - that we cannot upgrade until now) - old
Werkzeug
does
not support `scrypt` hashing algorithm and latest FAB version
defaulted
password hashing to scrypt - we have a workaround but we will
have to
make
a more complete fix with FAB provider. And I am sure Airflow 2
users
will
have more and more problems as the time passes.
I think there is a **real** chance with the Connexion team
working on
2.15.0 - https://pypi.org/project/connexion/2.15.0rc1/ that we
can finally get rid of it - in Both Airflow 2 and Airflow 3. But
we have one
problem ->
Connexion 2.15.0rc1 seems to require Flask 3 where we cannot
upgrade
to
Flask 3 because of the FAB <3 limit. I started a discussion about
it
here:
https://github.com/spec-first/connexion/pull/1992#issuecomment-2976
706491
and explained that it would be great if Connexion 2.15.0
supported
still
flask 2.
And it would be great if more people could support it and explain
that this
would be a major win for the Airflow community if they could
relax
this.
I do not think this is a big problem for them - the explanation
we
had from
them is "hey Flask 2 is really old" - but there is no "real"
reason.
On the other hand migrating FAB to Flask 3 would like be a very
complex and
risky thing (and Daniel already struggles with just SQLalchemy
upgrade and
FAB 5 so it would be too much to put the pressure on him).
Can you please help and upvote/comment on
https://github.com/spec-first/connexion/pull/1992#issuecomment-2976
706491
I would (and the whole community) really, really appreciate it.
J.
On Fri, Jun 13, 2025 at 11:16 AM Jarek Potiuk <[email protected]>
wrote:
Hello everyone,
As you might know, Airflow 2 has a long-time issue with not
being
able to
upgrade Werkzeug dependency to a non-vulnerable version and
that
raises a
lot of alarms for users who run CVE checks on Airflow.
We've been waiting for a long time for that - but it looks like
there is
a
light in a tunnel. We have two options that we can attempt:
1) Connexion 2.15.0.rc1
2) Releasing a package that will patch Werkzeug 2.2.3 with
backported CVE
fixes
Recently Google team attempted to back-port and test fixes to
older version of Werkzeug and I helped to get through to the
maintainers -
https://github.com/pallets/werkzeug/discussions/3034 - however
they are
not really willing to make that into regular release -
reasoning
explained
in the discussion.
However, after many months of discussions and at least 3
attempts
to bump
dependencies for Connexion - we seem to have an RC candidate
(2.15.0rc1
https://pypi.org/project/connexion/2.15.0rc1/) that lifts the
limit for
Werkzeug (released 4 days ago).
There were some breaking changes in Werkzeug that made it so
long
and
difficult but I think we should be able to release a 2.11.1
version
of
Airflow with it
I made first attempt to migrate - here:
https://github.com/apache/airflow/pull/51681 and while I was
able
to
work
out non-conflicting dependencies and bump Werkzeug, there are
some
things
to be fixed with session handling and there is still one
outstanding problem - FAB requires Flask < 3 and currently
Connexion 2.0.15rc1
requires
flask >= 3 - which FAB (even upcoming FAB 5) does not support.
And
likely
migrating to Flask 3 is **not** an option for us anyway.
I started discussion here with those who worked on the
Connexion
patch
for
Werkzeug to see if that is a "hard" limit..:
https://github.com/spec-first/connexion/pull/1992#issuecomment-2969
565640
Alternative option - patch package:
We also have a "last-resort" approach that we are looking at
with
the
Google team. We might want to release a "werkzeug-patch"
package
that
will
apply the CVE patches to Werkzeug 2.2.3
Option 1) is not clear yet if it is possible due to Flask 3 /
Flask
2 -
and it would only work for 2.11.1 - we need to make some fixes
and
change
dependencies for Airflow to make it work.
Option 2) Is hacky (I am talking to Werkzeug maintainers what
do
they
think about it as we would likely need to have at least a
comment
in the
CVE advisory that this package fixes it as well) . But it has
the
benefit
that it will **just work** by installing the patch on basically
all
past
Airflow versions
Just wanted to let everyone know it happens and ask if you have
any opinions on those.
J.
________________________________
Strike Technologies, LLC (“Strike”) is part of the GTS family of
companies. Strike is a technology solutions provider, and is not a broker
or dealer and does not transact any securities related business directly
whatsoever. This communication is the property of Strike and its
affiliates, and does not constitute an offer to sell or the solicitation of
an offer to buy any security in any jurisdiction. It is intended only for
the person to whom it is addressed and may contain information that is
privileged, confidential, or otherwise protected from disclosure.
Distribution or copying of this communication, or the information contained
herein, by anyone other than the intended recipient is prohibited. If you
have received this communication in error, please immediately notify Strike
at [email protected], and delete and destroy any copies
hereof.
________________________________
CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any
attachments are intended solely for the addressee. This transmission is
covered by the Electronic Communications Privacy Act, 18 U.S.C ''2510-2521.
The information contained in this transmission is confidential in nature
and protected from further use or disclosure under U.S. Pub. L. 106-102,
113 U.S. Stat. 1338 (1999), and may be subject to attorney-client or other
legal privilege. Your use or disclosure of this information for any purpose
other than that intended by its transmittal is strictly prohibited, and may
subject you to fines and/or penalties under federal and state law. If you
are not the intended recipient of this transmission, please DESTROY ALL
COPIES RECEIVED and confirm destruction to the sender via return
transmittal.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
