Echoing a full agreement to all statement before.
I also would emphasize the test pyramid. As we really have a low
coverage we should maybe also have a metric. I would also propose that
we focus more on testing coverage == hopefully also improved quality
preventing regression in 3.2. One option could a bit of force to force
all new changes applied to UI to request a unit test being added (all
changes TS/x files need to have a corresponding .test.ts/x file) except
for cases where it does not make sense we allow exclusion of the rule by
adding a label by maintainers. (just want to drop the idea, might be
many will hate it)
For language also have no strong preference but even though I am a TS
noob I see that testing UI is closer to UI development, to using Python
for TS UI testing might make a knowledge gap also for potential utility
code. So as most voices would propose TS (maybe then finally I get to
understand it)
Jens
On 24.09.25 18:33, Jarek Potiuk wrote:
Small poking at important things from holidays while having a coffee at a
place with internet :)
+1 on having both. I have no strong opinion on the language and Playwright.
But there are few things for consideration:
We should have many, many more unit tests than e2e tests. This is the
classic test pyramid case: The Practical Test Pyramid
https://share.google/YHVIt8EBptT1yYFur
The e2e tests are slow, brittle and difficult to maintain and we should
have rather small number of those - mostly green path on all major
functionalities. Possibly with some cross browser checks - but those are
very difficult to get right.
It will be a pain to maintain those. Like our k8s e2e tests - when they
break (rarely) people will have hard time to create and run the tests so
the tooling around that has to drive people who will be reproducing the
failures by hand - how to set it up and how to reproduce it and how to
iterate as fast as possible.
Keeping the tests stable and side-effect free will be a challenge - so
likely a good 'reset the environment' before each test is a must. Also they
need to be run with all UI changes in CI which will make UI PR run longer.
It's an imperative to get them run not only in canary builds but in all PRs
for ui and API and likely whenever any core model change. It will slow down
UI development and create frustration for contributors unfamiliar with the
test framework so we need to have at least a few people proficient in
intricacies of the test framework dedicated and willing to help those who
will struggle with issues.
Language is secondary - e2e tests will require all stack - Python (backend)
and Typescript and react experiences anyway. Most likely people who write
UI should be involved so Typescript is fine, and also even those who do not
know TypeScript will be able to fix some obvious cases.
J.
śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <[email protected]>
napisał:
This addition will be really nice.
Also, at this point of 3.x, I think UI has been relatively stable and
writing e2e would be a good starting point. Unit tests can go parallel.
It's right that people maintaining CI/CD would be first persons to see any
breakage, but I feel writing in ts will be easier for folks working on
frontend, and also for the new contributors who would like to contribute in
this UI e2e tests coming from cypress or similar experience, since majority
of other choices do come with ts/js support.
I would be happy to help with the e2e tests. We should probably go in
phases with P0/1/2.
Thanks.
On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <[email protected]>
wrote:
I also think that having a good unit test coverage + e2e tests would
really
help maintaining quality of releases and prevent regressions early,
making
all that much more manageable.
My two cents regarding the questions above:
-
Should we prioritize expanding unit test coverage first, or work on
unit
and E2E tests in parallel?
Both in parallel sounds good to me. They do not depend on each other
(At
least in my mind)
-
For E2E testing, is Playwright a good choice, or are there other
preferred tools we should explore?
I like playwright and I feel this is the modern solution to go to.
-
If Playwright, should we use TypeScript or Python as language for
tests?
Good arguments mentioned above for both sides. I personally think
(maybe
wrongly), that ideally people maintaining / developing e2e tests will be
people with
front-end knowledge. (events, identifiers, selectors, css styling,
browser local states, etc..), so I tend to think that those people will
most likely come from the front-end community and
having that in typescript would probably help them get started.
On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <[email protected]>
wrote:
Thanks for starting this discussion, Rahul.
I am aligned with your thoughts, we have seen many many UI issues
lately
which I believe we should prioritize fixing or at least reducing going
forward.
To answer your question(s) --
* I think the unit tests and e2e / user flow tests as orthogonal, so
they
can be
developed in parallel.
* While I do not have real experience with Playwright but from what I
hear
/ read on reddit,
github, and related forums, it looks like Playwright can do similar job
as
alternatives such as
Selenium, Cypress, while being more beginner friendly, which could play
better in our favour.
* Again, not very strong on this but I feel that having Typescript as
the
language of choice would limit
the number of contributors we get to help on this. Python would be my
weapon of choice, given
the "testing" is not limited by it and it potentially could encourage
broader participation.
(Folks active on CI/CD(atleast me), are not very comfortable with
Typescript tests compared to
Python)
One additional point: It would be valuable if we extract the e2e / user
flow testing into a GH job that
a release manager could run as "sort" of last validation before
releasing
to avoid any last
minute issues.
Thanks & Regards,
Amogh Desai
On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev <
[email protected]> wrote:
I agree that we need to improve UI tests. Happy to make that a
priority
during 3.1.x and 3.2 development.
Personally, I will start with expanding unit tests first. But that
doesn't
mean others can't get started on E2E tests.
I lean towards using typescript for playwright. With E2E we are
testing
the
UI more than we are testing the API so I think the code should sit
closer
to the UI code.
On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <[email protected]>
wrote:
Not answering any of these questions (sorry) but strong +1 on
adding
end-to-end (E2E) tests. This will significantly make our UI more
robust
but
also will speed up some of our work. An example is front-end
dependency
upgrades by dependabot. Today we have no way of knowing whether
these
PRs
break the UI, in order to merge these PRs we need to compile the
front-end
assets and test manually whether the UI work. Having end-to-end
tests
on
the front-end and making them run in the CI would speed up this
very
example (on top of increasing code robustness, increase our
confidence
etc).
On 2025/09/23 11:03:40 Rahul Vats wrote:
Hi all,
With Airflow 3.0, the UI has undergone a complete revamp, and new
changes
are naturally more prone to regressions. We've already seen more
UI
regressions that are hard to catch with our current limited test
coverage.
Manual testing is also challenging given the wide variety of
workflows
and
screens in the UI.
Currently, the Airflow UI has a few unit tests using Vitest and
React
Testing Library, but coverage is quite limited — only about 4
test
files
covering major pages like DagsList and TaskInstance. Many
critical
pages (Connections,
Variables, Pools, Dashboard, etc.) have no test coverage at all.
I’m exploring a two-pronged approach to improve UI testing:
1.
Expand unit test coverage for individual components and pages
2.
Add end-to-end (E2E) tests using Playwright for complete user
workflows
This would give us both component-level safety and full
user-journey
testing across browsers, helping catch regressions early.
A few questions:
-
Should we prioritize expanding unit test coverage first, or
work
on
unit
and E2E tests in parallel?
-
For E2E testing, is Playwright a good choice, or are there
other
preferred tools we should explore?
-
If Playwright, should we use TypeScript or Python as language
for
tests?
-
Any other suggestions to strengthen our UI testing strategy?
I’d love to hear your thoughts on whether this would be valuable
for
the
project before diving into implementation details.
Regards,
Rahul Vats
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
