Hi Everyone, Following up on our earlier discussions about UI E2E testing -- the initial framework is now in place and ready for community contributions!
*What's Been Merged:* - Playwright framework with TypeScript (#58548 <https://github.com/apache/airflow/pull/58548> ) - CI workflow integrated (#58901 <https://github.com/apache/airflow/pull/58901>) - Initial test covering login and DAG triggering *Meta Issue for Tracking:* #59028 <https://github.com/apache/airflow/issues/59028> *Sub-Issues Ready to Pick Up*: - https://github.com/apache/airflow/issues/59306 - https://github.com/apache/airflow/issues/59307 - https://github.com/apache/airflow/issues/59308 - https://github.com/apache/airflow/issues/59309 - https://github.com/apache/airflow/issues/59357 - https://github.com/apache/airflow/issues/59358 More sub-issues will be created and linked to the meta issue #59028 <https://github.com/apache/airflow/issues/59028> *Contribution Guide* https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/tests/e2e/README.md#contributing-new-tests *Slack Channel:* Join #sig-airflow-ui-e2e-tests <https://apache-airflow.slack.com/archives/C0A2QT7UUGP> for questions and discussions. Looking forward to contributions! Regards, Rahul Vats On Wed, 15 Oct 2025 at 10:45, Amogh Desai <[email protected]> wrote: > Thanks for the summary, Rahul. > > Sounds like a good plan, you could coordinate the work better either by > using > a Github Board or Slack channel or both too. > > Thanks & Regards, > Amogh Desai > > > On Fri, Oct 10, 2025 at 9:11 PM Rahul Vats <[email protected]> wrote: > > > Hi Everyone, > > > > Thank you all for the fantastic feedback and strong support! The > community > > consensus is clear and encouraging: > > > > - High priority for improving UI testing > > - Playwright + TypeScript for e2e testing > > - Parallel development of unit and e2e tests > > - Test pyramid approach with focused e2e scope > > - Multiple volunteers ready to contribute! > > > > > > *Addressing Key Concerns* > > Important points on e2e complexity are well taken. I completely agree > that > > we should follow the test pyramid. > > > > - E2E tests should cover only critical workflows (~10-15 scenarios) > > - Unit tests are the foundation and should be the primary focus and we > > should have an enforcement so that every UI change comes with a unit > test. > > - CI integration needs careful planning to avoid slowing development. > > > > *Next Steps* > > > > - I plan to work on a POC for E2E tests and will raise a PR for review > > soon. After that, I'll share a detailed implementation plan and > coordinate > > with volunteers who offered to help. > > - In the meantime, I suggest we should continue expanding unit test > > coverage as planned. > > > > > > It was great meeting you all at the Airflow Summit. Momentum here is > > fantastic, and I'm excited to work with everyone who offered to help make > > Airflow 3 UI much more reliable! > > > > Regards, > > Rahul Vats > > > > On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev < > [email protected] > > > > > wrote: > > > > > Rahul, > > > Thank you for starting this thread. > > > > > > Strong +1 from me as well. > > > I not only agree with this, but I believe this needs to be a high > > priority > > > going forward and before 3.2. > > > > > > I also agree with the comments on the "test pyramid" approach, of there > > > needing to be FAR more unit tests as compared to end to end tests. > > > I also believe that a certain level of rigor around this i.e. some form > > of > > > enforcement to ensure that all new UI changes come with a unit test. > > > > > > I would also add the Core API changes (for UI) to the mix here, since a > > > fair number of UI changes requires underlying API changes. > > > > > > Vikram > > > > > > > > > On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <[email protected]> > > > wrote: > > > > > > > Valid points. > > > > > > > > I was just expressing concern from a CI/CD failure perspective but in > > no > > > > way > > > > was I objecting to usage of typescript. > > > > > > > > Anything that works for the team that owns most of the tests works by > > me! > > > > > > > > > > > > Thanks & Regards, > > > > Amogh Desai > > > > > > > > > > > > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <[email protected]> > > > > wrote: > > > > > > > > > Echoing a full agreement to all statement before. > > > > > > > > > > I also would emphasize the test pyramid. As we really have a low > > > > > coverage we should maybe also have a metric. I would also propose > > that > > > > > we focus more on testing coverage == hopefully also improved > quality > > > > > preventing regression in 3.2. One option could a bit of force to > > force > > > > > all new changes applied to UI to request a unit test being added > (all > > > > > changes TS/x files need to have a corresponding .test.ts/x file) > > except > > > > > for cases where it does not make sense we allow exclusion of the > rule > > > by > > > > > adding a label by maintainers. (just want to drop the idea, might > be > > > > > many will hate it) > > > > > > > > > > For language also have no strong preference but even though I am a > TS > > > > > noob I see that testing UI is closer to UI development, to using > > Python > > > > > for TS UI testing might make a knowledge gap also for potential > > utility > > > > > code. So as most voices would propose TS (maybe then finally I get > to > > > > > understand it) > > > > > > > > > > Jens > > > > > > > > > > On 24.09.25 18:33, Jarek Potiuk wrote: > > > > > > Small poking at important things from holidays while having a > > coffee > > > > at a > > > > > > place with internet :) > > > > > > > > > > > > +1 on having both. I have no strong opinion on the language and > > > > > Playwright. > > > > > > > > > > > > But there are few things for consideration: > > > > > > > > > > > > We should have many, many more unit tests than e2e tests. This is > > the > > > > > > classic test pyramid case: The Practical Test Pyramid > > > > > > https://share.google/YHVIt8EBptT1yYFur > > > > > > > > > > > > The e2e tests are slow, brittle and difficult to maintain and we > > > should > > > > > > have rather small number of those - mostly green path on all > major > > > > > > functionalities. Possibly with some cross browser checks - but > > those > > > > are > > > > > > very difficult to get right. > > > > > > > > > > > > It will be a pain to maintain those. Like our k8s e2e tests - > when > > > they > > > > > > break (rarely) people will have hard time to create and run the > > tests > > > > so > > > > > > the tooling around that has to drive people who will be > > reproducing > > > > the > > > > > > failures by hand - how to set it up and how to reproduce it and > how > > > to > > > > > > iterate as fast as possible. > > > > > > > > > > > > Keeping the tests stable and side-effect free will be a > challenge - > > > so > > > > > > likely a good 'reset the environment' before each test is a must. > > > Also > > > > > they > > > > > > need to be run with all UI changes in CI which will make UI PR > run > > > > > longer. > > > > > > It's an imperative to get them run not only in canary builds but > in > > > all > > > > > PRs > > > > > > for ui and API and likely whenever any core model change. It will > > > slow > > > > > down > > > > > > UI development and create frustration for contributors unfamiliar > > > with > > > > > the > > > > > > test framework so we need to have at least a few people > proficient > > in > > > > > > intricacies of the test framework dedicated and willing to help > > those > > > > who > > > > > > will struggle with issues. > > > > > > > > > > > > Language is secondary - e2e tests will require all stack - Python > > > > > (backend) > > > > > > and Typescript and react experiences anyway. Most likely people > who > > > > write > > > > > > UI should be involved so Typescript is fine, and also even those > > who > > > do > > > > > not > > > > > > know TypeScript will be able to fix some obvious cases. > > > > > > > > > > > > J. > > > > > > > > > > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj < > > > [email protected]> > > > > > > napisał: > > > > > > > > > > > >> This addition will be really nice. > > > > > >> Also, at this point of 3.x, I think UI has been relatively > stable > > > and > > > > > >> writing e2e would be a good starting point. Unit tests can go > > > > parallel. > > > > > >> It's right that people maintaining CI/CD would be first persons > to > > > see > > > > > any > > > > > >> breakage, but I feel writing in ts will be easier for folks > > working > > > on > > > > > >> frontend, and also for the new contributors who would like to > > > > > contribute in > > > > > >> this UI e2e tests coming from cypress or similar experience, > since > > > > > majority > > > > > >> of other choices do come with ts/js support. > > > > > >> I would be happy to help with the e2e tests. We should probably > go > > > in > > > > > >> phases with P0/1/2. > > > > > >> > > > > > >> Thanks. > > > > > >> > > > > > >> > > > > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun < > > > > [email protected]> > > > > > >> wrote: > > > > > >> > > > > > >>> I also think that having a good unit test coverage + e2e tests > > > would > > > > > >> really > > > > > >>> help maintaining quality of releases and prevent regressions > > early, > > > > > >> making > > > > > >>> all that much more manageable. > > > > > >>> > > > > > >>> My two cents regarding the questions above: > > > > > >>> > > > > > >>> - > > > > > >>> Should we prioritize expanding unit test coverage first, or > > > work > > > > on > > > > > >> unit > > > > > >>> and E2E tests in parallel? > > > > > >>> > > > > > >>> Both in parallel sounds good to me. They do not depend on > > each > > > > > other > > > > > >> (At > > > > > >>> least in my mind) > > > > > >>> - > > > > > >>> For E2E testing, is Playwright a good choice, or are there > > > other > > > > > >>> preferred tools we should explore? > > > > > >>> > > > > > >>> I like playwright and I feel this is the modern solution to > > go > > > > to. > > > > > >>> - > > > > > >>> If Playwright, should we use TypeScript or Python as > language > > > for > > > > > >> tests? > > > > > >>> Good arguments mentioned above for both sides. I personally > > > think > > > > > >> (maybe > > > > > >>> wrongly), that ideally people maintaining / developing e2e > tests > > > will > > > > > be > > > > > >>> people with > > > > > >>> front-end knowledge. (events, identifiers, selectors, css > > > > styling, > > > > > >>> browser local states, etc..), so I tend to think that those > > people > > > > > will > > > > > >>> most likely come from the front-end community and > > > > > >>> having that in typescript would probably help them get > > started. > > > > > >>> > > > > > >>> > > > > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai < > > > [email protected]> > > > > > >>> wrote: > > > > > >>> > > > > > >>>> Thanks for starting this discussion, Rahul. > > > > > >>>> > > > > > >>>> I am aligned with your thoughts, we have seen many many UI > > issues > > > > > >> lately > > > > > >>>> which I believe we should prioritize fixing or at least > reducing > > > > going > > > > > >>>> forward. > > > > > >>>> > > > > > >>>> To answer your question(s) -- > > > > > >>>> > > > > > >>>> * I think the unit tests and e2e / user flow tests as > > orthogonal, > > > so > > > > > >> they > > > > > >>>> can be > > > > > >>>> developed in parallel. > > > > > >>>> > > > > > >>>> * While I do not have real experience with Playwright but from > > > what > > > > I > > > > > >>> hear > > > > > >>>> / read on reddit, > > > > > >>>> github, and related forums, it looks like Playwright can do > > > similar > > > > > job > > > > > >>> as > > > > > >>>> alternatives such as > > > > > >>>> Selenium, Cypress, while being more beginner friendly, which > > could > > > > > play > > > > > >>>> better in our favour. > > > > > >>>> > > > > > >>>> * Again, not very strong on this but I feel that having > > Typescript > > > > as > > > > > >> the > > > > > >>>> language of choice would limit > > > > > >>>> the number of contributors we get to help on this. Python > would > > be > > > > my > > > > > >>>> weapon of choice, given > > > > > >>>> the "testing" is not limited by it and it potentially could > > > > encourage > > > > > >>>> broader participation. > > > > > >>>> > > > > > >>>> (Folks active on CI/CD(atleast me), are not very comfortable > > with > > > > > >>>> Typescript tests compared to > > > > > >>>> Python) > > > > > >>>> > > > > > >>>> One additional point: It would be valuable if we extract the > > e2e / > > > > > user > > > > > >>>> flow testing into a GH job that > > > > > >>>> a release manager could run as "sort" of last validation > before > > > > > >> releasing > > > > > >>>> to avoid any last > > > > > >>>> minute issues. > > > > > >>>> > > > > > >>>> Thanks & Regards, > > > > > >>>> Amogh Desai > > > > > >>>> > > > > > >>>> > > > > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev < > > > > > >>>> [email protected]> wrote: > > > > > >>>> > > > > > >>>>> I agree that we need to improve UI tests. Happy to make that > a > > > > > >> priority > > > > > >>>>> during 3.1.x and 3.2 development. > > > > > >>>>> > > > > > >>>>> Personally, I will start with expanding unit tests first. But > > > that > > > > > >>>> doesn't > > > > > >>>>> mean others can't get started on E2E tests. > > > > > >>>>> > > > > > >>>>> I lean towards using typescript for playwright. With E2E we > are > > > > > >> testing > > > > > >>>> the > > > > > >>>>> UI more than we are testing the API so I think the code > should > > > sit > > > > > >>> closer > > > > > >>>>> to the UI code. > > > > > >>>>> > > > > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck < > > > [email protected]> > > > > > >>>> wrote: > > > > > >>>>>> Not answering any of these questions (sorry) but strong +1 > on > > > > > >> adding > > > > > >>>>>> end-to-end (E2E) tests. This will significantly make our UI > > more > > > > > >>> robust > > > > > >>>>> but > > > > > >>>>>> also will speed up some of our work. An example is front-end > > > > > >>> dependency > > > > > >>>>>> upgrades by dependabot. Today we have no way of knowing > > whether > > > > > >> these > > > > > >>>> PRs > > > > > >>>>>> break the UI, in order to merge these PRs we need to compile > > the > > > > > >>>>> front-end > > > > > >>>>>> assets and test manually whether the UI work. Having > > end-to-end > > > > > >> tests > > > > > >>>> on > > > > > >>>>>> the front-end and making them run in the CI would speed up > > this > > > > > >> very > > > > > >>>>>> example (on top of increasing code robustness, increase our > > > > > >>> confidence > > > > > >>>>> etc). > > > > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote: > > > > > >>>>>>> Hi all, > > > > > >>>>>>> > > > > > >>>>>>> With Airflow 3.0, the UI has undergone a complete revamp, > and > > > new > > > > > >>>>> changes > > > > > >>>>>>> are naturally more prone to regressions. We've already seen > > > more > > > > > >> UI > > > > > >>>>>>> regressions that are hard to catch with our current limited > > > test > > > > > >>>>>> coverage. > > > > > >>>>>>> Manual testing is also challenging given the wide variety > of > > > > > >>>> workflows > > > > > >>>>>> and > > > > > >>>>>>> screens in the UI. > > > > > >>>>>>> > > > > > >>>>>>> Currently, the Airflow UI has a few unit tests using Vitest > > and > > > > > >>> React > > > > > >>>>>>> Testing Library, but coverage is quite limited — only > about 4 > > > > > >> test > > > > > >>>>> files > > > > > >>>>>>> covering major pages like DagsList and TaskInstance. Many > > > > > >> critical > > > > > >>>>>>> pages (Connections, > > > > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test coverage at > > > all. > > > > > >>>>>>> > > > > > >>>>>>> I’m exploring a two-pronged approach to improve UI testing: > > > > > >>>>>>> > > > > > >>>>>>> 1. > > > > > >>>>>>> > > > > > >>>>>>> Expand unit test coverage for individual components and > > > pages > > > > > >>>>>>> > > > > > >>>>>>> 2. > > > > > >>>>>>> > > > > > >>>>>>> Add end-to-end (E2E) tests using Playwright for > complete > > > user > > > > > >>>>>> workflows > > > > > >>>>>>> > > > > > >>>>>>> This would give us both component-level safety and full > > > > > >>> user-journey > > > > > >>>>>>> testing across browsers, helping catch regressions early. > > > > > >>>>>>> > > > > > >>>>>>> A few questions: > > > > > >>>>>>> > > > > > >>>>>>> - > > > > > >>>>>>> > > > > > >>>>>>> Should we prioritize expanding unit test coverage > first, > > or > > > > > >> work > > > > > >>>> on > > > > > >>>>>> unit > > > > > >>>>>>> and E2E tests in parallel? > > > > > >>>>>>> > > > > > >>>>>>> - > > > > > >>>>>>> > > > > > >>>>>>> For E2E testing, is Playwright a good choice, or are > > there > > > > > >> other > > > > > >>>>>>> preferred tools we should explore? > > > > > >>>>>>> > > > > > >>>>>>> - > > > > > >>>>>>> > > > > > >>>>>>> If Playwright, should we use TypeScript or Python as > > > language > > > > > >>> for > > > > > >>>>>> tests? > > > > > >>>>>>> - > > > > > >>>>>>> > > > > > >>>>>>> Any other suggestions to strengthen our UI testing > > > strategy? > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> I’d love to hear your thoughts on whether this would be > > > valuable > > > > > >>> for > > > > > >>>>> the > > > > > >>>>>>> project before diving into implementation details. > > > > > >>>>>>> > > > > > >>>>>>> Regards, > > > > > >>>>>>> Rahul Vats > > > > > >>>>>>> > > > > > >>>>>> > > > > > >> > > > --------------------------------------------------------------------- > > > > > >>>>>> To unsubscribe, e-mail: [email protected] > > > > > >>>>>> For additional commands, e-mail: > [email protected] > > > > > >>>>>> > > > > > >>>>>> > > > > > >> > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > >
