Cool! On Fri, Dec 12, 2025 at 4:34 PM Vincent Beck <[email protected]> wrote:
> Nice! Thanks for driving it! Really excited about it! > > On 2025/12/12 10:02:18 Rahul Vats wrote: > > Hi Everyone, > > > > Following up on our earlier discussions about UI E2E testing -- the > initial > > framework is now in place and ready for community contributions! > > > > *What's Been Merged:* > > - Playwright framework with TypeScript (#58548 > > <https://github.com/apache/airflow/pull/58548> ) > > - CI workflow integrated (#58901 > > <https://github.com/apache/airflow/pull/58901>) > > - Initial test covering login and DAG triggering > > > > *Meta Issue for Tracking:* > > #59028 <https://github.com/apache/airflow/issues/59028> > > > > *Sub-Issues Ready to Pick Up*: > > > > > > - https://github.com/apache/airflow/issues/59306 > > - https://github.com/apache/airflow/issues/59307 > > - https://github.com/apache/airflow/issues/59308 > > - https://github.com/apache/airflow/issues/59309 > > - https://github.com/apache/airflow/issues/59357 > > - https://github.com/apache/airflow/issues/59358 > > > > > > More sub-issues will be created and linked to the meta issue #59028 > > <https://github.com/apache/airflow/issues/59028> > > > > *Contribution Guide* > > > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/tests/e2e/README.md#contributing-new-tests > > > > *Slack Channel:* > > Join #sig-airflow-ui-e2e-tests > > <https://apache-airflow.slack.com/archives/C0A2QT7UUGP> for questions > and > > discussions. > > > > Looking forward to contributions! > > > > Regards, > > Rahul Vats > > > > On Wed, 15 Oct 2025 at 10:45, Amogh Desai <[email protected]> wrote: > > > > > Thanks for the summary, Rahul. > > > > > > Sounds like a good plan, you could coordinate the work better either by > > > using > > > a Github Board or Slack channel or both too. > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > > > > On Fri, Oct 10, 2025 at 9:11 PM Rahul Vats <[email protected]> > wrote: > > > > > > > Hi Everyone, > > > > > > > > Thank you all for the fantastic feedback and strong support! The > > > community > > > > consensus is clear and encouraging: > > > > > > > > - High priority for improving UI testing > > > > - Playwright + TypeScript for e2e testing > > > > - Parallel development of unit and e2e tests > > > > - Test pyramid approach with focused e2e scope > > > > - Multiple volunteers ready to contribute! > > > > > > > > > > > > *Addressing Key Concerns* > > > > Important points on e2e complexity are well taken. I completely > agree > > > that > > > > we should follow the test pyramid. > > > > > > > > - E2E tests should cover only critical workflows (~10-15 scenarios) > > > > - Unit tests are the foundation and should be the primary focus and > we > > > > should have an enforcement so that every UI change comes with a unit > > > test. > > > > - CI integration needs careful planning to avoid slowing development. > > > > > > > > *Next Steps* > > > > > > > > - I plan to work on a POC for E2E tests and will raise a PR for > review > > > > soon. After that, I'll share a detailed implementation plan and > > > coordinate > > > > with volunteers who offered to help. > > > > - In the meantime, I suggest we should continue expanding unit test > > > > coverage as planned. > > > > > > > > > > > > It was great meeting you all at the Airflow Summit. Momentum here is > > > > fantastic, and I'm excited to work with everyone who offered to help > make > > > > Airflow 3 UI much more reliable! > > > > > > > > Regards, > > > > Rahul Vats > > > > > > > > On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev < > > > [email protected] > > > > > > > > > wrote: > > > > > > > > > Rahul, > > > > > Thank you for starting this thread. > > > > > > > > > > Strong +1 from me as well. > > > > > I not only agree with this, but I believe this needs to be a high > > > > priority > > > > > going forward and before 3.2. > > > > > > > > > > I also agree with the comments on the "test pyramid" approach, of > there > > > > > needing to be FAR more unit tests as compared to end to end tests. > > > > > I also believe that a certain level of rigor around this i.e. some > form > > > > of > > > > > enforcement to ensure that all new UI changes come with a unit > test. > > > > > > > > > > I would also add the Core API changes (for UI) to the mix here, > since a > > > > > fair number of UI changes requires underlying API changes. > > > > > > > > > > Vikram > > > > > > > > > > > > > > > On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai < > [email protected]> > > > > > wrote: > > > > > > > > > > > Valid points. > > > > > > > > > > > > I was just expressing concern from a CI/CD failure perspective > but in > > > > no > > > > > > way > > > > > > was I objecting to usage of typescript. > > > > > > > > > > > > Anything that works for the team that owns most of the tests > works by > > > > me! > > > > > > > > > > > > > > > > > > Thanks & Regards, > > > > > > Amogh Desai > > > > > > > > > > > > > > > > > > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler < > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Echoing a full agreement to all statement before. > > > > > > > > > > > > > > I also would emphasize the test pyramid. As we really have a > low > > > > > > > coverage we should maybe also have a metric. I would also > propose > > > > that > > > > > > > we focus more on testing coverage == hopefully also improved > > > quality > > > > > > > preventing regression in 3.2. One option could a bit of force > to > > > > force > > > > > > > all new changes applied to UI to request a unit test being > added > > > (all > > > > > > > changes TS/x files need to have a corresponding .test.ts/x > file) > > > > except > > > > > > > for cases where it does not make sense we allow exclusion of > the > > > rule > > > > > by > > > > > > > adding a label by maintainers. (just want to drop the idea, > might > > > be > > > > > > > many will hate it) > > > > > > > > > > > > > > For language also have no strong preference but even though I > am a > > > TS > > > > > > > noob I see that testing UI is closer to UI development, to > using > > > > Python > > > > > > > for TS UI testing might make a knowledge gap also for potential > > > > utility > > > > > > > code. So as most voices would propose TS (maybe then finally I > get > > > to > > > > > > > understand it) > > > > > > > > > > > > > > Jens > > > > > > > > > > > > > > On 24.09.25 18:33, Jarek Potiuk wrote: > > > > > > > > Small poking at important things from holidays while having a > > > > coffee > > > > > > at a > > > > > > > > place with internet :) > > > > > > > > > > > > > > > > +1 on having both. I have no strong opinion on the language > and > > > > > > > Playwright. > > > > > > > > > > > > > > > > But there are few things for consideration: > > > > > > > > > > > > > > > > We should have many, many more unit tests than e2e tests. > This is > > > > the > > > > > > > > classic test pyramid case: The Practical Test Pyramid > > > > > > > > https://share.google/YHVIt8EBptT1yYFur > > > > > > > > > > > > > > > > The e2e tests are slow, brittle and difficult to maintain > and we > > > > > should > > > > > > > > have rather small number of those - mostly green path on all > > > major > > > > > > > > functionalities. Possibly with some cross browser checks - > but > > > > those > > > > > > are > > > > > > > > very difficult to get right. > > > > > > > > > > > > > > > > It will be a pain to maintain those. Like our k8s e2e tests - > > > when > > > > > they > > > > > > > > break (rarely) people will have hard time to create and run > the > > > > tests > > > > > > so > > > > > > > > the tooling around that has to drive people who will be > > > > reproducing > > > > > > the > > > > > > > > failures by hand - how to set it up and how to reproduce it > and > > > how > > > > > to > > > > > > > > iterate as fast as possible. > > > > > > > > > > > > > > > > Keeping the tests stable and side-effect free will be a > > > challenge - > > > > > so > > > > > > > > likely a good 'reset the environment' before each test is a > must. > > > > > Also > > > > > > > they > > > > > > > > need to be run with all UI changes in CI which will make UI > PR > > > run > > > > > > > longer. > > > > > > > > It's an imperative to get them run not only in canary builds > but > > > in > > > > > all > > > > > > > PRs > > > > > > > > for ui and API and likely whenever any core model change. It > will > > > > > slow > > > > > > > down > > > > > > > > UI development and create frustration for contributors > unfamiliar > > > > > with > > > > > > > the > > > > > > > > test framework so we need to have at least a few people > > > proficient > > > > in > > > > > > > > intricacies of the test framework dedicated and willing to > help > > > > those > > > > > > who > > > > > > > > will struggle with issues. > > > > > > > > > > > > > > > > Language is secondary - e2e tests will require all stack - > Python > > > > > > > (backend) > > > > > > > > and Typescript and react experiences anyway. Most likely > people > > > who > > > > > > write > > > > > > > > UI should be involved so Typescript is fine, and also even > those > > > > who > > > > > do > > > > > > > not > > > > > > > > know TypeScript will be able to fix some obvious cases. > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj < > > > > > [email protected]> > > > > > > > > napisał: > > > > > > > > > > > > > > > >> This addition will be really nice. > > > > > > > >> Also, at this point of 3.x, I think UI has been relatively > > > stable > > > > > and > > > > > > > >> writing e2e would be a good starting point. Unit tests can > go > > > > > > parallel. > > > > > > > >> It's right that people maintaining CI/CD would be first > persons > > > to > > > > > see > > > > > > > any > > > > > > > >> breakage, but I feel writing in ts will be easier for folks > > > > working > > > > > on > > > > > > > >> frontend, and also for the new contributors who would like > to > > > > > > > contribute in > > > > > > > >> this UI e2e tests coming from cypress or similar experience, > > > since > > > > > > > majority > > > > > > > >> of other choices do come with ts/js support. > > > > > > > >> I would be happy to help with the e2e tests. We should > probably > > > go > > > > > in > > > > > > > >> phases with P0/1/2. > > > > > > > >> > > > > > > > >> Thanks. > > > > > > > >> > > > > > > > >> > > > > > > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun < > > > > > > [email protected]> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >>> I also think that having a good unit test coverage + e2e > tests > > > > > would > > > > > > > >> really > > > > > > > >>> help maintaining quality of releases and prevent > regressions > > > > early, > > > > > > > >> making > > > > > > > >>> all that much more manageable. > > > > > > > >>> > > > > > > > >>> My two cents regarding the questions above: > > > > > > > >>> > > > > > > > >>> - > > > > > > > >>> Should we prioritize expanding unit test coverage > first, or > > > > > work > > > > > > on > > > > > > > >> unit > > > > > > > >>> and E2E tests in parallel? > > > > > > > >>> > > > > > > > >>> Both in parallel sounds good to me. They do not depend > on > > > > each > > > > > > > other > > > > > > > >> (At > > > > > > > >>> least in my mind) > > > > > > > >>> - > > > > > > > >>> For E2E testing, is Playwright a good choice, or are > there > > > > > other > > > > > > > >>> preferred tools we should explore? > > > > > > > >>> > > > > > > > >>> I like playwright and I feel this is the modern > solution to > > > > go > > > > > > to. > > > > > > > >>> - > > > > > > > >>> If Playwright, should we use TypeScript or Python as > > > language > > > > > for > > > > > > > >> tests? > > > > > > > >>> Good arguments mentioned above for both sides. I > personally > > > > > think > > > > > > > >> (maybe > > > > > > > >>> wrongly), that ideally people maintaining / developing e2e > > > tests > > > > > will > > > > > > > be > > > > > > > >>> people with > > > > > > > >>> front-end knowledge. (events, identifiers, selectors, > css > > > > > > styling, > > > > > > > >>> browser local states, etc..), so I tend to think that > those > > > > people > > > > > > > will > > > > > > > >>> most likely come from the front-end community and > > > > > > > >>> having that in typescript would probably help them get > > > > started. > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai < > > > > > [email protected]> > > > > > > > >>> wrote: > > > > > > > >>> > > > > > > > >>>> Thanks for starting this discussion, Rahul. > > > > > > > >>>> > > > > > > > >>>> I am aligned with your thoughts, we have seen many many UI > > > > issues > > > > > > > >> lately > > > > > > > >>>> which I believe we should prioritize fixing or at least > > > reducing > > > > > > going > > > > > > > >>>> forward. > > > > > > > >>>> > > > > > > > >>>> To answer your question(s) -- > > > > > > > >>>> > > > > > > > >>>> * I think the unit tests and e2e / user flow tests as > > > > orthogonal, > > > > > so > > > > > > > >> they > > > > > > > >>>> can be > > > > > > > >>>> developed in parallel. > > > > > > > >>>> > > > > > > > >>>> * While I do not have real experience with Playwright but > from > > > > > what > > > > > > I > > > > > > > >>> hear > > > > > > > >>>> / read on reddit, > > > > > > > >>>> github, and related forums, it looks like Playwright can > do > > > > > similar > > > > > > > job > > > > > > > >>> as > > > > > > > >>>> alternatives such as > > > > > > > >>>> Selenium, Cypress, while being more beginner friendly, > which > > > > could > > > > > > > play > > > > > > > >>>> better in our favour. > > > > > > > >>>> > > > > > > > >>>> * Again, not very strong on this but I feel that having > > > > Typescript > > > > > > as > > > > > > > >> the > > > > > > > >>>> language of choice would limit > > > > > > > >>>> the number of contributors we get to help on this. Python > > > would > > > > be > > > > > > my > > > > > > > >>>> weapon of choice, given > > > > > > > >>>> the "testing" is not limited by it and it potentially > could > > > > > > encourage > > > > > > > >>>> broader participation. > > > > > > > >>>> > > > > > > > >>>> (Folks active on CI/CD(atleast me), are not very > comfortable > > > > with > > > > > > > >>>> Typescript tests compared to > > > > > > > >>>> Python) > > > > > > > >>>> > > > > > > > >>>> One additional point: It would be valuable if we extract > the > > > > e2e / > > > > > > > user > > > > > > > >>>> flow testing into a GH job that > > > > > > > >>>> a release manager could run as "sort" of last validation > > > before > > > > > > > >> releasing > > > > > > > >>>> to avoid any last > > > > > > > >>>> minute issues. > > > > > > > >>>> > > > > > > > >>>> Thanks & Regards, > > > > > > > >>>> Amogh Desai > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev < > > > > > > > >>>> [email protected]> wrote: > > > > > > > >>>> > > > > > > > >>>>> I agree that we need to improve UI tests. Happy to make > that > > > a > > > > > > > >> priority > > > > > > > >>>>> during 3.1.x and 3.2 development. > > > > > > > >>>>> > > > > > > > >>>>> Personally, I will start with expanding unit tests > first. But > > > > > that > > > > > > > >>>> doesn't > > > > > > > >>>>> mean others can't get started on E2E tests. > > > > > > > >>>>> > > > > > > > >>>>> I lean towards using typescript for playwright. With E2E > we > > > are > > > > > > > >> testing > > > > > > > >>>> the > > > > > > > >>>>> UI more than we are testing the API so I think the code > > > should > > > > > sit > > > > > > > >>> closer > > > > > > > >>>>> to the UI code. > > > > > > > >>>>> > > > > > > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck < > > > > > [email protected]> > > > > > > > >>>> wrote: > > > > > > > >>>>>> Not answering any of these questions (sorry) but strong > +1 > > > on > > > > > > > >> adding > > > > > > > >>>>>> end-to-end (E2E) tests. This will significantly make > our UI > > > > more > > > > > > > >>> robust > > > > > > > >>>>> but > > > > > > > >>>>>> also will speed up some of our work. An example is > front-end > > > > > > > >>> dependency > > > > > > > >>>>>> upgrades by dependabot. Today we have no way of knowing > > > > whether > > > > > > > >> these > > > > > > > >>>> PRs > > > > > > > >>>>>> break the UI, in order to merge these PRs we need to > compile > > > > the > > > > > > > >>>>> front-end > > > > > > > >>>>>> assets and test manually whether the UI work. Having > > > > end-to-end > > > > > > > >> tests > > > > > > > >>>> on > > > > > > > >>>>>> the front-end and making them run in the CI would speed > up > > > > this > > > > > > > >> very > > > > > > > >>>>>> example (on top of increasing code robustness, increase > our > > > > > > > >>> confidence > > > > > > > >>>>> etc). > > > > > > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote: > > > > > > > >>>>>>> Hi all, > > > > > > > >>>>>>> > > > > > > > >>>>>>> With Airflow 3.0, the UI has undergone a complete > revamp, > > > and > > > > > new > > > > > > > >>>>> changes > > > > > > > >>>>>>> are naturally more prone to regressions. We've already > seen > > > > > more > > > > > > > >> UI > > > > > > > >>>>>>> regressions that are hard to catch with our current > limited > > > > > test > > > > > > > >>>>>> coverage. > > > > > > > >>>>>>> Manual testing is also challenging given the wide > variety > > > of > > > > > > > >>>> workflows > > > > > > > >>>>>> and > > > > > > > >>>>>>> screens in the UI. > > > > > > > >>>>>>> > > > > > > > >>>>>>> Currently, the Airflow UI has a few unit tests using > Vitest > > > > and > > > > > > > >>> React > > > > > > > >>>>>>> Testing Library, but coverage is quite limited — only > > > about 4 > > > > > > > >> test > > > > > > > >>>>> files > > > > > > > >>>>>>> covering major pages like DagsList and TaskInstance. > Many > > > > > > > >> critical > > > > > > > >>>>>>> pages (Connections, > > > > > > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test > coverage at > > > > > all. > > > > > > > >>>>>>> > > > > > > > >>>>>>> I’m exploring a two-pronged approach to improve UI > testing: > > > > > > > >>>>>>> > > > > > > > >>>>>>> 1. > > > > > > > >>>>>>> > > > > > > > >>>>>>> Expand unit test coverage for individual > components and > > > > > pages > > > > > > > >>>>>>> > > > > > > > >>>>>>> 2. > > > > > > > >>>>>>> > > > > > > > >>>>>>> Add end-to-end (E2E) tests using Playwright for > > > complete > > > > > user > > > > > > > >>>>>> workflows > > > > > > > >>>>>>> > > > > > > > >>>>>>> This would give us both component-level safety and full > > > > > > > >>> user-journey > > > > > > > >>>>>>> testing across browsers, helping catch regressions > early. > > > > > > > >>>>>>> > > > > > > > >>>>>>> A few questions: > > > > > > > >>>>>>> > > > > > > > >>>>>>> - > > > > > > > >>>>>>> > > > > > > > >>>>>>> Should we prioritize expanding unit test coverage > > > first, > > > > or > > > > > > > >> work > > > > > > > >>>> on > > > > > > > >>>>>> unit > > > > > > > >>>>>>> and E2E tests in parallel? > > > > > > > >>>>>>> > > > > > > > >>>>>>> - > > > > > > > >>>>>>> > > > > > > > >>>>>>> For E2E testing, is Playwright a good choice, or > are > > > > there > > > > > > > >> other > > > > > > > >>>>>>> preferred tools we should explore? > > > > > > > >>>>>>> > > > > > > > >>>>>>> - > > > > > > > >>>>>>> > > > > > > > >>>>>>> If Playwright, should we use TypeScript or Python > as > > > > > language > > > > > > > >>> for > > > > > > > >>>>>> tests? > > > > > > > >>>>>>> - > > > > > > > >>>>>>> > > > > > > > >>>>>>> Any other suggestions to strengthen our UI testing > > > > > strategy? > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> I’d love to hear your thoughts on whether this would be > > > > > valuable > > > > > > > >>> for > > > > > > > >>>>> the > > > > > > > >>>>>>> project before diving into implementation details. > > > > > > > >>>>>>> > > > > > > > >>>>>>> Regards, > > > > > > > >>>>>>> Rahul Vats > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >> > > > > > > --------------------------------------------------------------------- > > > > > > > >>>>>> To unsubscribe, e-mail: > [email protected] > > > > > > > >>>>>> For additional commands, e-mail: > > > [email protected] > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >> > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
