Hey everyone, Thank you for attending the dev call earlier today. I updated our meeting notes on the Airflow wiki and the link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-Summary.35>
To everyone who attended the meeting, please check the summary and add anything I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please ask questions if something is unclear. Our next meeting is scheduled for the 23rd of April at the same time. It is scheduled for 8 a.m. Pacific Time. If you would like to discuss a particular topic, please let me know if you want to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-ProposedAgenda.37> . Best regards, Vikram -- Below is the summary from the call: - Catch-up on action items from last call - None - Airflow 3.2 Retrospective - Release Manager perspective (Rahul Vats) - Rahul opened by thanking all contributors who helped with development and testing for 3.2. - Rahul proposed introducing a recognition program for community members who file bugs during beta and RC phases, similar in spirit to the existing "PR of the Month," to incentivize earlier and broader community testing. - Jarek noted that the model used for provider releases works well: a GitHub issue is created that tags contributors and explicitly asks them to test the specific things they contributed. He suggested adapting this for minor releases, with a more compact version of the issue given the larger scope. He noted that for the most recent provider release, around 80% of tagged contributors confirmed they had tested their changes. Jarek also suggested reaching out directly to users who had filed issues that were fixed in the release, asking them to test the beta or RC on their staging environments — since those users are already motivated to see their fix land. - Jens highlighted that the key gap is getting testing done in production-like environments rather than artificial small-scale setups, since many issues only surface at real scale. He suggested building a more comprehensive integration test suite that covers known corner cases and can run longer, more complex scenarios. - Rahul agreed and noted that a few issues during RC1 were related to object storage on cloud — scenarios that exist in the E2E test suite but weren't being run against real cloud environments. He suggested reviewing gaps in E2E coverage as a follow-up. - Jens also noted that the Easter break timing reduced available testing resources significantly, and suggested that having an additional beta before cutting the RC, combined with targeted contributor outreach, would give the community more time to catch issues earlier. - Documentation-first proposal (Elad Kalif) - Elad raised two observations. First, that a large number of non-feature commits and bug fixes are bundled into minor releases, adding risk. He suggested considering whether some of those could go into a separate minor release to reduce scope. - Second, and more significantly, Elad noted that documentation for major features was arriving very late in the cycle, often close to the beta cut, making it very difficult to test features effectively. He proposed requiring that documentation be written upfront, even before implementation in some cases, so that testers can understand the intended behavior and verify against it. - Vikram said he was a strong fan of this idea, noting that with AI tools it is now very easy to generate and maintain documentation concurrently with code, and therefore there is little excuse for late docs. He suggested this should be a prerequisite for a feature being merged or included in the first beta, at least for significant features. - Jarek agreed and added that he has used an AI agent to continuously update documentation while developing, which works very well and can even generate screenshots. He noted this represents a shift in development mindset but is very achievable. - Rahul noted this would also reduce the last-minute documentation rush before RCs. - Action item: Elad to write up the proposal for requiring documentation alongside significant feature development as a prerequisite for beta inclusion, to be shared on the dev list. - Security landscape update (Jarek Potiuk) - Jarek shared a broader update on the security landscape that had affected the 3.2 release process. - A supply chain security incident occurred in recent weeks involving LiteLLM, where maintainers were impersonated. The Google provider had a dependency on LiteLLM and was briefly affected. The vulnerable version was pulled from PyPI within about two hours. - Separately, Jarek shared that Anthropic is investing $1.5M into ASF security initiatives, with $250K from Alpha Omega as a seed, as part of a new ASF Responsible AI initiative announced the day before the call. The goal is to raise $3M this year and $10M overall to fund tooling and access to advanced AI security models. He explained that the emerging security model involves AI that can proactively find and generate patches for security vulnerabilities, turning the security process from reactive to proactive. Bug bounty programs have already been suspended by several organizations as AI can now find vulnerabilities faster than human researchers. - ASF will receive early access to Anthropic models currently under embargo. The details of how access will be distributed are still being worked out at the ASF board level, with a decision expected at the board meeting in about a week and a half. A VP of AI Tooling role is expected to be appointed to lead the initiative. - Dheeraj asked whether ASF maintainers would receive direct access to Anthropic's preview models. Jarek confirmed that access is being arranged, either directly from Anthropic or through Alpha Omega, though it will be limited to those involved in security work due to embargo constraints. - Vikram noted that Airflow's cooldown protection is now in place broadly and asked the team to keep an eye on further developments. - UI / API feedback (Brent Bovenzi) - Brent noted that the increased PR velocity from the community on UI has been a mixed experience, helpful for getting features like additional filters and search shipped, but requiring more effort to reject low-quality PRs. He and Pierre have had to be stricter in their review standards. - Brent also noted that the pace of feature work has sometimes meant less time for thoughtful UX design. - Jens's upgrade issue follow-up - Vikram asked Jens about the GitHub issues he had planned to file from the Bosch upgrade experience. Jens said he had filed the easy ones and some had already been fixed in parallel, but had not completed the full list due to work pressures. - Discussion Topics: - AIP-103 Task State Management (Vikram) - Vikram noted he is looking forward to working with XD and Jake on AIP-103 Task State Management, with a target of the 3.3 release. He thanked everyone who gave feedback and voted on this. - Jake is also working on an updated incarnation of the Asset Watermarking AIP, rebasing it on the Task State Management work. - Vikram noted he plans to clean up the related Confluence pages, as there are currently around 4 to 5 related AIPs that are not well organized. The goal is to track them as a cohesive epic rather than unrelated items. - AIP-72 Multi-language support / Java Task SDK update (Vikram) - Vikram noted that TP has been making significant progress on the Java Task SDK, which represents the first delivery of non-Python multi-language support for the Task SDK. - It felt too rushed to include in 3.2, but Vikram has scheduled a demo for the next dev call in two weeks. He wanted to make sure the community was aware of this work, as it had not been discussed broadly as a group. - AIP-94 CLI Decoupling update (Bugra Ozturk) - Bugra shared that now that 3.2 is released, he has started creating tickets for AIP-94 (decoupling the CLI using Airflow CTL), all with the 3.3 milestone. He flagged this for awareness and asked the community to keep an eye out for new issues being created. - Vikram thanked Bugra and said he would update the wiki page in the next couple of days to reflect this. Vikram Koka Chief Strategy Officer Email: [email protected] <https://www.astronomer.io/>
