Hey everyone, Thank you for attending the dev call last week. I updated our meeting notes on the Airflow wiki and the link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-Summary.29>
To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is scheduled for tomorrow the 29th of January at the same time. The agenda is already populated, primarily with swim lane updates and Airflow 3.2 AIP updates. If you would like to keep this call to discuss a particular topic, please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886699#Airflow3.xDevCall:Meetingnotes-ProposedAgenda.31> . Best regards, Vikram -- Below is the summary from the call: - Development and swim lane updates - Macro update on swim lane numbers and focus areas (Vikram) - Vikram walked through the updated swim lane numbers as posted on the Airflow 3.x wiki page <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.x>. - Vikram shared that with agreement from Jens and Jed, he had combined the Kubernetes provider area with the Helm chart swim lane, since the expertise was common to resolve issues and approve PRs across both those areas. - Vikram also thanked all the tech leads for their focus on the swim lanes in the early part of the year, saying that he had a significant drop in the "open work items" in the first two weeks of the new year. - Vikram said that the "Logging" swim lane needed a volunteer to help lead efforts there, since it was limited by Ash's bandwidth at this time. - Feedback from the team was to: - Jarek asked to exclude draft PRs from all areas - Pierre said that the API issues were really two-fold: UI and Execution. Pierre proposed to break those apart for easier tracking by marking execution API issues as "Task SDK" and the team immediately agreed. - Jens had asked about looking at the details with each issue or PR. Vikram responded that it was possible, but he had run into GitHub rate limits and had therefore stayed at looking at the issue / PR labels. - Action item: Vikram to add the DAG (summarization script) to the repo for collaborative improvements on the scripts, especially for those areas which had multiple labels. - UI / API update (Brent / Pierre) - The UI / API area continues to be the largest backlog area with significant volume. Pierre and Brent both reported significant progress since the last dev call in closing issues and merging PRs - They also shared that the Grid view performance was significantly improved for large DAGs (tested with 1000+ tasks). - However, the Graph view still has issues with very large DAGs and more work is needed here. - API Server is now stable, but UX challenges remain for very large DAGs and DAG runs. - UI test framework update (Rahul Vats) - Rahul shared that we are about 30-35% coverage with over 40 test cases completed and 9 PRs merged. There are still 27 open issues still to be worked on. - A big focus is on pagination testing using the limit/offset approach for UI. - Also, he was spending a significant amount of time on manual validation of the tests locally before merging, by intentionally introducing bugs in the local brach to ensure that the tests are catching those bugs. - Action item: Rahul to continue end-to-end UI testing also with a focus on large DAG performance scenarios - Airflow 3.2 scope discussion - AIP-76 Asset Partitions <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-76+Asset+Partitions> updated scope (Daniel Standish) - Daniel walked through a presentation detailing the updated work on Asset Partitions highlighting the focus on date based partitions. - Daniel walked through the key components including: Partition-aware asset events, partition-driven DAG scheduling, and enabling DAGs to listen/respond to specific partition updates. - There was a significant discussion around the similarity / differences between date-based partitions and logical date based data-intervals. - Daniel emphasized the flexibility in keeping the above concepts separate going forward. Daniel shared the current status as having completed the listener functionality, and now working on the scheduler component. - AIP-67 Multi-team <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components> (experimental) updated scope (Niko / Vincent) - Niko shared a quick update about the multi-team work saying that the Core scheduling is working with multiple executors, of which the Local executor work is complete and the Celery Executor work is in progress. - This is expected to be in 3.2 as "experimental", without having any security isolation guarantees. - The Security isolation work will be continuing with an expected delivery in Airflow 3.3. Vincent said that David Blain offered to help with the Keycloak auth manager support for multi-team. - Action item: Niko to share a detailed scope of 3.2 vs. 3.3 via slack and follow up in the next dev call. - AIP-86 Deadline Alerts <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488182> Sync callbacks (Dennis / Ramit) - Dennis shared that time-based deadlines with all callback would be ready for Airflow 3.2 - Dennis also said that a large DB migration PR is pending review after the serialized DAG refactor work - Dennis also said that the asset-based deadline trigger work would be deferred to Airflow 3.3 - Action item: David to send deadline alerts completion / scope status via Slack. -- Vikram Koka Chief Strategy Officer Email: [email protected] <https://www.astronomer.io/>
