Hey everyone, I updated our meeting notes document in the Airflow wiki to capture the notes from our dev call last week. Apologies for the delay in getting this out. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.20>
It was great to see the progress over the last couple of weeks and the focus on polishing up the loose ends. To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is on the 20th of March at the same time. Please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-ProposedAgenda.4> . Best regards, Vikram -- Below is the summary from the call - Follow-up on action items from the last call: - Airflow 3.0 minimum version to be Python 3.10 (Ash Berlin-Taylor) - Moving to Python 3.10 is not a blocker. - Ash said that he didn't have time to make this change in time for the 3.0 release. - Therefore, decided to bump to after the 3.0 release and do this as part of the standard Python support cadence. - Development Updates and Presentations: - Update on AIP-72 Task Execution Interface aka Task SDK <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK> - Short circuit and Branch operators (Shahar Epstein) - Shahar walked through the changes required for these operators. - Specifically, the change from direct DB access to using the TaskSDK for updating XComs and the state of the task. - Test plan update (Rahul Vats) - Rahul walked through testing status with Beta1, including testing migrations (upgrades and downgrades). - Rahul said that confidence level with respect to the various AIPs was increasing and that he expected the Grid of confidence by AIP <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Test+Plan> to get a lot more colorful over the next week. - He said that since the "DB isolation will be enforced starting Beta2", he expected more test DAGs to start failing. - When asked about UI test automation, he said that the UI testing was manual and could not be automated yet because it was changing every day. - The team was very appreciative of Rahul's update. - Milestone update (Vikram Koka) - Feature completeness and road to RCs - Vikram shared the updated Airflow 3 dev milestones <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Development+Milestones> and said that he wanted the beta 4 release targeted for the week of March 17th, to be feature complete and the final beta release. - Vikram asked for all open PRs to be completed, tested, and merged before beta 4. He urged contributors to ask for help if needed to meet this milestone. - Vikram asked for patience for the community over the next few weeks, since open issues could not be allowed to linger and would be reassigned if the assignee was unresponsive. - There was general consensus from the team. - Vikram also brought up AIP-68 Extended Plugin interface for React Views <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+React+Views>, saying that this would need to be deferred to 3.1 because of timing. There was general agreement on this as well. - DevEx improvements as result of provider's distribution separation and devel-common (Jarek Potiuk) - Jarek shared a quick update on the recent DevEx improvements for Airflow development based on UV. - He shared how this would make Provider development a lot easier than previously the case as well as benefiting TaskSDK development going forward. - Discussion topics: - Packaging update - open items (Ash, Jed, Jarek) - A quick update shared was about making the Airflow repo a meta package, with Core Airflow becoming a project within the Airflow repo. - Edge provider naming issue (and PEP 752 as long term solution) (Jarek) - Jarek brought up the issue that this name is already taken on the PyPI registry and that this is related to the PEP 752 discussion. - When this PEP is implemented, ASF will take the "Apache" name, but this is still in progress with the POC to be implemented this week. - The immediate action is to hopefully get this name (Edge) from the PyPI registry. - FAB next steps (Jarek) - Jarek confirmed that he is working on enabling Airflow to use FAB 5.0, since this is holding us back from upgrading SqlAlchemy. - Jarek outlined multiple options: - Option 1: Wait for 5.0, but that is 2 to 3 months - Option 2: Take the needed parts of FAB and incorporate into the FAB provider; Then upgrade to SqlAlchemy 2. - Option 3: Stay with SqlAlchemy 1 for now, but the security issues with it are problematic - The team agreed that option 1 was a non-starter, and that we should on option 2, but be prepared to switch to option 3, if we run into issues with the SqlAlchemy upgrade being so close to release. Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
