Hey everyone, Thank you to everyone who made the dev call last week and apologies for the delay in getting these notes out. I updated our meeting notes document in the Airflow wiki to capture the notes. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.30>
To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. Our next meeting is scheduled for tomorrow, the 3rd of July at the same time. The agenda is pretty short for now: - Development Updates - CLI Update (Bugra) Please let me know if you would like to add anything to the agenda <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-ProposedAgenda.11> . Best regards, Vikram -- Below is the summary from the call: - Catch-up on action items from last call: - Docs site update (Jarek) - Jarek mentioned that there is an issue <https://github.com/apache/airflow/issues/51455> created for the work here. - The docs build process is a lot better now because of the work led by Pavan, but the actual result still needs work. - Jarek also spoke with Kamil about the original work done on the docs site and his availability to help. - Update regarding internationalization impact on release process (Shahar) - Shahar said there isn't any change to the process from prior discussions. - The expectation is for the Release Manager to announce that we will cut a new release and that serves as a notification for the code owner to finish translation by a particular date. - The medium term intent is to publish translations outside of "Core Airflow". The initial concept here is to have translations released similar to provider releases and not to be tied to the core Airflow software release. This could be just a file release or even a data update to the database, to be figured out. - There was a lot of enthusiasm from the team about how this project has happened so quickly and all the community involvement here. - Development Updates and Presentations (Please keep the updates to less than 10 minutes): - UI Plugins Update (Brent, Pierre) - Brent and Pierre did a quick demo of the UI plugins work and shared the progress to date. - This included the common abstraction for the UI plugin regardless if the UI should be in a new tab or an iFrame rendering - The team was enthusiastic about the great progress here. - Deadline Alerts: Update on conclusion for database tradeoff (Dennis) - Ramit shared that the development completion target for Deadline alerts is the end of July - Ramit also shared that right now when a DAG is created with a deadline alert, the deadline would be created and stored in the database - The scheduler loop change work is upcoming and the database change approach will be figured out as part of that. - Brent asked about needing to sync on the API endpoints for the same. Dennis to sync with Brent over the course of the next week. - Discussion topics: - Multi-team discussion from dev list - (Jarek) - There was long spirited discussion around the multi-team effort. Jarek opened by asking for comments. - Vikram went first, saying that he wanted to make sure that we were not trading off the simplicity of Airflow starter experience for individual data engineers with the complexity needed for large organizations wanting granular operational procedures. Vikram asserted that as an open source project, viral adoption has been the key to Airflow's success and that simplicity was important to maintain and in fact make better. - Jarek immediately agreed with that assertion and said that was a key point in his desire to also simplify this AIP to multi-team, from the original, far more complex, multi-tenancy proposal. - Vikram also said that he was surprised by the lack of reference to AIP-82 Event driven scheduling, since this was such a commonly used mechanism for segmenting Airflow deployments across teams from a ownership perspective, following classic distributed system principles. - Jens also chimed in saying that we should perhaps articulate this better in the context of how to set up Airflow for large enterprises. - There was a significant level of debate around the complexity vs. simplicity tradeoff, with Rajesh stating this is probably the most common requirement that MWAA hears from their customers. Niko also chimed in saying that we should be willing to accept some tradeoffs to simplicity. - There was a further discussion on what needs in priority order needed to be addressed here, with Kaxil raising the point that this raised significant operational considerations and he wanted to make sure that we were not getting in over our skis in complexity with quota management, etc. - This ran out of time without an explicit decision in the meeting itself. - Jarek concluded by asking people to comment back in the AIP soon, so that feedback could be addressed in writing. - Call for actions ...ah.. review! Example Dags! (Jens) - Jens raised a "call for action", asking people to help review the work on the Example DAGs and to contribute -- Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
