Hi all,
Here are the notes from our Airflow dev call last week. Thank you to all
who joined the call.
Here is a quick summary of the call.
*Summary Notes*:
- *Add a policy document around all our Policies / rules in the Github
Repo itself*
- This should cover any decision we take as PMC / Committers, for
example:
- Versioning Policy - Core, Providers and all others
- Committers have a binding vote on AIP Releases
- Lazy Consensus thread will be created on the mailing list for each
decision agreed on the Dev Call
- *Providers Release Policy*
- *Batch vs Ad-hoc*:
- Release Manager would default to release Providers in Batch
- ad-hoc releases are OK (i.e. if there is a critical bug that
needs fixing in a single provider)
- *Frequency:*
- For Batch release, we will release *every month *(starting of
the month - 1 to 7 most likely)
- Just a note that it generally takes around a week for the vote
to pass even though we have 72 hours minimum period
- *Testing*
- License and Signature Checks are mandatory (following the ASF
rules)
- For Providers, not all changes require strict testing -- you
make a judgement based on the changes for a particular provider
- For Airflow core it is different, smoke tests should be carried
out and we rely on community testing too (Kaxil to create a separate
Mailing list thread to discuss this further since there wasn't 100%
agreement)
- *Doc-only changes*
- When we have doc-only changes for Providers (during
batch-release), we should still release a new version.
Majority on the Dev
call agreed that releasing docs asap is good instead of waiting for
the next release with a code-change.
- *Feedback on AIPs*
- AIP-39 Richer scheduler_interval
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-39+Richer+scheduler_interval>
- Foreign keys between TaskInstance and DagRun will be changed
from execution_date and dag_id to dagrun_id.
- This and any other changes with columns will need migration that
might be slow for deployments with many Dags and large TIs or
DagRuns in
Metadata DB.
- Kevin Yang from Airbnb said that they have tried something
similar but instead of dagrun_id they used DagRun.id
(auto-incrementing
id for each DagRun) which had a performance impact.
- AIP-38 Modern Web Application
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application>
is
already being Voted on: link
<https://lists.apache.org/thread.html/r646c453087fa982e7277823d7b638a238314adbcb361fa93ac1230f6%40%3Cdev.airflow.apache.org%3E>
- *TaskGroups support in Tree View*
- We agreed that this would be good to have since we are planning to
remove SubDags for TaskGroups
- A separate Mailing list thread has been created for more details
and further discussion: link
<https://lists.apache.org/thread.html/r17626aff377aa4914edb8ed3ab9ce17193a8d1615e7388c77db76b4d%40%3Cdev.airflow.apache.org%3E>
Notes are also posted on Confluence too:
*Doc Link*:
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Dev+Calls
Best regards,
Kaxil
