jedcunningham commented on code in PR #36961:
URL: https://github.com/apache/airflow/pull/36961#discussion_r1462607332
##########
CONTRIBUTING.rst:
##########
@@ -15,23 +15,24 @@
specific language governing permissions and limitations
under the License.
+Contributions primer
+====================
+
.. contents:: :local:
-Contributions
-=============
+Contributions are welcome and are greatly appreciated! Every little bit helps,
and credit will always be given.
-Contributions are welcome and are greatly appreciated! Every little bit helps,
-and credit will always be given.
+This document explains general contribution rules and links to detailed
documentation that aim to explain
+the subject of contributions if you have not contributed to any Open Source
project, but it will also help
+people who have contributed to other projects learn about the rules of that
community.
Review Comment:
```suggestion
This document explains general contribution rules and links to detailed
documentation that aims to explain
the subject of contributions. This is particularly useful if you are new to
Open Source, but it will also help
people who have contributed to other projects learn about the rules of our
community.
```
Or something similar.
##########
contribution-docs/README.rst:
##########
@@ -0,0 +1,72 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Detailed contributing documentation
+-----------------------------------
+
+The following documents provide detailed information about contributing to
Airflow.
+
+* `Roles in Airflow Project <roles_in_airflow_project.rst>`__ describes
+ the roles in the Airflow project and how they relate to each other.
+
+* `How to communicate <how_to_communicate.rst>`__
+ describes how to communicate with the community and how to get help.
+
+* `Contributors quick start <contributors_quick_start.rst>`__ describes
+ how to set up your development environment and make your first contribution.
There are also more
+ detailed documents describing how to set up your development environment for
specific IDE/environment:
+
+ * `Contributors quick start for PyCharm
<contributors_quick_start_pycharm.rst>`__
+ * `Contributors quick start for VSCode
<contributors_quick_start_vscode.rst>`__
+ * `Contributors quick start for Codespaces
<contributors_quick_start_codespaces.rst>`__
+ * `Contributors quick start for GitPod
<contributors_quick_start_gitpod.rst>`__
+
+* `Creating issues and Pull requests
<creating_issues_and_pull_requests.rst>`__ describes
+ when and how you can create issues and submit pull requests as well as
describes the pull request guidelines
+ and conding standards.
+
+* `Contribution workflow <contribution_workflow.rst>`__ describes
+ the workflow for contributing to Airflow.
+
+* `Working with Git <working_with_git.rst>`__ describes what Git branches used
in Airflow,
Review Comment:
```suggestion
* `Working with Git <working_with_git.rst>`__ describes the Git branches
used in Airflow,
```
##########
CONTRIBUTING.rst:
##########
@@ -41,1770 +42,26 @@ of the community works and you might need to get some
mentorship from other memb
community - mostly Airflow committers (maintainers). Mentoring new members of
the community is part of
maintainers job so do not be afraid of asking them to help you. You can do it
via comments in your PR, asking on a devlist or via Slack. For your
convenience,
-we have a dedicated #development-first-pr-support Slack channel where you can
ask any questions
+we have a dedicated ``#development-first-pr-support`` Slack channel where you
can ask any questions
about making your first Pull Request (PR) contribution to the Airflow codebase
- it's a safe space
where it is expected that people asking questions do not know a lot Airflow
(yet!).
If you need help with Airflow see the Slack channel #troubleshooting.
To check on how mentoring works for the projects under Apache Software
Foundation's
`Apache Community Development - Mentoring
<https://community.apache.org/mentoring/>`_.
-Report Bugs
------------
-
-Report bugs through `GitHub <https://github.com/apache/airflow/issues>`__.
-
-Please report relevant information and preferably code that exhibits the
-problem.
-
-Fix Bugs
---------
-
-Look through the GitHub issues for bugs. Anything is open to whoever wants to
-implement it.
-
-Issue reporting and resolution process
---------------------------------------
-
-An unusual element of the Apache Airflow project is that you can open a PR to
-fix an issue or make an enhancement, without needing to open an issue first.
-This is intended to make it as easy as possible to contribute to the project.
-
-If you however feel the need to open an issue (usually a bug or feature
request)
-consider starting with a `GitHub Discussion
<https://github.com/apache/airflow/discussions>`_ instead.
-In the vast majority of cases discussions are better than issues - you should
only open
-issues if you are sure you found a bug and have a reproducible case,
-or when you want to raise a feature request that will not require a lot of
discussion.
-If you have a very important topic to discuss, start a discussion on the
-`Devlist <https://lists.apache.org/list.html?d...@airflow.apache.org>`_
instead.
-
-The Apache Airflow project uses a set of labels for tracking and triaging
issues, as
-well as a set of priorities and milestones to track how and when the
enhancements and bug
-fixes make it into an Airflow release. This is documented as part of
-the `Issue reporting and resolution process <ISSUE_TRIAGE_PROCESS.rst>`_,
-
-Implement Features
-------------------
-
-Look through the `GitHub issues labeled "kind:feature"
-<https://github.com/apache/airflow/labels/kind%3Afeature>`__ for features.
-
-Any unassigned feature request issue is open to whoever wants to implement it.
-
-We've created the operators, hooks, macros and executors we needed, but we've
-made sure that this part of Airflow is extensible. New operators, hooks, macros
-and executors are very welcomed!
-
-Improve Documentation
----------------------
-
-Airflow could always use better documentation, whether as part of the official
-Airflow docs, in docstrings, ``docs/*.rst`` or even on the web as blog posts or
-articles.
-
-See the `Docs README
<https://github.com/apache/airflow/blob/main/docs/README.rst>`__ for more
information about contributing to Airflow docs.
-
-Submit Feedback
----------------
-
-The best way to send feedback is to `open an issue on GitHub
<https://github.com/apache/airflow/issues/new/choose>`__.
-
-If you are proposing a new feature:
-
-- Explain in detail how it would work.
-- Keep the scope as narrow as possible to make it easier to implement.
-- Remember that this is a volunteer-driven project, and that contributions
are
- welcome :)
-
-
-Roles
-=============
-
-There are several roles within the Airflow Open-Source community.
-
-For detailed information for each role, see: `Committers and PMC's
<./COMMITTERS.rst>`__.
-
-PMC Member
------------
-
-The PMC (Project Management Committee) is a group of maintainers that drives
changes in the way that
-Airflow is managed as a project.
-
-Considering Apache, the role of the PMC is primarily to ensure that Airflow
conforms to Apache's processes
-and guidelines.
-
-Committers/Maintainers
-----------------------
-
-You will often see the term "committer" or "maintainer" in the context of the
Airflow project. This is a person
-who has write access to the Airflow repository and can merge pull requests.
Committers (also known as maintainers)
-are also responsible for reviewing pull requests and guiding contributors to
make their first contribution.
-They are also responsible for making sure that the project is moving forward
and that the quality of the
-code is maintained.
-
-The term "committer" and "maintainer" is used interchangeably. The term
"committer" is the official term used by the
-Apache Software Foundation, while "maintainer" is more commonly used in the
Open Source community and is used
-in context of GitHub in a number of guidelines and documentation, so this
document will mostly use "maintainer",
-when speaking about Github, Pull Request, Github Issues and Discussions. On
the other hand, "committer" is more
-often used in devlist discussions, official communications, Airflow website
and every time when we formally
-refer to the role.
-
-The official list of committers can be found `here
<https://airflow.apache.org/docs/apache-airflow/stable/project.html#committers>`__.
-
-Additionally, committers are listed in a few other places (some of these may
only be visible to existing committers):
-
-* https://whimsy.apache.org/roster/committee/airflow
-* https://github.com/orgs/apache/teams/airflow-committers/members
-
-Committers are responsible for:
-
-* Championing one or more items on the `Roadmap
<https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home>`__
-* Reviewing & Merging Pull-Requests
-* Scanning and responding to GitHub issues
-* Responding to questions on the dev mailing list (d...@airflow.apache.org)
-
-Contributors
-------------
-
-A contributor is anyone who wants to contribute code, documentation, tests,
ideas, or anything to the
-Apache Airflow project.
-
-Contributors are responsible for:
-
-* Fixing bugs
-* Adding features
-* Championing one or more items on the `Roadmap
<https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home>`__.
-
-Security Team
+Commit Policy
-------------
-Security issues in Airflow are handled by the Airflow Security Team. The team
consists
-of selected PMC members that are interested in looking at, discussing and
fixing
-security issues, but it can also include committers and non-committer
contributors that are
-not PMC members yet and have been approved by the PMC members in a vote. You
can request to
-be added to the team by sending a message to priv...@airflow.apache.org.
However, the team
-should be small and focused on solving security issues, so the requests will
be evaluated
-on a case-by-case basis and the team size will be kept relatively small,
limited to only actively
-security-focused contributors.
-
-There are certain expectations from the members of the security team:
-
-* They are supposed to be active in assessing, discussing, fixing and
releasing the
- security issues in Airflow. While it is perfectly understood that as
volunteers, we might have
- periods of lower activity, prolonged lack of activity and participation will
result in removal
- from the team, pending PMC decision (the decision on removal can be taken by
`LAZY CONSENSUS <https://community.apache.org/committers/lazyConsensus.html>`_
among
- all the PMC members on priv...@airflow.apache.org mailing list).
-
-* They are not supposed to reveal the information about pending and unfixed
security issues to anyone
- (including their employers) unless specifically authorised by the security
team members, specifically
- if diagnosing and solving the issue might involve the need of external
experts - for example security
- experts that are available through Airflow stakeholders. The intent about
involving 3rd parties has
- to be discussed and agreed upon at secur...@airflow.apache.org.
-
-* They have to have an `ICLA
<https://www.apache.org/licenses/contributor-agreements.html>`_ signed with
- Apache Software Foundation.
-
-* The security team members might inform 3rd parties about fixes, for example
in order to assess if the fix
- is solving the problem or in order to assess its applicability to be applied
by 3rd parties, as soon
- as a PR solving the issue is opened in the public airflow repository.
-
-* In case of critical security issues, the members of the security team might
iterate on a fix in a
- private repository and only open the PR in the public repository once the
fix is ready to be released,
- with the intent of minimizing the time between the fix being available and
the fix being released. In this
- case the PR might be sent to review and comment to the PMC members on
private list, in order to request
- an expedited voting on the release. The voting for such release might be
done on the
- priv...@airflow.apache.org mailing list and should be made public at the
d...@apache.airflow.org
- mailing list as soon as the release is ready to be announced.
-
-* The security team members working on the fix might be mentioned as
remediation developers in the CVE
- including their job affiliation if they want to.
-
-* Community members acting as release managers are by default members of the
security team and unless they
- want to, they do not have to be involved in discussing and solving the
issues. They are responsible for
- releasing the CVE information (announcement and publishing to security
indexes) as part of the
- release process. This is facilitated by the security tool provided by the
Apache Software Foundation.
-
-* Severity of the issue is determined based on the criteria described in the
- `Severity Rating blog post
<https://security.apache.org/blog/severityrating/>`_ by the Apache Software
- Foundation Security team.
-
-Periodic Security team rotation
--------------------------------
-
-Handling security issues is something of a chore, it takes vigilance, requires
quick reaction and responses
-and often requires to act outside of the regular "day" job. This means that
not everyone can keep up with
-being part of the security team for long while being engaged and active. While
we do not expect all the
-security team members to be active all the time, and - since we are
volunteers, it's perfectly understandable
-that work, personal life, family and generally life might not help with being
active. And this is not a
-considered as being failure, it's more stating the fact of life.
-
-Also prolonged time of being exposed to handling "other's" problems and
discussing similar kinds of problem
-and responses might be tiring and might lead to burnout.
-
-However, for those who have never done that before, participation in the
security team might be an interesting
-experience and a way to learn a lot about security and security issue
handling. We have a lot of
-established processes and tools that make the work of the security team
members easier, so this can be
-treated as a great learning experience for some community members. And knowing
that this is not
-a "lifetime" assignment, but rather a temporary engagement might make it
easier for people to decide to
-join the security team.
-
-That's why introduced rotation of the security team members.
-
-Periodically - every 3-4 months (depending on actual churn of the security
issues that are reported to us),
-we re-evaluate the engagement and activity of the security team members, and
we ask them if they want to
-continue being part of the security team, taking into account their engagement
since the last team refinement.
-Generally speaking if the engagement during the last period was marginal, the
person is considered as a
-candidate for removing from the team and it requires a deliberate confirmation
of re-engagement to take
-the person off-the-list.
-
-At the same time we open up the possibility to other people in the community
to join the team and make
-a "call for new security team members" where community members can volunteer
to join the security team.
-Such volunteering should happen on the private@ list. The current members of
the security team as well
-as PMC members can also nominate other community members to join the team and
those new team members
-have to be well recognized and trusted by the community and accepted by the
PMC.
-
-The proposal of team refinement is passed to the PMC as LAZY CONSENSUS (or
VOTE if consensus cannot
-be reached). In case the consensus cannot be reached for the whole list, we
can split it and ask for
-lazy consensus for each person separately.
-
-Contribution Workflow
-=====================
-
-Typically, you start your first contribution by reviewing open tickets
-at `GitHub issues <https://github.com/apache/airflow/issues>`__.
-
-If you create pull-request, you don't have to create an issue first, but if
you want, you can do it.
-Creating an issue will allow you to collect feedback or share plans with other
people.
-
-For example, you want to have the following sample ticket assigned to you:
-`#7782: Add extra CC: to the emails sent by Airflow
<https://github.com/apache/airflow/issues/7782>`_.
-
-In general, your contribution includes the following stages:
-
-.. image:: images/workflow.png
- :align: center
- :alt: Contribution Workflow
-
-1. Make your own `fork
<https://help.github.com/en/github/getting-started-with-github/fork-a-repo>`__
of
- the Apache Airflow `main repository <https://github.com/apache/airflow>`__.
-
-2. Create a `local virtualenv <LOCAL_VIRTUALENV.rst>`_,
- initialize the `Breeze environment <dev/breeze/doc/breeze.rst>`__, and
- install `pre-commit framework <STATIC_CODE_CHECKS.rst#pre-commit-hooks>`__.
- If you want to add more changes in the future, set up your fork and enable
GitHub Actions.
-
-3. Join `devlist
<https://lists.apache.org/list.html?d...@airflow.apache.org>`__
- and set up a `Slack account <https://s.apache.org/airflow-slack>`__.
-
-4. Make the change and create a `Pull Request (PR) from your fork
<https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork>`__.
-
-5. Ping @ #development slack, comment @people. Be annoying. Be considerate.
-
-Step 1: Fork the Apache Airflow Repo
-------------------------------------
-From the `apache/airflow <https://github.com/apache/airflow>`_ repo,
-`create a fork
<https://help.github.com/en/github/getting-started-with-github/fork-a-repo>`_:
-
-.. image:: images/fork.png
- :align: center
- :alt: Creating a fork
-
-
-Step 2: Configure Your Environment
-----------------------------------
-
-You can use several development environments for Airflow. If you prefer to
have development environments
-on your local machine, you might choose Local Virtualenv, or dockerized Breeze
environment, however we
-also have support for popular remote development environments: GitHub
Codespaces and GitPodify.
-You can see the differences between the various environments
-`here
<https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#development-environments>`__.
-
-The local env instructions can be found in full in the `LOCAL_VIRTUALENV.rst
<https://github.com/apache/airflow/blob/main/LOCAL_VIRTUALENV.rst>`_ file.
-
-The Breeze Docker Compose env is to maintain a consistent and common
development environment so that you
-can replicate CI failures locally and work on solving them locally rather by
pushing to CI.
-
-The Breeze instructions can be found in full in the
-`Breeze
<https://github.com/apache/airflow/blob/main/dev/breeze/doc/breeze.rst>`_ file.
-
-You can configure the Docker-based Breeze development environment as follows:
-
-1. Install the latest versions of the `Docker Community Edition
<https://docs.docker.com/get-docker/>`_ and `Docker Compose
<https://docs.docker.com/compose/install/#install-compose>`_ and add them to
the PATH.
-
-2. Install `jq`_ on your machine. The exact command depends on the operating
system (or Linux distribution) you use.
-
-.. _jq: https://stedolan.github.io/jq/
-
-For example, on Ubuntu:
-
-.. code-block:: bash
-
- sudo apt install jq
-
-or on macOS with `Homebrew <https://formulae.brew.sh/formula/jq>`_
-
-.. code-block:: bash
-
- brew install jq
-
-3. Enter Breeze, and run the following in the Airflow source code directory:
-
-.. code-block:: bash
-
- breeze
-
-Breeze starts with downloading the Airflow CI image from
-the Docker Hub and installing all required dependencies.
-
-This will enter the Docker environment and mount your local sources
-to make them immediately visible in the environment.
-
-4. Create a local virtualenv, for example:
-
-.. code-block:: bash
-
- mkvirtualenv myenv --python=python3.9
-
-5. Initialize the created environment:
-
-.. code-block:: bash
-
- ./scripts/tools/initialize_virtualenv.py
-
-
-6. Open your IDE (for example, PyCharm) and select the virtualenv you created
- as the project's default virtualenv in your IDE.
-
-Step 3: Connect with People
----------------------------
-
-For effective collaboration, make sure to join the following Airflow groups:
-
-- Mailing lists:
-
- - Developer's mailing list `<dev-subscr...@airflow.apache.org>`_
- (quite substantial traffic on this list)
-
- - All commits mailing list: `<commits-subscr...@airflow.apache.org>`_
- (very high traffic on this list)
-
- - Airflow users mailing list: `<users-subscr...@airflow.apache.org>`_
- (reasonably small traffic on this list)
-
-- `Issues on GitHub <https://github.com/apache/airflow/issues>`__
-
-- `Slack (chat) <https://s.apache.org/airflow-slack>`__
-
-Step 4: Prepare PR
-------------------
-
-1. Update the local sources to address the issue.
-
- For example, to address this example issue, do the following:
-
- * Read about `email configuration in Airflow
</docs/apache-airflow/howto/email-config.rst>`__.
-
- * Find the class you should modify. For the example GitHub issue,
- this is `email.py
<https://github.com/apache/airflow/blob/main/airflow/utils/email.py>`__.
-
- * Find the test class where you should add tests. For the example ticket,
- this is `test_email.py
<https://github.com/apache/airflow/blob/main/tests/utils/test_email.py>`__.
-
- * Make sure your fork's main is synced with Apache Airflow's main before
you create a branch. See
- `How to sync your fork <#how-to-sync-your-fork>`_ for details.
-
- * Create a local branch for your development. Make sure to use latest
- ``apache/main`` as base for the branch. See `How to Rebase PR
<#how-to-rebase-pr>`_ for some details
- on setting up the ``apache`` remote. Note, some people develop their
changes directly in their own
- ``main`` branches - this is OK and you can make PR from your main to
``apache/main`` but we
- recommend to always create a local branch for your development. This
allows you to easily compare
- changes, have several changes that you work on at the same time and many
more.
- If you have ``apache`` set as remote then you can make sure that you have
latest changes in your main
- by ``git pull apache main`` when you are in the local ``main`` branch. If
you have conflicts and
- want to override your locally changed main you can override your local
changes with
- ``git fetch apache; git reset --hard apache/main``.
-
- * Modify the class and add necessary code and unit tests.
-
- * Run the unit tests from the `IDE
<TESTING.rst#running-unit-tests-from-ide>`__
- or `local virtualenv
<TESTING.rst#running-unit-tests-from-local-virtualenv>`__ as you see fit.
-
- * Run the tests in `Breeze
<TESTING.rst#running-unit-tests-inside-breeze>`__.
-
- * Run and fix all the `static checks <STATIC_CODE_CHECKS.rst>`__. If you
have
- `pre-commits installed <STATIC_CODE_CHECKS.rst#pre-commit-hooks>`__,
- this step is automatically run while you are committing your code. If
not, you can do it manually
- via ``git add`` and then ``pre-commit run``.
-
- * Consider adding a newsfragment to your PR so you can add an entry in the
release notes.
- The following newsfragment types are supported:
-
- * `significant`
- * `feature`
- * `improvement`
- * `bugfix`
- * `doc`
- * `misc`
-
- To add a newsfragment, create an ``rst`` file named
``{pr_number}.{type}.rst`` (e.g. ``1234.bugfix.rst``)
- and place in either `newsfragments
<https://github.com/apache/airflow/blob/main/newsfragments>`__ for core
newsfragments,
- or `chart/newsfragments
<https://github.com/apache/airflow/blob/main/chart/newsfragments>`__ for helm
chart newsfragments.
-
- In general newsfragments must be one line. For newsfragment type
``significant``, you may include summary and body separated by a blank line,
similar to ``git`` commit messages.
-
-2. Rebase your fork, squash commits, and resolve all conflicts. See `How to
rebase PR <#how-to-rebase-pr>`_
- if you need help with rebasing your change. Remember to rebase often if
your PR takes a lot of time to
- review/fix. This will make rebase process much easier and less painful and
the more often you do it,
- the more comfortable you will feel doing it.
-
-3. Re-run static code checks again.
-
-4. Make sure your commit has a good title and description of the context of
your change, enough
- for maintainers reviewing it to understand why you are proposing a change.
Make sure to follow other
- PR guidelines described in `Pull Request guidelines
<#pull-request-guidelines>`_.
- Create Pull Request! Make yourself ready for the discussion!
-
-5. The ``static checks`` and ``tests`` in your PR serve as a
first-line-of-check, whether the PR
- passes the quality bar for Airflow. It basically means that until you get
your PR green, it is not
- likely to get reviewed by maintainers unless you specifically ask for it
and explain that you would like
- to get first pass of reviews and explain why achieving ``green`` status for
it is not easy/feasible/desired.
- Similarly if your PR contains ``[WIP]`` in the title or it is marked as
``Draft`` it is not likely to get
- reviewed by maintainers unless you specifically ask for it and explain why
and what specifically you want
- to get reviewed before it reaches ``Ready for review`` status. This might
happen if you want to get initial
- feedback on the direction of your PR or if you want to get feedback on the
design of your PR.
-
-6. Avoid @-mentioning individual maintainers in your PR, unless you have good
reason to believe that they are
- available, have time and/or interest in your PR. Generally speaking there
are no "exclusive" reviewers for
- different parts of the code. Reviewers review PRs and respond when they
have some free time to spare and
- when they feel they can provide some valuable feedback. If you want to get
attention of maintainers, you can just
- follow-up on your PR and ask for review in general, however be considerate
and do not expect "immediate"
- reviews. People review when they have time, most of the maintainers do such
reviews in their
- free time, which is taken away from their families and other interests, so
allow sufficient time before you
- follow-up - but if you see no reaction in several days, do follow-up, as
with the number of PRs we have
- daily, some of them might simply fall through the cracks, and following up
shows your interest in completing
- the PR as well as puts it at the top of "Recently commented" PRs. However,
be considerate and mindful of
- the time zones, holidays, busy periods, and expect that some discussions
and conversation might take time
- and get stalled occasionally. Generally speaking it's the author's
responsibility to follow-up on the PR when
- they want to get it reviewed and merged.
-
-
-Step 5: Pass PR Review
-----------------------
-
-.. image:: images/review.png
- :align: center
- :alt: PR Review
-
-Note that maintainers will use **Squash and Merge** instead of **Rebase and
Merge**
-when merging PRs and your commit will be squashed to single commit.
-
-When a reviewer starts a conversation it is expected that you respond to
questions, suggestions, doubts,
-and generally it's great if all such conversations seem to converge to a
common understanding. You do not
-necessarily have to apply all the suggestions (often they are just opinions
and suggestions even if they are
-coming from seasoned maintainers) - it's perfectly ok that you respond to it
with your own opinions and
-understanding of the problem and your approach and if you have good arguments,
presenting them is a good idea.
-
-The reviewers might leave several types of responses:
-
-* ``General PR comment`` - which usually means that there is a
question/opinion/suggestion on how the PR can be
- improved, or it's an ask to explain how you understand the PR. You can
usually quote some parts of such
- general comment and respond to it in your comments. Often comments that are
raising questions in general
- might lead to different discussions, even a request to move the discussion
to the devlist or even lead to
- completely new PRs created as a spin-off of the discussion.
-
-* ``Comment/Conversation around specific lines of code`` - such conversation
usually flags a potential
- improvement, or a potential problem with the code. It's a good idea to
respond to such comments and explain
- your approach and understanding of the problem. The whole idea of a
conversation is try to reach a consensus
- on a good way to address the problem. As an author you can resolve the
conversation if you think the
- problem raised in the comment is resolved or ask the reviewer to re-review,
confirm If you do not understand
- the comment, you can ask for clarifications. Generally assume good intention
of the person who is reviewing
- your code and resolve conversations also having good intentions. Understand
that it's not a person that
- is criticised or argued with, but rather the code and the approach. The
important thing is to take care
- about quality of the the code and the project and want to make sure that the
code is good.
-
- It's ok to mark the conversation resolved by anyone who can do it - it could
be the author, who thinks
- the arguments are changes implemented make the conversation resolved, or the
maintainer/person who
- started the conversation or it can be even marked as resolved by the
maintainer who attempts to merge the
- PR and thinks that all conversations are resolved. However if you want to
make sure attention and decision
- on merging the PR is given by maintainer, make sure you monitor, follow-up
and close the conversations when
- you think they are resolved (ideally explaining why you think the
conversation is resolved).
-
-* ``Request changes`` - this is where maintainer is pretty sure that you
should make a change to your PR
- because it contains serious flaw, design misconception, or a bug or it is
just not in-line with the common
- approach Airflow community took on the issue. Usually you should respond to
such request and either fix
- the problem or convince the maintainer that they were wrong (it happens more
often than you think).
- Sometimes even if you do not agree with the request, it's a good idea to
make the change anyway, because
- it might be a good idea to follow the common approach in the project.
Sometimes it might even happen that
- two maintainers will have completely different opinions on the same issue
and you will have to lead the
- discussion to try to achieve consensus. If you cannot achieve consensus and
you think it's an important
- issue, you can ask for a vote on the issue by raising a devlist discussion -
where you explain your case
- and follow up the discussion with a vote when you cannot achieve consensus
there. The ``Request changes``
- status can be withdrawn by the maintainer, but if they don't - such PR
cannot be merged - maintainers have
- the right to veto any code modification according to the `Apache Software
Foundation rules
<https://www.apache.org/foundation/voting.html#votes-on-code-modification>`_.
-
-* ``Approval`` - this is given by a maintainer after the code has been
reviewed and the maintainer agrees that
- it is a good idea to merge it. There might still be some unresolved
conversations, requests and questions on
- such PR and you are expected to resolve them before the PR is merged. But
the ``Approval`` status is a sign
- of trust from the maintainer who gave the approval that they think the PR is
good enough as long as their
- comments will be resolved and they put the trust in the hands of the author
and - possibly - other
- maintainers who will merge the request that they can do that without
follow-up re-review and verification.
-
-
-You need to have ``Approval`` of at least one maintainer (if you are
maintainer yourself, it has to be
-another maintainer). Ideally you should have 2 or more maintainers reviewing
the code that touches
-the core of Airflow - we do not have enforcement about ``2+`` reviewers
required for Core of Airflow,
-but maintainers will generally ask in the PR if they think second review is
needed.
-
-Your PR can be merged by a maintainer who will see that the PR is approved,
all conversations are resolved
-and the code looks good. The criteria for PR being merge-able are:
-
-* ``green status for static checks and tests``
-* ``conversations resolved``
-* ``approval from 1 (or more for core changes) maintainers``
-* no unresolved ``Request changes``
-
-Once you reach the status, you do not need to do anything to get the PR
merged. One of the maintainers
-will merge such PRs. However if you see that for a few days such a PR is not
merged, do not hesitate to comment
-on your PR and mention that you think it is ready to be merged. Also, it's a
good practice to rebase your PR
-to latest ``main``, because there could be other changes merged in the
meantime that might cause conflicts or
-fail tests or static checks, so by rebasing a PR that has been build few days
ago you make sure that it
-still passes the tests and static checks today.
-
-
-.. note:: |experimental|
-
- In December 2023 we enabled - experimentally - the requirement to resolve
all the open conversations in a
- PR in order to make it merge-able. You will see in the status of the PR
that it needs to have all the
- conversations resolved before it can be merged.
-
- This is an experiment and we will evaluate by the end of January 2024. If
it turns out to be a good idea,
- we will keep it enabled in the future.
-
- The goal of this experiment is to make it easier to see when there are some
conversations that are not
- resolved for everyone involved in the PR - author, reviewers and
maintainers who try to figure out if
- the PR is ready to merge and - eventually - merge it. The goal is also to
use conversations more as a "soft" way
- to request changes and limit the use of ``Request changes`` status to only
those cases when the maintainer
- is sure that the PR should not be merged in the current state. That should
lead to faster review/merge
- cycle and less problems with stalled PRs that have ``Request changes``
status but all the issues are
- already solved (assuming that maintainers will start treating the
conversations this way).
-
-
-Pull Request guidelines
-=======================
-
-Before you submit a Pull Request (PR) from your forked repo, check that it
meets
-these guidelines:
-
-- Include tests, either as doctests, unit tests, or both, to your pull
request.
-
- The airflow repo uses `GitHub Actions
<https://help.github.com/en/actions>`__ to
- run the tests and `codecov <https://codecov.io/gh/apache/airflow>`__ to
track
- coverage. You can set up both for free on your fork. It will help you make
sure you do not
- break the build with your PR and that you help increase coverage.
- Also we advise to install locally `pre-commit hooks
<STATIC_CODE_CHECKS.rst#pre-commit-hooks>`__ to
- apply various checks, code generation and formatting at the time you make
a local commit - which
- gives you near-immediate feedback on things you need to fix before you
push your code to the PR, or in
- many case it will even fix it for you locally so that you can add and
commit it straight away.
-
-- Follow our project's `Coding style and best practices`_. Usually we
attempt to enforce the practices by
- having appropriate pre-commits. There are checks amongst them that aren't
currently enforced
- programmatically (either because they are too hard or just not yet done).
-
-- We prefer that you ``rebase`` your PR (and do it quite often) rather than
merge. It leads to
- easier reviews and cleaner changes where you know exactly what changes
you've done. You can learn more
- about rebase vs. merge workflow in `Rebase and merge your pull request
<https://github.blog/2016-09-26-rebase-and-merge-pull-requests/>`__
- and `Rebase your fork <http://stackoverflow.com/a/7244456/1110993>`__.
Make sure to resolve all conflicts
- during rebase.
-
-- When merging PRs, Maintainer will use **Squash and Merge** which means
then your PR will be merged as one
- commit, regardless of the number of commits in your PR. During the review
cycle, you can keep a commit
- history for easier review, but if you need to, you can also squash all
commits to reduce the
- maintenance burden during rebase.
-
-- Add an `Apache License <http://www.apache.org/legal/src-headers.html>`__
header to all new files. If you
- have ``pre-commit`` installed, pre-commit will do it automatically for
you. If you hesitate to install
- pre-commit for your local repository - for example because it takes a few
seconds to commit your changes,
- this one thing might be a good reason to convince anyone to install
pre-commit.
-
-- If your PR adds functionality, make sure to update the docs as part of the
same PR, not only
- code and tests. Docstring is often sufficient. Make sure to follow the
Sphinx compatible standards.
-
-- Make sure your code fulfills all the
- `static code checks <STATIC_CODE_CHECKS.rst#static-code-checks>`__ we have
in our code. The easiest way
- to make sure of that is - again - to install `pre-commit hooks
<STATIC_CODE_CHECKS.rst#pre-commit-hooks>`__
-
-- Make sure your PR is small and focused on one change only - avoid adding
unrelated changes, mixing
- adding features and refactoring. Keeping to that rule will make it easier
to review your PR and will make
- it easier for release managers if they decide that your change should be
cherry-picked to release it in a
- bug-fix release of Airflow. If you want to add a new feature and refactor
the code, it's better to split the
- PR to several smaller PRs. It's also quite a good and common idea to keep
a big ``Draft`` PR if you have
- a bigger change that you want to make and then create smaller PRs from it
that are easier to review and
- merge and cherry-pick. It takes a long time (and a lot of attention and
focus of a reviewer to review
- big PRs so by splitting it to smaller PRs you actually speed up the review
process and make it easier
- for your change to be eventually merged.
-
-- Run relevant tests locally before opening PR. Often tests are placed in
the files that are corresponding
- to the changed code (for example for ``airflow/cli/cli_parser.py`` changes
you have tests in
- ``tests/cli/test_cli_parser.py``). However there are a number of cases
where the tests that should run
- are placed elsewhere - you can either run tests for the whole
``TEST_TYPE`` that is relevant (see
- ``breeze testing tests --help`` output for available test types) or you
can run all tests, or eventually
- you can push your code to PR and see results of the tests in the CI.
-
-- You can use any supported python version to run the tests, but the best is
to check
- if it works for the oldest supported version (Python 3.8 currently). In
rare cases
- tests might fail with the oldest version when you use features that are
available in newer Python
- versions. For that purpose we have ``airflow.compat`` package where we
keep back-ported
- useful features from newer versions.
-
-- Adhere to guidelines for commit messages described in this `article
<http://chris.beams.io/posts/git-commit/>`__.
- This makes the lives of those who come after you (and your future self) a
lot easier.
-
-Airflow Git Branches
-====================
-
-All new development in Airflow happens in the ``main`` branch. All PRs should
target that branch.
-
-We also have a ``v2-*-test`` branches that are used to test ``2.*.x`` series
of Airflow and where maintainers
-cherry-pick selected commits from the main branch.
-
-Cherry-picking is done with the ``-x`` flag.
-
-The ``v2-*-test`` branch might be broken at times during testing. Expect
force-pushes there so
-maintainers should coordinate between themselves on who is working on the
``v2-*-test`` branch -
-usually these are developers with the release manager permissions.
-
-The ``v2-*-stable`` branch is rather stable - there are minimum changes coming
from approved PRs that
-passed the tests. This means that the branch is rather, well, "stable".
-
-Once the ``v2-*-test`` branch stabilises, the ``v2-*-stable`` branch is
synchronized with ``v2-*-test``.
-The ``v2-*-stable`` branches are used to release ``2.*.x`` releases.
-
-The general approach is that cherry-picking a commit that has already had a PR
and unit tests run
-against main is done to ``v2-*-test`` branches, but PRs from contributors
towards 2.0 should target
-``v2-*-stable`` branches.
-
-The ``v2-*-test`` branches and ``v2-*-stable`` ones are merged just before the
release and that's the
-time when they converge.
-
-The production images are released in DockerHub from:
-
-* main branch for development
-* ``2.*.*``, ``2.*.*rc*`` releases from the ``v2-*-stable`` branch when we
prepare release candidates and
- final releases.
-
-Development Environments
-========================
-
-There are two environments, available on Linux and macOS, that you can use to
-develop Apache Airflow:
-
-- `Local virtualenv development environment <LOCAL_VIRTUALENV.rst>`_
- that supports running unit tests and can be used in your IDE.
-
-- `Breeze Docker-based development environment <dev/breeze/doc/breeze.rst>`_
that provides
- an end-to-end CI solution with all software dependencies covered.
-
-The table below summarizes differences between the environments:
-
-
-========================= ================================
===================================== ========================================
-**Property** **Local virtualenv** **Breeze
environment** **GitHub Codespaces**
-========================= ================================
===================================== ========================================
-Dev machine needed - (-) You need a dev PC - (-) You need a
dev PC (+) Works with remote setup
-------------------------- --------------------------------
------------------------------------- ----------------------------------------
-Test coverage - (-) unit tests only - (+) integration
and unit tests (*/-) integration tests (extra config)
-------------------------- --------------------------------
------------------------------------- ----------------------------------------
-Setup - (+) automated with breeze cmd - (+) automated
with breeze cmd (+) automated with VSCode
-------------------------- --------------------------------
------------------------------------- ----------------------------------------
-Installation difficulty - (-) depends on the OS setup - (+) works
whenever Docker works (+) works in a modern browser/VSCode
-------------------------- --------------------------------
------------------------------------- ----------------------------------------
-Team synchronization - (-) difficult to achieve - (+) reproducible
within team (+) reproducible within team
-------------------------- --------------------------------
------------------------------------- ----------------------------------------
-Reproducing CI failures - (-) not possible in many cases - (+) fully
reproducible (+) reproduce CI failures
-------------------------- --------------------------------
------------------------------------- ----------------------------------------
-Ability to update - (-) requires manual updates - (+) automated
update via breeze cmd (+/-) can be rebuild on demand
-------------------------- --------------------------------
------------------------------------- ----------------------------------------
-Disk space and CPU usage - (+) relatively lightweight - (-) uses GBs of
disk and many CPUs (-) integration tests (extra config)
-------------------------- --------------------------------
------------------------------------- ----------------------------------------
-IDE integration - (+) straightforward - (-) via remote
debugging only (-) integration tests (extra config)
-========================= ================================
===================================== ----------------------------------------
-
-
-Typically, you are recommended to use both of these environments depending on
your needs.
-
-Local virtualenv Development Environment
-----------------------------------------
-
-All details about using and running local virtualenv environment for Airflow
can be found
-in `LOCAL_VIRTUALENV.rst <LOCAL_VIRTUALENV.rst>`__.
-
-Benefits:
-
-- Packages are installed locally. No container environment is required.
-
-- You can benefit from local debugging within your IDE.
-
-- With the virtualenv in your IDE, you can benefit from autocompletion and
running tests directly from the IDE.
-
-Limitations:
-
-- You have to maintain your dependencies and local environment consistent
with
- other development environments that you have on your local machine.
-
-- You cannot run tests that require external components, such as mysql,
- postgres database, hadoop, mongo, cassandra, redis, etc.
-
- The tests in Airflow are a mixture of unit and integration tests and some
of
- them require these components to be set up. Local virtualenv supports only
- real unit tests. Technically, to run integration tests, you can configure
- and install the dependencies on your own, but it is usually complex.
- Instead, you are recommended to use
- `Breeze development environment <dev/breeze/doc/breeze.rst>`__ with all
required packages
- pre-installed.
-
-- You need to make sure that your local environment is consistent with other
- developer environments. This often leads to a "works for me" syndrome. The
- Breeze container-based solution provides a reproducible environment that is
- consistent with other developers.
-
-- You are **STRONGLY** encouraged to also install and use `pre-commit hooks
<STATIC_CODE_CHECKS.rst#pre-commit-hooks>`_
- for your local virtualenv development environment.
- Pre-commit hooks can speed up your development cycle a lot.
-
-Breeze Development Environment
-------------------------------
-
-All details about using and running Airflow Breeze can be found in
-`Breeze <dev/breeze/doc/breeze.rst>`__.
-
-The Airflow Breeze solution is intended to ease your local development as
"*It's
-a Breeze to develop Airflow*".
-
-Benefits:
-
-- Breeze is a complete environment that includes external components, such as
- mysql database, hadoop, mongo, cassandra, redis, etc., required by some of
- Airflow tests. Breeze provides a preconfigured Docker Compose environment
- where all these services are available and can be used by tests
- automatically.
-
-- Breeze environment is almost the same as used in the CI automated builds.
- So, if the tests run in your Breeze environment, they will work in the CI
as well.
- See `<CI.rst>`_ for details about Airflow CI.
-
-Limitations:
-
-- Breeze environment takes significant space in your local Docker cache.
There
- are separate environments for different Python and Airflow versions, and
- each of the images takes around 3GB in total.
-
-- Though Airflow Breeze setup is automated, it takes time. The Breeze
- environment uses pre-built images from DockerHub and it takes time to
- download and extract those images. Building the environment for a
particular
- Python version takes less than 10 minutes.
-
-- Breeze environment runs in the background taking precious resources, such
as
- disk space and CPU. You can stop the environment manually after you use it
- or even use a ``bare`` environment to decrease resource usage.
-
-
-
-.. note::
-
- Breeze CI images are not supposed to be used in production environments.
- They are optimized for repeatability of tests, maintainability and speed of
building rather
- than production performance. The production images are not yet officially
published.
-
-
-
-Airflow dependencies
-====================
-
-.. note::
-
- Only ``pip`` installation is currently officially supported.
-
- While there are some successes with using other tools like `poetry
<https://python-poetry.org/>`_ or
- `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the
same workflow as
- ``pip`` - especially when it comes to constraint vs. requirements
management.
- Installing via ``Poetry`` or ``pip-tools`` is not currently supported.
-
- There are known issues with ``bazel`` that might lead to circular
dependencies when using it to install
- Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel``
community works on fixing
- the problem in `this PR
<https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that
- newer versions of ``bazel`` will handle it.
-
- If you wish to install airflow using those tools you should use the
constraint files and convert
- them to appropriate format and workflow that your tool requires.
-
-
-Extras
-------
-
-There are a number of extras that can be specified when installing Airflow.
Those
-extras can be specified after the usual pip install - for example ``pip
install -e.[ssh]`` for editable
-installation. Note that there are two kinds of those extras - ``regular``
extras (used when you install
-airflow as a user, but in ``editable`` mode you can also install ``devel``
extras that are necessary if
-you want to run airflow locally for testing and ``doc`` extras that install
tools needed to build
-the documentation.
-
-This is the full list of those extras:
-
-Devel extras
-.............
-
-The ``devel`` extras are not available in the released packages. They are only
available when you install
-Airflow from sources in ``editable`` installation - i.e. one that you are
usually using to contribute to
-Airflow. They provide tools such as ``pytest`` and ``mypy`` for general
purpose development and testing, also
-some providers have their own development-related extras tbat allow to install
tools necessary to run tests,
-where the tools are specific for the provider.
-
-
- .. START DEVEL EXTRAS HERE
-devel, devel-all, devel-all-dbs, devel-ci, devel-debuggers, devel-devscripts,
devel-duckdb, devel-
-hadoop, devel-mypy, devel-sentry, devel-static-checks, devel-tests
- .. END DEVEL EXTRAS HERE
-
-Doc extras
-...........
-
-The ``doc`` extras are not available in the released packages. They are only
available when you install
-Airflow from sources in ``editable`` installation - i.e. one that you are
usually using to contribute to
-Airflow. They provide tools needed when you want to build Airflow
documentation (note that you also need
-``devel`` extras installed for airflow and providers in order to build
documentation for airflow and
-provider packages respectively). The ``doc`` package is enough to build
regular documentation, where
-``doc_gen`` is needed to generate ER diagram we have describing our database.
-
- .. START DOC EXTRAS HERE
-doc, doc-gen
- .. END DOC EXTRAS HERE
-
-
-Regular extras
-..............
-
-Those extras are available as regular Airflow extras and are targeted to be
used by Airflow users and
-contributors to select features of Airflow they want to use They might install
additional providers or
-just install dependencies that are necessary to enable the feature.
+The following commit policy passed by a vote 8(binding) FOR to 0 against on
May 27, 2016 on the dev list
+and slightly modified and consensus reached in October 2020:
- .. START REGULAR EXTRAS HERE
-aiobotocore, airbyte, alibaba, all, all-core, all-dbs, amazon, apache-atlas,
apache-beam, apache-
-cassandra, apache-drill, apache-druid, apache-flink, apache-hdfs, apache-hive,
apache-impala,
-apache-kafka, apache-kylin, apache-livy, apache-pig, apache-pinot,
apache-spark, apache-webhdfs,
-apprise, arangodb, asana, async, atlas, atlassian-jira, aws, azure, cassandra,
celery, cgroups,
-cloudant, cncf-kubernetes, cohere, common-io, common-sql, crypto, databricks,
datadog, dbt-cloud,
-deprecated-api, dingding, discord, docker, druid, elasticsearch, exasol, fab,
facebook, ftp, gcp,
-gcp_api, github, github-enterprise, google, google-auth, graphviz, grpc,
hashicorp, hdfs, hive,
-http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes, ldap, leveldb,
microsoft-azure,
-microsoft-mssql, microsoft-psrp, microsoft-winrm, mongo, mssql, mysql, neo4j,
odbc, openai,
-openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty, pandas,
papermill, password,
-pgvector, pinecone, pinot, postgres, presto, rabbitmq, redis, s3, s3fs,
salesforce, samba, saml,
-segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake, spark,
sqlite, ssh, statsd,
-tableau, tabular, telegram, trino, vertica, virtualenv, weaviate, webhdfs,
winrm, yandex, zendesk
- .. END REGULAR EXTRAS HERE
+* Commits need a +1 vote from a committer who is not the author
+* Do not merge a PR that regresses linting or does not pass CI tests (unless
we have
+ justification such as clearly transient error).
+* When we do AIP voting, both PMC and committer +1s are considered as binding
vote.
Review Comment:
```suggestion
* When we do AIP voting, both PMC and committer +1s are considered a binding
vote.
```
##########
contribution-docs/airflow_dependencies_and_extras.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Airflow dependencies
+====================
+
+Airflow is not a standard python project. Most of the python projects fall
into one of two types -
+application or library. As described in
+`this StackOverflow question
<https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions>`_,
+the decision whether to pin (freeze) dependency versions for a python project
depends on the type. For
+applications, dependencies should be pinned, but for libraries, they should be
open.
+
+For application, pinning the dependencies makes it more stable to install in
the future - because new
Review Comment:
```suggestion
For applications, pinning the dependencies makes it more stable to install
in the future - because new
```
##########
contribution-docs/airflow_dependencies_and_extras.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Airflow dependencies
+====================
+
+Airflow is not a standard python project. Most of the python projects fall
into one of two types -
+application or library. As described in
+`this StackOverflow question
<https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions>`_,
+the decision whether to pin (freeze) dependency versions for a python project
depends on the type. For
+applications, dependencies should be pinned, but for libraries, they should be
open.
+
+For application, pinning the dependencies makes it more stable to install in
the future - because new
+(even transitive) dependencies might cause installation to fail. For libraries
- the dependencies should
+be open to allow several different libraries with the same requirements to be
installed at the same time.
+
+The problem is that Apache Airflow is a bit of both - application to install
and library to be used when
+you are developing your own operators and DAGs.
+
+This - seemingly unsolvable - puzzle is solved by having pinned constraints
files.
+
+.. contents:: :local:
+
+Pinned constraint files
+-----------------------
+
+.. note::
+
+ Only ``pip`` installation is officially supported.
+
+ While it is possible to install Airflow with tools like `poetry
<https://python-poetry.org/>`_ or
+ `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the
same workflow as
+ ``pip`` - especially when it comes to constraint vs. requirements
management.
+ Installing via ``Poetry`` or ``pip-tools`` is not currently supported.
+
+ There are known issues with ``bazel`` that might lead to circular
dependencies when using it to install
+ Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel``
community works on fixing
+ the problem in `this PR
<https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that
+ newer versions of ``bazel`` will handle it.
+
+ If you wish to install airflow using those tools you should use the
constraint files and convert
+ them to appropriate format and workflow that your tool requires.
+
+
+By default when you install ``apache-airflow`` package - the dependencies are
as open as possible while
+still allowing the apache-airflow package to install. This means that
``apache-airflow`` package might fail to
+install in case a direct or transitive dependency is released that breaks the
installation. In such case
Review Comment:
```suggestion
install when a direct or transitive dependency is released that breaks the
installation. In that case,
```
##########
contribution-docs/airflow_dependencies_and_extras.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Airflow dependencies
+====================
+
+Airflow is not a standard python project. Most of the python projects fall
into one of two types -
+application or library. As described in
+`this StackOverflow question
<https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions>`_,
+the decision whether to pin (freeze) dependency versions for a python project
depends on the type. For
+applications, dependencies should be pinned, but for libraries, they should be
open.
+
+For application, pinning the dependencies makes it more stable to install in
the future - because new
+(even transitive) dependencies might cause installation to fail. For libraries
- the dependencies should
+be open to allow several different libraries with the same requirements to be
installed at the same time.
+
+The problem is that Apache Airflow is a bit of both - application to install
and library to be used when
+you are developing your own operators and DAGs.
+
+This - seemingly unsolvable - puzzle is solved by having pinned constraints
files.
+
+.. contents:: :local:
+
+Pinned constraint files
+-----------------------
+
+.. note::
+
+ Only ``pip`` installation is officially supported.
+
+ While it is possible to install Airflow with tools like `poetry
<https://python-poetry.org/>`_ or
+ `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the
same workflow as
+ ``pip`` - especially when it comes to constraint vs. requirements
management.
+ Installing via ``Poetry`` or ``pip-tools`` is not currently supported.
+
+ There are known issues with ``bazel`` that might lead to circular
dependencies when using it to install
+ Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel``
community works on fixing
+ the problem in `this PR
<https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that
+ newer versions of ``bazel`` will handle it.
+
+ If you wish to install airflow using those tools you should use the
constraint files and convert
+ them to appropriate format and workflow that your tool requires.
+
+
+By default when you install ``apache-airflow`` package - the dependencies are
as open as possible while
+still allowing the apache-airflow package to install. This means that
``apache-airflow`` package might fail to
Review Comment:
```suggestion
still allowing the ``apache-airflow`` package to install. This means that
the ``apache-airflow`` package might fail to
```
##########
contribution-docs/airflow_dependencies_and_extras.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Airflow dependencies
+====================
+
+Airflow is not a standard python project. Most of the python projects fall
into one of two types -
+application or library. As described in
+`this StackOverflow question
<https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions>`_,
+the decision whether to pin (freeze) dependency versions for a python project
depends on the type. For
+applications, dependencies should be pinned, but for libraries, they should be
open.
+
+For application, pinning the dependencies makes it more stable to install in
the future - because new
+(even transitive) dependencies might cause installation to fail. For libraries
- the dependencies should
+be open to allow several different libraries with the same requirements to be
installed at the same time.
+
+The problem is that Apache Airflow is a bit of both - application to install
and library to be used when
+you are developing your own operators and DAGs.
+
+This - seemingly unsolvable - puzzle is solved by having pinned constraints
files.
+
+.. contents:: :local:
+
+Pinned constraint files
+-----------------------
+
+.. note::
+
+ Only ``pip`` installation is officially supported.
+
+ While it is possible to install Airflow with tools like `poetry
<https://python-poetry.org/>`_ or
+ `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the
same workflow as
+ ``pip`` - especially when it comes to constraint vs. requirements
management.
+ Installing via ``Poetry`` or ``pip-tools`` is not currently supported.
+
+ There are known issues with ``bazel`` that might lead to circular
dependencies when using it to install
+ Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel``
community works on fixing
Review Comment:
```suggestion
Airflow. Please switch to ``pip`` if you encounter such problems. The
``Bazel`` community is fixing
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
