feluelle commented on a change in pull request #10380: URL: https://github.com/apache/airflow/pull/10380#discussion_r474459881
########## File path: CI.rst ########## @@ -188,32 +578,104 @@ The issues that gets updated are different for different branches: * master: `Quarantine tests master <https://github.com/apache/airflow/issues/10118>`_ * v1-10-stable: `Quarantine tests v1-10-stable <https://github.com/apache/airflow/issues/10127>`_ -* v1-10-test: `Quarantine tests v1-10-test <hhttps://github.com/apache/airflow/issues/10128>`_ - -+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+ -| Job | Description | Pull Request Run | Direct Push/Merge Run | Scheduled Run | -+===========================+================================================================================================================+====================================+=================================+======================================================================+ -| Cancel previous workflow | Cancels the previously running workflow run if there is one running | Yes | Yes | Yes * | -+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+ -| Trigger tests | Checks if tests should be triggered | Yes | Yes | Yes * | -+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+ -| Quarantined tests | Those are tests that are flaky and we need to fix them | Yes (if tests-triggered) | Yes (Updates quarantine issue) | Yes * (updates quarantine issue) | -+---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+ - -Cancel other workflow runs workflow ------------------------------------ - -This workflow is run only on schedule (every 5 minutes) it's only purpose is to cancel other running -``CI Build`` workflows if important jobs failed in those runs. This is to save runners for other runs -in case we know that the build will not succeed anyway without some basic fixes to static checks or -documentation - effectively implementing missing "fail-fast" (on a job level) in Github Actions -similar to fail-fast in matrix strategy. - -The jobs that are considered as "fail-fast" are: - -* Static checks -* Docs -* Prepare Backport packages -* Helm tests -* Build Prod Image -* TTest OpenAPI client gen +* v1-10-test: `Quarantine tests v1-10-test <https://github.com/apache/airflow/issues/10128>`_ + +Force sync master from apache/airflow +------------------------------------- + +This is manually triggered workflow (via GitHub UI manual run) that should only be run in GitHub forks. +When triggered, it will force-push the "apache/airflow" master to the fork's master. It's the easiest +way to sync your fork master to the Apache Airflow's one. + +Naming conventions for stored images +==================================== + +The images produced during the CI builds are stored in the +`GitHub Registry <https://github.com/apache/airflow/packages>`_ + +The images are stored with both "latest" tag (for last master push image that passes all the tests as well +with the tags indicating the origin of the image. + +The image names follow the patterns: + ++--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+ +| Image | Name pattern | Tag for format | Comment | ++==============+============================+================================+============================================================================================+ +| Python image | python | <X.Y>-slim-buster-<RUN_ID> | Base python image used by both production and CI image. | +| | | <X.Y>-slim-buster-<COMMIT_SHA> | Python maintainer release new versions of those image with security fixes every few weeks. | ++--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+ +| CI image | <BRANCH>-python<X.Y>-ci | <RUN_ID> | CI image - this is the image used for most of the tests. | +| | | <COMMIT_SHA> | | ++--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+ +| PROD Build | <BRANCH>-python<X.Y>-build | <RUN_ID> | Production Build image - this is the "build" segment of production image. | +| image | | <COMMIT_SHA> | It contains build-essentials and all necessary packages to install PIP packages. | ++--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+ +| PROD image | <BRANCH>-python<X.Y> | <RUN_ID> | Production image. This is the actual production image - optimized for size. | +| | | <COMMIT_SHA> | It contains only compiled libraries and minimal set of dependencies to run Airflow. | ++--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+ + +* <BRANCH> might be either "master" or "v1-10-test" +* <X.Y> - Python version (Major + Minor). For "master" it should be in ["3.6", "3.7", "3.8"]. For + v1-10-test it should be in ["2.7", "3.5", "3.6". "3.7", "3.8"]. +* <RUN_ID> - Github Actions RUN_ID. You can get it from CI action job outputs (run id is printed in + logs and displayed as part of the step name. All PRs belong to some RUN_ID and this way you can + pull the very exact version of image used in that RUN_ID +* <COMMIT_SHA> - for images that get merged to "master" of "v1-10-test" the images are also tagged + with the commit SHA of that particular commit. This way you can easily find the image that was used + for testing for that "master" or "v1-10-test" test run. + +Reproducing CI Runs locally +=========================== + +Since we store images from every CI run, you should be able easily reproduce any of the CI build problems +locally. You can do it by pulling and using the right image and running it with the right docker command, +For example knowing that the CI build had 210056909 RUN_ID (you can find it from GitHub CI logs): + +.. code-block:: bash + + docker pull docker.pkg.github.com/apache/airflow/master-python3.6-ci:210056909 + + docker run -it docker.pkg.github.com/apache/airflow/master-python3.6-ci:210056909 + + +But you usually need to pass more variables amd complex setup if you want to connect to a database or +enable some integrations. Therefore it is easiest to use `Breeze <BREEZE.rs>`_ for that. For example if Review comment: ```suggestion enable some integrations. Therefore it is easiest to use `Breeze <BREEZE.rst>`_ for that. For example if ``` ########## File path: CI.rst ########## @@ -17,6 +17,10 @@ .. contents:: :local: +NOTE! This is the target architecture we are aspiring to. We have a +`<work in progress PR https://github.com/apache/airflow/pull/10368>`_ which implements this +architecture but the documentation generation and plan is merged separately as "future" change. + Review comment: This is not up to date anymore, is it? ########## File path: images/ci/pull_request_ci_flow.mermaid ########## @@ -0,0 +1,108 @@ +%% Licensed to the Apache Software Foundation (ASF) under one +%% or more contributor license agreements. See the NOTICE file +%% distributed with this work for additional information +%% regarding copyright ownership. The ASF licenses this file +%% to you under the Apache License, Version 2.0 (the +%% "License"); you may not use this file except in compliance +%% with the License. You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, +%% software distributed under the License is distributed on an +%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +%% KIND, either express or implied. See the License for the +%% specific language governing permissions and limitations +%% under the License. + +sequenceDiagram + Contributor-->>Fork: Submit PR Review comment: ```suggestion Contributor -->> Fork: Submit PR ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
