[airflow] branch v1-10-test updated (01eb5d4 -> 430c047)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard 01eb5d4 Simplifies check whether the CI image should be rebuilt (#12181) discard 42b3491 For v1-10-test PRs and pushes, use target branch scripts for images (#12339) discard 23151fd Deploy was not working from Breeze (#12319) discard f1f58f7 Python base image is shared between CI and PROD image (#12280) discard 533ea3c Added k9s as integrated tool to help with kubernetes testing (#12163) new 2807875 Added k9s as integrated tool to help with kubernetes testing (#12163) new b284205 Python base image is shared between CI and PROD image (#12280) new 0a51dfd Deploy was not working from Breeze (#12319) new a90131c For v1-10-test PRs and pushes, use target branch scripts for images (#12339) new 430c047 Simplifies check whether the CI image should be rebuilt (#12181) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (01eb5d4) \ N -- N -- N refs/heads/v1-10-test (430c047) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: scripts/ci/libraries/_kind.sh | 2 -- 1 file changed, 2 deletions(-)
[airflow] 04/05: For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a90131c0ae098c2a063cd94fdc1f36ace818b606
Author: Jarek Potiuk
AuthorDate: Fri Nov 13 14:28:36 2020 +0100
For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
Previously, always master scripts were used to build images
for workflow_run, because workflow_run always runs from master
branch. However that causes some surprising effects becuase the
sripts from master had to support both master and 1.10.
This change utilises a new feature in the "get-workflow-origin"
action - to get the target branch of PR and uses ci scripts from that
target branch.
This is perfectly secure, because both v1-10-test, v1-10-stable
and future 2-0 branches can only be updated by committers,
either by direct push or by merge.
(cherry picked from commit 7c4fe19e41ae02a1df1c0a217501cae2e0e84819)
---
.github/workflows/build-images-workflow-run.yml| 12 +--
.github/workflows/ci.yml | 38 +++---
.../workflows/label_when_reviewed_workflow_run.yml | 2 +-
3 files changed, 22 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index a3277a7..af71710 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -53,13 +53,14 @@ jobs:
targetCommitSha: ${{ steps.source-run-info.outputs.targetCommitSha }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
+ targetBranch: ${{ steps.source-run-info.outputs.targetBranch }}
sourceEvent: ${{ steps.source-run-info.outputs.sourceEvent }}
cacheDirective: ${{ steps.cache-directive.outputs.docker-cache }}
buildImages: ${{ steps.build-images.outputs.buildImages }}
upgradeToLatestConstraints: ${{
steps.upgrade-constraints.outputs.upgradeToLatestConstraints }}
steps:
- name: "Get information about the original trigger of the run"
- uses:
potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+ uses:
potiuk/get-workflow-origin@588cc14f9f1cdf1b8be3db816855e96422204fec # v1_3
id: source-run-info
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -326,17 +327,22 @@ jobs:
[Image Build](https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }})
for details" }
if: steps.defaults.outputs.proceed == 'true'
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} ) to
'main-airflow' to use main scripts"
+ - name: >
+ Checkout "${{ needs.cancel-workflow-runs.outputs.targetBranch }}"
branch to 'main-airflow' folder
+ to use ci/scripts from there.
uses: actions/checkout@v2
with:
path: "main-airflow"
+ ref: "${{ needs.cancel-workflow-runs.outputs.targetBranch }}"
if: steps.defaults.outputs.proceed == 'true'
- name: "Setup python"
uses: actions/setup-python@v2
with:
python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
if: steps.defaults.outputs.proceed == 'true'
- - name: "Override 'scripts/ci' with the ${{ github.ref }} version so
that the PR cannot override it."
+ - name: >
+ Override "scripts/ci" with the "${{
needs.cancel-workflow-runs.outputs.targetBranch }}" branch
+ so that the PR does not override it
# We should not override those scripts which become part of the image
as they will not be
# changed in the image built - we should only override those that are
executed to build
# the image.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 81890a7..df5f53e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -95,27 +95,15 @@ jobs:
image-build: ${{ steps.selective-checks.outputs.image-build }}
docs-build: ${{ steps.selective-checks.outputs.docs-build }}
needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
- needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
steps:
- name: "Get information about the PR"
-uses:
potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+uses:
potiuk/ge
[airflow] branch v1-10-test updated (430c047 -> 09fe8f1)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard 430c047 Simplifies check whether the CI image should be rebuilt (#12181) discard a90131c For v1-10-test PRs and pushes, use target branch scripts for images (#12339) discard 0a51dfd Deploy was not working from Breeze (#12319) discard b284205 Python base image is shared between CI and PROD image (#12280) discard 2807875 Added k9s as integrated tool to help with kubernetes testing (#12163) discard 301e5e4 Docker context files should be available earlier (#12219) discard d6e48ef Fixes continuous image rebuilding with Breeze (#12256) discard 1e6f091 Add Markdown linting to pre-commit (#11465) discard cb4eb46 Beautify Output of setup-installation pre-commit (#12218) discard 04b9037 Fix permissions of mounted /tmp directory for Breeze (#12157) discard 2076ea4 Remove popd which is a remnant from past (#12211) discard 262bc04 Fixes timeout in helm chart tests (#12209) discard 0f06d6f Fixed path of the test_core.py file in docs (#12191) discard e32ff8b Adds extra check while the selective checks are run (#12178) discard 40a1471 Uses always the same Python base image as used for CI image (#12177) discard d2628f6 Fixes "--force-clean-images" flag in Breeze (#12156) discard bb5175f Fixes undefined variables (#12155) discard 6848f9b Update to new helm stable repo (#12137) discard 5e805d6 Work properly if some variables are not defined (#12135) discard 5baa6b6 Add Kubernetes files to selective checks (#12114) discard bd318b5 Update install_mysql.sh (#12101) discard 70e00bf Fix proper SHA in check preventing accidentally merging PR (#12083) discard 2abe5c0 Use sys.exit() instead of exit() (#12084) discard 40256da Fixes problem with building a PROD image (#12080) discard 17ce4f1 If we build a new image, we should run more than basic checks (#12070) discard 1dcf44f Adds a forgotten word in a README.md (#12066) discard 47a9bc2 Uses DOCKER_TAG when building image in DockerHub (#12050) discard fa7cbec Fixes documentation-only selective checks (#12038) discard c523148 Checks if all the libraries in setup.py are listed in installation.rst file (#12023) discard fc3f8fa Turns failure of PR label when approved action into warning (#12017) discard 170eed1 Fix canceling of CodeQL workflow (#12024) discard 7fdef93 Fixes problem with non-iterable data returned by GH API (#12021) discard d50eb18 Revise "Project Focus" copy (#12011) discard 0bc73ae Adds more aggressive cancelling of duplicate Build Image jobs (#12018) discard 3553206 Add contributor-targeted description of the PR workflow. (#12016) discard 12e151b Adds documentation about the optimized PR workflow (#12006) discard 380a43a Fix K8S CI job name rendering (#12007) discard 3d35d89 Switches to "cancel-all-duplicates' mode of cancelling. (#12004) discard a4ff4dd Implements canceling of future duplicate runs (but the latest) (#11980) discard ee15306 Move Project focus and Principles higher in the README (#11973) discard aada43c Pin `kubernetes` to a max version of 11.0.0. (#11974) discard 8e1cc25 Updated documentation for the CI with mermaid sequence diagrams (#10380) discard b16fc03 Migrate from helm-unittest to python unittest (#11827) discard 91bd628 Remove archived link from README.md (#11945) discard 2a6be33 ci: Fix CodeQL Workflow for Javascript (#11941) discard a5f049f The PRs which are not approved run subset of tests (#11828) discard 44795b6 Simplify "scheduled" conditons to follow today's change in GA (#11876) discard 228c511 Fix Helm Chart Testing guide (#11909) discard 9c8d1c5 Constraints job depends on CI images (#11904) discard 1b80e6b Add Python Helm testing framework (#11693) discard 2058c86 Fixes a problem with checked-out version of the selective check (#11891) discard c7cf4ff Fix broken link in README.md (#11885) discard 66ea0b1 Fixes the doc pattern in selective checks (#11834) discard e0b63fc Constraints generation runs regardless from test status (#11838) discard a3f7725 Retrieve PR labels from the associated PR (#11820) discard 4514e24 Occasional docker-compose errors will be easier to diagnose (#11835) discard f2a02b6 Local Executor is used by default for MySQL/Postgres breeze (#11792) discard 3651a9a Removes duplicates from DISABLED_INTEGRATIONS variable (#11831) discard 58aef2c Fix typo in scripts/in_container/entrypoint_ci.sh (#11824) discard 4e8ad3c Switch postgres from 10 to 13 (#11785) discard cd8a303 Use LocalExecutor by default with tmux + Breeze (#11791) discard 6f12ebe Update download url for Airflow Version (#11800) discard b33134b Add Project URLs for PyPI page (#11801) discard 7806742 The .pypirc file is read from docker-context-files (#11779) discard 49c1277 Add
[airflow] branch v1-10-test updated (01eb5d4 -> 430c047)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard 01eb5d4 Simplifies check whether the CI image should be rebuilt (#12181) discard 42b3491 For v1-10-test PRs and pushes, use target branch scripts for images (#12339) discard 23151fd Deploy was not working from Breeze (#12319) discard f1f58f7 Python base image is shared between CI and PROD image (#12280) discard 533ea3c Added k9s as integrated tool to help with kubernetes testing (#12163) new 2807875 Added k9s as integrated tool to help with kubernetes testing (#12163) new b284205 Python base image is shared between CI and PROD image (#12280) new 0a51dfd Deploy was not working from Breeze (#12319) new a90131c For v1-10-test PRs and pushes, use target branch scripts for images (#12339) new 430c047 Simplifies check whether the CI image should be rebuilt (#12181) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (01eb5d4) \ N -- N -- N refs/heads/v1-10-test (430c047) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: scripts/ci/libraries/_kind.sh | 2 -- 1 file changed, 2 deletions(-)
[airflow] branch v1-10-test updated (09fe8f1 -> 0b8bb3e)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. from 09fe8f1 Simplifies check whether the CI image should be rebuilt (#12181) add 0b8bb3e Synchronize INTHEWILD.md with master No new revisions were added by this update. Summary of changes: INTHEWILD.md | 20 +++- 1 file changed, 19 insertions(+), 1 deletion(-)
[airflow] 05/05: Simplifies check whether the CI image should be rebuilt (#12181)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 430c047a0605125e463fdd3b45e7a848e2d26c13
Author: Jarek Potiuk
AuthorDate: Fri Nov 13 22:21:39 2020 +0100
Simplifies check whether the CI image should be rebuilt (#12181)
Rather than counting changed layers in the image (which was
enigmatic, difficult and prone to some magic number) we rely now
on random file generated while building the image.
We are using the docker image caching mechanism here. The random
file will be regenerated only when the previous layer (which is
about installling Airflow dependencies for the first time) gets
rebuild. And for us this is the indication, that the building
the image will take quite some time. This layer should be
relatively static - even if setup.py changes the CI image is
designed in the way that the first time installation of Airflow
dependencies is not invalidated.
This should lead to faster and less frequent rebuild for people
using Breeze and static checks.
(cherry picked from commit 167b9b9889ac5481b21cb35c6cdef5869b8ab713)
---
Dockerfile.ci | 9 +-
breeze | 6 +-
manifests/.gitignore| 2 +-
scripts/ci/libraries/_build_images.sh | 163
scripts/ci/libraries/_initialization.sh | 19 +++-
5 files changed, 106 insertions(+), 93 deletions(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 7699706..f06087b 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -132,7 +132,9 @@ ARG RUNTIME_APT_DEPS="\
sqlite3 \
tmux \
unzip \
- vim"
+ vim \
+ xxd"
+ENV RUNTIME_APT_DEP=${RUNTIME_APT_DEPS}
ARG ADDITIONAL_RUNTIME_APT_DEPS=""
ENV ADDITIONAL_RUNTIME_APT_DEPS=${ADDITIONAL_RUNTIME_APT_DEPS}
@@ -275,6 +277,11 @@ RUN if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" ]];
then \
fi
+# Generate random hex dump file so that we can determine whether it's faster
to rebuild the image
+# using current cache (when our dump is the same as the remote onb) or better
to pull
+# the new image (when it is different)
+RUN head -c 30 /dev/urandom | xxd -ps >/build-cache-hash
+
# Link dumb-init for backwards compatibility (so that older images also work)
RUN ln -sf /usr/bin/dumb-init /usr/local/bin/dumb-init
diff --git a/breeze b/breeze
index 55412d0..175a4ab 100755
--- a/breeze
+++ b/breeze
@@ -2873,11 +2873,9 @@ function breeze::run_build_command() {
build_images::prepare_prod_build
build_images::build_prod_images
else
+
build_images::prepare_ci_build
-md5sum::calculate_md5sum_for_all_files
-build_images::build_ci_image
-md5sum::update_all_md5
-build_images::build_ci_image_manifest
+build_images::rebuild_ci_image_if_needed
fi
;;
cleanup_image | run_exec)
diff --git a/manifests/.gitignore b/manifests/.gitignore
index a6c57f5..72e8ffc 100644
--- a/manifests/.gitignore
+++ b/manifests/.gitignore
@@ -1 +1 @@
-*.json
+*
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index a2577cb..52b8a84 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -136,10 +136,18 @@ function build_images::confirm_image_rebuild() {
;;
esac
elif [[ -t 0 ]]; then
+echo
+echo
+echo "Make sure that you rebased to latest master before rebuilding!"
+echo
# Check if this script is run interactively with stdin open and
terminal attached
"${AIRFLOW_SOURCES}/confirm" "${ACTION} image
${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}"
RES=$?
elif [[ ${DETECTED_TERMINAL:=$(tty)} != "not a tty" ]]; then
+echo > "${DETECTED_TERMINAL}"
+echo > "${DETECTED_TERMINAL}"
+echo "Make sure that you rebased to latest master before rebuilding!"
> "${DETECTED_TERMINAL}"
+echo > "${DETECTED_TERMINAL}"
# Make sure to use output of tty rather than stdin/stdout when
available - this way confirm
# will works also in case of pre-commits (git does not pass
stdin/stdout to pre-commit hooks)
# shellcheck disable=SC2094
@@ -151,6 +159,10 @@ function build_images::confirm_image_rebuild() {
export DETECTED_TERMINAL=/dev/tty
# Make sure to use /dev/tty first rather than stdin/stdout when
available - this way confirm
# will works also in case of pre-commits (git does not pass
stdin/stdout to pre-commit hooks)
+echo > "${DETECTED_TERMINAL}"
+echo > &qu
[airflow] branch v1-10-test updated (01eb5d4 -> 430c047)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard 01eb5d4 Simplifies check whether the CI image should be rebuilt (#12181) discard 42b3491 For v1-10-test PRs and pushes, use target branch scripts for images (#12339) discard 23151fd Deploy was not working from Breeze (#12319) discard f1f58f7 Python base image is shared between CI and PROD image (#12280) discard 533ea3c Added k9s as integrated tool to help with kubernetes testing (#12163) new 2807875 Added k9s as integrated tool to help with kubernetes testing (#12163) new b284205 Python base image is shared between CI and PROD image (#12280) new 0a51dfd Deploy was not working from Breeze (#12319) new a90131c For v1-10-test PRs and pushes, use target branch scripts for images (#12339) new 430c047 Simplifies check whether the CI image should be rebuilt (#12181) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (01eb5d4) \ N -- N -- N refs/heads/v1-10-test (430c047) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: scripts/ci/libraries/_kind.sh | 2 -- 1 file changed, 2 deletions(-)
[airflow] 04/05: For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a90131c0ae098c2a063cd94fdc1f36ace818b606
Author: Jarek Potiuk
AuthorDate: Fri Nov 13 14:28:36 2020 +0100
For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
Previously, always master scripts were used to build images
for workflow_run, because workflow_run always runs from master
branch. However that causes some surprising effects becuase the
sripts from master had to support both master and 1.10.
This change utilises a new feature in the "get-workflow-origin"
action - to get the target branch of PR and uses ci scripts from that
target branch.
This is perfectly secure, because both v1-10-test, v1-10-stable
and future 2-0 branches can only be updated by committers,
either by direct push or by merge.
(cherry picked from commit 7c4fe19e41ae02a1df1c0a217501cae2e0e84819)
---
.github/workflows/build-images-workflow-run.yml| 12 +--
.github/workflows/ci.yml | 38 +++---
.../workflows/label_when_reviewed_workflow_run.yml | 2 +-
3 files changed, 22 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index a3277a7..af71710 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -53,13 +53,14 @@ jobs:
targetCommitSha: ${{ steps.source-run-info.outputs.targetCommitSha }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
+ targetBranch: ${{ steps.source-run-info.outputs.targetBranch }}
sourceEvent: ${{ steps.source-run-info.outputs.sourceEvent }}
cacheDirective: ${{ steps.cache-directive.outputs.docker-cache }}
buildImages: ${{ steps.build-images.outputs.buildImages }}
upgradeToLatestConstraints: ${{
steps.upgrade-constraints.outputs.upgradeToLatestConstraints }}
steps:
- name: "Get information about the original trigger of the run"
- uses:
potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+ uses:
potiuk/get-workflow-origin@588cc14f9f1cdf1b8be3db816855e96422204fec # v1_3
id: source-run-info
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -326,17 +327,22 @@ jobs:
[Image Build](https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }})
for details" }
if: steps.defaults.outputs.proceed == 'true'
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} ) to
'main-airflow' to use main scripts"
+ - name: >
+ Checkout "${{ needs.cancel-workflow-runs.outputs.targetBranch }}"
branch to 'main-airflow' folder
+ to use ci/scripts from there.
uses: actions/checkout@v2
with:
path: "main-airflow"
+ ref: "${{ needs.cancel-workflow-runs.outputs.targetBranch }}"
if: steps.defaults.outputs.proceed == 'true'
- name: "Setup python"
uses: actions/setup-python@v2
with:
python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
if: steps.defaults.outputs.proceed == 'true'
- - name: "Override 'scripts/ci' with the ${{ github.ref }} version so
that the PR cannot override it."
+ - name: >
+ Override "scripts/ci" with the "${{
needs.cancel-workflow-runs.outputs.targetBranch }}" branch
+ so that the PR does not override it
# We should not override those scripts which become part of the image
as they will not be
# changed in the image built - we should only override those that are
executed to build
# the image.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 81890a7..df5f53e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -95,27 +95,15 @@ jobs:
image-build: ${{ steps.selective-checks.outputs.image-build }}
docs-build: ${{ steps.selective-checks.outputs.docs-build }}
needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
- needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
steps:
- name: "Get information about the PR"
-uses:
potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+uses:
potiuk/ge
[airflow] 03/05: Deploy was not working from Breeze (#12319)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0a51dfd78c3bf58eaacf4c1136228bb50388a7ce
Author: Jarek Potiuk
AuthorDate: Thu Nov 12 19:53:35 2020 +0100
Deploy was not working from Breeze (#12319)
The get_cluster_name was called twice resulting in redonly
error after rebasing/fixing CI failure in #12163.
This PR is fxing it.
(cherry picked from commit af19b126e94876c371553f6a7cfae6b1102f79fd)
---
scripts/ci/libraries/_kind.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index defa4de..3a170cb 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -173,7 +173,6 @@ function kind::perform_kind_cluster_operation() {
echo "Deploying Airflow to KinD"
echo
kind::build_image_for_kubernetes_tests
-kind::get_kind_cluster_name
kind::load_image_to_kind_cluster
kind::deploy_airflow_with_helm
kind::deploy_test_kubernetes_resources
[airflow] 02/05: Python base image is shared between CI and PROD image (#12280)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b28420576048a6bedaa89e5bde9aec4277595cf1
Author: Jarek Potiuk
AuthorDate: Thu Nov 12 12:31:14 2020 +0100
Python base image is shared between CI and PROD image (#12280)
When you are building CI images locally you use the CI
base images from apache:airflow/python* now to maintain
consistency and avoid often rebuilds. But when you build
prod images, you would accidentaly override it with the
python base image available in python repo which might be
different (newer and not yet tested in CI). This PR
changes it to use the same base image which is now
tagged in Apache Airflow's dockerhub repository.
(cherry picked from commit 3c2c29187afffb1a1dd1327803097651cdc079ee)
---
scripts/ci/libraries/_push_pull_remove_images.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh
b/scripts/ci/libraries/_push_pull_remove_images.sh
index 4c51f0a..7c65db1 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -154,7 +154,8 @@ function
push_pull_remove_images::pull_prod_images_if_needed() {
fi
push_pull_remove_images::pull_image_github_dockerhub
"${PYTHON_BASE_IMAGE}"
"${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
else
-docker pull "${PYTHON_BASE_IMAGE}"
+docker pull "${AIRFLOW_CI_PYTHON_IMAGE}"
+docker tag "${AIRFLOW_CI_PYTHON_IMAGE}" "${PYTHON_BASE_IMAGE}"
fi
echo
fi
[airflow] 01/05: Added k9s as integrated tool to help with kubernetes testing (#12163)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 280787593252d6cf9e60c2cd078f9b6abb3d57cd
Author: Jarek Potiuk
AuthorDate: Wed Nov 11 17:15:02 2020 +0100
Added k9s as integrated tool to help with kubernetes testing (#12163)
The K9s is fantastic tool that helps to debug a running k8s
instance. It is terminal-based windowed CLI that makes you
several times more productive comparing to using kubectl
commands. We've integrated k9s (it is run as a docker container
and downloaded on demand). We've also separated out KUBECONFIG
of the integrated kind cluster so that it does not mess with
kubernetes configuration you might already have.
Also - together with that the "surrounding" of the kubernetes
tests were simplified and improved so that the k9s integration
can be utilized well. Instead of kubectl port forwarding (which
caused multitude of problems) we are now utilizing kind's
portMapping feature + custom NodePort resource that maps
port 8080 to 30007 NodePort which in turn maps it to 8080
port of the Webserver. This way we do not have to establish
an external kubectl port forward which is prone to error and
management - everything is brought up when Airflow gets
deployed to the Kind Cluster and shuts down when the Kind
cluster is stopped.
Yet another problem fixed was killing of postgres by one of the
kubernetes tests ('test_integration_run_dag_with_scheduler_failure').
Instead of just killing the scheduler it killed all pods - including
the Postgres one (it was named 'airflow-postgres.*'). That caused
various problems, as the database could be left in a strange state.
I changed the tests to do what it claimed was doing - so killing only the
scheduler during the test. This seemed to improve the stability
of tests immensely in my local setup.
(cherry picked from commit 21999dd56e9dbe9f7f9e25961954c5677c3c7c58)
---
.github/workflows/ci.yml | 17 +-
BREEZE.rst | 11 +-
TESTING.rst| 151 -
breeze | 9 +
breeze-complete| 5 +-
chart/requirements.lock| 4 +-
images/testing/k9s.png | Bin 0 -> 238713 bytes
images/testing/kubeconfig-env.png | Bin 0 -> 231280 bytes
images/testing/kubernetes-virtualenv.png | Bin 0 -> 110011 bytes
images/testing/pytest-runner.png | Bin 0 -> 131589 bytes
images/testing/run-test.png| Bin 0 -> 140728 bytes
kubernetes_tests/test_kubernetes_executor.py | 7 +-
kubernetes_tests/test_kubernetes_pod_operator.py | 673 -
scripts/ci/kubernetes/ci_run_kubernetes_tests.sh | 2 -
...up_cluster_and_deploy_airflow_to_kubernetes.sh} | 3 +-
scripts/ci/kubernetes/kind-cluster-conf.yaml | 5 +
.../{kind-cluster-conf.yaml => nodeport.yaml} | 30 +-
...oy_app_to_kubernetes.sh => redeploy_airflow.sh} | 6 +-
scripts/ci/libraries/_kind.sh | 126 ++--
19 files changed, 512 insertions(+), 537 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 6c854a1..81890a7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -636,23 +636,14 @@ jobs:
python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
- name: "Free space"
run: ./scripts/ci/tools/ci_free_space_on_ci.sh
- - name: "Setup Kind Cluster ${{ env.KIND_VERSION }}"
-uses: engineerd/setup-kind@v0.4.0
-with:
- version: "${{ env.KIND_VERSION }}"
- name:
airflow-python-${{matrix.python-version}}-${{matrix.kubernetes-version}}
- config: "scripts/ci/kubernetes/kind-cluster-conf.yaml"
- name: "Prepare PROD Image"
run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
- - name: "Deploy airflow to cluster"
-id: deploy-app
-run: ./scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh
+ - name: "Setup cluster and deploy Airflow"
+id: setp-cluster-deploy-app
+run:
./scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
env:
# We have the right image pulled already by the previous step
SKIP_BUILDING_PROD_IMAGE: "true"
- # due to some instabilities, in CI we try to increase port numbers
when trying to establish
- # port forwarding
- INCREASE_PORT_NUMBER_FOR_KUBERNETES: "true"
- name: "Cache
[airflow] branch v1-10-test updated (430c047 -> 09fe8f1)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard 430c047 Simplifies check whether the CI image should be rebuilt (#12181) discard a90131c For v1-10-test PRs and pushes, use target branch scripts for images (#12339) discard 0a51dfd Deploy was not working from Breeze (#12319) discard b284205 Python base image is shared between CI and PROD image (#12280) discard 2807875 Added k9s as integrated tool to help with kubernetes testing (#12163) discard 301e5e4 Docker context files should be available earlier (#12219) discard d6e48ef Fixes continuous image rebuilding with Breeze (#12256) discard 1e6f091 Add Markdown linting to pre-commit (#11465) discard cb4eb46 Beautify Output of setup-installation pre-commit (#12218) discard 04b9037 Fix permissions of mounted /tmp directory for Breeze (#12157) discard 2076ea4 Remove popd which is a remnant from past (#12211) discard 262bc04 Fixes timeout in helm chart tests (#12209) discard 0f06d6f Fixed path of the test_core.py file in docs (#12191) discard e32ff8b Adds extra check while the selective checks are run (#12178) discard 40a1471 Uses always the same Python base image as used for CI image (#12177) discard d2628f6 Fixes "--force-clean-images" flag in Breeze (#12156) discard bb5175f Fixes undefined variables (#12155) discard 6848f9b Update to new helm stable repo (#12137) discard 5e805d6 Work properly if some variables are not defined (#12135) discard 5baa6b6 Add Kubernetes files to selective checks (#12114) discard bd318b5 Update install_mysql.sh (#12101) discard 70e00bf Fix proper SHA in check preventing accidentally merging PR (#12083) discard 2abe5c0 Use sys.exit() instead of exit() (#12084) discard 40256da Fixes problem with building a PROD image (#12080) discard 17ce4f1 If we build a new image, we should run more than basic checks (#12070) discard 1dcf44f Adds a forgotten word in a README.md (#12066) discard 47a9bc2 Uses DOCKER_TAG when building image in DockerHub (#12050) discard fa7cbec Fixes documentation-only selective checks (#12038) discard c523148 Checks if all the libraries in setup.py are listed in installation.rst file (#12023) discard fc3f8fa Turns failure of PR label when approved action into warning (#12017) discard 170eed1 Fix canceling of CodeQL workflow (#12024) discard 7fdef93 Fixes problem with non-iterable data returned by GH API (#12021) discard d50eb18 Revise "Project Focus" copy (#12011) discard 0bc73ae Adds more aggressive cancelling of duplicate Build Image jobs (#12018) discard 3553206 Add contributor-targeted description of the PR workflow. (#12016) discard 12e151b Adds documentation about the optimized PR workflow (#12006) discard 380a43a Fix K8S CI job name rendering (#12007) discard 3d35d89 Switches to "cancel-all-duplicates' mode of cancelling. (#12004) discard a4ff4dd Implements canceling of future duplicate runs (but the latest) (#11980) discard ee15306 Move Project focus and Principles higher in the README (#11973) discard aada43c Pin `kubernetes` to a max version of 11.0.0. (#11974) discard 8e1cc25 Updated documentation for the CI with mermaid sequence diagrams (#10380) discard b16fc03 Migrate from helm-unittest to python unittest (#11827) discard 91bd628 Remove archived link from README.md (#11945) discard 2a6be33 ci: Fix CodeQL Workflow for Javascript (#11941) discard a5f049f The PRs which are not approved run subset of tests (#11828) discard 44795b6 Simplify "scheduled" conditons to follow today's change in GA (#11876) discard 228c511 Fix Helm Chart Testing guide (#11909) discard 9c8d1c5 Constraints job depends on CI images (#11904) discard 1b80e6b Add Python Helm testing framework (#11693) discard 2058c86 Fixes a problem with checked-out version of the selective check (#11891) discard c7cf4ff Fix broken link in README.md (#11885) discard 66ea0b1 Fixes the doc pattern in selective checks (#11834) discard e0b63fc Constraints generation runs regardless from test status (#11838) discard a3f7725 Retrieve PR labels from the associated PR (#11820) discard 4514e24 Occasional docker-compose errors will be easier to diagnose (#11835) discard f2a02b6 Local Executor is used by default for MySQL/Postgres breeze (#11792) discard 3651a9a Removes duplicates from DISABLED_INTEGRATIONS variable (#11831) discard 58aef2c Fix typo in scripts/in_container/entrypoint_ci.sh (#11824) discard 4e8ad3c Switch postgres from 10 to 13 (#11785) discard cd8a303 Use LocalExecutor by default with tmux + Breeze (#11791) discard 6f12ebe Update download url for Airflow Version (#11800) discard b33134b Add Project URLs for PyPI page (#11801) discard 7806742 The .pypirc file is read from docker-context-files (#11779) discard 49c1277 Add
[airflow] branch v1-10-test updated (09fe8f1 -> 0b8bb3e)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. from 09fe8f1 Simplifies check whether the CI image should be rebuilt (#12181) add 0b8bb3e Synchronize INTHEWILD.md with master No new revisions were added by this update. Summary of changes: INTHEWILD.md | 20 +++- 1 file changed, 19 insertions(+), 1 deletion(-)
[airflow] branch v1-10-test updated (430c047 -> 09fe8f1)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard 430c047 Simplifies check whether the CI image should be rebuilt (#12181) discard a90131c For v1-10-test PRs and pushes, use target branch scripts for images (#12339) discard 0a51dfd Deploy was not working from Breeze (#12319) discard b284205 Python base image is shared between CI and PROD image (#12280) discard 2807875 Added k9s as integrated tool to help with kubernetes testing (#12163) discard 301e5e4 Docker context files should be available earlier (#12219) discard d6e48ef Fixes continuous image rebuilding with Breeze (#12256) discard 1e6f091 Add Markdown linting to pre-commit (#11465) discard cb4eb46 Beautify Output of setup-installation pre-commit (#12218) discard 04b9037 Fix permissions of mounted /tmp directory for Breeze (#12157) discard 2076ea4 Remove popd which is a remnant from past (#12211) discard 262bc04 Fixes timeout in helm chart tests (#12209) discard 0f06d6f Fixed path of the test_core.py file in docs (#12191) discard e32ff8b Adds extra check while the selective checks are run (#12178) discard 40a1471 Uses always the same Python base image as used for CI image (#12177) discard d2628f6 Fixes "--force-clean-images" flag in Breeze (#12156) discard bb5175f Fixes undefined variables (#12155) discard 6848f9b Update to new helm stable repo (#12137) discard 5e805d6 Work properly if some variables are not defined (#12135) discard 5baa6b6 Add Kubernetes files to selective checks (#12114) discard bd318b5 Update install_mysql.sh (#12101) discard 70e00bf Fix proper SHA in check preventing accidentally merging PR (#12083) discard 2abe5c0 Use sys.exit() instead of exit() (#12084) discard 40256da Fixes problem with building a PROD image (#12080) discard 17ce4f1 If we build a new image, we should run more than basic checks (#12070) discard 1dcf44f Adds a forgotten word in a README.md (#12066) discard 47a9bc2 Uses DOCKER_TAG when building image in DockerHub (#12050) discard fa7cbec Fixes documentation-only selective checks (#12038) discard c523148 Checks if all the libraries in setup.py are listed in installation.rst file (#12023) discard fc3f8fa Turns failure of PR label when approved action into warning (#12017) discard 170eed1 Fix canceling of CodeQL workflow (#12024) discard 7fdef93 Fixes problem with non-iterable data returned by GH API (#12021) discard d50eb18 Revise "Project Focus" copy (#12011) discard 0bc73ae Adds more aggressive cancelling of duplicate Build Image jobs (#12018) discard 3553206 Add contributor-targeted description of the PR workflow. (#12016) discard 12e151b Adds documentation about the optimized PR workflow (#12006) discard 380a43a Fix K8S CI job name rendering (#12007) discard 3d35d89 Switches to "cancel-all-duplicates' mode of cancelling. (#12004) discard a4ff4dd Implements canceling of future duplicate runs (but the latest) (#11980) discard ee15306 Move Project focus and Principles higher in the README (#11973) discard aada43c Pin `kubernetes` to a max version of 11.0.0. (#11974) discard 8e1cc25 Updated documentation for the CI with mermaid sequence diagrams (#10380) discard b16fc03 Migrate from helm-unittest to python unittest (#11827) discard 91bd628 Remove archived link from README.md (#11945) discard 2a6be33 ci: Fix CodeQL Workflow for Javascript (#11941) discard a5f049f The PRs which are not approved run subset of tests (#11828) discard 44795b6 Simplify "scheduled" conditons to follow today's change in GA (#11876) discard 228c511 Fix Helm Chart Testing guide (#11909) discard 9c8d1c5 Constraints job depends on CI images (#11904) discard 1b80e6b Add Python Helm testing framework (#11693) discard 2058c86 Fixes a problem with checked-out version of the selective check (#11891) discard c7cf4ff Fix broken link in README.md (#11885) discard 66ea0b1 Fixes the doc pattern in selective checks (#11834) discard e0b63fc Constraints generation runs regardless from test status (#11838) discard a3f7725 Retrieve PR labels from the associated PR (#11820) discard 4514e24 Occasional docker-compose errors will be easier to diagnose (#11835) discard f2a02b6 Local Executor is used by default for MySQL/Postgres breeze (#11792) discard 3651a9a Removes duplicates from DISABLED_INTEGRATIONS variable (#11831) discard 58aef2c Fix typo in scripts/in_container/entrypoint_ci.sh (#11824) discard 4e8ad3c Switch postgres from 10 to 13 (#11785) discard cd8a303 Use LocalExecutor by default with tmux + Breeze (#11791) discard 6f12ebe Update download url for Airflow Version (#11800) discard b33134b Add Project URLs for PyPI page (#11801) discard 7806742 The .pypirc file is read from docker-context-files (#11779) discard 49c1277 Add
[airflow] branch v1-10-test updated (09fe8f1 -> 0b8bb3e)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. from 09fe8f1 Simplifies check whether the CI image should be rebuilt (#12181) add 0b8bb3e Synchronize INTHEWILD.md with master No new revisions were added by this update. Summary of changes: INTHEWILD.md | 20 +++- 1 file changed, 19 insertions(+), 1 deletion(-)
[airflow] 15/44: Checks if all the libraries in setup.py are listed in installation.rst file (#12023)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 43c72f2f4c7d0b72d9fcbbfb17135d9be59eb4c9 Author: SZN AuthorDate: Mon Nov 2 14:17:41 2020 +0100 Checks if all the libraries in setup.py are listed in installation.rst file (#12023) (cherry picked from commit 2354bd2be381bcfe6db132990af1ac34df52b9b4) --- .pre-commit-config.yaml| 6 + BREEZE.rst | 9 +- CONTRIBUTING.rst | 16 +- INSTALL| 16 +- STATIC_CODE_CHECKS.rst | 2 + breeze-complete| 1 + docs/installation.rst | 216 +++-- .../pre_commit_check_setup_installation.py | 108 +++ setup.py | 16 ++ 9 files changed, 314 insertions(+), 76 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 6763164..cb5c753 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -213,6 +213,12 @@ repos: files: ^setup.py$ pass_filenames: false entry: ./scripts/ci/pre_commit/pre_commit_check_order_setup.py + - id: setup-installation +name: Checks if all the libraries in setup.py are listed in installation.rst file +language: python +files: ^setup.py$|^docs/installation.rst$ +pass_filenames: false +entry: ./scripts/ci/pre_commit/pre_commit_check_setup_installation.py - id: update-breeze-file name: Update output of breeze command in BREEZE.rst entry: "./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.sh" diff --git a/BREEZE.rst b/BREEZE.rst index cf119ba..5e481f7 100644 --- a/BREEZE.rst +++ b/BREEZE.rst @@ -395,7 +395,7 @@ run ``airflow webserver``, ``airflow scheduler``, ``airflow worker`` in separate This can be achieved either via ``tmux`` or via exec-ing into the running container from the host. Tmux is installed inside the container and you can launch it with ``tmux`` command. Tmux provides you with the capability of creating multiple virtual terminals and multiplex between them. More about ``tmux`` can be -found at `tmux github wiki page <https://github.com/tmux/tmux/wiki>`_ . Tmux has several useful shortcuts +found at `tmux GitHub wiki page <https://github.com/tmux/tmux/wiki>`_ . Tmux has several useful shortcuts that allow you to split the terminals, open new tabs etc - it's pretty useful to learn it. .. raw:: html @@ -1001,7 +1001,7 @@ Managing Dependencies - If you need to change apt dependencies in the ``Dockerfile.ci``, add Python packages in ``setup.py`` or -add javascript dependencies in ``package.json``, you can either add dependencies temporarily for a single +add JavaScript dependencies in ``package.json``, you can either add dependencies temporarily for a single Breeze session or permanently in ``setup.py``, ``Dockerfile.ci``, or ``package.json`` files. Installing Dependencies for a Single Breeze Session @@ -1865,8 +1865,9 @@ This is the current syntax for `./breeze <./breeze>`_: flake8 forbid-tabs helm-lint incorrect-use-of-LoggingMixin insert-license language-matters lint-dockerfile lint-openapi mixed-line-ending mypy mypy-helm no-relative-imports pre-commit-descriptions pydevd python2-compile python2-fastcheck - python-no-log-warn rst-backticks setup-order shellcheck sort-in-the-wild - trailing-whitespace update-breeze-file update-extras update-local-yml-file yamllint + python-no-log-warn rst-backticks setup-order setup-installation shellcheck + sort-in-the-wild trailing-whitespace update-breeze-file update-extras + update-local-yml-file yamllint You can pass extra arguments including options to to the pre-commit framework as passed after --. For example: diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 6d3aa91..6d34026 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -540,13 +540,15 @@ This is the full list of those extras: .. START EXTRAS HERE -all, all_dbs, async, atlas, aws, azure, azure_blob_storage, azure_container_instances, azure_cosmos, -azure_data_lake, azure_secrets, cassandra, celery, cgroups, cloudant, crypto, dask, databricks, -datadog, devel, devel_azure, devel_ci, devel_hadoop, doc, docker, druid, elasticsearch, emr, gcp, -gcp_api, github_enterprise, google_auth, grpc, hashicorp, hdfs, hive, jdbc, jira, kerberos, -kubernetes, ldap, mongo, mssql, mysql, oracle, papermill, password, pinot, postgres, presto, qds, -rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentr
[airflow] 08/44: Adds documentation about the optimized PR workflow (#12006)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 80845847d5971e64a0b19a589a9998cf330abe07 Author: Jarek Potiuk AuthorDate: Sun Nov 1 00:20:38 2020 +0100 Adds documentation about the optimized PR workflow (#12006) We had a lot of problems recently about the queues in Github Actions. This documentations explains the motivation and approach we have taken for optimizing our PR workflow. (cherry picked from commit d85a31f2d88b92b480438d86aa8e3e79e6c3614d) --- CI.rst | 102 ++-- PULL_REQUEST_WORKFLOW.rst| 260 +++ images/pr/pr-full-tests-needed.png | Bin 0 -> 88512 bytes images/pr/pr-likely-ok-to-merge.png | Bin 0 -> 98362 bytes images/pr/pr-no-tests-needed-comment.png | Bin 0 -> 80852 bytes images/pr/selective_checks.md5 | 1 + images/pr/selective_checks.mermaid | 35 + images/pr/selective_checks.png | Bin 0 -> 64501 bytes 8 files changed, 309 insertions(+), 89 deletions(-) diff --git a/CI.rst b/CI.rst index dc1cdf2..f4b5294 100644 --- a/CI.rst +++ b/CI.rst @@ -35,7 +35,7 @@ the CI jobs. And we have a number of variables determine build behaviour. -Github Actions runs +GitHub Actions runs --- Our builds on CI are highly optimized. They utilise some of the latest features provided by GitHub Actions @@ -65,7 +65,7 @@ utilise the WRITE access to Apache Airflow repository via an external Pull Reque Thanks to the WRITE access and fact that the 'workflow_run' by default uses the 'master' version of the sources, we can safely run some logic there will checkout the incoming Pull Request, build the container -image from the sources from the incoming PR and push such image to an Github Docker Registry - so that +image from the sources from the incoming PR and push such image to an GitHub Docker Registry - so that this image can be built only once and used by all the jobs running tests. The image is tagged with unique ``RUN_ID`` of the incoming Pull Request and the tests run in the Pull Request can simply pull such image rather than build it from the scratch. Pulling such image takes ~ 1 minute, thanks to that we are saving @@ -92,7 +92,7 @@ connected with the run. You can read more about it in `BREEZE.rst `_ and `TESTING.rst `_ -Difference between local runs and Github Action workflows +Difference between local runs and GitHub Action workflows - Depending whether the scripts are run locally (most often via `Breeze `_) or whether they @@ -470,7 +470,13 @@ The main purpose of those jobs is to check if PR builds cleanly, if the test run the PR is ready to review and merge. The runs are using cached images from the Private GitHub registry - CI, Production Images as well as base Python images that are also cached in the Private GitHub registry. Also for those builds we only execute Python tests if important files changed (so for example if it is -doc-only change, no tests will be executed. +"no-code" change, no tests will be executed. + +The workflow involved in Pull Requests review and approval is a bit more complex than simple workflows +in most of other projects because we've implemented some optimizations related to efficient use +of queue slots we share with other Apache Software Foundation projects. More details about it +can be found in `PULL_REQUEST_WORKFLOW.rst `_. + Direct Push/Merge Run - @@ -641,7 +647,7 @@ Comments: (1) CRON jobs builds images from scratch - to test if everything works properly for clean builds (2) The tests are run when the Trigger Tests job determine that important files change (this allows - for example doc-only changes to build much faster) + for example "no-code" changes to build much faster) (3) The jobs wait for CI images if ``GITHUB_REGISTRY_WAIT_FOR_IMAGE`` variable is set to "true". You can set it to "false" to disable using shared images - this is slower though as the images are rebuilt in every job that needs them. You can also set your own fork's secret @@ -685,7 +691,7 @@ way to sync your fork master to the Apache Airflow's one. Delete old artifacts -This workflow is introduced, to delete old artifacts from the Github Actions build. We set it to +This workflow is introduced, to delete old artifacts from the GitHub Actions build. We set it to delete old artifacts that are > 7 days old. It only runs for the 'apache/airflow' repository. We also have a script that can help to clean-up the old artifacts: @@ -695,89 +701,7 @@ CodeQL scan --- The CodeQL security scan uses Git
[airflow] 07/44: Fix K8S CI job name rendering (#12007)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 33a14e7e666c6c9b92027a18c7c5941e52ad506f
Author: Xiaodong DENG
AuthorDate: Sun Nov 1 00:09:19 2020 +0100
Fix K8S CI job name rendering (#12007)
(cherry picked from commit 72f3fcb293bbaee405133a77a8c1d6cab87b9e62)
---
.github/workflows/ci.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 596ab3e..630b5d6 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -613,7 +613,7 @@ jobs:
tests-kubernetes:
timeout-minutes: 50
-name: "K8s: ${{matrix.python-version}} ${{matrix.kubernetes-version}}
${{matrix.kubernetes-mode}}"
+name: K8s ${{matrix.python-version}} ${{matrix.kubernetes-version}}
${{matrix.kubernetes-mode}}
runs-on: ubuntu-latest
needs: [build-info, prod-images]
strategy:
[airflow] 05/44: Implements canceling of future duplicate runs (but the latest) (#11980)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ad5679025d84378a54ac22a9203c3cc75f7aedd8
Author: Jarek Potiuk
AuthorDate: Fri Oct 30 21:48:40 2020 +0100
Implements canceling of future duplicate runs (but the latest) (#11980)
* Implements canceling of future duplicate runs (but the latest)
Previous version of the cancel-workflow-runs action implemented
canceling of only past duplicates, but when there are queues
involved, some future "cancel-workflow-runs" might be in a queue
for long time. This change has the effect that cancel-workflow-runs
for duplicates will also allow future runs of the same branch/repo
sparing only the most recent run - no matter if the duplicates
were older than my own run.
This should handle the case where we have queues blocking the
"cancel-workflow-runs" from running.
* Update .github/workflows/build-images-workflow-run.yml
Co-authored-by: Kaxil Naik
Co-authored-by: Kaxil Naik
(cherry picked from commit 36b5cc6df0c7b49bcd9d38c37c2ed4b881089d6d)
---
.github/workflows/build-images-workflow-run.yml | 19 ---
1 file changed, 8 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index a4f2dbe..81c4fb4 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -21,7 +21,6 @@ on: # yamllint disable-line rule:truthy
workflow_run:
workflows: ["CI Build"]
types: ['requested']
-
env:
MOUNT_LOCAL_SOURCES: "false"
MOUNT_FILES: "true"
@@ -66,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
-uses:
potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: duplicates
@@ -84,7 +83,7 @@ jobs:
# in GitHub Actions, we have to use Job names to match
Event/Repo/Branch from the
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -93,9 +92,7 @@ jobs:
[".*Event: ${{ steps.source-run-info.outputs.sourceEvent }}
Repo: ${{ steps.source-run-info.outputs.sourceHeadRepo }}
Branch: ${{ steps.source-run-info.outputs.sourceHeadBranch }}.*"]
-if: >
- env.BUILD_IMAGES == 'true' &&
steps.source-run-info.outputs.sourceEvent != 'schedule'
- && steps.source-run-info.outputs.sourceEvent != 'push'
+if: env.BUILD_IMAGES == 'true'
- name: "Cancel all 'CI Build' runs where some jobs failed"
# We find any of the "CI Build" workflow runs, where any of the
important jobs
@@ -104,7 +101,7 @@ jobs:
# can cancel all the matching "Build Images" workflow runs in the two
following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -138,14 +135,14 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run
Id:
# followed by one of the run ids. Yes I know it's super complex
¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' &&
steps.source-run-info-failed.outputs.cancelledRuns != '[]'
-uses:
potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{
steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
-uses:
potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
id: cancel
[airflow] 04/44: Move Project focus and Principles higher in the README (#11973)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 6f6b5600c371783d91a1f1234c627f895a3bbef9 Author: Ry Walker <4283+...@users.noreply.github.com> AuthorDate: Fri Oct 30 15:10:30 2020 -0400 Move Project focus and Principles higher in the README (#11973) (cherry picked from commit 3c723e35a58b274962dc47e21cbb05389263d97a) --- README.md | 34 +- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 5e638a5..ccac802 100644 --- a/README.md +++ b/README.md @@ -40,13 +40,13 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The **Table of contents** +- [Project Focus](#project-focus) +- [Principles](#principles) - [Requirements](#requirements) - [Getting started](#getting-started) - [Installing from PyPI](#installing-from-pypi) - [Official source code](#official-source-code) - [Convenience packages](#convenience-packages) -- [Project Focus](#project-focus) -- [Principles](#principles) - [User Interface](#user-interface) - [Contributing](#contributing) - [Who uses Apache Airflow?](#who-uses-apache-airflow) @@ -57,6 +57,21 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The +## Project Focus + +Airflow works best with workflows that are mostly static and slowly changing. When the structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). + +Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent, and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work. + +Airflow **is not** a streaming solution. Airflow is not in the [Spark Streaming](http://spark.apache.org/streaming/) or [Storm](https://storm.apache.org/) space. + +## Principles + +- **Dynamic**: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically. +- **Extensible**: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment. +- **Elegant**: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine. +- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. + ## Requirements Apache Airflow is tested with: @@ -150,21 +165,6 @@ All those artifacts are not official releases, but they are prepared using offic Some of those artifacts are "development" or "pre-release" ones, and they are clearly marked as such following the ASF Policy. -## Project Focus - -Airflow works best with workflows that are mostly static and slowly changing. When the structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). - -Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent, and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work. - -Airflow **is not** a streaming solution. Airflow is not in the [Spark Streaming](http://spark.apache.org/streaming/) or [Storm](https://storm.apache.org/) space. - -## Principles - -- **Dynamic**: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically. -- **Extensible**: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment. -- **Elegant**: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine. -- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. - ## User Interface - **
[airflow] branch v1-10-test updated (0b8bb3e -> 00f320f)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. omit 0b8bb3e Synchronize INTHEWILD.md with master omit 09fe8f1 Simplifies check whether the CI image should be rebuilt (#12181) omit a950aac For v1-10-test PRs and pushes, use target branch scripts for images (#12339) omit 2f59506 Deploy was not working from Breeze (#12319) omit ca91871 Python base image is shared between CI and PROD image (#12280) omit 330e4d3 Added k9s as integrated tool to help with kubernetes testing (#12163) omit f1b8c7e Docker context files should be available earlier (#12219) omit b40568e Fixes continuous image rebuilding with Breeze (#12256) omit 07298ba Add Markdown linting to pre-commit (#11465) omit c89257c Beautify Output of setup-installation pre-commit (#12218) omit 7d40471 Fix permissions of mounted /tmp directory for Breeze (#12157) omit 7eac201 Remove popd which is a remnant from past (#12211) omit 4f8473d Fixes timeout in helm chart tests (#12209) omit 330f0b9 Fixed path of the test_core.py file in docs (#12191) omit 6370a91 Adds extra check while the selective checks are run (#12178) omit c7d64d5 Uses always the same Python base image as used for CI image (#12177) omit ae033b9 Fixes "--force-clean-images" flag in Breeze (#12156) omit 67f2c97 Fixes undefined variables (#12155) omit 4166311 Update to new helm stable repo (#12137) omit 61346ca Work properly if some variables are not defined (#12135) omit 5914d12 Add Kubernetes files to selective checks (#12114) omit 4c8f860 Update install_mysql.sh (#12101) omit 96f0a77 Fix proper SHA in check preventing accidentally merging PR (#12083) omit 6b1078b Use sys.exit() instead of exit() (#12084) omit b1b608e Fixes problem with building a PROD image (#12080) omit cdc26fc If we build a new image, we should run more than basic checks (#12070) omit ea58242 Adds a forgotten word in a README.md (#12066) omit 4e0b9fe Uses DOCKER_TAG when building image in DockerHub (#12050) omit 5ab6cd5 Fixes documentation-only selective checks (#12038) omit 56727d0 Checks if all the libraries in setup.py are listed in installation.rst file (#12023) omit 46d778a Turns failure of PR label when approved action into warning (#12017) omit 56d5960 Fix canceling of CodeQL workflow (#12024) omit d10eece Fixes problem with non-iterable data returned by GH API (#12021) omit 86c7b7b Revise "Project Focus" copy (#12011) omit 9051a87 Adds more aggressive cancelling of duplicate Build Image jobs (#12018) omit 08c0275 Add contributor-targeted description of the PR workflow. (#12016) omit a8d602b Adds documentation about the optimized PR workflow (#12006) omit 310cbd6 Fix K8S CI job name rendering (#12007) omit c45c045 Switches to "cancel-all-duplicates' mode of cancelling. (#12004) omit 6454b94 Implements canceling of future duplicate runs (but the latest) (#11980) omit 4acdc8e Move Project focus and Principles higher in the README (#11973) omit b310082 Pin `kubernetes` to a max version of 11.0.0. (#11974) omit eb11556 Updated documentation for the CI with mermaid sequence diagrams (#10380) omit a13366d Migrate from helm-unittest to python unittest (#11827) new 572387d Migrate from helm-unittest to python unittest (#11827) new 132e899 Updated documentation for the CI with mermaid sequence diagrams (#10380) new d5e7b51 Pin `kubernetes` to a max version of 11.0.0. (#11974) new 6f6b560 Move Project focus and Principles higher in the README (#11973) new ad56790 Implements canceling of future duplicate runs (but the latest) (#11980) new d3ab3df Switches to "cancel-all-duplicates' mode of cancelling. (#12004) new 33a14e7 Fix K8S CI job name rendering (#12007) new 8084584 Adds documentation about the optimized PR workflow (#12006) new b2633ff Add contributor-targeted description of the PR workflow. (#12016) new 99aa883 Adds more aggressive cancelling of duplicate Build Image jobs (#12018) new 887c231 Revise "Project Focus" copy (#12011) new e18d675 Fixes problem with non-iterable data returned by GH API (#12021) new df68fb7 Fix canceling of CodeQL workflow (#12024) new 8e58291 Turns failure of PR label when approved action into warning (#12017) new 43c72f2 Checks if all the libraries in setup.py are listed in installation.rst file (#12023) new ad8aa65 Fixes documentation-only selective checks (#12038) new 7512344 Uses DOCKER_TAG when building image in DockerHub (#12050) new 26d001f Adds a forgotten word in a README.md (#12066) new 6f64e98 If we build a new image, we should run more than basic che
[airflow] 01/44: Migrate from helm-unittest to python unittest (#11827)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 572387dbeb9d83a2cc4fbde5f32a03b1f08f
Author: Kamil Breguła
AuthorDate: Fri Oct 30 15:48:22 2020 +0100
Migrate from helm-unittest to python unittest (#11827)
* Migrate from helm-unittest to python unittest
* fixup! Migrate from helm-unittest to python unittest
* fixup! fixup! Migrate from helm-unittest to python unittest
(cherry picked from commit 9322f3e46c3a06d2e5b891e399cc054e9b76ae72)
---
.github/workflows/ci.yml | 15 +-
.pre-commit-config.yaml| 17 +-
airflow/www_rbac/webpack.config.js | 2 +-
chart/tests/conftest.py| 6 +-
chart/tests/dags-persistent-volume-claim_test.yaml | 64 -
chart/tests/git-sync-webserver_test.yaml | 66 -
chart/tests/git-sync-worker_test.yaml | 70 -
chart/tests/pod-template-file_test.yaml| 149 --
chart/tests/test_basic_helm_chart.py | 2 +-
chart/tests/test_celery_kubernetes_executor.py | 47
.../test_celery_kubernetes_pod_launcher_role.py| 23 +-
chart/tests/test_dags_persistent_volume_claim.py | 72 +
chart/tests/test_git_sync_scheduler.py | 311 -
chart/tests/test_git_sync_webserver.py | 62
chart/tests/test_git_sync_worker.py| 75 +
.../tests/test_migrate_database_job.py | 23 +-
chart/tests/test_pod_template_file.py | 185
chart/tests/test_scheduler.py | 40 +++
chart/tests/test_worker.py | 40 +++
docs/static/exampleinclude.css | 2 +-
docs/static/jira-links.js | 2 +-
21 files changed, 686 insertions(+), 587 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 2295bd9..596ab3e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -153,19 +153,6 @@ jobs:
./scripts/ci/selective_ci_checks.sh
fi
- helm-tests:
-timeout-minutes: 5
-name: "Checks: Helm tests. Will soon be replaced with python tests"
-runs-on: ubuntu-latest
-needs: [build-info]
-if: >
- needs.build-info.outputs.needs-helm-tests == 'true'
-steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit
${{ github.merge_commit_sha }} )"
-uses: actions/checkout@v2
- - name: "Helm Tests"
-run: ./scripts/ci/kubernetes/ci_run_helm_testing.sh
-
ci-images:
timeout-minutes: 120
name: "Wait for CI images"
@@ -295,7 +282,7 @@ jobs:
name: airflow-documentation
path: "./files/documentation"
- helm-python-tests:
+ tests-helm:
timeout-minutes: 5
name: "Python unit tests for helm chart"
runs-on: ubuntu-latest
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index e4c0a86..1b5c436 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -58,12 +58,12 @@ repos:
- --fuzzy-match-generates-todo
files: \.rst$
- id: insert-license
-name: Add license for all JS/CSS files
-files: \.(js|css)$
+name: Add license for all JS/CSS/PUML files
+files: \.(js|css|puml)$
exclude:
^\.github/.*$|^airflow/www/static/.*|^airflow/www_rbac/static/.*$
args:
- --comment-style
- - "/**| *| */"
+ - "/*!| *| */"
- --license-filepath
- license-templates/LICENSE.txt
- --fuzzy-match-generates-todo
@@ -121,7 +121,7 @@ repos:
- id: insert-license
name: Add license for all md files
files: \.md$
-exclude: ^\.github/.*$
+exclude: ^\.github/.*$|PROVIDER_CHANGES.*\.md
args:
- --comment-style
- ""
@@ -133,7 +133,7 @@ repos:
hooks:
- id: doctoc
name: Add TOC for md files
-files: ^README\.md$|^CONTRIBUTING\.md$|^UPDATING.md$|^dev/README.md$
+files:
^README\.md$|^CONTRIBUTING\.md$|^UPDATING.*.md$|^dev/README\.md$|^dev/PROVIDER_PACKAGES.md$
args:
- "--maxlevel"
- "2"
@@ -141,7 +141,7 @@ repos:
hooks:
- id: check-hooks-apply
- repo: https://github.com/pre-commit/pre-commit-hooks
-rev: v3.2.0
+rev: v3.3.0
hooks:
- id: check-merge-conflict
- id: debug-statements
@@ -157,13 +157,14 @@ repos:
- id: rst-backticks
- id: python-no-log-warn
- repo: https://github.com/adrienverge/yamllint
-rev: v1.24.2
+rev: v1.25.0
hooks:
- id: yamllint
[airflow] 02/44: Updated documentation for the CI with mermaid sequence diagrams (#10380)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 132e899a1132988e99281215fd6d9e9c37bd0369 Author: Jarek Potiuk AuthorDate: Mon Aug 24 22:45:28 2020 +0200 Updated documentation for the CI with mermaid sequence diagrams (#10380) (cherry picked from commit f2da6b419fde71d335d133031f2c0fac28b87289) --- .pre-commit-config.yaml | 11 ++- .rat-excludes | 1 + images/ci/CI.png| Bin 0 -> 243004 bytes images/ci/pull_request_ci_flow.md5 | 1 + images/ci/pull_request_ci_flow.mermaid | 108 + images/ci/pull_request_ci_flow.png | Bin 0 -> 195487 bytes images/ci/push_ci_flow.md5 | 1 + images/ci/push_ci_flow.mermaid | 117 images/ci/push_ci_flow.png | Bin 0 -> 209013 bytes images/ci/scheduled_ci_flow.md5 | 1 + images/ci/scheduled_ci_flow.mermaid | 111 ++ images/ci/scheduled_ci_flow.png | Bin 0 -> 201540 bytes scripts/ci/pre_commit/pre_commit_mermaid.sh | 97 +++ 13 files changed, 447 insertions(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 1b5c436..6763164 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -128,6 +128,15 @@ repos: - --license-filepath - license-templates/LICENSE.txt - --fuzzy-match-generates-todo + - id: insert-license +name: Add license for all mermaid files +args: + - --comment-style + - "|%%|" + - --license-filepath + - license-templates/LICENSE.txt + - --fuzzy-match-generates-todo +files: \.mermaid$ - repo: https://github.com/thlorenz/doctoc.git rev: v1.4.0 hooks: @@ -277,7 +286,7 @@ repos: entry: ./scripts/ci/pre_commit/pre_commit_check_integrations.sh language: system pass_filenames: false -files: ^airflow/scripts/ci/libraries/_initialization.sh$|^breeze-complete$ +files: ^common/_common_values.sh$|^breeze-complete$ - id: check-apache-license name: Check if licenses are OK for Apache entry: "./scripts/ci/pre_commit/pre_commit_check_license.sh" diff --git a/.rat-excludes b/.rat-excludes index 497d7ed..69c8ccc 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -28,6 +28,7 @@ metastore_db .*sql .*svg .*csv +.*md5 CHANGELOG.txt .*zip .*lock diff --git a/images/ci/CI.png b/images/ci/CI.png new file mode 100644 index 000..7123519 Binary files /dev/null and b/images/ci/CI.png differ diff --git a/images/ci/pull_request_ci_flow.md5 b/images/ci/pull_request_ci_flow.md5 new file mode 100644 index 000..3fe05dd --- /dev/null +++ b/images/ci/pull_request_ci_flow.md5 @@ -0,0 +1 @@ +e3aa93e8b46ca97b4aad06de23a10ad5 images/ci/pull_request_ci_flow.mermaid diff --git a/images/ci/pull_request_ci_flow.mermaid b/images/ci/pull_request_ci_flow.mermaid new file mode 100644 index 000..2e4bf9c --- /dev/null +++ b/images/ci/pull_request_ci_flow.mermaid @@ -0,0 +1,108 @@ +%% Licensed to the Apache Software Foundation (ASF) under one +%% or more contributor license agreements. See the NOTICE file +%% distributed with this work for additional information +%% regarding copyright ownership. The ASF licenses this file +%% to you under the Apache License, Version 2.0 (the +%% "License"); you may not use this file except in compliance +%% with the License. You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, +%% software distributed under the License is distributed on an +%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +%% KIND, either express or implied. See the License for the +%% specific language governing permissions and limitations +%% under the License. + +sequenceDiagram +Contributor -->> Fork: Submit PR +Fork -->> Airflow Repo: Start Pull Request +activate Airflow Repo +Note over Airflow Repo: Trigger PR build +Note over CI Build: pull_request[Read Token] +Note over Build Images: workflow_run[Write Token] +Airflow Repo -->> CI Build: Trigger 'pull_request' event +activate CI Build +CI Build -->> Build Images: Trigger 'workflow_run' event +activate Build Images +Note over Build Images: Find Duplicate[CI Build] +opt Duplicated builds +Build Images -->> CI Build: Cancel +end +Note over Build Images: Find Duplicates[Build Image] +opt Duplicated builds +Build Images -->> Build Images: Cancel +end +Note over Build Images:
[airflow] 13/44: Fix canceling of CodeQL workflow (#12024)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit df68fb7cd84626abe5675443f5254649147b446b
Author: Jarek Potiuk
AuthorDate: Sun Nov 1 22:05:45 2020 +0100
Fix canceling of CodeQL workflow (#12024)
The previous update for 4.3 version of the action also broke
CodeQL cancelling. This PR fixes it.
(cherry picked from commit 5f9792cee26c729f319e57fd09538fa6703929b4)
---
.github/workflows/build-images-workflow-run.yml | 16
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index 752e577..6099727 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
- uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: allDuplicates
@@ -82,7 +82,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately
# and then we try to find and cancel all the jobs with the same Event
+ Repo + Branch as the
# current Event/Repo/Branch combination.
- uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -98,7 +98,7 @@ jobs:
# We also produce list of canceled "CI Build' runs as output, so that
we
# can cancel all the matching "Build Images" workflow runs in the two
following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -131,14 +131,14 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run
Id:
# followed by one of the run ids. Yes I know it's super complex
¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' &&
steps.cancel-failed.outputs.cancelledRuns != '[]'
-uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{
steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
-uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -167,7 +167,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately and then
we try to match
# all the jobs with the same Event + Repo + Branch match and cancel
all the duplicates for those
# This might cancel own run, so this is the last step in the job
-uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
cancelMode: allDuplicatedNamedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -382,7 +382,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -397,7 +397,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
-uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
[airflow] 09/44: Add contributor-targeted description of the PR workflow. (#12016)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit b2633ff5231198363fcf6ac64ed388bc4125a12f Author: Jarek Potiuk AuthorDate: Sun Nov 1 14:22:23 2020 +0100 Add contributor-targeted description of the PR workflow. (#12016) * Add contributor-targeted description of the PR workflow. Simpler/shorter description of the PR workflow targeted for new contributors and contributors who did not follow the recent changes in the PR workflow. * Update CONTRIBUTING.rst Co-authored-by: Xiaodong DENG * Update CONTRIBUTING.rst Co-authored-by: Xiaodong DENG Co-authored-by: Xiaodong DENG (cherry picked from commit 6c8c3089e9f7fc00debe031b8c1ff576470a) --- CONTRIBUTING.rst | 552 --- 1 file changed, 285 insertions(+), 267 deletions(-) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 40e4f89..6d3aa91 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -42,8 +42,7 @@ to follow it and apply to the programme and follow up with the community. Report Bugs --- -Report bugs through `Apache -JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__. +Report bugs through `GitHub <https://github.com/apache/airflow/issues>`__. Please report relevant information and preferably code that exhibits the problem. @@ -51,16 +50,16 @@ problem. Fix Bugs -Look through the JIRA issues for bugs. Anything is open to whoever wants to +Look through the GitHub issues for bugs. Anything is open to whoever wants to implement it. Implement Features -- -Look through the `Apache -JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__ for features. +Look through the `GitHub issues labeled "kind:feature" +<https://github.com/apache/airflow/labels/kind%3Afeature>`__ for features. -Any unassigned "Improvement" issue is open to whoever wants to implement it. +Any unassigned feature request issue is open to whoever wants to implement it. We've created the operators, hooks, macros and executors we needed, but we've made sure that this part of Airflow is extensible. New operators, hooks, macros @@ -76,8 +75,7 @@ articles. Submit Feedback --- -The best way to send feedback is to open an issue on `Apache -JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__. +The best way to send feedback is to `open an issue on GitHub <https://github.com/apache/airflow/issues/new/choose>`__. If you are proposing a new feature: @@ -150,69 +148,6 @@ Contributors are responsible for: * Adding features * Championing one or more items on the `Roadmap <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home>`__. -= - -There are several roles within the Airflow Open-Source community. - - -PMC Member -The PMC (Project Management Committee) is a group of maintainers that drives changes in the way that -Airflow is managed as a project. - -Considering Apache, the role of the PMC is primarily to ensure that Airflow conforms to Apache's processes -and guidelines. - -Committers/Maintainers --- - -Committers are community members that have write access to the project’s repositories, i.e., they can modify the code, -documentation, and website by themselves and also accept other contributions. - -The official list of committers can be found `here <https://airflow.apache.org/docs/stable/project.html#committers>`__. - -Additionally, committers are listed in a few other places (some of these may only be visible to existing committers): - -* https://whimsy.apache.org/roster/ppmc/airflow -* https://github.com/orgs/apache/teams/airflow-committers/members - -Committers are responsible for: - -* Championing one or more items on the `Roadmap <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home>`__ -* Reviewing & Merging Pull-Requests -* Scanning and responding to Github issues -* Responding to questions on the dev mailing list (d...@airflow.apache.org) - -Becoming a Committer - - -There is no strict protocol for becoming a committer. -Candidates for new committers are typically people that are active contributors and community members. - -The key aspects of a committer are: - -* Consistent contributions over the past 6 months -* Understanding of Airflow Core or has displayed a holistic understanding of a particular part and made - contributions towards a more strategic goal -* Understanding of contributor/committer guidelines: `Contributors' Guide <https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst>`__ -* Quality of the commits -* Visibility in community discussions (dev mailing list, Slack and Github) -* Testing Release Candidate
[airflow] 12/44: Fixes problem with non-iterable data returned by GH API (#12021)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e18d6752814ea17cf596048cb8196224c46586a7
Author: Jarek Potiuk
AuthorDate: Sun Nov 1 19:25:29 2020 +0100
Fixes problem with non-iterable data returned by GH API (#12021)
The action to cancel workflow switched from deprecated
method of retrieving jobs to a 'better' one but it caused
some unexpected failures as some of the job data is not iterable and
failures in 'failedJobs" matching
Version 4.6 fixed the problem.
(cherry picked from commit aa5213b3609388b7b77e362dd9ad0bb05a8b5bc6)
---
.github/workflows/build-images-workflow-run.yml | 16
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index a3880e7..752e577 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: allDuplicates
@@ -82,7 +82,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately
# and then we try to find and cancel all the jobs with the same Event
+ Repo + Branch as the
# current Event/Repo/Branch combination.
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -98,7 +98,7 @@ jobs:
# We also produce list of canceled "CI Build' runs as output, so that
we
# can cancel all the matching "Build Images" workflow runs in the two
following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -131,14 +131,14 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run
Id:
# followed by one of the run ids. Yes I know it's super complex
¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' &&
steps.cancel-failed.outputs.cancelledRuns != '[]'
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{
steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
- uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -167,7 +167,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately and then
we try to match
# all the jobs with the same Event + Repo + Branch match and cancel
all the duplicates for those
# This might cancel own run, so this is the last step in the job
- uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
cancelMode: allDuplicatedNamedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -382,7 +382,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -397,7 +397,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea516
[airflow] 11/44: Revise "Project Focus" copy (#12011)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 887c231332d2d425d7f81ba0973dc12c508d8a2c Author: Ry Walker <4283+...@users.noreply.github.com> AuthorDate: Sun Nov 1 12:34:34 2020 -0500 Revise "Project Focus" copy (#12011) (cherry picked from commit 77ae32e311ebd9f6ab236cbeba42f7de3dd1e211) --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index ccac802..70bf2d8 100644 --- a/README.md +++ b/README.md @@ -59,11 +59,11 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The ## Project Focus -Airflow works best with workflows that are mostly static and slowly changing. When the structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). +Airflow works best with workflows that are mostly static and slowly changing. When DAG structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](https://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). -Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent, and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work. +Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent (i.e. results of the task will be the same, and will not create duplicated data in a destination system), and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external servic [...] -Airflow **is not** a streaming solution. Airflow is not in the [Spark Streaming](http://spark.apache.org/streaming/) or [Storm](https://storm.apache.org/) space. +Airflow is not a streaming solution, but it is often used to process real-time data, pulling data off streams in batches. ## Principles
[airflow] 06/44: Switches to "cancel-all-duplicates' mode of cancelling. (#12004)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d3ab3df2eaad211f2ce24f73e354de4ec904716d
Author: Jarek Potiuk
AuthorDate: Sat Oct 31 23:39:54 2020 +0100
Switches to "cancel-all-duplicates' mode of cancelling. (#12004)
The cancel-workflow-runs action in version 4.1 got the capability
of cancelling not only duplicates of own run (including future,
queued duplicates) but also cancelling all duplicates from all
running worklfows - regardless if they were triggered by my own
PR or some other PRs. This will be even more helpful with handling
the queues and optimising our builds, because in case ANY of
the build image workflows starts to run, it will cancel ALL
duplicates immediately.
(cherry picked from commit 21350aa3cf8952b605713257ae94e1ed648dd00b)
---
.github/workflows/build-images-workflow-run.yml | 19 +--
1 file changed, 9 insertions(+), 10 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index 81c4fb4..972195a 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,12 +65,11 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
token: ${{ secrets.GITHUB_TOKEN }}
- cancelMode: duplicates
+ cancelMode: allDuplicates
sourceRunId: ${{ github.event.workflow_run.id }}
- notifyPRCancel: true
- name: "Output BUILD_IMAGES"
id: build-images
run: |
@@ -83,7 +82,7 @@ jobs:
# in GitHub Actions, we have to use Job names to match
Event/Repo/Branch from the
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -101,7 +100,7 @@ jobs:
# can cancel all the matching "Build Images" workflow runs in the two
following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -135,18 +134,18 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run
Id:
# followed by one of the run ids. Yes I know it's super complex
¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' &&
steps.source-run-info-failed.outputs.cancelledRuns != '[]'
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{
steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
- uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
- cancelMode: duplicates
+ cancelMode: allDuplicates
workflowFileName: 'codeql-analysis.yml'
- name: "Set Docker Cache Directive"
id: cache-directive
@@ -373,7 +372,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -388,7 +387,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
[airflow] 16/44: Fixes documentation-only selective checks (#12038)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ad8aa653f874141084973b00d1f73b1f1f6e5402
Author: Jarek Potiuk
AuthorDate: Mon Nov 2 15:16:24 2020 +0100
Fixes documentation-only selective checks (#12038)
There was a problem that documentation-only checks triggered
selective checks without docs build (they resulted in
basic-checks-only and no images being built.
This occured for example in #12025
This PR fixes it by adding image-build and docs-build as two
separate outputs.
(cherry picked from commit adbf764ade6916b505c3238697bac10f98bfa6eb)
---
.github/workflows/build-images-workflow-run.yml | 4 +-
.github/workflows/ci.yml| 41 +--
scripts/ci/selective_ci_checks.sh | 69 +
3 files changed, 63 insertions(+), 51 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index 6099727..a3277a7 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -201,7 +201,7 @@ jobs:
defaultPythonVersion: ${{
steps.selective-checks.outputs.default-python-version }}
run-tests: ${{ steps.selective-checks.outputs.run-tests }}
run-kubernetes-tests: ${{
steps.selective-checks.outputs.run-kubernetes-tests }}
- basic-checks-only: ${{ steps.selective-checks.outputs.basic-checks-only
}}
+ image-build: ${{ steps.selective-checks.outputs.image-build }}
if: >
needs.cancel-workflow-runs.outputs.buildImages == 'true'
steps:
@@ -266,7 +266,7 @@ jobs:
image-type: [CI, PROD]
fail-fast: true
if: >
- needs.build-info.outputs.basic-checks-only == 'false' &&
+ needs.build-info.outputs.image-build == 'true' &&
needs.cancel-workflow-runs.outputs.buildImages == 'true'
env:
BACKEND: postgres
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a5457c0..c217bdc 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -92,6 +92,8 @@ jobs:
run-tests: ${{ steps.selective-checks.outputs.run-tests }}
run-kubernetes-tests: ${{
steps.selective-checks.outputs.run-kubernetes-tests }}
basic-checks-only: ${{ steps.selective-checks.outputs.basic-checks-only
}}
+ image-build: ${{ steps.selective-checks.outputs.image-build }}
+ docs-build: ${{ steps.selective-checks.outputs.docs-build }}
needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
@@ -158,8 +160,7 @@ jobs:
name: "Wait for CI images"
runs-on: ubuntu-latest
needs: [build-info]
-if: >
- needs.build-info.outputs.basic-checks-only == 'false'
+if: needs.build-info.outputs.image-build == 'true'
env:
BACKEND: sqlite
steps:
@@ -196,8 +197,7 @@ jobs:
env:
MOUNT_LOCAL_SOURCES: "true"
PYTHON_MAJOR_MINOR_VERSION:
${{needs.build-info.outputs.defaultPythonVersion}}
-if: >
- needs.build-info.outputs.basic-checks-only == 'false'
+if: needs.build-info.outputs.basic-checks-only == 'false'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit
${{ github.merge_commit_sha }} )"
uses: actions/checkout@v2
@@ -233,8 +233,7 @@ jobs:
SKIP: "build,mypy,flake8,pylint,bats-in-container-tests"
MOUNT_LOCAL_SOURCES: "true"
PYTHON_MAJOR_MINOR_VERSION:
${{needs.build-info.outputs.defaultPythonVersion}}
-if: >
- needs.build-info.outputs.basic-checks-only == 'true'
+if: needs.build-info.outputs.basic-checks-only == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -266,8 +265,7 @@ jobs:
name: "Build docs"
runs-on: ubuntu-latest
needs: [build-info, ci-images]
-if: >
- needs.build-info.outputs.basic-checks-only == 'false'
+if: needs.build-info.outputs.docs-build == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit
${{ github.merge_commit_sha }})"
uses: actions/checkout@v2
@@ -348,8 +346,7 @@ jobs:
RUN_TESTS: true
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
TEST_TYPE: ""
-if: >
-needs.build-info.outputs.run-tests == 'true'
+if: needs.build-info.outputs.run-tests == '
[airflow] 26/44: Update to new helm stable repo (#12137)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2ddafd037a1f919aaabff4708c69b9766d2225b2
Author: Ash Berlin-Taylor
AuthorDate: Fri Nov 6 16:05:18 2020 +
Update to new helm stable repo (#12137)
Switch out deprecated helm repo for new stable repo.
-
https://www.cncf.io/blog/2020/11/05/helm-chart-repository-deprecation-update/
-
https://helm.sh/docs/faq/#i-am-getting-a-warning-about-unable-to-get-an-update-from-the-stable-chart-repository
(cherry picked from commit 128c9918b5f79cb46a563b77e803c29548c4319c)
---
chart/README.md | 2 +-
chart/requirements.lock | 6 +++---
chart/tests/conftest.py | 4 +---
scripts/ci/libraries/_kind.sh | 2 +-
4 files changed, 6 insertions(+), 8 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index e7ef973..7d23ce3 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -37,7 +37,7 @@ cluster using the [Helm](https://helm.sh) package manager.
To install this repository from source (using helm 3)
```bash
kubectl create namespace airflow
-helm repo add stable https://kubernetes-charts.storage.googleapis.com
+helm repo add stable https://charts.helm.sh/stable/
helm dep update
helm install airflow . --namespace airflow
```
diff --git a/chart/requirements.lock b/chart/requirements.lock
index 715458e..3f3c34a 100644
--- a/chart/requirements.lock
+++ b/chart/requirements.lock
@@ -1,6 +1,6 @@
dependencies:
- name: postgresql
- repository: https://kubernetes-charts.storage.googleapis.com/
+ repository: https://charts.helm.sh/stable/
version: 6.3.12
-digest: sha256:e8d53453861c590e6ae176331634c9268a11cf894be17ed580fa2b347101be97
-generated: "2020-10-27T21:16:13.0063538Z"
+digest: sha256:58d88cf56e78b2380091e9e16cc6ccf58b88b3abe4a1886dd47cd9faef5309af
+generated: "2020-11-04T15:59:36.967913-08:00"
diff --git a/chart/tests/conftest.py b/chart/tests/conftest.py
index db1f22f..789f311 100644
--- a/chart/tests/conftest.py
+++ b/chart/tests/conftest.py
@@ -26,7 +26,5 @@ def upgrade_helm():
"""
Upgrade Helm repo
"""
-subprocess.check_output(
-["helm", "repo", "add", "stable",
"https://kubernetes-charts.storage.googleapis.com/";]
-)
+subprocess.check_output(["helm", "repo", "add", "stable",
"https://charts.helm.sh/stable/";])
subprocess.check_output(["helm", "dep", "update", sys.path[0]])
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index 41048fc..6194742 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -336,7 +336,7 @@ function kind::deploy_airflow_with_helm() {
kubectl create namespace "${HELM_AIRFLOW_NAMESPACE}"
kubectl create namespace "test-namespace"
pushd "${AIRFLOW_SOURCES}/chart" || exit 1
-helm repo add stable https://kubernetes-charts.storage.googleapis.com
+helm repo add stable https://charts.helm.sh/stable/
helm dep update
helm install airflow . --namespace "${HELM_AIRFLOW_NAMESPACE}" \
--set "defaultAirflowRepository=${DOCKERHUB_USER}/${DOCKERHUB_REPO}" \
[airflow] 17/44: Uses DOCKER_TAG when building image in DockerHub (#12050)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7512344582888106d46fb871ecd9804d2c706f6b
Author: Jarek Potiuk
AuthorDate: Mon Nov 2 22:00:51 2020 +0100
Uses DOCKER_TAG when building image in DockerHub (#12050)
DockerHub uses `hooks/build` to build the image and it passes
DOCKER_TAG variable when the script is called.
This PR makes the DOCKER_TAG to provide the default valuei for tag
that is calculated from sources (taking the default branch and
python version). Since it is only set in the DockerHub build, it
should be safe.
Fixes #11937
(cherry picked from commit 5c199fbddfaf9f83915e84225313169a0486c3a6)
---
scripts/ci/libraries/_build_images.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index 444fff9..387372a 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -327,7 +327,7 @@ function build_images::get_docker_image_names() {
export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
# CI image base tag
-export
AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
+export
AIRFLOW_CI_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci}"
# CI image to build
export
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
@@ -336,7 +336,7 @@ function build_images::get_docker_image_names() {
export
AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
# Base production image tag - used to build kubernetes tag as well
-export
AIRFLOW_PROD_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
+export
AIRFLOW_PROD_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}}"
# PROD image to build
export
AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
[airflow] 20/44: Fixes problem with building a PROD image (#12080)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f4323092e6544cf1c28e97c3f461ef0ed282e3ef
Author: Jarek Potiuk
AuthorDate: Wed Nov 4 09:31:00 2020 +0100
Fixes problem with building a PROD image (#12080)
The change #12050 that aimed at automation of Docker images
building in DockerHub had an undesired effect of overriding the
production image tag with the CI one.
This is fixed by this PR.
(cherry picked from commit d971c1c0e141f285a3cdd3f029028ab62df6f81a)
---
scripts/ci/images/ci_build_dockerhub.sh | 6 +-
scripts/ci/libraries/_build_images.sh | 9 ++---
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/scripts/ci/images/ci_build_dockerhub.sh
b/scripts/ci/images/ci_build_dockerhub.sh
index 7d92abc..ac7764f 100755
--- a/scripts/ci/images/ci_build_dockerhub.sh
+++ b/scripts/ci/images/ci_build_dockerhub.sh
@@ -39,7 +39,6 @@ echo
echo "DOCKER_TAG=${DOCKER_TAG}"
-
[[ ${DOCKER_TAG:=} =~ ${DEFAULT_BRANCH}-python([0-9.]*)(.*) ]] && export
PYTHON_MAJOR_MINOR_VERSION=${BASH_REMATCH[1]}
: "${PYTHON_MAJOR_MINOR_VERSION:?"The tag '${DOCKER_TAG}' should follow the
pattern .*-pythonX.Y[-ci]"}"
@@ -47,6 +46,11 @@ echo "DOCKER_TAG=${DOCKER_TAG}"
echo "Detected PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}"
echo
+FORCE_AIRFLOW_PROD_BASE_TAG="${DOCKER_TAG}"
+export FORCE_AIRFLOW_PROD_BASE_TAG
+
+readonly FORCE_AIRFLOW_PROD_BASE_TAG
+
# shellcheck source=scripts/ci/libraries/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index 387372a..a6dbae0 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -327,8 +327,7 @@ function build_images::get_docker_image_names() {
export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
# CI image base tag
-export
AIRFLOW_CI_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci}"
-
+export
AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
# CI image to build
export
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
@@ -336,7 +335,11 @@ function build_images::get_docker_image_names() {
export
AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
# Base production image tag - used to build kubernetes tag as well
-export
AIRFLOW_PROD_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}}"
+if [[ ${FORCE_AIRFLOW_PROD_BASE_TAG=} == "" ]]; then
+export
AIRFLOW_PROD_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
+else
+export AIRFLOW_PROD_BASE_TAG="${FORCE_AIRFLOW_PROD_BASE_TAG}"
+fi
# PROD image to build
export
AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
[airflow] 24/44: Add Kubernetes files to selective checks (#12114)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 49430a0b6478b80b720a2ea278cfd428a23d1f95
Author: Daniel Imberman
AuthorDate: Thu Nov 5 11:54:49 2020 -0800
Add Kubernetes files to selective checks (#12114)
* Add Kubernetes files to selective checks
There are multiple kubernetes-related files that require
running the k8s integration tests. This PR adds those to the
run_selective_tests
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Kaxil Naik
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk
Co-authored-by: Kaxil Naik
Co-authored-by: Jarek Potiuk
(cherry picked from commit 054de0703aa99fc425028d4cbe1f7b363e0cbeca)
---
scripts/ci/selective_ci_checks.sh | 6 +-
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/scripts/ci/selective_ci_checks.sh
b/scripts/ci/selective_ci_checks.sh
index b9a6d74..3c7132d 100755
--- a/scripts/ci/selective_ci_checks.sh
+++ b/scripts/ci/selective_ci_checks.sh
@@ -179,7 +179,6 @@ ALL_TESTS="Core Integration Heisentests"
readonly ALL_TESTS
function set_outputs_run_everything_and_exit() {
-needs_api_tests "true"
needs_helm_tests "true"
needs_javascript_scans "true"
needs_python_scans "true"
@@ -201,7 +200,6 @@ function set_outputs_run_all_tests() {
}
function set_output_skip_all_tests_and_docs_and_exit() {
-needs_api_tests "false"
needs_helm_tests "false"
needs_javascript_scans "false"
needs_python_scans "false"
@@ -215,7 +213,6 @@ function set_output_skip_all_tests_and_docs_and_exit() {
}
function set_output_skip_tests_but_build_images_and_exit() {
-needs_api_tests "false"
needs_helm_tests "false"
needs_javascript_scans "false"
needs_python_scans "false"
@@ -378,9 +375,8 @@ function get_count_kubernetes_files() {
echo "Count Kubernetes files"
echo
local pattern_array=(
-"^airflow/kubernetes"
"^chart"
-"^tests/kubernetes_tests"
+"^kubernetes_tests"
)
show_changed_files
COUNT_KUBERNETES_CHANGED_FILES=$(count_changed_files)
[airflow] 10/44: Adds more aggressive cancelling of duplicate Build Image jobs (#12018)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 99aa8839732795038288adcfb031a726700ff930
Author: Jarek Potiuk
AuthorDate: Sun Nov 1 17:54:37 2020 +0100
Adds more aggressive cancelling of duplicate Build Image jobs (#12018)
This change adds even more aggressive cancelling of duplicates of
'Build Image' jobs. it's not an obvious task to know which
Build Image jobs are duplicates, we are matching those duplicates
based on specially crafted "build-info" job names. We add
Event, Branch, Repo to the job names and assume that two
runs with the same event + branch + repo are duplicates.
It also disables self-preservation for this step because
it is perfectly ok to cancel itself in case there is a newer
in-progress Build Image job.
Unfortunately even this will not work perfectly well. Those job
names are resolved only for the jobs that are runnning rather than
the queued ones, so in case we have several duplicates of the
same build image job in the queue, they will not be found/cancelled.
The cancelling will only happen if both duplicates are already
running.
It's good enough for now and we cannot do much more until there
is a missing feature added to GitHub API that allows to link
the workflow_run with the run that triggered it. This issue has
been raised to GitHub Support and internal engineering ticket
has been apparently opened to add this feature.
More detailed status for the missing feature is kept at #11294
(cherry picked from commit 1d14e74e33efbcb17e5553dba92baf9c8b0fc7c8)
---
.github/workflows/build-images-workflow-run.yml | 40 +++--
.github/workflows/ci.yml| 2 +-
2 files changed, 26 insertions(+), 16 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index 972195a..a3880e7 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
-uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: allDuplicates
@@ -77,12 +77,12 @@ jobs:
#
https://github.community/t/how-to-set-and-access-a-workflow-variable/17335/16
echo "::set-output name=buildImages::${BUILD_IMAGES}"
- name: "Cancel duplicated 'Build Image' runs"
-
# We find duplicates of our own "Build Image" runs - due to a missing
feature
-# in GitHub Actions, we have to use Job names to match
Event/Repo/Branch from the
-# build-info step there to find the duplicates ¯\_(ツ)_/¯.
-
-uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+# in GitHub Actions, we have to use Job names to match
Event/Repo/Branch matching
+# trick ¯\_(ツ)_/¯. We name the build-info job appropriately
+# and then we try to find and cancel all the jobs with the same Event
+ Repo + Branch as the
+# current Event/Repo/Branch combination.
+uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -93,14 +93,12 @@ jobs:
Branch: ${{ steps.source-run-info.outputs.sourceHeadBranch }}.*"]
if: env.BUILD_IMAGES == 'true'
- name: "Cancel all 'CI Build' runs where some jobs failed"
-
# We find any of the "CI Build" workflow runs, where any of the
important jobs
# failed. The important jobs are selected by the regexp array below.
# We also produce list of canceled "CI Build' runs as output, so that
we
# can cancel all the matching "Build Images" workflow runs in the two
following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-
-uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -111,7 +109,6 @@ jobs:
["^Static checks.*", "^Build docs$", "^Spell check docs$",
"^Backport packages$",
"^Checks: Helm tests$&
[airflow] 23/44: Update install_mysql.sh (#12101)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b062b9259b81ba9c042a523f5454beb366e134ce
Author: J. Daniel Medeiros <3796667+jdmedei...@users.noreply.github.com>
AuthorDate: Thu Nov 5 15:32:39 2020 -0100
Update install_mysql.sh (#12101)
After Debian 9 and according to the manual
https://manpages.debian.org/stretch/apt/apt-key.8.en.html, after Debian 9
instead of using "apt-key add" a keyring should be placed directly in the
/etc/apt/trusted.gpg.d/ directory with a descriptive name and either "gpg" or
"asc" as file extension. Also added better redirection on the apt-key list
command.
(cherry picked from commit ded3dbbff0c8e4fabcee62a677394dec0db1aa45)
---
scripts/docker/install_mysql.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/docker/install_mysql.sh b/scripts/docker/install_mysql.sh
index 1ddc76b..0fbb1da 100755
--- a/scripts/docker/install_mysql.sh
+++ b/scripts/docker/install_mysql.sh
@@ -46,10 +46,10 @@ if [[ ${INSTALL_MYSQL_CLIENT:="true"} == "true" ]]; then
gpg --keyserver "${keyserver}" --recv-keys "${KEY}" && break
done
set -e
-gpg --export "${KEY}" | apt-key add -
+gpg --export "${KEY}" > /etc/apt/trusted.gpg.d/mysql.gpg
gpgconf --kill all
rm -rf "${GNUPGHOME}"
-apt-key list > /dev/null
+apt-key list > /dev/null 2>&1
echo "deb http://repo.mysql.com/apt/debian/ buster mysql-5.7" | tee -a
/etc/apt/sources.list.d/mysql.list
apt-get update
apt-get install --no-install-recommends -y "${packages[@]}"
[airflow] 03/44: Pin `kubernetes` to a max version of 11.0.0. (#11974)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit d5e7b5115055fd3ccdb7896bb45dfe48db83c6da Author: Jed Cunningham <66968678+jedcunning...@users.noreply.github.com> AuthorDate: Fri Oct 30 11:56:58 2020 -0600 Pin `kubernetes` to a max version of 11.0.0. (#11974) 12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper' object` when serializing V1Pod's in `executor_config`. (cherry picked from commit 9687b3bc812a394f9f1debe48ea17bbfe78c270f) --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 50818d4..3ace091 100644 --- a/setup.py +++ b/setup.py @@ -305,7 +305,7 @@ kerberos = [ ] kubernetes = [ 'cryptography>=2.0.0', -'kubernetes>=3.0.0', +'kubernetes>=3.0.0, <12.0.0', ] ldap = [ 'ldap3>=2.5.1',
[airflow] 33/44: Remove popd which is a remnant from past (#12211)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 73187015e2dc64a76bf0d221e05ba7322eb436b8
Author: Jarek Potiuk
AuthorDate: Mon Nov 9 21:58:37 2020 +0100
Remove popd which is a remnant from past (#12211)
(cherry picked from commit d8abee690823b7973e5561f2725dc7b79467bfdf)
---
scripts/ci/libraries/_start_end.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/ci/libraries/_start_end.sh
b/scripts/ci/libraries/_start_end.sh
index f4d8bfe..8fc7e5d 100644
--- a/scripts/ci/libraries/_start_end.sh
+++ b/scripts/ci/libraries/_start_end.sh
@@ -53,7 +53,6 @@ function start_end::dump_container_logs() {
echo
"###"
docker logs "${container}" > "${dump_file}"
echo " Container ${container} logs dumped to
${dump_file}"
-popd || exit 1
echo
"###"
}
[airflow] 29/44: Uses always the same Python base image as used for CI image (#12177)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a27701a396f803e6c5434f2c2062393efa55f62a
Author: Jarek Potiuk
AuthorDate: Sun Nov 8 11:20:31 2020 +0100
Uses always the same Python base image as used for CI image (#12177)
When new Python version is released (bugfixes), we rebuild the CI image
and replace it with the new one, however releasing of the python
image and CI image is often hours or even days apart (we only
release the CI image when tests pass in master with the new python
image). We already use a better approach for Github - we simply
push the new python image to our registry together with the CI
image and the CI jobs are always pulling them from our registry
knowing that the two - python and CI image are in sync.
This PR introduces the same approach. We not only push CI image
but also the corresponding Python image to our registry. This has
no ill effect - DockerHub handles it automatically and reuses
the layers of the image directly from the Python one so it is
merely a label that is stored in our registry that points to the
exact Python image that was used by the last pushed CI image.
(cherry picked from commit 75bdfaeb9b2d7f47599e951ccaccc515a180ca19)
---
IMAGES.rst | 11
scripts/ci/libraries/_build_images.sh| 11 +++-
scripts/ci/libraries/_push_pull_remove_images.sh | 35 +++-
3 files changed, 19 insertions(+), 38 deletions(-)
diff --git a/IMAGES.rst b/IMAGES.rst
index 45aa09ab..8c913db 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -46,6 +46,17 @@ where:
* The ``-ci`` suffix is added for CI images
* The ``-manifest`` is added for manifest images (see below for explanation of
manifest images)
+We also store (to increase speed of local build/pulls) python images that were
used to build
+the CI images. Each CI image, when built uses current python version of the
base images. Those
+python images are regularly updated (with bugfixes/security fixes), so for
example python3.8 from
+last week might be a different image than python3.8 today. Therefore whenever
we push CI image
+to airflow repository, we also push the python image that was used to build it
this image is stored
+as ``apache/airflow:python-3.8-``.
+
+Since those are simply snapshots of the existing python images, DockerHub does
not create a separate
+copy of those images - all layers are mounted from the original python images
and those are merely
+labels pointing to those.
+
Building docker images
==
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index cbd01c1..a2577cb 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -330,9 +330,11 @@ function build_images::get_docker_image_names() {
export
AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
# CI image to build
export
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
-
# Default CI image
-export
AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
+export
AIRFLOW_CI_PYTHON_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:python${PYTHON_MAJOR_MINOR_VERSION}-${BRANCH_NAME}"
+# CI image to build
+export
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
+
# Base production image tag - used to build kubernetes tag as well
if [[ ${FORCE_AIRFLOW_PROD_BASE_TAG=} == "" ]]; then
@@ -374,11 +376,6 @@ function build_images::prepare_ci_build() {
export
GITHUB_REGISTRY_AIRFLOW_CI_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/${AIRFLOW_CI_BASE_TAG}"
export
GITHUB_REGISTRY_PYTHON_BASE_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
fi
-if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" ==
"${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
-export DEFAULT_PROD_IMAGE="${AIRFLOW_CI_IMAGE_DEFAULT}"
-else
-export DEFAULT_PROD_IMAGE=""
-fi
export THE_IMAGE_TYPE="CI"
export IMAGE_DESCRIPTION="Airflow CI"
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh
b/scripts/ci/libraries/_push_pull_remove_images.sh
index fae0807..e853f42 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -104,7 +104,6 @@ function
push_pull_remove_images::pull_image_github_dockerhub() {
# Pulls CI image in case caching strategy is "pulled" and the image needs to
be pulled
function push_pull_remove_images::pull_
[airflow] 27/44: Fixes undefined variables (#12155)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit fc19025960f659b86d8d989b8b2db22010c9ebfd
Author: Jarek Potiuk
AuthorDate: Sat Nov 7 12:18:14 2020 +0100
Fixes undefined variables (#12155)
There are few more variables that (if not defined) prevent
from using the CI image directly without breeze or the
CI scripts.
With this change you can run:
`docker run -it apache/airflow:master-python3.6-ci`
and enter the image without errors.
(cherry picked from commit c7f34104516095c487cd6729f830df373ab78fbd)
---
scripts/ci/libraries/_initialization.sh | 4 ++--
scripts/in_container/entrypoint_ci.sh | 2 +-
scripts/in_container/run_init_script.sh | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/scripts/ci/libraries/_initialization.sh
b/scripts/ci/libraries/_initialization.sh
index e0b4a8a..b58fa2e 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -566,10 +566,10 @@ Detected CI build environment:
Initialization variables:
-INIT_SCRIPT_FILE: ${INIT_SCRIPT_FILE}
+INIT_SCRIPT_FILE: ${INIT_SCRIPT_FILE=}
LOAD_DEFAULT_CONNECTIONS: ${LOAD_DEFAULT_CONNECTIONS}
LOAD_EXAMPLES: ${LOAD_EXAMPLES}
-INSTALL_WHEELS: ${INSTALL_WHEELS}
+INSTALL_WHEELS: ${INSTALL_WHEELS=}
DISABLE_RBAC: ${DISABLE_RBAC}
Test variables:
diff --git a/scripts/in_container/entrypoint_ci.sh
b/scripts/in_container/entrypoint_ci.sh
index 8a7b683..37d3199 100755
--- a/scripts/in_container/entrypoint_ci.sh
+++ b/scripts/in_container/entrypoint_ci.sh
@@ -97,7 +97,7 @@ else
install_released_airflow_version "${INSTALL_AIRFLOW_VERSION}"
fi
-if [[ ${INSTALL_WHEELS} == "true" ]]; then
+if [[ ${INSTALL_WHEELS=} == "true" ]]; then
pip install /dist/*.whl || true
fi
diff --git a/scripts/in_container/run_init_script.sh
b/scripts/in_container/run_init_script.sh
index a4b030a..0bd685c 100755
--- a/scripts/in_container/run_init_script.sh
+++ b/scripts/in_container/run_init_script.sh
@@ -23,7 +23,7 @@ if [ -z "${AIRFLOW_BREEZE_CONFIG_DIR+x}" ]; then
export AIRFLOW_BREEZE_CONFIG_DIR="${FILES_DIR}/airflow-breeze-config"
fi
-if [ -z "${INIT_SCRIPT_FILE}" ]; then
+if [ -z "${INIT_SCRIPT_FILE=}" ]; then
export INIT_SCRIPT_FILE="init.sh"
fi
[airflow] 14/44: Turns failure of PR label when approved action into warning (#12017)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8e582911f9576fad1827d613a7980700c3c0080e
Author: Jarek Potiuk
AuthorDate: Mon Nov 2 00:01:20 2020 +0100
Turns failure of PR label when approved action into warning (#12017)
Sometimes (quite often really) when PR gets approved, the PR
gets merged rather quickly, without waiting for result of this
action. Or a new PR gets pushed quickly. In those cases PR will
not be found. But this is usually not a problem then and rather
than failing, we should simply print a warning and exit.
(cherry picked from commit a7a7cf22172ca92edfb62c373993ed974a298496)
---
.github/workflows/label_when_reviewed_workflow_run.yml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/label_when_reviewed_workflow_run.yml
b/.github/workflows/label_when_reviewed_workflow_run.yml
index 0fcab12..56bc835 100644
--- a/.github/workflows/label_when_reviewed_workflow_run.yml
+++ b/.github/workflows/label_when_reviewed_workflow_run.yml
@@ -72,7 +72,7 @@ jobs:
./scripts/ci/selective_ci_checks.sh
fi
- name: "Label when approved by committers for PRs that require full
tests"
-uses:
TobKed/label-when-approved-action@7872312da76508d29f98d4fa68843ea91754cc59 #
v1.2
+uses:
TobKed/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194 #
v1.3
id: label-full-test-prs-when-approved-by-commiters
if: >
steps.selective-checks.outputs.run-tests == 'true' &&
@@ -101,7 +101,7 @@ jobs:
[the run](https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }})
"}
- name: "Label when approved by committers for PRs that do not require
full tests"
-uses:
TobKed/label-when-approved-action@7872312da76508d29f98d4fa68843ea91754cc59 #
v1.2
+uses:
TobKed/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194 #
v1.3
id: label-simple-test-prs-when-approved-by-commiters
if: >
steps.selective-checks.outputs.run-tests == 'true' &&
@@ -116,7 +116,7 @@ jobs:
Airflow. The committers might merge it or can add a label 'full
tests needed' and re-run it
to run all tests if they see it is needed!
- name: "Label when approved by committers for PRs that do not require
tests at all"
-uses:
TobKed/label-when-approved-action@7872312da76508d29f98d4fa68843ea91754cc59 #
v1.2
+uses:
TobKed/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194 #
v1.3
id: label-no-test-prs-when-approved-by-commiters
if: steps.selective-checks.outputs.run-tests != 'true'
with:
[airflow] 34/44: Fix permissions of mounted /tmp directory for Breeze (#12157)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 40ebe6eb41a935cb83bff4ef80106347aa929f8c
Author: Jarek Potiuk
AuthorDate: Mon Nov 9 22:01:58 2020 +0100
Fix permissions of mounted /tmp directory for Breeze (#12157)
The "tmp" directory is mounted from the host (from tmp folder
in the source airflow directory). This is needed to get some
of our docker-in-docker tools (such as gcloud/aws/java) and
get them working on demand. Thanks to that we do not have
to increase the size of CI image unnecessarily.
Those tools were introduced and made to work in #9376
However this causes some of the standard tools (such as apt-get)
to not work inside the container unless the mounted /tmp
folder has write permission for groups/other.
This PR fixes it.
(cherry picked from commit a42bbe21c899f4048b0861df2c84eb2ba9c5eeb5)
---
scripts/in_container/entrypoint_ci.sh | 5 +
1 file changed, 5 insertions(+)
diff --git a/scripts/in_container/entrypoint_ci.sh
b/scripts/in_container/entrypoint_ci.sh
index 37d3199..057d630 100755
--- a/scripts/in_container/entrypoint_ci.sh
+++ b/scripts/in_container/entrypoint_ci.sh
@@ -31,6 +31,11 @@ function disable_rbac_if_requested() {
# shellcheck source=scripts/in_container/_in_container_script_init.sh
. /opt/airflow/scripts/in_container/_in_container_script_init.sh
+# Add "other" and "group" write permission to the tmp folder
+# Note that it will also change permissions in the /tmp folder on the host
+# but this is necessary to enable some of our CLI tools to work without errors
+chmod 1777 /tmp
+
AIRFLOW_SOURCES=$(cd "${IN_CONTAINER_DIR}/../.." || exit 1; pwd)
PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:=3.6}
[airflow] 44/44: Synchronize INTHEWILD.md with master
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 00f320fb50f72c8458e139340ab14ab526f88a81 Author: Jarek Potiuk AuthorDate: Sat Nov 14 16:07:25 2020 +0100 Synchronize INTHEWILD.md with master --- INTHEWILD.md | 20 +++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/INTHEWILD.md b/INTHEWILD.md index d1c914c..fe65c23 100644 --- a/INTHEWILD.md +++ b/INTHEWILD.md @@ -40,10 +40,12 @@ Currently, **officially** using Airflow: 1. [Airbnb](http://airbnb.io/) [[@mistercrunch](https://github.com/mistercrunch), [@artwr](https://github.com/artwr)] 1. [Airfinity](https://www.airfinity.com) [[@sibowyer](https://github.com/sibowyer)] 1. [Airtel](https://www.airtel.in/) [[@harishbisht](https://github.com/harishbisht)] +1. [Akamai](https://www.akamai.com/) [[@anirudhbagri](https://github.com/anirudhbagri)] 1. [Akamas](https://akamas.io) [[@GiovanniPaoloGibilisco](https://github.com/GiovanniPaoloGibilisco), [@lucacavazzana](https://github.com/lucacavazzana)] 1. [Alan](https://alan.eu) [[@charles-go](https://github.com/charles-go)] 1. [AloPeyk](https://alopeyk.com) [[@blcksrx](https://github.com/blcksrx), [@AloPeyk](https://github.com/AloPeyk)] 1. [AltX](https://www.getaltx.com/about) [[@pedromduarte](https://github.com/pedromduarte)] +1. [American Family Insurance](https://www.amfam.com/about) [[@di1eep](https://github.com/di1eep)] 1. [Apigee](https://apigee.com) [[@btallman](https://github.com/btallman)] 1. [Arquivei](https://www.arquivei.com.br/) [[@arquivei](https://github.com/arquivei)] 1. [Arrive](https://www.arrive.com/) @@ -51,7 +53,9 @@ Currently, **officially** using Airflow: 1. [Astronomer](http://www.astronomer.io) [[@schnie](https://github.com/schnie), [@ashb](https://github.com/ashb), [@kaxil](https://github.com/kaxil), [@dimberman](https://github.com/dimberman), [@andriisoldatenko](https://github.com/andriisoldatenko), [@ryw](https://github.com/ryw), [@ryanahamilton](https://github.com/ryanahamilton), [@jhtimmins](https://github.com/jhtimmins), [@vikramkoka](https://github.com/vikramkoka)] 1. [Auth0](https://auth0.com) [[@scottypate](https://github.com/scottypate)], [[@dm03514](https://github.com/dm03514)], [[@karangale](https://github.com/karangale)] 1. [Automattic](https://automattic.com/) [[@anandnalya](https://github.com/anandnalya), [@bperson](https://github.com/bperson), [@khrol](https://github.com/Khrol), [@xyu](https://github.com/xyu)] +1. [Avesta Technologies](https://avestatechnologies.com) [[@TheRum](https://github.com/TheRum)] 1. [Away](https://awaytravel.com) [[@trunsky](https://github.com/trunsky)] +1. [Axesor designeted activity company](https://www.axesor.es/) 1. [Azri Solutions](http://www.azrisolutions.com/) [[@userimack](https://github.com/userimack)] 1. [BBM](https://www.bbm.com/) 1. [Bagelcode](https://site.bagelcode.com/) @@ -71,6 +75,7 @@ Currently, **officially** using Airflow: 1. [Blacklane](https://www.blacklane.com) [[@serkef](https://github.com/serkef)] 1. [Bloc](https://www.bloc.io) [[@dpaola2](https://github.com/dpaola2)] 1. [Bloomberg](https://www.techatbloomberg.com) [[@dimberman](https://github.com/dimberman)] +1. [Bloomreach](https://www.bloomreach.com/) [[@neelborooah](https://github.com/neelborooah) & [@debodirno](https://github.com/debodirno) & [@ayushmnnit](https://github.com/ayushmnnit)] 1. [Blue Yonder](http://www.blue-yonder.com) [[@blue-yonder](https://github.com/blue-yonder)] 1. [BlueApron](https://www.blueapron.com) [[@jasonjho](https://github.com/jasonjho) & [@matthewdavidhauser](https://github.com/matthewdavidhauser)] 1. [Bluecore](https://www.bluecore.com) [[@JLDLaughlin](https://github.com/JLDLaughlin)] @@ -102,6 +107,7 @@ Currently, **officially** using Airflow: 1. [Clairvoyant](https://clairvoyantsoft.com) [[@shekharv](https://github.com/shekharv)] 1. [Classmethod, Inc.](https://classmethod.jp/) [[@shoito](https://github.com/shoito)] 1. [Cleartax](https://cleartax.in/) [[@anks](https://github.com/anks) & [@codebuff](https://github.com/codebuff)] +1. [Cloudera](https://www.cloudera.com/) [[@phraniiac](https://github.com/phraniiac) & [@VivekPemawat](https://github.com/VivekPemawat)] 1. [Clover Health](https://www.cloverhealth.com) [[@gwax](https://github.com/gwax) & [@vansivallab](https://github.com/vansivallab)] 1. [Colgate-Palmolive](https://www.colgatepalmolive.com/) [[@fhoda](https://github.com/fhoda)] 1. [Collectivehealth Inc.](https://www.collectivehealth.com) [[@retornam](https://github.com/retornam)] @@ -125,6 +131,7 @@ Currently, **officially** using Airflow: 1. [DataFox](https://www.datafox.com/) [[@sudowork](https://github.com/sudowork)] 1. [DataSprints](https://datasprints.com/) [[@lopesdiego12](https://github.com/lopesdiego12) & [@rafaelsantanaep](https://github.com/rafaelsanta
[airflow] 43/44: Simplifies check whether the CI image should be rebuilt (#12181)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 23c62063e11b0cb0f1a9ab763fd0e3c106711904
Author: Jarek Potiuk
AuthorDate: Fri Nov 13 22:21:39 2020 +0100
Simplifies check whether the CI image should be rebuilt (#12181)
Rather than counting changed layers in the image (which was
enigmatic, difficult and prone to some magic number) we rely now
on random file generated while building the image.
We are using the docker image caching mechanism here. The random
file will be regenerated only when the previous layer (which is
about installling Airflow dependencies for the first time) gets
rebuild. And for us this is the indication, that the building
the image will take quite some time. This layer should be
relatively static - even if setup.py changes the CI image is
designed in the way that the first time installation of Airflow
dependencies is not invalidated.
This should lead to faster and less frequent rebuild for people
using Breeze and static checks.
(cherry picked from commit 167b9b9889ac5481b21cb35c6cdef5869b8ab713)
---
Dockerfile.ci | 9 +-
breeze | 6 +-
manifests/.gitignore| 2 +-
scripts/ci/libraries/_build_images.sh | 163
scripts/ci/libraries/_initialization.sh | 19 +++-
5 files changed, 106 insertions(+), 93 deletions(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 7699706..f06087b 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -132,7 +132,9 @@ ARG RUNTIME_APT_DEPS="\
sqlite3 \
tmux \
unzip \
- vim"
+ vim \
+ xxd"
+ENV RUNTIME_APT_DEP=${RUNTIME_APT_DEPS}
ARG ADDITIONAL_RUNTIME_APT_DEPS=""
ENV ADDITIONAL_RUNTIME_APT_DEPS=${ADDITIONAL_RUNTIME_APT_DEPS}
@@ -275,6 +277,11 @@ RUN if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" ]];
then \
fi
+# Generate random hex dump file so that we can determine whether it's faster
to rebuild the image
+# using current cache (when our dump is the same as the remote onb) or better
to pull
+# the new image (when it is different)
+RUN head -c 30 /dev/urandom | xxd -ps >/build-cache-hash
+
# Link dumb-init for backwards compatibility (so that older images also work)
RUN ln -sf /usr/bin/dumb-init /usr/local/bin/dumb-init
diff --git a/breeze b/breeze
index 55412d0..175a4ab 100755
--- a/breeze
+++ b/breeze
@@ -2873,11 +2873,9 @@ function breeze::run_build_command() {
build_images::prepare_prod_build
build_images::build_prod_images
else
+
build_images::prepare_ci_build
-md5sum::calculate_md5sum_for_all_files
-build_images::build_ci_image
-md5sum::update_all_md5
-build_images::build_ci_image_manifest
+build_images::rebuild_ci_image_if_needed
fi
;;
cleanup_image | run_exec)
diff --git a/manifests/.gitignore b/manifests/.gitignore
index a6c57f5..72e8ffc 100644
--- a/manifests/.gitignore
+++ b/manifests/.gitignore
@@ -1 +1 @@
-*.json
+*
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index a2577cb..52b8a84 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -136,10 +136,18 @@ function build_images::confirm_image_rebuild() {
;;
esac
elif [[ -t 0 ]]; then
+echo
+echo
+echo "Make sure that you rebased to latest master before rebuilding!"
+echo
# Check if this script is run interactively with stdin open and
terminal attached
"${AIRFLOW_SOURCES}/confirm" "${ACTION} image
${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}"
RES=$?
elif [[ ${DETECTED_TERMINAL:=$(tty)} != "not a tty" ]]; then
+echo > "${DETECTED_TERMINAL}"
+echo > "${DETECTED_TERMINAL}"
+echo "Make sure that you rebased to latest master before rebuilding!"
> "${DETECTED_TERMINAL}"
+echo > "${DETECTED_TERMINAL}"
# Make sure to use output of tty rather than stdin/stdout when
available - this way confirm
# will works also in case of pre-commits (git does not pass
stdin/stdout to pre-commit hooks)
# shellcheck disable=SC2094
@@ -151,6 +159,10 @@ function build_images::confirm_image_rebuild() {
export DETECTED_TERMINAL=/dev/tty
# Make sure to use /dev/tty first rather than stdin/stdout when
available - this way confirm
# will works also in case of pre-commits (git does not pass
stdin/stdout to pre-commit hooks)
+echo > "${DETECTED_TERMINAL}"
+echo > &qu
[airflow] 19/44: If we build a new image, we should run more than basic checks (#12070)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6f64e98543f230f23b87d3adef72d2a29d5a9b40
Author: Ash Berlin-Taylor
AuthorDate: Tue Nov 3 17:42:01 2020 +
If we build a new image, we should run more than basic checks (#12070)
This lead to bases such as in #11699 where despite there being changes,
and an image being build, the pre-commit tests were not being run.
(cherry picked from commit 8000ab7375d319e389b6ea973acf8b1085c46bb1)
---
scripts/ci/selective_ci_checks.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/ci/selective_ci_checks.sh
b/scripts/ci/selective_ci_checks.sh
index 0faa541..b9a6d74 100755
--- a/scripts/ci/selective_ci_checks.sh
+++ b/scripts/ci/selective_ci_checks.sh
@@ -450,7 +450,6 @@ if [[ ${image_build_needed} == "true" ]]; then
else
set_basic_checks_only "true"
fi
-set_basic_checks_only "${image_build_needed}"
set_docs_build "${docs_build_needed}"
run_tests "${tests_needed}"
run_kubernetes_tests "${kubernetes_tests_needed}"
[airflow] 32/44: Fixes timeout in helm chart tests (#12209)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 6e74f1b6d0c3d188aed013633c0666955e07e14a Author: Jarek Potiuk AuthorDate: Mon Nov 9 21:18:58 2020 +0100 Fixes timeout in helm chart tests (#12209) (cherry picked from commit e77867aeab1899a87e3d212ff69ccae0a8447adb) --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c217bdc..6c854a1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -281,7 +281,7 @@ jobs: path: "./files/documentation" tests-helm: -timeout-minutes: 5 +timeout-minutes: 20 name: "Python unit tests for helm chart" runs-on: ubuntu-latest needs: [build-info, ci-images]
[airflow] 41/44: Deploy was not working from Breeze (#12319)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 891a01d0ee3b68c8a91b60c6f82fe907aac0c4b1
Author: Jarek Potiuk
AuthorDate: Thu Nov 12 19:53:35 2020 +0100
Deploy was not working from Breeze (#12319)
The get_cluster_name was called twice resulting in redonly
error after rebasing/fixing CI failure in #12163.
This PR is fxing it.
(cherry picked from commit af19b126e94876c371553f6a7cfae6b1102f79fd)
---
scripts/ci/libraries/_kind.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index defa4de..3a170cb 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -173,7 +173,6 @@ function kind::perform_kind_cluster_operation() {
echo "Deploying Airflow to KinD"
echo
kind::build_image_for_kubernetes_tests
-kind::get_kind_cluster_name
kind::load_image_to_kind_cluster
kind::deploy_airflow_with_helm
kind::deploy_test_kubernetes_resources
[airflow] 31/44: Fixed path of the test_core.py file in docs (#12191)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 10700af913b3415a97acc6bd7161dd1f2a3c2885
Author: Jarek Potiuk
AuthorDate: Mon Nov 9 10:34:06 2020 +0100
Fixed path of the test_core.py file in docs (#12191)
The test_core.py has been used as example in Breeze and it's
location changed to tests/core folder. This PR fixes references
to the changed location.
(cherry picked from commit 57b273a0b1b8af30ed017c2b24c498deb9010247)
---
BREEZE.rst | 2 +-
TESTING.rst | 8
breeze | 2 +-
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 5e481f7..b66c7d2 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1894,7 +1894,7 @@ This is the current syntax for `./breeze <./breeze>`_:
as additional options passed to pytest. You can pass 'tests' as target
to
run all tests. For example:
-'breeze tests tests/test_core.py -- --logging-level=DEBUG'
+'breeze tests tests/core/test_core.py -- --logging-level=DEBUG'
'breeze tests tests
Flags:
diff --git a/TESTING.rst b/TESTING.rst
index f4f790b..c8b170b 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -111,20 +111,20 @@ This can also be done by specifying a full path to the
test:
.. code-block:: bash
-pytest tests/test_core.py::TestCore::test_check_operators
+pytest tests/core/test_core.py::TestCore::test_check_operators
To run the whole test class, enter:
.. code-block:: bash
-pytest tests/test_core.py::TestCore
+pytest tests/core/test_core.py::TestCore
You can use all available ``pytest`` flags. For example, to increase a log
level
for debugging purposes, enter:
.. code-block:: bash
-pytest --log-level=DEBUG tests/test_core.py::TestCore
+pytest --log-level=DEBUG tests/core/test_core.py::TestCore
Running Tests for a Specified Target Using Breeze from the Host
@@ -149,7 +149,7 @@ You can also specify individual tests or a group of tests:
.. code-block:: bash
-./breeze tests --db-reset tests/test_core.py::TestCore
+./breeze tests --db-reset tests/core/test_core.py::TestCore
Running Tests of a specified type from the Host
diff --git a/breeze b/breeze
index 1d8ffea..3498c64 100755
--- a/breeze
+++ b/breeze
@@ -1763,7 +1763,7 @@ ${CMDNAME} tests [FLAGS] [TEST_TARGET ..] [--
]
as additional options passed to pytest. You can pass 'tests' as target to
run all tests. For example:
- '${CMDNAME} tests tests/test_core.py -- --logging-level=DEBUG'
+ '${CMDNAME} tests tests/core/test_core.py -- --logging-level=DEBUG'
'${CMDNAME} tests tests
Flags:
[airflow] 37/44: Fixes continuous image rebuilding with Breeze (#12256)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4ca85a6a315c5c46aad95a67482e09de707c0293
Author: Jarek Potiuk
AuthorDate: Tue Nov 10 17:34:52 2020 +0100
Fixes continuous image rebuilding with Breeze (#12256)
There was a problem that even if we pulled the right image
from the Airflow repository, we have not tagged it properly.
Also added protection for people who have not yet at all pulled
the Python image from airflow, to force pull for the first time.
(cherry picked from commit 09febee4c1caf4e430fcbad2386f80aab52a0b15)
---
scripts/ci/libraries/_push_pull_remove_images.sh | 6 ++
1 file changed, 6 insertions(+)
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh
b/scripts/ci/libraries/_push_pull_remove_images.sh
index e853f42..4c51f0a 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -105,6 +105,11 @@ function
push_pull_remove_images::pull_image_github_dockerhub() {
# Pulls CI image in case caching strategy is "pulled" and the image needs to
be pulled
function push_pull_remove_images::pull_ci_images_if_needed() {
if [[ "${DOCKER_CACHE}" == "pulled" ]]; then
+local python_image_hash
+python_image_hash=$(docker images -q "${AIRFLOW_CI_PYTHON_IMAGE}" 2>
/dev/null || true)
+if [[ -z "${python_image_hash=}" ]]; then
+FORCE_PULL_IMAGES="true"
+fi
if [[ "${FORCE_PULL_IMAGES}" == "true" ]]; then
echo
echo "Force pull base image ${PYTHON_BASE_IMAGE}"
@@ -122,6 +127,7 @@ Docker pulling ${PYTHON_BASE_IMAGE}.
push_pull_remove_images::pull_image_github_dockerhub
"${PYTHON_BASE_IMAGE}"
"${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
else
docker pull "${AIRFLOW_CI_PYTHON_IMAGE}"
+docker tag "${AIRFLOW_CI_PYTHON_IMAGE}" "${PYTHON_BASE_IMAGE}"
fi
echo
fi
[airflow] 25/44: Work properly if some variables are not defined (#12135)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 38d55820057aa4ae9f3750a819435b2193406e6e
Author: Jarek Potiuk
AuthorDate: Fri Nov 6 16:56:43 2020 +0100
Work properly if some variables are not defined (#12135)
Those variables are defined in GitHub environment so when they
were recently addded it was not obvious that they will fail when
running kubernetes tests locally.
This PR fixes that.
(cherry picked from commit 5351f0d99605738b5efa0134bb51eb2dbb6eba46)
---
scripts/ci/libraries/_initialization.sh | 4 +++-
scripts/ci/libraries/_kind.sh | 2 +-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/libraries/_initialization.sh
b/scripts/ci/libraries/_initialization.sh
index 316a92c..e0b4a8a 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -771,7 +771,9 @@ function initialization::ga_output() {
}
function initialization::ga_env() {
-echo "${1}=${2}" >> "${GITHUB_ENV}"
+if [[ ${GITHUB_ENV=} != "" ]]; then
+echo "${1}=${2}" >> "${GITHUB_ENV}"
+fi
}
function initialization::set_mysql_encoding() {
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index 451c370..41048fc 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -302,7 +302,7 @@ function kind::forward_port_to_kind_webserver() {
echo
echo "Trying to establish port forwarding to 'airflow webserver'"
echo
-if [[ ${INCREASE_PORT_NUMBER_FOR_KUBERNETES} == "true" ]] ; then
+if [[ ${INCREASE_PORT_NUMBER_FOR_KUBERNETES=} == "true" ]] ; then
forwarded_port_number=$(( forwarded_port_number + 1 ))
fi
if [[ ${num_tries} == "${MAX_NUM_TRIES_FOR_PORT_FORWARD}" ]]; then
[airflow] 22/44: Fix proper SHA in check preventing accidentally merging PR (#12083)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 64393ad508119197907f8a7aa61afa82dd506095
Author: Jarek Potiuk
AuthorDate: Wed Nov 4 13:51:04 2020 +0100
Fix proper SHA in check preventing accidentally merging PR (#12083)
The SHA in check was not working for PRs from forks.
(cherry picked from commit d559da14b0a99b369ba027b4b6261321b6b8e03c)
---
.github/workflows/label_when_reviewed_workflow_run.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/label_when_reviewed_workflow_run.yml
b/.github/workflows/label_when_reviewed_workflow_run.yml
index 56bc835..cdcbca5 100644
--- a/.github/workflows/label_when_reviewed_workflow_run.yml
+++ b/.github/workflows/label_when_reviewed_workflow_run.yml
@@ -93,7 +93,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
name: "Please rebase or re-run to run full tests"
status: "in_progress"
- sha: ${{ github.event.pull_request.head.sha }}
+ sha: ${{ steps.source-run-info.outputs.sourceHeadSha }}
details_url: https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }}
output: >
{"summary":
[airflow] 21/44: Use sys.exit() instead of exit() (#12084)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2be4d56a78b2b49b08866ac9114a89ab5db7eef3
Author: Kaxil Naik
AuthorDate: Wed Nov 4 11:50:52 2020 +
Use sys.exit() instead of exit() (#12084)
The `exit` and `quit` functions are actually `site.Quitter` objects and are
loaded, at interpreter start up, from site.py. However, if the interpreter is
started with the `-S` flag, or a custom site.py is used then exit and quit may
not be present. It is recommended to use `sys.exit()` which is built into the
interpreter and is guaranteed to be present.
(cherry picked from commit bec9f3b29fd42ecd1beae3db75784b9a726caf15)
---
scripts/ci/pre_commit/pre_commit_check_setup_installation.py | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
index 2fdeca6..b4f3281 100755
--- a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
+++ b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
@@ -22,6 +22,7 @@ Checks if all the libraries in setup.py are listed in
installation.rst file
import os
import re
+import sys
from os.path import dirname
from typing import Dict, List
@@ -90,8 +91,8 @@ if __name__ == '__main__':
if f"'{extras}'" not in setup_packages_str:
output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras, "",
"V")
-if(output_table == ""):
-exit(0)
+if output_table == "":
+sys.exit(0)
print(f"""
ERROR
@@ -105,4 +106,4 @@ documented although not used.
print(".{:_^22}.{:_^12}.{:_^12}.".format("NAME", "SETUP", "INSTALLATION"))
print(output_table)
-exit(1)
+sys.exit(1)
[airflow] 30/44: Adds extra check while the selective checks are run (#12178)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 20512aff626a04a418ccef5861b6856ac0c3e215
Author: Jarek Potiuk
AuthorDate: Sun Nov 8 13:25:32 2020 +0100
Adds extra check while the selective checks are run (#12178)
The selective checks are run in "workflow_run" because
they need to be able to set label and make comments, however
status of those checks are not displayed in GitHub and in
cases of small PRs the "merge" button might be green before
the status complete.
This PR adds additional check that is always completed after
the "worfklow_run" finishes it's job. This will prevent
accidental merges before the check completes.
(cherry picked from commit 63ac07d9c735b1ccc8aa5fa974a260cc944cc539)
---
.../workflows/label_when_reviewed_workflow_run.yml | 29 ++
1 file changed, 29 insertions(+)
diff --git a/.github/workflows/label_when_reviewed_workflow_run.yml
b/.github/workflows/label_when_reviewed_workflow_run.yml
index cdcbca5..c2359c8 100644
--- a/.github/workflows/label_when_reviewed_workflow_run.yml
+++ b/.github/workflows/label_when_reviewed_workflow_run.yml
@@ -35,6 +35,20 @@ jobs:
with:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
+ - name: Initiate Selective Build check
+uses:
LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 # v1.1.0
+id: selective-build-check
+with:
+ token: ${{ secrets.GITHUB_TOKEN }}
+ name: "Selective build check"
+ status: "in_progress"
+ sha: ${{ steps.source-run-info.outputs.sourceHeadSha }}
+ details_url: https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }}
+ output: >
+{"summary":
+"Checking selective status of the build in
+[the run](https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }})
+"}
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: >
@@ -125,3 +139,18 @@ jobs:
pullRequestNumber: ${{
steps.source-run-info.outputs.pullRequestNumber }}
require_committers_approval: 'true'
comment: "The PR is ready to be merged. No tests are needed!"
+ - name: Update Selective Build check
+uses:
LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 # v1.1.0
+if: always()
+with:
+ token: ${{ secrets.GITHUB_TOKEN }}
+ check_id: ${{ steps.selective-build-check.outputs.check_id }}
+ status: "completed"
+ sha: ${{ steps.source-run-info.outputs.sourceHeadSha }}
+ conclusion: ${{ job.status }}
+ details_url: https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }}
+ output: >
+{"summary":
+"Checking selective status of the build completed in
+[the run](https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }})
+"}
[airflow] 35/44: Beautify Output of setup-installation pre-commit (#12218)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit eb7d914492d580dd06c05b5c114b9f9792b6de38
Author: Kaxil Naik
AuthorDate: Tue Nov 10 00:54:47 2020 +
Beautify Output of setup-installation pre-commit (#12218)
(cherry picked from commit 08d67add52c29f9f29f6e786e2f4e6591cdf)
---
.pre-commit-config.yaml| 1 +
.../pre_commit_check_setup_installation.py | 29 ++
2 files changed, 19 insertions(+), 11 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index cb5c753..e626cd4 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -219,6 +219,7 @@ repos:
files: ^setup.py$|^docs/installation.rst$
pass_filenames: false
entry: ./scripts/ci/pre_commit/pre_commit_check_setup_installation.py
+additional_dependencies: ['rich==9.2.0']
- id: update-breeze-file
name: Update output of breeze command in BREEZE.rst
entry: "./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.sh"
diff --git a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
index b4f3281..e386461 100755
--- a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
+++ b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
@@ -26,6 +26,10 @@ import sys
from os.path import dirname
from typing import Dict, List
+from rich import print as rprint
+from rich.console import Console
+from rich.table import Table
+
AIRFLOW_SOURCES_DIR = os.path.join(dirname(__file__), os.pardir, os.pardir,
os.pardir)
SETUP_PY_FILE = 'setup.py'
DOCS_FILE = 'installation.rst'
@@ -80,30 +84,33 @@ if __name__ == '__main__':
setup_packages = get_extras_from_setup()
docs_packages = get_extras_from_docs()
-output_table = ""
+table = Table()
+table.add_column("NAME", justify="right", style="cyan")
+table.add_column("SETUP", justify="center", style="magenta")
+table.add_column("INSTALLATION", justify="center", style="green")
for extras in sorted(setup_packages.keys()):
if not set(setup_packages[extras]).intersection(docs_packages):
-output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras,
"V", "")
+table.add_row(extras, "V", "")
setup_packages_str = str(setup_packages)
for extras in sorted(docs_packages):
if f"'{extras}'" not in setup_packages_str:
-output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras, "",
"V")
+table.add_row(extras, "", "V")
-if output_table == "":
+if table.row_count == 0:
sys.exit(0)
-print(f"""
-ERROR
+rprint(f"""\
+[red bold]ERROR!![/red bold]
-"EXTRAS_REQUIREMENTS" section in {SETUP_PY_FILE} should be synchronized
-with "Extra Packages" section in documentation file doc/{DOCS_FILE}.
+"EXTRAS_REQUIREMENTS" section in [bold yellow]{SETUP_PY_FILE}[/bold yellow]
should be synchronized
+with "Extra Packages" section in documentation file [bold
yellow]doc/{DOCS_FILE}[/bold yellow].
-here is a list of packages that are used but are not documented, or
+Here is a list of packages that are used but are not documented, or
documented although not used.
""")
-print(".{:_^22}.{:_^12}.{:_^12}.".format("NAME", "SETUP", "INSTALLATION"))
-print(output_table)
+console = Console()
+console.print(table)
sys.exit(1)
[airflow] 28/44: Fixes "--force-clean-images" flag in Breeze (#12156)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 815f18b06c8711208971d9766350856d892b695c
Author: Jarek Potiuk
AuthorDate: Sat Nov 7 13:51:01 2020 +0100
Fixes "--force-clean-images" flag in Breeze (#12156)
The flag was broken - bad cache parameter value was passed.
This PR fixes it.
(cherry picked from commit 5c601578194739286097a75f082af2d83539581f)
---
breeze| 2 +-
scripts/ci/libraries/_build_images.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/breeze b/breeze
index 7838c88..1d8ffea 100755
--- a/breeze
+++ b/breeze
@@ -884,7 +884,7 @@ function breeze::parse_arguments() {
-C | --force-clean-images)
echo "Clean build of images without cache"
echo
-export DOCKER_CACHE="no-cache"
+export DOCKER_CACHE="disabled"
# if not set here, docker cached is determined later, depending on
type of image to be build
readonly DOCKER_CACHE
export FORCE_BUILD_IMAGES="true"
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index a6dbae0..cbd01c1 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -536,7 +536,7 @@ function build_images::build_ci_image() {
)
else
echo >&2
-echo >&2 "Error - thee ${DOCKER_CACHE} cache is unknown!"
+echo >&2 "Error - the ${DOCKER_CACHE} cache is unknown!"
echo >&2
exit 1
fi
[airflow] 39/44: Added k9s as integrated tool to help with kubernetes testing (#12163)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d8db36a1b1bee478af4b4ca3e8ac6471da557a96
Author: Jarek Potiuk
AuthorDate: Wed Nov 11 17:15:02 2020 +0100
Added k9s as integrated tool to help with kubernetes testing (#12163)
The K9s is fantastic tool that helps to debug a running k8s
instance. It is terminal-based windowed CLI that makes you
several times more productive comparing to using kubectl
commands. We've integrated k9s (it is run as a docker container
and downloaded on demand). We've also separated out KUBECONFIG
of the integrated kind cluster so that it does not mess with
kubernetes configuration you might already have.
Also - together with that the "surrounding" of the kubernetes
tests were simplified and improved so that the k9s integration
can be utilized well. Instead of kubectl port forwarding (which
caused multitude of problems) we are now utilizing kind's
portMapping feature + custom NodePort resource that maps
port 8080 to 30007 NodePort which in turn maps it to 8080
port of the Webserver. This way we do not have to establish
an external kubectl port forward which is prone to error and
management - everything is brought up when Airflow gets
deployed to the Kind Cluster and shuts down when the Kind
cluster is stopped.
Yet another problem fixed was killing of postgres by one of the
kubernetes tests ('test_integration_run_dag_with_scheduler_failure').
Instead of just killing the scheduler it killed all pods - including
the Postgres one (it was named 'airflow-postgres.*'). That caused
various problems, as the database could be left in a strange state.
I changed the tests to do what it claimed was doing - so killing only the
scheduler during the test. This seemed to improve the stability
of tests immensely in my local setup.
(cherry picked from commit 21999dd56e9dbe9f7f9e25961954c5677c3c7c58)
---
.github/workflows/ci.yml | 17 +-
BREEZE.rst | 11 +-
TESTING.rst| 151 -
breeze | 9 +
breeze-complete| 5 +-
chart/requirements.lock| 4 +-
images/testing/k9s.png | Bin 0 -> 238713 bytes
images/testing/kubeconfig-env.png | Bin 0 -> 231280 bytes
images/testing/kubernetes-virtualenv.png | Bin 0 -> 110011 bytes
images/testing/pytest-runner.png | Bin 0 -> 131589 bytes
images/testing/run-test.png| Bin 0 -> 140728 bytes
kubernetes_tests/test_kubernetes_executor.py | 7 +-
kubernetes_tests/test_kubernetes_pod_operator.py | 672 -
scripts/ci/kubernetes/ci_run_kubernetes_tests.sh | 7 +-
...up_cluster_and_deploy_airflow_to_kubernetes.sh} | 3 +-
scripts/ci/kubernetes/kind-cluster-conf.yaml | 5 +
.../{kind-cluster-conf.yaml => nodeport.yaml} | 30 +-
...oy_app_to_kubernetes.sh => redeploy_airflow.sh} | 6 +-
scripts/ci/libraries/_kind.sh | 126 ++--
19 files changed, 516 insertions(+), 537 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 6c854a1..81890a7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -636,23 +636,14 @@ jobs:
python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
- name: "Free space"
run: ./scripts/ci/tools/ci_free_space_on_ci.sh
- - name: "Setup Kind Cluster ${{ env.KIND_VERSION }}"
-uses: engineerd/setup-kind@v0.4.0
-with:
- version: "${{ env.KIND_VERSION }}"
- name:
airflow-python-${{matrix.python-version}}-${{matrix.kubernetes-version}}
- config: "scripts/ci/kubernetes/kind-cluster-conf.yaml"
- name: "Prepare PROD Image"
run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
- - name: "Deploy airflow to cluster"
-id: deploy-app
-run: ./scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh
+ - name: "Setup cluster and deploy Airflow"
+id: setp-cluster-deploy-app
+run:
./scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
env:
# We have the right image pulled already by the previous step
SKIP_BUILDING_PROD_IMAGE: "true"
- # due to some instabilities, in CI we try to increase port numbers
when trying to establish
- # port forwarding
- INCREASE_PORT_NUMBER_FOR_KUBERNETES: "true"
- name: "Cache
[airflow] 42/44: For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4ab76df0eb547fc2af670473d38536ab9b1774b1
Author: Jarek Potiuk
AuthorDate: Fri Nov 13 14:28:36 2020 +0100
For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
Previously, always master scripts were used to build images
for workflow_run, because workflow_run always runs from master
branch. However that causes some surprising effects becuase the
sripts from master had to support both master and 1.10.
This change utilises a new feature in the "get-workflow-origin"
action - to get the target branch of PR and uses ci scripts from that
target branch.
This is perfectly secure, because both v1-10-test, v1-10-stable
and future 2-0 branches can only be updated by committers,
either by direct push or by merge.
(cherry picked from commit 7c4fe19e41ae02a1df1c0a217501cae2e0e84819)
---
.github/workflows/build-images-workflow-run.yml| 12 +--
.github/workflows/ci.yml | 38 +++---
.../workflows/label_when_reviewed_workflow_run.yml | 2 +-
3 files changed, 22 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index a3277a7..af71710 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -53,13 +53,14 @@ jobs:
targetCommitSha: ${{ steps.source-run-info.outputs.targetCommitSha }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
+ targetBranch: ${{ steps.source-run-info.outputs.targetBranch }}
sourceEvent: ${{ steps.source-run-info.outputs.sourceEvent }}
cacheDirective: ${{ steps.cache-directive.outputs.docker-cache }}
buildImages: ${{ steps.build-images.outputs.buildImages }}
upgradeToLatestConstraints: ${{
steps.upgrade-constraints.outputs.upgradeToLatestConstraints }}
steps:
- name: "Get information about the original trigger of the run"
- uses:
potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+ uses:
potiuk/get-workflow-origin@588cc14f9f1cdf1b8be3db816855e96422204fec # v1_3
id: source-run-info
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -326,17 +327,22 @@ jobs:
[Image Build](https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }})
for details" }
if: steps.defaults.outputs.proceed == 'true'
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} ) to
'main-airflow' to use main scripts"
+ - name: >
+ Checkout "${{ needs.cancel-workflow-runs.outputs.targetBranch }}"
branch to 'main-airflow' folder
+ to use ci/scripts from there.
uses: actions/checkout@v2
with:
path: "main-airflow"
+ ref: "${{ needs.cancel-workflow-runs.outputs.targetBranch }}"
if: steps.defaults.outputs.proceed == 'true'
- name: "Setup python"
uses: actions/setup-python@v2
with:
python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
if: steps.defaults.outputs.proceed == 'true'
- - name: "Override 'scripts/ci' with the ${{ github.ref }} version so
that the PR cannot override it."
+ - name: >
+ Override "scripts/ci" with the "${{
needs.cancel-workflow-runs.outputs.targetBranch }}" branch
+ so that the PR does not override it
# We should not override those scripts which become part of the image
as they will not be
# changed in the image built - we should only override those that are
executed to build
# the image.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 81890a7..df5f53e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -95,27 +95,15 @@ jobs:
image-build: ${{ steps.selective-checks.outputs.image-build }}
docs-build: ${{ steps.selective-checks.outputs.docs-build }}
needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
- needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
steps:
- name: "Get information about the PR"
-uses:
potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+uses:
potiuk/ge
[airflow] 38/44: Docker context files should be available earlier (#12219)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 10b6c48b67ec2c23b256897a6e5b562178a1eb9b
Author: Jarek Potiuk
AuthorDate: Wed Nov 11 11:00:16 2020 +0100
Docker context files should be available earlier (#12219)
If you want to override constraints with local version,
the docker-context-files should be earlier in the Dockerfile
(cherry picked from commit 9b7e7603c4a01c6c943fa68df3b20a8859cdde1e)
---
.gitignore| 2 ++
Dockerfile.ci | 8
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/.gitignore b/.gitignore
index 27b6f0c..6b2650c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -188,6 +188,8 @@ dmypy.json
log.txt*
/backport_packages/CHANGELOG.txt
+# Docker context files
+/docker-context-files
# Local .terraform directories
**/.terraform/*
diff --git a/Dockerfile.ci b/Dockerfile.ci
index fae86b2..7699706 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -260,6 +260,10 @@ ENV AIRFLOW_LOCAL_PIP_WHEELS=${AIRFLOW_LOCAL_PIP_WHEELS}
ARG INSTALL_AIRFLOW_VIA_PIP="true"
ENV INSTALL_AIRFLOW_VIA_PIP=${INSTALL_AIRFLOW_VIA_PIP}
+# If wheel files are found in /docker-context-files during installation
+# they are also installed additionally to whatever is installed from Airflow.
+COPY docker-context-files /docker-context-files
+
# In case of CI builds we want to pre-install master version of airflow
dependencies so that
# We do not have to always reinstall it from the scratch.
# This can be reinstalled from latest master by increasing
PIP_DEPENDENCIES_EPOCH_NUMBER.
@@ -311,10 +315,6 @@ RUN if [[ ${INSTALL_AIRFLOW_VIA_PIP} == "true" ]]; then \
fi; \
fi
-# If wheel files are found in /docker-context-files sduring installation
-# they are also installed additionally to whatever is installed from Airflow.
-COPY docker-context-files /docker-context-files
-
RUN if [[ ${AIRFLOW_LOCAL_PIP_WHEELS} != "true" ]]; then \
if ls /docker-context-files/*.whl 1> /dev/null 2>&1; then \
pip install --no-deps /docker-context-files/*.whl; \
[airflow] 36/44: Add Markdown linting to pre-commit (#11465)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 64af9450e37faeb5b08e9568938d6cdf2d214f6d Author: John Bampton AuthorDate: Tue Nov 10 12:37:45 2020 +1000 Add Markdown linting to pre-commit (#11465) (cherry picked from commit 7463b6bcc7a823a9d02c7026d9b03b81c6421654) --- .markdownlint.yml | 77 +++ .pre-commit-config.yaml | 32 +- BREEZE.rst |8 +- STATIC_CODE_CHECKS.rst |4 +- breeze-complete |2 + chart/README.md |2 +- dev/README.md | 1522 --- 7 files changed, 1552 insertions(+), 95 deletions(-) diff --git a/.markdownlint.yml b/.markdownlint.yml new file mode 100644 index 000..dae8217 --- /dev/null +++ b/.markdownlint.yml @@ -0,0 +1,77 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +# MD003/heading-style/header-style +MD003: false + +# MD004/ul-style +MD004: false + +# MD007/ul-indent +MD007: false + +# MD012/no-multiple-blanks +MD012: false + +# MD013 Line length +MD013: false + +# MD014/commands-show-output +MD014: false + +# MD022/blanks-around-headings/blanks-around-headers +MD022: false + +# MD024/no-duplicate-heading/no-duplicate-header +MD024: false + +# MD026/no-trailing-punctuation +MD026: false + +# MD029/ol-prefix +MD029: false + +# MD030/list-marker-space +MD030: false + +# MD031/blanks-around-fences +MD031: false + +# MD032/blanks-around-lists +MD032: false + +# MD033/no-inline-html +MD033: false + +# MD034/no-bare-urls +MD034: false + +# MD036/no-emphasis-as-heading/no-emphasis-as-header +MD036: false + +# MD040/fenced-code-language +MD040: false + +# MD041/first-line-heading/first-line-h1 +MD041: false + +# MD045/no-alt-text +MD045: false + +# MD046/code-block-style +MD046: false diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index e626cd4..4dbf84c 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -333,6 +333,26 @@ repos: files: ^breeze$|^breeze-complete$|\.sh$|\.bash$|\.bats$ exclude: ^tests/bats/in_container/.*bats$|^scripts/in_container/.*sh$ pass_filenames: false + - id: pre-commit-descriptions +name: Check if pre-commits are described +entry: ./scripts/ci/pre_commit/pre_commit_check_pre_commits.sh +language: system +files: ^.pre-commit-config.yaml$|^STATIC_CODE_CHECKS.rst|^breeze-complete$ +require_serial: true + - id: sort-in-the-wild +name: Sort INTHEWILD.md alphabetically +entry: ./scripts/ci/pre_commit/pre_commit_sort_in_the_wild.sh +language: system +files: ^.pre-commit-config.yaml$|^INTHEWILD.md$ +require_serial: true + - id: markdownlint +name: Run markdownlint +description: "Checks the style of Markdown files." +entry: markdownlint +language: node +types: [markdown] +files: \.(md|mdown|markdown)$ +additional_dependencies: ['markdownlint-cli'] - id: build name: Check if image build is needed entry: ./scripts/ci/pre_commit/pre_commit_ci_build.sh 3.6 false @@ -364,15 +384,3 @@ repos: entry: "./scripts/ci/pre_commit/pre_commit_in_container_bats_test.sh" files: ^tests/bats/in_container/.*.bats$|^scripts/in_container/.*sh pass_filenames: false - - id: pre-commit-descriptions -name: Check if pre-commits are described -entry: ./scripts/ci/pre_commit/pre_commit_check_pre_commits.sh -language: system -files: ^.pre-commit-config.yaml$|^STATIC_CODE_CHECKS.rst|^breeze-complete$ -require_serial: true - - id: sort-in-the-wild -name: Sort INTHEWILD.md alphabetically -entry: ./scripts/ci/pre_commit/pre_commit_sort_in_the_wild.sh -language: system -files: ^.pre-commit-config.yaml$|^INTHEWILD.md$ -require_serial: true diff --git a/BREEZE.rst b/BREEZE.rst index b66c7d2..c3c2d95 100644 --- a/BREEZE.rst +++ b/BREEZE.rst @@ -1863,10 +1863,10 @@ This is the current syntax for `./breeze
[airflow] 40/44: Python base image is shared between CI and PROD image (#12280)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 76f51e15fd4afbaad77b22f01768cc1a795e988b
Author: Jarek Potiuk
AuthorDate: Thu Nov 12 12:31:14 2020 +0100
Python base image is shared between CI and PROD image (#12280)
When you are building CI images locally you use the CI
base images from apache:airflow/python* now to maintain
consistency and avoid often rebuilds. But when you build
prod images, you would accidentaly override it with the
python base image available in python repo which might be
different (newer and not yet tested in CI). This PR
changes it to use the same base image which is now
tagged in Apache Airflow's dockerhub repository.
(cherry picked from commit 3c2c29187afffb1a1dd1327803097651cdc079ee)
---
scripts/ci/libraries/_push_pull_remove_images.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh
b/scripts/ci/libraries/_push_pull_remove_images.sh
index 4c51f0a..7c65db1 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -154,7 +154,8 @@ function
push_pull_remove_images::pull_prod_images_if_needed() {
fi
push_pull_remove_images::pull_image_github_dockerhub
"${PYTHON_BASE_IMAGE}"
"${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
else
-docker pull "${PYTHON_BASE_IMAGE}"
+docker pull "${AIRFLOW_CI_PYTHON_IMAGE}"
+docker tag "${AIRFLOW_CI_PYTHON_IMAGE}" "${PYTHON_BASE_IMAGE}"
fi
echo
fi
[airflow] branch v1-10-test updated (0b8bb3e -> 00f320f)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. omit 0b8bb3e Synchronize INTHEWILD.md with master omit 09fe8f1 Simplifies check whether the CI image should be rebuilt (#12181) omit a950aac For v1-10-test PRs and pushes, use target branch scripts for images (#12339) omit 2f59506 Deploy was not working from Breeze (#12319) omit ca91871 Python base image is shared between CI and PROD image (#12280) omit 330e4d3 Added k9s as integrated tool to help with kubernetes testing (#12163) omit f1b8c7e Docker context files should be available earlier (#12219) omit b40568e Fixes continuous image rebuilding with Breeze (#12256) omit 07298ba Add Markdown linting to pre-commit (#11465) omit c89257c Beautify Output of setup-installation pre-commit (#12218) omit 7d40471 Fix permissions of mounted /tmp directory for Breeze (#12157) omit 7eac201 Remove popd which is a remnant from past (#12211) omit 4f8473d Fixes timeout in helm chart tests (#12209) omit 330f0b9 Fixed path of the test_core.py file in docs (#12191) omit 6370a91 Adds extra check while the selective checks are run (#12178) omit c7d64d5 Uses always the same Python base image as used for CI image (#12177) omit ae033b9 Fixes "--force-clean-images" flag in Breeze (#12156) omit 67f2c97 Fixes undefined variables (#12155) omit 4166311 Update to new helm stable repo (#12137) omit 61346ca Work properly if some variables are not defined (#12135) omit 5914d12 Add Kubernetes files to selective checks (#12114) omit 4c8f860 Update install_mysql.sh (#12101) omit 96f0a77 Fix proper SHA in check preventing accidentally merging PR (#12083) omit 6b1078b Use sys.exit() instead of exit() (#12084) omit b1b608e Fixes problem with building a PROD image (#12080) omit cdc26fc If we build a new image, we should run more than basic checks (#12070) omit ea58242 Adds a forgotten word in a README.md (#12066) omit 4e0b9fe Uses DOCKER_TAG when building image in DockerHub (#12050) omit 5ab6cd5 Fixes documentation-only selective checks (#12038) omit 56727d0 Checks if all the libraries in setup.py are listed in installation.rst file (#12023) omit 46d778a Turns failure of PR label when approved action into warning (#12017) omit 56d5960 Fix canceling of CodeQL workflow (#12024) omit d10eece Fixes problem with non-iterable data returned by GH API (#12021) omit 86c7b7b Revise "Project Focus" copy (#12011) omit 9051a87 Adds more aggressive cancelling of duplicate Build Image jobs (#12018) omit 08c0275 Add contributor-targeted description of the PR workflow. (#12016) omit a8d602b Adds documentation about the optimized PR workflow (#12006) omit 310cbd6 Fix K8S CI job name rendering (#12007) omit c45c045 Switches to "cancel-all-duplicates' mode of cancelling. (#12004) omit 6454b94 Implements canceling of future duplicate runs (but the latest) (#11980) omit 4acdc8e Move Project focus and Principles higher in the README (#11973) omit b310082 Pin `kubernetes` to a max version of 11.0.0. (#11974) omit eb11556 Updated documentation for the CI with mermaid sequence diagrams (#10380) omit a13366d Migrate from helm-unittest to python unittest (#11827) new 572387d Migrate from helm-unittest to python unittest (#11827) new 132e899 Updated documentation for the CI with mermaid sequence diagrams (#10380) new d5e7b51 Pin `kubernetes` to a max version of 11.0.0. (#11974) new 6f6b560 Move Project focus and Principles higher in the README (#11973) new ad56790 Implements canceling of future duplicate runs (but the latest) (#11980) new d3ab3df Switches to "cancel-all-duplicates' mode of cancelling. (#12004) new 33a14e7 Fix K8S CI job name rendering (#12007) new 8084584 Adds documentation about the optimized PR workflow (#12006) new b2633ff Add contributor-targeted description of the PR workflow. (#12016) new 99aa883 Adds more aggressive cancelling of duplicate Build Image jobs (#12018) new 887c231 Revise "Project Focus" copy (#12011) new e18d675 Fixes problem with non-iterable data returned by GH API (#12021) new df68fb7 Fix canceling of CodeQL workflow (#12024) new 8e58291 Turns failure of PR label when approved action into warning (#12017) new 43c72f2 Checks if all the libraries in setup.py are listed in installation.rst file (#12023) new ad8aa65 Fixes documentation-only selective checks (#12038) new 7512344 Uses DOCKER_TAG when building image in DockerHub (#12050) new 26d001f Adds a forgotten word in a README.md (#12066) new 6f64e98 If we build a new image, we should run more than basic che
[airflow] 06/44: Switches to "cancel-all-duplicates' mode of cancelling. (#12004)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d3ab3df2eaad211f2ce24f73e354de4ec904716d
Author: Jarek Potiuk
AuthorDate: Sat Oct 31 23:39:54 2020 +0100
Switches to "cancel-all-duplicates' mode of cancelling. (#12004)
The cancel-workflow-runs action in version 4.1 got the capability
of cancelling not only duplicates of own run (including future,
queued duplicates) but also cancelling all duplicates from all
running worklfows - regardless if they were triggered by my own
PR or some other PRs. This will be even more helpful with handling
the queues and optimising our builds, because in case ANY of
the build image workflows starts to run, it will cancel ALL
duplicates immediately.
(cherry picked from commit 21350aa3cf8952b605713257ae94e1ed648dd00b)
---
.github/workflows/build-images-workflow-run.yml | 19 +--
1 file changed, 9 insertions(+), 10 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index 81c4fb4..972195a 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,12 +65,11 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
token: ${{ secrets.GITHUB_TOKEN }}
- cancelMode: duplicates
+ cancelMode: allDuplicates
sourceRunId: ${{ github.event.workflow_run.id }}
- notifyPRCancel: true
- name: "Output BUILD_IMAGES"
id: build-images
run: |
@@ -83,7 +82,7 @@ jobs:
# in GitHub Actions, we have to use Job names to match
Event/Repo/Branch from the
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -101,7 +100,7 @@ jobs:
# can cancel all the matching "Build Images" workflow runs in the two
following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -135,18 +134,18 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run
Id:
# followed by one of the run ids. Yes I know it's super complex
¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' &&
steps.source-run-info-failed.outputs.cancelledRuns != '[]'
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{
steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
- uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
- cancelMode: duplicates
+ cancelMode: allDuplicates
workflowFileName: 'codeql-analysis.yml'
- name: "Set Docker Cache Directive"
id: cache-directive
@@ -373,7 +372,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -388,7 +387,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
-uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
[airflow] 02/44: Updated documentation for the CI with mermaid sequence diagrams (#10380)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 132e899a1132988e99281215fd6d9e9c37bd0369 Author: Jarek Potiuk AuthorDate: Mon Aug 24 22:45:28 2020 +0200 Updated documentation for the CI with mermaid sequence diagrams (#10380) (cherry picked from commit f2da6b419fde71d335d133031f2c0fac28b87289) --- .pre-commit-config.yaml | 11 ++- .rat-excludes | 1 + images/ci/CI.png| Bin 0 -> 243004 bytes images/ci/pull_request_ci_flow.md5 | 1 + images/ci/pull_request_ci_flow.mermaid | 108 + images/ci/pull_request_ci_flow.png | Bin 0 -> 195487 bytes images/ci/push_ci_flow.md5 | 1 + images/ci/push_ci_flow.mermaid | 117 images/ci/push_ci_flow.png | Bin 0 -> 209013 bytes images/ci/scheduled_ci_flow.md5 | 1 + images/ci/scheduled_ci_flow.mermaid | 111 ++ images/ci/scheduled_ci_flow.png | Bin 0 -> 201540 bytes scripts/ci/pre_commit/pre_commit_mermaid.sh | 97 +++ 13 files changed, 447 insertions(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 1b5c436..6763164 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -128,6 +128,15 @@ repos: - --license-filepath - license-templates/LICENSE.txt - --fuzzy-match-generates-todo + - id: insert-license +name: Add license for all mermaid files +args: + - --comment-style + - "|%%|" + - --license-filepath + - license-templates/LICENSE.txt + - --fuzzy-match-generates-todo +files: \.mermaid$ - repo: https://github.com/thlorenz/doctoc.git rev: v1.4.0 hooks: @@ -277,7 +286,7 @@ repos: entry: ./scripts/ci/pre_commit/pre_commit_check_integrations.sh language: system pass_filenames: false -files: ^airflow/scripts/ci/libraries/_initialization.sh$|^breeze-complete$ +files: ^common/_common_values.sh$|^breeze-complete$ - id: check-apache-license name: Check if licenses are OK for Apache entry: "./scripts/ci/pre_commit/pre_commit_check_license.sh" diff --git a/.rat-excludes b/.rat-excludes index 497d7ed..69c8ccc 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -28,6 +28,7 @@ metastore_db .*sql .*svg .*csv +.*md5 CHANGELOG.txt .*zip .*lock diff --git a/images/ci/CI.png b/images/ci/CI.png new file mode 100644 index 000..7123519 Binary files /dev/null and b/images/ci/CI.png differ diff --git a/images/ci/pull_request_ci_flow.md5 b/images/ci/pull_request_ci_flow.md5 new file mode 100644 index 000..3fe05dd --- /dev/null +++ b/images/ci/pull_request_ci_flow.md5 @@ -0,0 +1 @@ +e3aa93e8b46ca97b4aad06de23a10ad5 images/ci/pull_request_ci_flow.mermaid diff --git a/images/ci/pull_request_ci_flow.mermaid b/images/ci/pull_request_ci_flow.mermaid new file mode 100644 index 000..2e4bf9c --- /dev/null +++ b/images/ci/pull_request_ci_flow.mermaid @@ -0,0 +1,108 @@ +%% Licensed to the Apache Software Foundation (ASF) under one +%% or more contributor license agreements. See the NOTICE file +%% distributed with this work for additional information +%% regarding copyright ownership. The ASF licenses this file +%% to you under the Apache License, Version 2.0 (the +%% "License"); you may not use this file except in compliance +%% with the License. You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, +%% software distributed under the License is distributed on an +%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +%% KIND, either express or implied. See the License for the +%% specific language governing permissions and limitations +%% under the License. + +sequenceDiagram +Contributor -->> Fork: Submit PR +Fork -->> Airflow Repo: Start Pull Request +activate Airflow Repo +Note over Airflow Repo: Trigger PR build +Note over CI Build: pull_request[Read Token] +Note over Build Images: workflow_run[Write Token] +Airflow Repo -->> CI Build: Trigger 'pull_request' event +activate CI Build +CI Build -->> Build Images: Trigger 'workflow_run' event +activate Build Images +Note over Build Images: Find Duplicate[CI Build] +opt Duplicated builds +Build Images -->> CI Build: Cancel +end +Note over Build Images: Find Duplicates[Build Image] +opt Duplicated builds +Build Images -->> Build Images: Cancel +end +Note over Build Images:
[airflow] 04/44: Move Project focus and Principles higher in the README (#11973)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 6f6b5600c371783d91a1f1234c627f895a3bbef9 Author: Ry Walker <4283+...@users.noreply.github.com> AuthorDate: Fri Oct 30 15:10:30 2020 -0400 Move Project focus and Principles higher in the README (#11973) (cherry picked from commit 3c723e35a58b274962dc47e21cbb05389263d97a) --- README.md | 34 +- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 5e638a5..ccac802 100644 --- a/README.md +++ b/README.md @@ -40,13 +40,13 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The **Table of contents** +- [Project Focus](#project-focus) +- [Principles](#principles) - [Requirements](#requirements) - [Getting started](#getting-started) - [Installing from PyPI](#installing-from-pypi) - [Official source code](#official-source-code) - [Convenience packages](#convenience-packages) -- [Project Focus](#project-focus) -- [Principles](#principles) - [User Interface](#user-interface) - [Contributing](#contributing) - [Who uses Apache Airflow?](#who-uses-apache-airflow) @@ -57,6 +57,21 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The +## Project Focus + +Airflow works best with workflows that are mostly static and slowly changing. When the structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). + +Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent, and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work. + +Airflow **is not** a streaming solution. Airflow is not in the [Spark Streaming](http://spark.apache.org/streaming/) or [Storm](https://storm.apache.org/) space. + +## Principles + +- **Dynamic**: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically. +- **Extensible**: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment. +- **Elegant**: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine. +- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. + ## Requirements Apache Airflow is tested with: @@ -150,21 +165,6 @@ All those artifacts are not official releases, but they are prepared using offic Some of those artifacts are "development" or "pre-release" ones, and they are clearly marked as such following the ASF Policy. -## Project Focus - -Airflow works best with workflows that are mostly static and slowly changing. When the structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). - -Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent, and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work. - -Airflow **is not** a streaming solution. Airflow is not in the [Spark Streaming](http://spark.apache.org/streaming/) or [Storm](https://storm.apache.org/) space. - -## Principles - -- **Dynamic**: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically. -- **Extensible**: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment. -- **Elegant**: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine. -- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. - ## User Interface - **
[airflow] 03/44: Pin `kubernetes` to a max version of 11.0.0. (#11974)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit d5e7b5115055fd3ccdb7896bb45dfe48db83c6da Author: Jed Cunningham <66968678+jedcunning...@users.noreply.github.com> AuthorDate: Fri Oct 30 11:56:58 2020 -0600 Pin `kubernetes` to a max version of 11.0.0. (#11974) 12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper' object` when serializing V1Pod's in `executor_config`. (cherry picked from commit 9687b3bc812a394f9f1debe48ea17bbfe78c270f) --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 50818d4..3ace091 100644 --- a/setup.py +++ b/setup.py @@ -305,7 +305,7 @@ kerberos = [ ] kubernetes = [ 'cryptography>=2.0.0', -'kubernetes>=3.0.0', +'kubernetes>=3.0.0, <12.0.0', ] ldap = [ 'ldap3>=2.5.1',
[airflow] 01/44: Migrate from helm-unittest to python unittest (#11827)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 572387dbeb9d83a2cc4fbde5f32a03b1f08f
Author: Kamil Breguła
AuthorDate: Fri Oct 30 15:48:22 2020 +0100
Migrate from helm-unittest to python unittest (#11827)
* Migrate from helm-unittest to python unittest
* fixup! Migrate from helm-unittest to python unittest
* fixup! fixup! Migrate from helm-unittest to python unittest
(cherry picked from commit 9322f3e46c3a06d2e5b891e399cc054e9b76ae72)
---
.github/workflows/ci.yml | 15 +-
.pre-commit-config.yaml| 17 +-
airflow/www_rbac/webpack.config.js | 2 +-
chart/tests/conftest.py| 6 +-
chart/tests/dags-persistent-volume-claim_test.yaml | 64 -
chart/tests/git-sync-webserver_test.yaml | 66 -
chart/tests/git-sync-worker_test.yaml | 70 -
chart/tests/pod-template-file_test.yaml| 149 --
chart/tests/test_basic_helm_chart.py | 2 +-
chart/tests/test_celery_kubernetes_executor.py | 47
.../test_celery_kubernetes_pod_launcher_role.py| 23 +-
chart/tests/test_dags_persistent_volume_claim.py | 72 +
chart/tests/test_git_sync_scheduler.py | 311 -
chart/tests/test_git_sync_webserver.py | 62
chart/tests/test_git_sync_worker.py| 75 +
.../tests/test_migrate_database_job.py | 23 +-
chart/tests/test_pod_template_file.py | 185
chart/tests/test_scheduler.py | 40 +++
chart/tests/test_worker.py | 40 +++
docs/static/exampleinclude.css | 2 +-
docs/static/jira-links.js | 2 +-
21 files changed, 686 insertions(+), 587 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 2295bd9..596ab3e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -153,19 +153,6 @@ jobs:
./scripts/ci/selective_ci_checks.sh
fi
- helm-tests:
-timeout-minutes: 5
-name: "Checks: Helm tests. Will soon be replaced with python tests"
-runs-on: ubuntu-latest
-needs: [build-info]
-if: >
- needs.build-info.outputs.needs-helm-tests == 'true'
-steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit
${{ github.merge_commit_sha }} )"
-uses: actions/checkout@v2
- - name: "Helm Tests"
-run: ./scripts/ci/kubernetes/ci_run_helm_testing.sh
-
ci-images:
timeout-minutes: 120
name: "Wait for CI images"
@@ -295,7 +282,7 @@ jobs:
name: airflow-documentation
path: "./files/documentation"
- helm-python-tests:
+ tests-helm:
timeout-minutes: 5
name: "Python unit tests for helm chart"
runs-on: ubuntu-latest
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index e4c0a86..1b5c436 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -58,12 +58,12 @@ repos:
- --fuzzy-match-generates-todo
files: \.rst$
- id: insert-license
-name: Add license for all JS/CSS files
-files: \.(js|css)$
+name: Add license for all JS/CSS/PUML files
+files: \.(js|css|puml)$
exclude:
^\.github/.*$|^airflow/www/static/.*|^airflow/www_rbac/static/.*$
args:
- --comment-style
- - "/**| *| */"
+ - "/*!| *| */"
- --license-filepath
- license-templates/LICENSE.txt
- --fuzzy-match-generates-todo
@@ -121,7 +121,7 @@ repos:
- id: insert-license
name: Add license for all md files
files: \.md$
-exclude: ^\.github/.*$
+exclude: ^\.github/.*$|PROVIDER_CHANGES.*\.md
args:
- --comment-style
- ""
@@ -133,7 +133,7 @@ repos:
hooks:
- id: doctoc
name: Add TOC for md files
-files: ^README\.md$|^CONTRIBUTING\.md$|^UPDATING.md$|^dev/README.md$
+files:
^README\.md$|^CONTRIBUTING\.md$|^UPDATING.*.md$|^dev/README\.md$|^dev/PROVIDER_PACKAGES.md$
args:
- "--maxlevel"
- "2"
@@ -141,7 +141,7 @@ repos:
hooks:
- id: check-hooks-apply
- repo: https://github.com/pre-commit/pre-commit-hooks
-rev: v3.2.0
+rev: v3.3.0
hooks:
- id: check-merge-conflict
- id: debug-statements
@@ -157,13 +157,14 @@ repos:
- id: rst-backticks
- id: python-no-log-warn
- repo: https://github.com/adrienverge/yamllint
-rev: v1.24.2
+rev: v1.25.0
hooks:
- id: yamllint
[airflow] 05/44: Implements canceling of future duplicate runs (but the latest) (#11980)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ad5679025d84378a54ac22a9203c3cc75f7aedd8
Author: Jarek Potiuk
AuthorDate: Fri Oct 30 21:48:40 2020 +0100
Implements canceling of future duplicate runs (but the latest) (#11980)
* Implements canceling of future duplicate runs (but the latest)
Previous version of the cancel-workflow-runs action implemented
canceling of only past duplicates, but when there are queues
involved, some future "cancel-workflow-runs" might be in a queue
for long time. This change has the effect that cancel-workflow-runs
for duplicates will also allow future runs of the same branch/repo
sparing only the most recent run - no matter if the duplicates
were older than my own run.
This should handle the case where we have queues blocking the
"cancel-workflow-runs" from running.
* Update .github/workflows/build-images-workflow-run.yml
Co-authored-by: Kaxil Naik
Co-authored-by: Kaxil Naik
(cherry picked from commit 36b5cc6df0c7b49bcd9d38c37c2ed4b881089d6d)
---
.github/workflows/build-images-workflow-run.yml | 19 ---
1 file changed, 8 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index a4f2dbe..81c4fb4 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -21,7 +21,6 @@ on: # yamllint disable-line rule:truthy
workflow_run:
workflows: ["CI Build"]
types: ['requested']
-
env:
MOUNT_LOCAL_SOURCES: "false"
MOUNT_FILES: "true"
@@ -66,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
-uses:
potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: duplicates
@@ -84,7 +83,7 @@ jobs:
# in GitHub Actions, we have to use Job names to match
Event/Repo/Branch from the
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -93,9 +92,7 @@ jobs:
[".*Event: ${{ steps.source-run-info.outputs.sourceEvent }}
Repo: ${{ steps.source-run-info.outputs.sourceHeadRepo }}
Branch: ${{ steps.source-run-info.outputs.sourceHeadBranch }}.*"]
-if: >
- env.BUILD_IMAGES == 'true' &&
steps.source-run-info.outputs.sourceEvent != 'schedule'
- && steps.source-run-info.outputs.sourceEvent != 'push'
+if: env.BUILD_IMAGES == 'true'
- name: "Cancel all 'CI Build' runs where some jobs failed"
# We find any of the "CI Build" workflow runs, where any of the
important jobs
@@ -104,7 +101,7 @@ jobs:
# can cancel all the matching "Build Images" workflow runs in the two
following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -138,14 +135,14 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run
Id:
# followed by one of the run ids. Yes I know it's super complex
¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' &&
steps.source-run-info-failed.outputs.cancelledRuns != '[]'
-uses:
potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{
steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
-uses:
potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+uses:
potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
id: cancel
[airflow] 10/44: Adds more aggressive cancelling of duplicate Build Image jobs (#12018)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 99aa8839732795038288adcfb031a726700ff930
Author: Jarek Potiuk
AuthorDate: Sun Nov 1 17:54:37 2020 +0100
Adds more aggressive cancelling of duplicate Build Image jobs (#12018)
This change adds even more aggressive cancelling of duplicates of
'Build Image' jobs. it's not an obvious task to know which
Build Image jobs are duplicates, we are matching those duplicates
based on specially crafted "build-info" job names. We add
Event, Branch, Repo to the job names and assume that two
runs with the same event + branch + repo are duplicates.
It also disables self-preservation for this step because
it is perfectly ok to cancel itself in case there is a newer
in-progress Build Image job.
Unfortunately even this will not work perfectly well. Those job
names are resolved only for the jobs that are runnning rather than
the queued ones, so in case we have several duplicates of the
same build image job in the queue, they will not be found/cancelled.
The cancelling will only happen if both duplicates are already
running.
It's good enough for now and we cannot do much more until there
is a missing feature added to GitHub API that allows to link
the workflow_run with the run that triggered it. This issue has
been raised to GitHub Support and internal engineering ticket
has been apparently opened to add this feature.
More detailed status for the missing feature is kept at #11294
(cherry picked from commit 1d14e74e33efbcb17e5553dba92baf9c8b0fc7c8)
---
.github/workflows/build-images-workflow-run.yml | 40 +++--
.github/workflows/ci.yml| 2 +-
2 files changed, 26 insertions(+), 16 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index 972195a..a3880e7 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
-uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: allDuplicates
@@ -77,12 +77,12 @@ jobs:
#
https://github.community/t/how-to-set-and-access-a-workflow-variable/17335/16
echo "::set-output name=buildImages::${BUILD_IMAGES}"
- name: "Cancel duplicated 'Build Image' runs"
-
# We find duplicates of our own "Build Image" runs - due to a missing
feature
-# in GitHub Actions, we have to use Job names to match
Event/Repo/Branch from the
-# build-info step there to find the duplicates ¯\_(ツ)_/¯.
-
-uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+# in GitHub Actions, we have to use Job names to match
Event/Repo/Branch matching
+# trick ¯\_(ツ)_/¯. We name the build-info job appropriately
+# and then we try to find and cancel all the jobs with the same Event
+ Repo + Branch as the
+# current Event/Repo/Branch combination.
+uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -93,14 +93,12 @@ jobs:
Branch: ${{ steps.source-run-info.outputs.sourceHeadBranch }}.*"]
if: env.BUILD_IMAGES == 'true'
- name: "Cancel all 'CI Build' runs where some jobs failed"
-
# We find any of the "CI Build" workflow runs, where any of the
important jobs
# failed. The important jobs are selected by the regexp array below.
# We also produce list of canceled "CI Build' runs as output, so that
we
# can cancel all the matching "Build Images" workflow runs in the two
following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-
-uses:
potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -111,7 +109,6 @@ jobs:
["^Static checks.*", "^Build docs$", "^Spell check docs$",
"^Backport packages$",
"^Checks: Helm tests$&
[airflow] 24/44: Add Kubernetes files to selective checks (#12114)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 49430a0b6478b80b720a2ea278cfd428a23d1f95
Author: Daniel Imberman
AuthorDate: Thu Nov 5 11:54:49 2020 -0800
Add Kubernetes files to selective checks (#12114)
* Add Kubernetes files to selective checks
There are multiple kubernetes-related files that require
running the k8s integration tests. This PR adds those to the
run_selective_tests
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Kaxil Naik
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk
Co-authored-by: Kaxil Naik
Co-authored-by: Jarek Potiuk
(cherry picked from commit 054de0703aa99fc425028d4cbe1f7b363e0cbeca)
---
scripts/ci/selective_ci_checks.sh | 6 +-
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/scripts/ci/selective_ci_checks.sh
b/scripts/ci/selective_ci_checks.sh
index b9a6d74..3c7132d 100755
--- a/scripts/ci/selective_ci_checks.sh
+++ b/scripts/ci/selective_ci_checks.sh
@@ -179,7 +179,6 @@ ALL_TESTS="Core Integration Heisentests"
readonly ALL_TESTS
function set_outputs_run_everything_and_exit() {
-needs_api_tests "true"
needs_helm_tests "true"
needs_javascript_scans "true"
needs_python_scans "true"
@@ -201,7 +200,6 @@ function set_outputs_run_all_tests() {
}
function set_output_skip_all_tests_and_docs_and_exit() {
-needs_api_tests "false"
needs_helm_tests "false"
needs_javascript_scans "false"
needs_python_scans "false"
@@ -215,7 +213,6 @@ function set_output_skip_all_tests_and_docs_and_exit() {
}
function set_output_skip_tests_but_build_images_and_exit() {
-needs_api_tests "false"
needs_helm_tests "false"
needs_javascript_scans "false"
needs_python_scans "false"
@@ -378,9 +375,8 @@ function get_count_kubernetes_files() {
echo "Count Kubernetes files"
echo
local pattern_array=(
-"^airflow/kubernetes"
"^chart"
-"^tests/kubernetes_tests"
+"^kubernetes_tests"
)
show_changed_files
COUNT_KUBERNETES_CHANGED_FILES=$(count_changed_files)
[airflow] 25/44: Work properly if some variables are not defined (#12135)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 38d55820057aa4ae9f3750a819435b2193406e6e
Author: Jarek Potiuk
AuthorDate: Fri Nov 6 16:56:43 2020 +0100
Work properly if some variables are not defined (#12135)
Those variables are defined in GitHub environment so when they
were recently addded it was not obvious that they will fail when
running kubernetes tests locally.
This PR fixes that.
(cherry picked from commit 5351f0d99605738b5efa0134bb51eb2dbb6eba46)
---
scripts/ci/libraries/_initialization.sh | 4 +++-
scripts/ci/libraries/_kind.sh | 2 +-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/libraries/_initialization.sh
b/scripts/ci/libraries/_initialization.sh
index 316a92c..e0b4a8a 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -771,7 +771,9 @@ function initialization::ga_output() {
}
function initialization::ga_env() {
-echo "${1}=${2}" >> "${GITHUB_ENV}"
+if [[ ${GITHUB_ENV=} != "" ]]; then
+echo "${1}=${2}" >> "${GITHUB_ENV}"
+fi
}
function initialization::set_mysql_encoding() {
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index 451c370..41048fc 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -302,7 +302,7 @@ function kind::forward_port_to_kind_webserver() {
echo
echo "Trying to establish port forwarding to 'airflow webserver'"
echo
-if [[ ${INCREASE_PORT_NUMBER_FOR_KUBERNETES} == "true" ]] ; then
+if [[ ${INCREASE_PORT_NUMBER_FOR_KUBERNETES=} == "true" ]] ; then
forwarded_port_number=$(( forwarded_port_number + 1 ))
fi
if [[ ${num_tries} == "${MAX_NUM_TRIES_FOR_PORT_FORWARD}" ]]; then
[airflow] 08/44: Adds documentation about the optimized PR workflow (#12006)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 80845847d5971e64a0b19a589a9998cf330abe07 Author: Jarek Potiuk AuthorDate: Sun Nov 1 00:20:38 2020 +0100 Adds documentation about the optimized PR workflow (#12006) We had a lot of problems recently about the queues in Github Actions. This documentations explains the motivation and approach we have taken for optimizing our PR workflow. (cherry picked from commit d85a31f2d88b92b480438d86aa8e3e79e6c3614d) --- CI.rst | 102 ++-- PULL_REQUEST_WORKFLOW.rst| 260 +++ images/pr/pr-full-tests-needed.png | Bin 0 -> 88512 bytes images/pr/pr-likely-ok-to-merge.png | Bin 0 -> 98362 bytes images/pr/pr-no-tests-needed-comment.png | Bin 0 -> 80852 bytes images/pr/selective_checks.md5 | 1 + images/pr/selective_checks.mermaid | 35 + images/pr/selective_checks.png | Bin 0 -> 64501 bytes 8 files changed, 309 insertions(+), 89 deletions(-) diff --git a/CI.rst b/CI.rst index dc1cdf2..f4b5294 100644 --- a/CI.rst +++ b/CI.rst @@ -35,7 +35,7 @@ the CI jobs. And we have a number of variables determine build behaviour. -Github Actions runs +GitHub Actions runs --- Our builds on CI are highly optimized. They utilise some of the latest features provided by GitHub Actions @@ -65,7 +65,7 @@ utilise the WRITE access to Apache Airflow repository via an external Pull Reque Thanks to the WRITE access and fact that the 'workflow_run' by default uses the 'master' version of the sources, we can safely run some logic there will checkout the incoming Pull Request, build the container -image from the sources from the incoming PR and push such image to an Github Docker Registry - so that +image from the sources from the incoming PR and push such image to an GitHub Docker Registry - so that this image can be built only once and used by all the jobs running tests. The image is tagged with unique ``RUN_ID`` of the incoming Pull Request and the tests run in the Pull Request can simply pull such image rather than build it from the scratch. Pulling such image takes ~ 1 minute, thanks to that we are saving @@ -92,7 +92,7 @@ connected with the run. You can read more about it in `BREEZE.rst `_ and `TESTING.rst `_ -Difference between local runs and Github Action workflows +Difference between local runs and GitHub Action workflows - Depending whether the scripts are run locally (most often via `Breeze `_) or whether they @@ -470,7 +470,13 @@ The main purpose of those jobs is to check if PR builds cleanly, if the test run the PR is ready to review and merge. The runs are using cached images from the Private GitHub registry - CI, Production Images as well as base Python images that are also cached in the Private GitHub registry. Also for those builds we only execute Python tests if important files changed (so for example if it is -doc-only change, no tests will be executed. +"no-code" change, no tests will be executed. + +The workflow involved in Pull Requests review and approval is a bit more complex than simple workflows +in most of other projects because we've implemented some optimizations related to efficient use +of queue slots we share with other Apache Software Foundation projects. More details about it +can be found in `PULL_REQUEST_WORKFLOW.rst `_. + Direct Push/Merge Run - @@ -641,7 +647,7 @@ Comments: (1) CRON jobs builds images from scratch - to test if everything works properly for clean builds (2) The tests are run when the Trigger Tests job determine that important files change (this allows - for example doc-only changes to build much faster) + for example "no-code" changes to build much faster) (3) The jobs wait for CI images if ``GITHUB_REGISTRY_WAIT_FOR_IMAGE`` variable is set to "true". You can set it to "false" to disable using shared images - this is slower though as the images are rebuilt in every job that needs them. You can also set your own fork's secret @@ -685,7 +691,7 @@ way to sync your fork master to the Apache Airflow's one. Delete old artifacts -This workflow is introduced, to delete old artifacts from the Github Actions build. We set it to +This workflow is introduced, to delete old artifacts from the GitHub Actions build. We set it to delete old artifacts that are > 7 days old. It only runs for the 'apache/airflow' repository. We also have a script that can help to clean-up the old artifacts: @@ -695,89 +701,7 @@ CodeQL scan --- The CodeQL security scan uses Git
[airflow] 29/44: Uses always the same Python base image as used for CI image (#12177)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a27701a396f803e6c5434f2c2062393efa55f62a
Author: Jarek Potiuk
AuthorDate: Sun Nov 8 11:20:31 2020 +0100
Uses always the same Python base image as used for CI image (#12177)
When new Python version is released (bugfixes), we rebuild the CI image
and replace it with the new one, however releasing of the python
image and CI image is often hours or even days apart (we only
release the CI image when tests pass in master with the new python
image). We already use a better approach for Github - we simply
push the new python image to our registry together with the CI
image and the CI jobs are always pulling them from our registry
knowing that the two - python and CI image are in sync.
This PR introduces the same approach. We not only push CI image
but also the corresponding Python image to our registry. This has
no ill effect - DockerHub handles it automatically and reuses
the layers of the image directly from the Python one so it is
merely a label that is stored in our registry that points to the
exact Python image that was used by the last pushed CI image.
(cherry picked from commit 75bdfaeb9b2d7f47599e951ccaccc515a180ca19)
---
IMAGES.rst | 11
scripts/ci/libraries/_build_images.sh| 11 +++-
scripts/ci/libraries/_push_pull_remove_images.sh | 35 +++-
3 files changed, 19 insertions(+), 38 deletions(-)
diff --git a/IMAGES.rst b/IMAGES.rst
index 45aa09ab..8c913db 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -46,6 +46,17 @@ where:
* The ``-ci`` suffix is added for CI images
* The ``-manifest`` is added for manifest images (see below for explanation of
manifest images)
+We also store (to increase speed of local build/pulls) python images that were
used to build
+the CI images. Each CI image, when built uses current python version of the
base images. Those
+python images are regularly updated (with bugfixes/security fixes), so for
example python3.8 from
+last week might be a different image than python3.8 today. Therefore whenever
we push CI image
+to airflow repository, we also push the python image that was used to build it
this image is stored
+as ``apache/airflow:python-3.8-``.
+
+Since those are simply snapshots of the existing python images, DockerHub does
not create a separate
+copy of those images - all layers are mounted from the original python images
and those are merely
+labels pointing to those.
+
Building docker images
==
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index cbd01c1..a2577cb 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -330,9 +330,11 @@ function build_images::get_docker_image_names() {
export
AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
# CI image to build
export
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
-
# Default CI image
-export
AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
+export
AIRFLOW_CI_PYTHON_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:python${PYTHON_MAJOR_MINOR_VERSION}-${BRANCH_NAME}"
+# CI image to build
+export
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
+
# Base production image tag - used to build kubernetes tag as well
if [[ ${FORCE_AIRFLOW_PROD_BASE_TAG=} == "" ]]; then
@@ -374,11 +376,6 @@ function build_images::prepare_ci_build() {
export
GITHUB_REGISTRY_AIRFLOW_CI_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/${AIRFLOW_CI_BASE_TAG}"
export
GITHUB_REGISTRY_PYTHON_BASE_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
fi
-if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" ==
"${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
-export DEFAULT_PROD_IMAGE="${AIRFLOW_CI_IMAGE_DEFAULT}"
-else
-export DEFAULT_PROD_IMAGE=""
-fi
export THE_IMAGE_TYPE="CI"
export IMAGE_DESCRIPTION="Airflow CI"
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh
b/scripts/ci/libraries/_push_pull_remove_images.sh
index fae0807..e853f42 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -104,7 +104,6 @@ function
push_pull_remove_images::pull_image_github_dockerhub() {
# Pulls CI image in case caching strategy is "pulled" and the image needs to
be pulled
function push_pull_remove_images::pull_
[airflow] 19/44: If we build a new image, we should run more than basic checks (#12070)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6f64e98543f230f23b87d3adef72d2a29d5a9b40
Author: Ash Berlin-Taylor
AuthorDate: Tue Nov 3 17:42:01 2020 +
If we build a new image, we should run more than basic checks (#12070)
This lead to bases such as in #11699 where despite there being changes,
and an image being build, the pre-commit tests were not being run.
(cherry picked from commit 8000ab7375d319e389b6ea973acf8b1085c46bb1)
---
scripts/ci/selective_ci_checks.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/ci/selective_ci_checks.sh
b/scripts/ci/selective_ci_checks.sh
index 0faa541..b9a6d74 100755
--- a/scripts/ci/selective_ci_checks.sh
+++ b/scripts/ci/selective_ci_checks.sh
@@ -450,7 +450,6 @@ if [[ ${image_build_needed} == "true" ]]; then
else
set_basic_checks_only "true"
fi
-set_basic_checks_only "${image_build_needed}"
set_docs_build "${docs_build_needed}"
run_tests "${tests_needed}"
run_kubernetes_tests "${kubernetes_tests_needed}"
[airflow] 20/44: Fixes problem with building a PROD image (#12080)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f4323092e6544cf1c28e97c3f461ef0ed282e3ef
Author: Jarek Potiuk
AuthorDate: Wed Nov 4 09:31:00 2020 +0100
Fixes problem with building a PROD image (#12080)
The change #12050 that aimed at automation of Docker images
building in DockerHub had an undesired effect of overriding the
production image tag with the CI one.
This is fixed by this PR.
(cherry picked from commit d971c1c0e141f285a3cdd3f029028ab62df6f81a)
---
scripts/ci/images/ci_build_dockerhub.sh | 6 +-
scripts/ci/libraries/_build_images.sh | 9 ++---
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/scripts/ci/images/ci_build_dockerhub.sh
b/scripts/ci/images/ci_build_dockerhub.sh
index 7d92abc..ac7764f 100755
--- a/scripts/ci/images/ci_build_dockerhub.sh
+++ b/scripts/ci/images/ci_build_dockerhub.sh
@@ -39,7 +39,6 @@ echo
echo "DOCKER_TAG=${DOCKER_TAG}"
-
[[ ${DOCKER_TAG:=} =~ ${DEFAULT_BRANCH}-python([0-9.]*)(.*) ]] && export
PYTHON_MAJOR_MINOR_VERSION=${BASH_REMATCH[1]}
: "${PYTHON_MAJOR_MINOR_VERSION:?"The tag '${DOCKER_TAG}' should follow the
pattern .*-pythonX.Y[-ci]"}"
@@ -47,6 +46,11 @@ echo "DOCKER_TAG=${DOCKER_TAG}"
echo "Detected PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}"
echo
+FORCE_AIRFLOW_PROD_BASE_TAG="${DOCKER_TAG}"
+export FORCE_AIRFLOW_PROD_BASE_TAG
+
+readonly FORCE_AIRFLOW_PROD_BASE_TAG
+
# shellcheck source=scripts/ci/libraries/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index 387372a..a6dbae0 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -327,8 +327,7 @@ function build_images::get_docker_image_names() {
export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
# CI image base tag
-export
AIRFLOW_CI_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci}"
-
+export
AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
# CI image to build
export
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
@@ -336,7 +335,11 @@ function build_images::get_docker_image_names() {
export
AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
# Base production image tag - used to build kubernetes tag as well
-export
AIRFLOW_PROD_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}}"
+if [[ ${FORCE_AIRFLOW_PROD_BASE_TAG=} == "" ]]; then
+export
AIRFLOW_PROD_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
+else
+export AIRFLOW_PROD_BASE_TAG="${FORCE_AIRFLOW_PROD_BASE_TAG}"
+fi
# PROD image to build
export
AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
[airflow] 21/44: Use sys.exit() instead of exit() (#12084)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2be4d56a78b2b49b08866ac9114a89ab5db7eef3
Author: Kaxil Naik
AuthorDate: Wed Nov 4 11:50:52 2020 +
Use sys.exit() instead of exit() (#12084)
The `exit` and `quit` functions are actually `site.Quitter` objects and are
loaded, at interpreter start up, from site.py. However, if the interpreter is
started with the `-S` flag, or a custom site.py is used then exit and quit may
not be present. It is recommended to use `sys.exit()` which is built into the
interpreter and is guaranteed to be present.
(cherry picked from commit bec9f3b29fd42ecd1beae3db75784b9a726caf15)
---
scripts/ci/pre_commit/pre_commit_check_setup_installation.py | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
index 2fdeca6..b4f3281 100755
--- a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
+++ b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
@@ -22,6 +22,7 @@ Checks if all the libraries in setup.py are listed in
installation.rst file
import os
import re
+import sys
from os.path import dirname
from typing import Dict, List
@@ -90,8 +91,8 @@ if __name__ == '__main__':
if f"'{extras}'" not in setup_packages_str:
output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras, "",
"V")
-if(output_table == ""):
-exit(0)
+if output_table == "":
+sys.exit(0)
print(f"""
ERROR
@@ -105,4 +106,4 @@ documented although not used.
print(".{:_^22}.{:_^12}.{:_^12}.".format("NAME", "SETUP", "INSTALLATION"))
print(output_table)
-exit(1)
+sys.exit(1)
[airflow] 07/44: Fix K8S CI job name rendering (#12007)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 33a14e7e666c6c9b92027a18c7c5941e52ad506f
Author: Xiaodong DENG
AuthorDate: Sun Nov 1 00:09:19 2020 +0100
Fix K8S CI job name rendering (#12007)
(cherry picked from commit 72f3fcb293bbaee405133a77a8c1d6cab87b9e62)
---
.github/workflows/ci.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 596ab3e..630b5d6 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -613,7 +613,7 @@ jobs:
tests-kubernetes:
timeout-minutes: 50
-name: "K8s: ${{matrix.python-version}} ${{matrix.kubernetes-version}}
${{matrix.kubernetes-mode}}"
+name: K8s ${{matrix.python-version}} ${{matrix.kubernetes-version}}
${{matrix.kubernetes-mode}}
runs-on: ubuntu-latest
needs: [build-info, prod-images]
strategy:
[airflow] 33/44: Remove popd which is a remnant from past (#12211)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 73187015e2dc64a76bf0d221e05ba7322eb436b8
Author: Jarek Potiuk
AuthorDate: Mon Nov 9 21:58:37 2020 +0100
Remove popd which is a remnant from past (#12211)
(cherry picked from commit d8abee690823b7973e5561f2725dc7b79467bfdf)
---
scripts/ci/libraries/_start_end.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/ci/libraries/_start_end.sh
b/scripts/ci/libraries/_start_end.sh
index f4d8bfe..8fc7e5d 100644
--- a/scripts/ci/libraries/_start_end.sh
+++ b/scripts/ci/libraries/_start_end.sh
@@ -53,7 +53,6 @@ function start_end::dump_container_logs() {
echo
"###"
docker logs "${container}" > "${dump_file}"
echo " Container ${container} logs dumped to
${dump_file}"
-popd || exit 1
echo
"###"
}
[airflow] 32/44: Fixes timeout in helm chart tests (#12209)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 6e74f1b6d0c3d188aed013633c0666955e07e14a Author: Jarek Potiuk AuthorDate: Mon Nov 9 21:18:58 2020 +0100 Fixes timeout in helm chart tests (#12209) (cherry picked from commit e77867aeab1899a87e3d212ff69ccae0a8447adb) --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c217bdc..6c854a1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -281,7 +281,7 @@ jobs: path: "./files/documentation" tests-helm: -timeout-minutes: 5 +timeout-minutes: 20 name: "Python unit tests for helm chart" runs-on: ubuntu-latest needs: [build-info, ci-images]
[airflow] 11/44: Revise "Project Focus" copy (#12011)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 887c231332d2d425d7f81ba0973dc12c508d8a2c Author: Ry Walker <4283+...@users.noreply.github.com> AuthorDate: Sun Nov 1 12:34:34 2020 -0500 Revise "Project Focus" copy (#12011) (cherry picked from commit 77ae32e311ebd9f6ab236cbeba42f7de3dd1e211) --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index ccac802..70bf2d8 100644 --- a/README.md +++ b/README.md @@ -59,11 +59,11 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The ## Project Focus -Airflow works best with workflows that are mostly static and slowly changing. When the structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). +Airflow works best with workflows that are mostly static and slowly changing. When DAG structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](https://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). -Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent, and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work. +Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent (i.e. results of the task will be the same, and will not create duplicated data in a destination system), and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external servic [...] -Airflow **is not** a streaming solution. Airflow is not in the [Spark Streaming](http://spark.apache.org/streaming/) or [Storm](https://storm.apache.org/) space. +Airflow is not a streaming solution, but it is often used to process real-time data, pulling data off streams in batches. ## Principles
[airflow] 13/44: Fix canceling of CodeQL workflow (#12024)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit df68fb7cd84626abe5675443f5254649147b446b
Author: Jarek Potiuk
AuthorDate: Sun Nov 1 22:05:45 2020 +0100
Fix canceling of CodeQL workflow (#12024)
The previous update for 4.3 version of the action also broke
CodeQL cancelling. This PR fixes it.
(cherry picked from commit 5f9792cee26c729f319e57fd09538fa6703929b4)
---
.github/workflows/build-images-workflow-run.yml | 16
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index 752e577..6099727 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
- uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: allDuplicates
@@ -82,7 +82,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately
# and then we try to find and cancel all the jobs with the same Event
+ Repo + Branch as the
# current Event/Repo/Branch combination.
- uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -98,7 +98,7 @@ jobs:
# We also produce list of canceled "CI Build' runs as output, so that
we
# can cancel all the matching "Build Images" workflow runs in the two
following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -131,14 +131,14 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run
Id:
# followed by one of the run ids. Yes I know it's super complex
¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' &&
steps.cancel-failed.outputs.cancelledRuns != '[]'
-uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{
steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
-uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -167,7 +167,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately and then
we try to match
# all the jobs with the same Event + Repo + Branch match and cancel
all the duplicates for those
# This might cancel own run, so this is the last step in the job
-uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
cancelMode: allDuplicatedNamedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -382,7 +382,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -397,7 +397,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
-uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+uses:
potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
[airflow] 09/44: Add contributor-targeted description of the PR workflow. (#12016)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit b2633ff5231198363fcf6ac64ed388bc4125a12f Author: Jarek Potiuk AuthorDate: Sun Nov 1 14:22:23 2020 +0100 Add contributor-targeted description of the PR workflow. (#12016) * Add contributor-targeted description of the PR workflow. Simpler/shorter description of the PR workflow targeted for new contributors and contributors who did not follow the recent changes in the PR workflow. * Update CONTRIBUTING.rst Co-authored-by: Xiaodong DENG * Update CONTRIBUTING.rst Co-authored-by: Xiaodong DENG Co-authored-by: Xiaodong DENG (cherry picked from commit 6c8c3089e9f7fc00debe031b8c1ff576470a) --- CONTRIBUTING.rst | 552 --- 1 file changed, 285 insertions(+), 267 deletions(-) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 40e4f89..6d3aa91 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -42,8 +42,7 @@ to follow it and apply to the programme and follow up with the community. Report Bugs --- -Report bugs through `Apache -JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__. +Report bugs through `GitHub <https://github.com/apache/airflow/issues>`__. Please report relevant information and preferably code that exhibits the problem. @@ -51,16 +50,16 @@ problem. Fix Bugs -Look through the JIRA issues for bugs. Anything is open to whoever wants to +Look through the GitHub issues for bugs. Anything is open to whoever wants to implement it. Implement Features -- -Look through the `Apache -JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__ for features. +Look through the `GitHub issues labeled "kind:feature" +<https://github.com/apache/airflow/labels/kind%3Afeature>`__ for features. -Any unassigned "Improvement" issue is open to whoever wants to implement it. +Any unassigned feature request issue is open to whoever wants to implement it. We've created the operators, hooks, macros and executors we needed, but we've made sure that this part of Airflow is extensible. New operators, hooks, macros @@ -76,8 +75,7 @@ articles. Submit Feedback --- -The best way to send feedback is to open an issue on `Apache -JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__. +The best way to send feedback is to `open an issue on GitHub <https://github.com/apache/airflow/issues/new/choose>`__. If you are proposing a new feature: @@ -150,69 +148,6 @@ Contributors are responsible for: * Adding features * Championing one or more items on the `Roadmap <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home>`__. -= - -There are several roles within the Airflow Open-Source community. - - -PMC Member -The PMC (Project Management Committee) is a group of maintainers that drives changes in the way that -Airflow is managed as a project. - -Considering Apache, the role of the PMC is primarily to ensure that Airflow conforms to Apache's processes -and guidelines. - -Committers/Maintainers --- - -Committers are community members that have write access to the project’s repositories, i.e., they can modify the code, -documentation, and website by themselves and also accept other contributions. - -The official list of committers can be found `here <https://airflow.apache.org/docs/stable/project.html#committers>`__. - -Additionally, committers are listed in a few other places (some of these may only be visible to existing committers): - -* https://whimsy.apache.org/roster/ppmc/airflow -* https://github.com/orgs/apache/teams/airflow-committers/members - -Committers are responsible for: - -* Championing one or more items on the `Roadmap <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home>`__ -* Reviewing & Merging Pull-Requests -* Scanning and responding to Github issues -* Responding to questions on the dev mailing list (d...@airflow.apache.org) - -Becoming a Committer - - -There is no strict protocol for becoming a committer. -Candidates for new committers are typically people that are active contributors and community members. - -The key aspects of a committer are: - -* Consistent contributions over the past 6 months -* Understanding of Airflow Core or has displayed a holistic understanding of a particular part and made - contributions towards a more strategic goal -* Understanding of contributor/committer guidelines: `Contributors' Guide <https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst>`__ -* Quality of the commits -* Visibility in community discussions (dev mailing list, Slack and Github) -* Testing Release Candidate
[airflow] 17/44: Uses DOCKER_TAG when building image in DockerHub (#12050)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7512344582888106d46fb871ecd9804d2c706f6b
Author: Jarek Potiuk
AuthorDate: Mon Nov 2 22:00:51 2020 +0100
Uses DOCKER_TAG when building image in DockerHub (#12050)
DockerHub uses `hooks/build` to build the image and it passes
DOCKER_TAG variable when the script is called.
This PR makes the DOCKER_TAG to provide the default valuei for tag
that is calculated from sources (taking the default branch and
python version). Since it is only set in the DockerHub build, it
should be safe.
Fixes #11937
(cherry picked from commit 5c199fbddfaf9f83915e84225313169a0486c3a6)
---
scripts/ci/libraries/_build_images.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index 444fff9..387372a 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -327,7 +327,7 @@ function build_images::get_docker_image_names() {
export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
# CI image base tag
-export
AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
+export
AIRFLOW_CI_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci}"
# CI image to build
export
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
@@ -336,7 +336,7 @@ function build_images::get_docker_image_names() {
export
AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
# Base production image tag - used to build kubernetes tag as well
-export
AIRFLOW_PROD_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
+export
AIRFLOW_PROD_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}}"
# PROD image to build
export
AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
[airflow] 34/44: Fix permissions of mounted /tmp directory for Breeze (#12157)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 40ebe6eb41a935cb83bff4ef80106347aa929f8c
Author: Jarek Potiuk
AuthorDate: Mon Nov 9 22:01:58 2020 +0100
Fix permissions of mounted /tmp directory for Breeze (#12157)
The "tmp" directory is mounted from the host (from tmp folder
in the source airflow directory). This is needed to get some
of our docker-in-docker tools (such as gcloud/aws/java) and
get them working on demand. Thanks to that we do not have
to increase the size of CI image unnecessarily.
Those tools were introduced and made to work in #9376
However this causes some of the standard tools (such as apt-get)
to not work inside the container unless the mounted /tmp
folder has write permission for groups/other.
This PR fixes it.
(cherry picked from commit a42bbe21c899f4048b0861df2c84eb2ba9c5eeb5)
---
scripts/in_container/entrypoint_ci.sh | 5 +
1 file changed, 5 insertions(+)
diff --git a/scripts/in_container/entrypoint_ci.sh
b/scripts/in_container/entrypoint_ci.sh
index 37d3199..057d630 100755
--- a/scripts/in_container/entrypoint_ci.sh
+++ b/scripts/in_container/entrypoint_ci.sh
@@ -31,6 +31,11 @@ function disable_rbac_if_requested() {
# shellcheck source=scripts/in_container/_in_container_script_init.sh
. /opt/airflow/scripts/in_container/_in_container_script_init.sh
+# Add "other" and "group" write permission to the tmp folder
+# Note that it will also change permissions in the /tmp folder on the host
+# but this is necessary to enable some of our CLI tools to work without errors
+chmod 1777 /tmp
+
AIRFLOW_SOURCES=$(cd "${IN_CONTAINER_DIR}/../.." || exit 1; pwd)
PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:=3.6}
[airflow] 30/44: Adds extra check while the selective checks are run (#12178)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 20512aff626a04a418ccef5861b6856ac0c3e215
Author: Jarek Potiuk
AuthorDate: Sun Nov 8 13:25:32 2020 +0100
Adds extra check while the selective checks are run (#12178)
The selective checks are run in "workflow_run" because
they need to be able to set label and make comments, however
status of those checks are not displayed in GitHub and in
cases of small PRs the "merge" button might be green before
the status complete.
This PR adds additional check that is always completed after
the "worfklow_run" finishes it's job. This will prevent
accidental merges before the check completes.
(cherry picked from commit 63ac07d9c735b1ccc8aa5fa974a260cc944cc539)
---
.../workflows/label_when_reviewed_workflow_run.yml | 29 ++
1 file changed, 29 insertions(+)
diff --git a/.github/workflows/label_when_reviewed_workflow_run.yml
b/.github/workflows/label_when_reviewed_workflow_run.yml
index cdcbca5..c2359c8 100644
--- a/.github/workflows/label_when_reviewed_workflow_run.yml
+++ b/.github/workflows/label_when_reviewed_workflow_run.yml
@@ -35,6 +35,20 @@ jobs:
with:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
+ - name: Initiate Selective Build check
+uses:
LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 # v1.1.0
+id: selective-build-check
+with:
+ token: ${{ secrets.GITHUB_TOKEN }}
+ name: "Selective build check"
+ status: "in_progress"
+ sha: ${{ steps.source-run-info.outputs.sourceHeadSha }}
+ details_url: https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }}
+ output: >
+{"summary":
+"Checking selective status of the build in
+[the run](https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }})
+"}
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: >
@@ -125,3 +139,18 @@ jobs:
pullRequestNumber: ${{
steps.source-run-info.outputs.pullRequestNumber }}
require_committers_approval: 'true'
comment: "The PR is ready to be merged. No tests are needed!"
+ - name: Update Selective Build check
+uses:
LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 # v1.1.0
+if: always()
+with:
+ token: ${{ secrets.GITHUB_TOKEN }}
+ check_id: ${{ steps.selective-build-check.outputs.check_id }}
+ status: "completed"
+ sha: ${{ steps.source-run-info.outputs.sourceHeadSha }}
+ conclusion: ${{ job.status }}
+ details_url: https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }}
+ output: >
+{"summary":
+"Checking selective status of the build completed in
+[the run](https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }})
+"}
[airflow] 14/44: Turns failure of PR label when approved action into warning (#12017)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8e582911f9576fad1827d613a7980700c3c0080e
Author: Jarek Potiuk
AuthorDate: Mon Nov 2 00:01:20 2020 +0100
Turns failure of PR label when approved action into warning (#12017)
Sometimes (quite often really) when PR gets approved, the PR
gets merged rather quickly, without waiting for result of this
action. Or a new PR gets pushed quickly. In those cases PR will
not be found. But this is usually not a problem then and rather
than failing, we should simply print a warning and exit.
(cherry picked from commit a7a7cf22172ca92edfb62c373993ed974a298496)
---
.github/workflows/label_when_reviewed_workflow_run.yml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/label_when_reviewed_workflow_run.yml
b/.github/workflows/label_when_reviewed_workflow_run.yml
index 0fcab12..56bc835 100644
--- a/.github/workflows/label_when_reviewed_workflow_run.yml
+++ b/.github/workflows/label_when_reviewed_workflow_run.yml
@@ -72,7 +72,7 @@ jobs:
./scripts/ci/selective_ci_checks.sh
fi
- name: "Label when approved by committers for PRs that require full
tests"
-uses:
TobKed/label-when-approved-action@7872312da76508d29f98d4fa68843ea91754cc59 #
v1.2
+uses:
TobKed/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194 #
v1.3
id: label-full-test-prs-when-approved-by-commiters
if: >
steps.selective-checks.outputs.run-tests == 'true' &&
@@ -101,7 +101,7 @@ jobs:
[the run](https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }})
"}
- name: "Label when approved by committers for PRs that do not require
full tests"
-uses:
TobKed/label-when-approved-action@7872312da76508d29f98d4fa68843ea91754cc59 #
v1.2
+uses:
TobKed/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194 #
v1.3
id: label-simple-test-prs-when-approved-by-commiters
if: >
steps.selective-checks.outputs.run-tests == 'true' &&
@@ -116,7 +116,7 @@ jobs:
Airflow. The committers might merge it or can add a label 'full
tests needed' and re-run it
to run all tests if they see it is needed!
- name: "Label when approved by committers for PRs that do not require
tests at all"
-uses:
TobKed/label-when-approved-action@7872312da76508d29f98d4fa68843ea91754cc59 #
v1.2
+uses:
TobKed/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194 #
v1.3
id: label-no-test-prs-when-approved-by-commiters
if: steps.selective-checks.outputs.run-tests != 'true'
with:
[airflow] 35/44: Beautify Output of setup-installation pre-commit (#12218)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit eb7d914492d580dd06c05b5c114b9f9792b6de38
Author: Kaxil Naik
AuthorDate: Tue Nov 10 00:54:47 2020 +
Beautify Output of setup-installation pre-commit (#12218)
(cherry picked from commit 08d67add52c29f9f29f6e786e2f4e6591cdf)
---
.pre-commit-config.yaml| 1 +
.../pre_commit_check_setup_installation.py | 29 ++
2 files changed, 19 insertions(+), 11 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index cb5c753..e626cd4 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -219,6 +219,7 @@ repos:
files: ^setup.py$|^docs/installation.rst$
pass_filenames: false
entry: ./scripts/ci/pre_commit/pre_commit_check_setup_installation.py
+additional_dependencies: ['rich==9.2.0']
- id: update-breeze-file
name: Update output of breeze command in BREEZE.rst
entry: "./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.sh"
diff --git a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
index b4f3281..e386461 100755
--- a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
+++ b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
@@ -26,6 +26,10 @@ import sys
from os.path import dirname
from typing import Dict, List
+from rich import print as rprint
+from rich.console import Console
+from rich.table import Table
+
AIRFLOW_SOURCES_DIR = os.path.join(dirname(__file__), os.pardir, os.pardir,
os.pardir)
SETUP_PY_FILE = 'setup.py'
DOCS_FILE = 'installation.rst'
@@ -80,30 +84,33 @@ if __name__ == '__main__':
setup_packages = get_extras_from_setup()
docs_packages = get_extras_from_docs()
-output_table = ""
+table = Table()
+table.add_column("NAME", justify="right", style="cyan")
+table.add_column("SETUP", justify="center", style="magenta")
+table.add_column("INSTALLATION", justify="center", style="green")
for extras in sorted(setup_packages.keys()):
if not set(setup_packages[extras]).intersection(docs_packages):
-output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras,
"V", "")
+table.add_row(extras, "V", "")
setup_packages_str = str(setup_packages)
for extras in sorted(docs_packages):
if f"'{extras}'" not in setup_packages_str:
-output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras, "",
"V")
+table.add_row(extras, "", "V")
-if output_table == "":
+if table.row_count == 0:
sys.exit(0)
-print(f"""
-ERROR
+rprint(f"""\
+[red bold]ERROR!![/red bold]
-"EXTRAS_REQUIREMENTS" section in {SETUP_PY_FILE} should be synchronized
-with "Extra Packages" section in documentation file doc/{DOCS_FILE}.
+"EXTRAS_REQUIREMENTS" section in [bold yellow]{SETUP_PY_FILE}[/bold yellow]
should be synchronized
+with "Extra Packages" section in documentation file [bold
yellow]doc/{DOCS_FILE}[/bold yellow].
-here is a list of packages that are used but are not documented, or
+Here is a list of packages that are used but are not documented, or
documented although not used.
""")
-print(".{:_^22}.{:_^12}.{:_^12}.".format("NAME", "SETUP", "INSTALLATION"))
-print(output_table)
+console = Console()
+console.print(table)
sys.exit(1)
[airflow] 36/44: Add Markdown linting to pre-commit (#11465)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 64af9450e37faeb5b08e9568938d6cdf2d214f6d Author: John Bampton AuthorDate: Tue Nov 10 12:37:45 2020 +1000 Add Markdown linting to pre-commit (#11465) (cherry picked from commit 7463b6bcc7a823a9d02c7026d9b03b81c6421654) --- .markdownlint.yml | 77 +++ .pre-commit-config.yaml | 32 +- BREEZE.rst |8 +- STATIC_CODE_CHECKS.rst |4 +- breeze-complete |2 + chart/README.md |2 +- dev/README.md | 1522 --- 7 files changed, 1552 insertions(+), 95 deletions(-) diff --git a/.markdownlint.yml b/.markdownlint.yml new file mode 100644 index 000..dae8217 --- /dev/null +++ b/.markdownlint.yml @@ -0,0 +1,77 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +# MD003/heading-style/header-style +MD003: false + +# MD004/ul-style +MD004: false + +# MD007/ul-indent +MD007: false + +# MD012/no-multiple-blanks +MD012: false + +# MD013 Line length +MD013: false + +# MD014/commands-show-output +MD014: false + +# MD022/blanks-around-headings/blanks-around-headers +MD022: false + +# MD024/no-duplicate-heading/no-duplicate-header +MD024: false + +# MD026/no-trailing-punctuation +MD026: false + +# MD029/ol-prefix +MD029: false + +# MD030/list-marker-space +MD030: false + +# MD031/blanks-around-fences +MD031: false + +# MD032/blanks-around-lists +MD032: false + +# MD033/no-inline-html +MD033: false + +# MD034/no-bare-urls +MD034: false + +# MD036/no-emphasis-as-heading/no-emphasis-as-header +MD036: false + +# MD040/fenced-code-language +MD040: false + +# MD041/first-line-heading/first-line-h1 +MD041: false + +# MD045/no-alt-text +MD045: false + +# MD046/code-block-style +MD046: false diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index e626cd4..4dbf84c 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -333,6 +333,26 @@ repos: files: ^breeze$|^breeze-complete$|\.sh$|\.bash$|\.bats$ exclude: ^tests/bats/in_container/.*bats$|^scripts/in_container/.*sh$ pass_filenames: false + - id: pre-commit-descriptions +name: Check if pre-commits are described +entry: ./scripts/ci/pre_commit/pre_commit_check_pre_commits.sh +language: system +files: ^.pre-commit-config.yaml$|^STATIC_CODE_CHECKS.rst|^breeze-complete$ +require_serial: true + - id: sort-in-the-wild +name: Sort INTHEWILD.md alphabetically +entry: ./scripts/ci/pre_commit/pre_commit_sort_in_the_wild.sh +language: system +files: ^.pre-commit-config.yaml$|^INTHEWILD.md$ +require_serial: true + - id: markdownlint +name: Run markdownlint +description: "Checks the style of Markdown files." +entry: markdownlint +language: node +types: [markdown] +files: \.(md|mdown|markdown)$ +additional_dependencies: ['markdownlint-cli'] - id: build name: Check if image build is needed entry: ./scripts/ci/pre_commit/pre_commit_ci_build.sh 3.6 false @@ -364,15 +384,3 @@ repos: entry: "./scripts/ci/pre_commit/pre_commit_in_container_bats_test.sh" files: ^tests/bats/in_container/.*.bats$|^scripts/in_container/.*sh pass_filenames: false - - id: pre-commit-descriptions -name: Check if pre-commits are described -entry: ./scripts/ci/pre_commit/pre_commit_check_pre_commits.sh -language: system -files: ^.pre-commit-config.yaml$|^STATIC_CODE_CHECKS.rst|^breeze-complete$ -require_serial: true - - id: sort-in-the-wild -name: Sort INTHEWILD.md alphabetically -entry: ./scripts/ci/pre_commit/pre_commit_sort_in_the_wild.sh -language: system -files: ^.pre-commit-config.yaml$|^INTHEWILD.md$ -require_serial: true diff --git a/BREEZE.rst b/BREEZE.rst index b66c7d2..c3c2d95 100644 --- a/BREEZE.rst +++ b/BREEZE.rst @@ -1863,10 +1863,10 @@ This is the current syntax for `./breeze
[airflow] 38/44: Docker context files should be available earlier (#12219)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 10b6c48b67ec2c23b256897a6e5b562178a1eb9b
Author: Jarek Potiuk
AuthorDate: Wed Nov 11 11:00:16 2020 +0100
Docker context files should be available earlier (#12219)
If you want to override constraints with local version,
the docker-context-files should be earlier in the Dockerfile
(cherry picked from commit 9b7e7603c4a01c6c943fa68df3b20a8859cdde1e)
---
.gitignore| 2 ++
Dockerfile.ci | 8
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/.gitignore b/.gitignore
index 27b6f0c..6b2650c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -188,6 +188,8 @@ dmypy.json
log.txt*
/backport_packages/CHANGELOG.txt
+# Docker context files
+/docker-context-files
# Local .terraform directories
**/.terraform/*
diff --git a/Dockerfile.ci b/Dockerfile.ci
index fae86b2..7699706 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -260,6 +260,10 @@ ENV AIRFLOW_LOCAL_PIP_WHEELS=${AIRFLOW_LOCAL_PIP_WHEELS}
ARG INSTALL_AIRFLOW_VIA_PIP="true"
ENV INSTALL_AIRFLOW_VIA_PIP=${INSTALL_AIRFLOW_VIA_PIP}
+# If wheel files are found in /docker-context-files during installation
+# they are also installed additionally to whatever is installed from Airflow.
+COPY docker-context-files /docker-context-files
+
# In case of CI builds we want to pre-install master version of airflow
dependencies so that
# We do not have to always reinstall it from the scratch.
# This can be reinstalled from latest master by increasing
PIP_DEPENDENCIES_EPOCH_NUMBER.
@@ -311,10 +315,6 @@ RUN if [[ ${INSTALL_AIRFLOW_VIA_PIP} == "true" ]]; then \
fi; \
fi
-# If wheel files are found in /docker-context-files sduring installation
-# they are also installed additionally to whatever is installed from Airflow.
-COPY docker-context-files /docker-context-files
-
RUN if [[ ${AIRFLOW_LOCAL_PIP_WHEELS} != "true" ]]; then \
if ls /docker-context-files/*.whl 1> /dev/null 2>&1; then \
pip install --no-deps /docker-context-files/*.whl; \
[airflow] 31/44: Fixed path of the test_core.py file in docs (#12191)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 10700af913b3415a97acc6bd7161dd1f2a3c2885
Author: Jarek Potiuk
AuthorDate: Mon Nov 9 10:34:06 2020 +0100
Fixed path of the test_core.py file in docs (#12191)
The test_core.py has been used as example in Breeze and it's
location changed to tests/core folder. This PR fixes references
to the changed location.
(cherry picked from commit 57b273a0b1b8af30ed017c2b24c498deb9010247)
---
BREEZE.rst | 2 +-
TESTING.rst | 8
breeze | 2 +-
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 5e481f7..b66c7d2 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1894,7 +1894,7 @@ This is the current syntax for `./breeze <./breeze>`_:
as additional options passed to pytest. You can pass 'tests' as target
to
run all tests. For example:
-'breeze tests tests/test_core.py -- --logging-level=DEBUG'
+'breeze tests tests/core/test_core.py -- --logging-level=DEBUG'
'breeze tests tests
Flags:
diff --git a/TESTING.rst b/TESTING.rst
index f4f790b..c8b170b 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -111,20 +111,20 @@ This can also be done by specifying a full path to the
test:
.. code-block:: bash
-pytest tests/test_core.py::TestCore::test_check_operators
+pytest tests/core/test_core.py::TestCore::test_check_operators
To run the whole test class, enter:
.. code-block:: bash
-pytest tests/test_core.py::TestCore
+pytest tests/core/test_core.py::TestCore
You can use all available ``pytest`` flags. For example, to increase a log
level
for debugging purposes, enter:
.. code-block:: bash
-pytest --log-level=DEBUG tests/test_core.py::TestCore
+pytest --log-level=DEBUG tests/core/test_core.py::TestCore
Running Tests for a Specified Target Using Breeze from the Host
@@ -149,7 +149,7 @@ You can also specify individual tests or a group of tests:
.. code-block:: bash
-./breeze tests --db-reset tests/test_core.py::TestCore
+./breeze tests --db-reset tests/core/test_core.py::TestCore
Running Tests of a specified type from the Host
diff --git a/breeze b/breeze
index 1d8ffea..3498c64 100755
--- a/breeze
+++ b/breeze
@@ -1763,7 +1763,7 @@ ${CMDNAME} tests [FLAGS] [TEST_TARGET ..] [--
]
as additional options passed to pytest. You can pass 'tests' as target to
run all tests. For example:
- '${CMDNAME} tests tests/test_core.py -- --logging-level=DEBUG'
+ '${CMDNAME} tests tests/core/test_core.py -- --logging-level=DEBUG'
'${CMDNAME} tests tests
Flags:
[airflow] 27/44: Fixes undefined variables (#12155)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit fc19025960f659b86d8d989b8b2db22010c9ebfd
Author: Jarek Potiuk
AuthorDate: Sat Nov 7 12:18:14 2020 +0100
Fixes undefined variables (#12155)
There are few more variables that (if not defined) prevent
from using the CI image directly without breeze or the
CI scripts.
With this change you can run:
`docker run -it apache/airflow:master-python3.6-ci`
and enter the image without errors.
(cherry picked from commit c7f34104516095c487cd6729f830df373ab78fbd)
---
scripts/ci/libraries/_initialization.sh | 4 ++--
scripts/in_container/entrypoint_ci.sh | 2 +-
scripts/in_container/run_init_script.sh | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/scripts/ci/libraries/_initialization.sh
b/scripts/ci/libraries/_initialization.sh
index e0b4a8a..b58fa2e 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -566,10 +566,10 @@ Detected CI build environment:
Initialization variables:
-INIT_SCRIPT_FILE: ${INIT_SCRIPT_FILE}
+INIT_SCRIPT_FILE: ${INIT_SCRIPT_FILE=}
LOAD_DEFAULT_CONNECTIONS: ${LOAD_DEFAULT_CONNECTIONS}
LOAD_EXAMPLES: ${LOAD_EXAMPLES}
-INSTALL_WHEELS: ${INSTALL_WHEELS}
+INSTALL_WHEELS: ${INSTALL_WHEELS=}
DISABLE_RBAC: ${DISABLE_RBAC}
Test variables:
diff --git a/scripts/in_container/entrypoint_ci.sh
b/scripts/in_container/entrypoint_ci.sh
index 8a7b683..37d3199 100755
--- a/scripts/in_container/entrypoint_ci.sh
+++ b/scripts/in_container/entrypoint_ci.sh
@@ -97,7 +97,7 @@ else
install_released_airflow_version "${INSTALL_AIRFLOW_VERSION}"
fi
-if [[ ${INSTALL_WHEELS} == "true" ]]; then
+if [[ ${INSTALL_WHEELS=} == "true" ]]; then
pip install /dist/*.whl || true
fi
diff --git a/scripts/in_container/run_init_script.sh
b/scripts/in_container/run_init_script.sh
index a4b030a..0bd685c 100755
--- a/scripts/in_container/run_init_script.sh
+++ b/scripts/in_container/run_init_script.sh
@@ -23,7 +23,7 @@ if [ -z "${AIRFLOW_BREEZE_CONFIG_DIR+x}" ]; then
export AIRFLOW_BREEZE_CONFIG_DIR="${FILES_DIR}/airflow-breeze-config"
fi
-if [ -z "${INIT_SCRIPT_FILE}" ]; then
+if [ -z "${INIT_SCRIPT_FILE=}" ]; then
export INIT_SCRIPT_FILE="init.sh"
fi
[airflow] 16/44: Fixes documentation-only selective checks (#12038)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ad8aa653f874141084973b00d1f73b1f1f6e5402
Author: Jarek Potiuk
AuthorDate: Mon Nov 2 15:16:24 2020 +0100
Fixes documentation-only selective checks (#12038)
There was a problem that documentation-only checks triggered
selective checks without docs build (they resulted in
basic-checks-only and no images being built.
This occured for example in #12025
This PR fixes it by adding image-build and docs-build as two
separate outputs.
(cherry picked from commit adbf764ade6916b505c3238697bac10f98bfa6eb)
---
.github/workflows/build-images-workflow-run.yml | 4 +-
.github/workflows/ci.yml| 41 +--
scripts/ci/selective_ci_checks.sh | 69 +
3 files changed, 63 insertions(+), 51 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index 6099727..a3277a7 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -201,7 +201,7 @@ jobs:
defaultPythonVersion: ${{
steps.selective-checks.outputs.default-python-version }}
run-tests: ${{ steps.selective-checks.outputs.run-tests }}
run-kubernetes-tests: ${{
steps.selective-checks.outputs.run-kubernetes-tests }}
- basic-checks-only: ${{ steps.selective-checks.outputs.basic-checks-only
}}
+ image-build: ${{ steps.selective-checks.outputs.image-build }}
if: >
needs.cancel-workflow-runs.outputs.buildImages == 'true'
steps:
@@ -266,7 +266,7 @@ jobs:
image-type: [CI, PROD]
fail-fast: true
if: >
- needs.build-info.outputs.basic-checks-only == 'false' &&
+ needs.build-info.outputs.image-build == 'true' &&
needs.cancel-workflow-runs.outputs.buildImages == 'true'
env:
BACKEND: postgres
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a5457c0..c217bdc 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -92,6 +92,8 @@ jobs:
run-tests: ${{ steps.selective-checks.outputs.run-tests }}
run-kubernetes-tests: ${{
steps.selective-checks.outputs.run-kubernetes-tests }}
basic-checks-only: ${{ steps.selective-checks.outputs.basic-checks-only
}}
+ image-build: ${{ steps.selective-checks.outputs.image-build }}
+ docs-build: ${{ steps.selective-checks.outputs.docs-build }}
needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
@@ -158,8 +160,7 @@ jobs:
name: "Wait for CI images"
runs-on: ubuntu-latest
needs: [build-info]
-if: >
- needs.build-info.outputs.basic-checks-only == 'false'
+if: needs.build-info.outputs.image-build == 'true'
env:
BACKEND: sqlite
steps:
@@ -196,8 +197,7 @@ jobs:
env:
MOUNT_LOCAL_SOURCES: "true"
PYTHON_MAJOR_MINOR_VERSION:
${{needs.build-info.outputs.defaultPythonVersion}}
-if: >
- needs.build-info.outputs.basic-checks-only == 'false'
+if: needs.build-info.outputs.basic-checks-only == 'false'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit
${{ github.merge_commit_sha }} )"
uses: actions/checkout@v2
@@ -233,8 +233,7 @@ jobs:
SKIP: "build,mypy,flake8,pylint,bats-in-container-tests"
MOUNT_LOCAL_SOURCES: "true"
PYTHON_MAJOR_MINOR_VERSION:
${{needs.build-info.outputs.defaultPythonVersion}}
-if: >
- needs.build-info.outputs.basic-checks-only == 'true'
+if: needs.build-info.outputs.basic-checks-only == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -266,8 +265,7 @@ jobs:
name: "Build docs"
runs-on: ubuntu-latest
needs: [build-info, ci-images]
-if: >
- needs.build-info.outputs.basic-checks-only == 'false'
+if: needs.build-info.outputs.docs-build == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit
${{ github.merge_commit_sha }})"
uses: actions/checkout@v2
@@ -348,8 +346,7 @@ jobs:
RUN_TESTS: true
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
TEST_TYPE: ""
-if: >
-needs.build-info.outputs.run-tests == 'true'
+if: needs.build-info.outputs.run-tests == '
[airflow] 23/44: Update install_mysql.sh (#12101)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b062b9259b81ba9c042a523f5454beb366e134ce
Author: J. Daniel Medeiros <3796667+jdmedei...@users.noreply.github.com>
AuthorDate: Thu Nov 5 15:32:39 2020 -0100
Update install_mysql.sh (#12101)
After Debian 9 and according to the manual
https://manpages.debian.org/stretch/apt/apt-key.8.en.html, after Debian 9
instead of using "apt-key add" a keyring should be placed directly in the
/etc/apt/trusted.gpg.d/ directory with a descriptive name and either "gpg" or
"asc" as file extension. Also added better redirection on the apt-key list
command.
(cherry picked from commit ded3dbbff0c8e4fabcee62a677394dec0db1aa45)
---
scripts/docker/install_mysql.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/docker/install_mysql.sh b/scripts/docker/install_mysql.sh
index 1ddc76b..0fbb1da 100755
--- a/scripts/docker/install_mysql.sh
+++ b/scripts/docker/install_mysql.sh
@@ -46,10 +46,10 @@ if [[ ${INSTALL_MYSQL_CLIENT:="true"} == "true" ]]; then
gpg --keyserver "${keyserver}" --recv-keys "${KEY}" && break
done
set -e
-gpg --export "${KEY}" | apt-key add -
+gpg --export "${KEY}" > /etc/apt/trusted.gpg.d/mysql.gpg
gpgconf --kill all
rm -rf "${GNUPGHOME}"
-apt-key list > /dev/null
+apt-key list > /dev/null 2>&1
echo "deb http://repo.mysql.com/apt/debian/ buster mysql-5.7" | tee -a
/etc/apt/sources.list.d/mysql.list
apt-get update
apt-get install --no-install-recommends -y "${packages[@]}"
[airflow] 22/44: Fix proper SHA in check preventing accidentally merging PR (#12083)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 64393ad508119197907f8a7aa61afa82dd506095
Author: Jarek Potiuk
AuthorDate: Wed Nov 4 13:51:04 2020 +0100
Fix proper SHA in check preventing accidentally merging PR (#12083)
The SHA in check was not working for PRs from forks.
(cherry picked from commit d559da14b0a99b369ba027b4b6261321b6b8e03c)
---
.github/workflows/label_when_reviewed_workflow_run.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/label_when_reviewed_workflow_run.yml
b/.github/workflows/label_when_reviewed_workflow_run.yml
index 56bc835..cdcbca5 100644
--- a/.github/workflows/label_when_reviewed_workflow_run.yml
+++ b/.github/workflows/label_when_reviewed_workflow_run.yml
@@ -93,7 +93,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
name: "Please rebase or re-run to run full tests"
status: "in_progress"
- sha: ${{ github.event.pull_request.head.sha }}
+ sha: ${{ steps.source-run-info.outputs.sourceHeadSha }}
details_url: https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }}
output: >
{"summary":
[airflow] 15/44: Checks if all the libraries in setup.py are listed in installation.rst file (#12023)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 43c72f2f4c7d0b72d9fcbbfb17135d9be59eb4c9 Author: SZN AuthorDate: Mon Nov 2 14:17:41 2020 +0100 Checks if all the libraries in setup.py are listed in installation.rst file (#12023) (cherry picked from commit 2354bd2be381bcfe6db132990af1ac34df52b9b4) --- .pre-commit-config.yaml| 6 + BREEZE.rst | 9 +- CONTRIBUTING.rst | 16 +- INSTALL| 16 +- STATIC_CODE_CHECKS.rst | 2 + breeze-complete| 1 + docs/installation.rst | 216 +++-- .../pre_commit_check_setup_installation.py | 108 +++ setup.py | 16 ++ 9 files changed, 314 insertions(+), 76 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 6763164..cb5c753 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -213,6 +213,12 @@ repos: files: ^setup.py$ pass_filenames: false entry: ./scripts/ci/pre_commit/pre_commit_check_order_setup.py + - id: setup-installation +name: Checks if all the libraries in setup.py are listed in installation.rst file +language: python +files: ^setup.py$|^docs/installation.rst$ +pass_filenames: false +entry: ./scripts/ci/pre_commit/pre_commit_check_setup_installation.py - id: update-breeze-file name: Update output of breeze command in BREEZE.rst entry: "./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.sh" diff --git a/BREEZE.rst b/BREEZE.rst index cf119ba..5e481f7 100644 --- a/BREEZE.rst +++ b/BREEZE.rst @@ -395,7 +395,7 @@ run ``airflow webserver``, ``airflow scheduler``, ``airflow worker`` in separate This can be achieved either via ``tmux`` or via exec-ing into the running container from the host. Tmux is installed inside the container and you can launch it with ``tmux`` command. Tmux provides you with the capability of creating multiple virtual terminals and multiplex between them. More about ``tmux`` can be -found at `tmux github wiki page <https://github.com/tmux/tmux/wiki>`_ . Tmux has several useful shortcuts +found at `tmux GitHub wiki page <https://github.com/tmux/tmux/wiki>`_ . Tmux has several useful shortcuts that allow you to split the terminals, open new tabs etc - it's pretty useful to learn it. .. raw:: html @@ -1001,7 +1001,7 @@ Managing Dependencies - If you need to change apt dependencies in the ``Dockerfile.ci``, add Python packages in ``setup.py`` or -add javascript dependencies in ``package.json``, you can either add dependencies temporarily for a single +add JavaScript dependencies in ``package.json``, you can either add dependencies temporarily for a single Breeze session or permanently in ``setup.py``, ``Dockerfile.ci``, or ``package.json`` files. Installing Dependencies for a Single Breeze Session @@ -1865,8 +1865,9 @@ This is the current syntax for `./breeze <./breeze>`_: flake8 forbid-tabs helm-lint incorrect-use-of-LoggingMixin insert-license language-matters lint-dockerfile lint-openapi mixed-line-ending mypy mypy-helm no-relative-imports pre-commit-descriptions pydevd python2-compile python2-fastcheck - python-no-log-warn rst-backticks setup-order shellcheck sort-in-the-wild - trailing-whitespace update-breeze-file update-extras update-local-yml-file yamllint + python-no-log-warn rst-backticks setup-order setup-installation shellcheck + sort-in-the-wild trailing-whitespace update-breeze-file update-extras + update-local-yml-file yamllint You can pass extra arguments including options to to the pre-commit framework as passed after --. For example: diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 6d3aa91..6d34026 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -540,13 +540,15 @@ This is the full list of those extras: .. START EXTRAS HERE -all, all_dbs, async, atlas, aws, azure, azure_blob_storage, azure_container_instances, azure_cosmos, -azure_data_lake, azure_secrets, cassandra, celery, cgroups, cloudant, crypto, dask, databricks, -datadog, devel, devel_azure, devel_ci, devel_hadoop, doc, docker, druid, elasticsearch, emr, gcp, -gcp_api, github_enterprise, google_auth, grpc, hashicorp, hdfs, hive, jdbc, jira, kerberos, -kubernetes, ldap, mongo, mssql, mysql, oracle, papermill, password, pinot, postgres, presto, qds, -rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentr
[airflow] 28/44: Fixes "--force-clean-images" flag in Breeze (#12156)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 815f18b06c8711208971d9766350856d892b695c
Author: Jarek Potiuk
AuthorDate: Sat Nov 7 13:51:01 2020 +0100
Fixes "--force-clean-images" flag in Breeze (#12156)
The flag was broken - bad cache parameter value was passed.
This PR fixes it.
(cherry picked from commit 5c601578194739286097a75f082af2d83539581f)
---
breeze| 2 +-
scripts/ci/libraries/_build_images.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/breeze b/breeze
index 7838c88..1d8ffea 100755
--- a/breeze
+++ b/breeze
@@ -884,7 +884,7 @@ function breeze::parse_arguments() {
-C | --force-clean-images)
echo "Clean build of images without cache"
echo
-export DOCKER_CACHE="no-cache"
+export DOCKER_CACHE="disabled"
# if not set here, docker cached is determined later, depending on
type of image to be build
readonly DOCKER_CACHE
export FORCE_BUILD_IMAGES="true"
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index a6dbae0..cbd01c1 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -536,7 +536,7 @@ function build_images::build_ci_image() {
)
else
echo >&2
-echo >&2 "Error - thee ${DOCKER_CACHE} cache is unknown!"
+echo >&2 "Error - the ${DOCKER_CACHE} cache is unknown!"
echo >&2
exit 1
fi
[airflow] 12/44: Fixes problem with non-iterable data returned by GH API (#12021)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e18d6752814ea17cf596048cb8196224c46586a7
Author: Jarek Potiuk
AuthorDate: Sun Nov 1 19:25:29 2020 +0100
Fixes problem with non-iterable data returned by GH API (#12021)
The action to cancel workflow switched from deprecated
method of retrieving jobs to a 'better' one but it caused
some unexpected failures as some of the job data is not iterable and
failures in 'failedJobs" matching
Version 4.6 fixed the problem.
(cherry picked from commit aa5213b3609388b7b77e362dd9ad0bb05a8b5bc6)
---
.github/workflows/build-images-workflow-run.yml | 16
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index a3880e7..752e577 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: allDuplicates
@@ -82,7 +82,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately
# and then we try to find and cancel all the jobs with the same Event
+ Repo + Branch as the
# current Event/Repo/Branch combination.
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -98,7 +98,7 @@ jobs:
# We also produce list of canceled "CI Build' runs as output, so that
we
# can cancel all the matching "Build Images" workflow runs in the two
following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -131,14 +131,14 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run
Id:
# followed by one of the run ids. Yes I know it's super complex
¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' &&
steps.cancel-failed.outputs.cancelledRuns != '[]'
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{
steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
- uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -167,7 +167,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately and then
we try to match
# all the jobs with the same Event + Repo + Branch match and cancel
all the duplicates for those
# This might cancel own run, so this is the last step in the job
- uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
cancelMode: allDuplicatedNamedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -382,7 +382,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+uses:
potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -397,7 +397,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
-uses:
potiuk/cancel-workflow-runs@f06d03cd576a179ea516
[airflow] 37/44: Fixes continuous image rebuilding with Breeze (#12256)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4ca85a6a315c5c46aad95a67482e09de707c0293
Author: Jarek Potiuk
AuthorDate: Tue Nov 10 17:34:52 2020 +0100
Fixes continuous image rebuilding with Breeze (#12256)
There was a problem that even if we pulled the right image
from the Airflow repository, we have not tagged it properly.
Also added protection for people who have not yet at all pulled
the Python image from airflow, to force pull for the first time.
(cherry picked from commit 09febee4c1caf4e430fcbad2386f80aab52a0b15)
---
scripts/ci/libraries/_push_pull_remove_images.sh | 6 ++
1 file changed, 6 insertions(+)
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh
b/scripts/ci/libraries/_push_pull_remove_images.sh
index e853f42..4c51f0a 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -105,6 +105,11 @@ function
push_pull_remove_images::pull_image_github_dockerhub() {
# Pulls CI image in case caching strategy is "pulled" and the image needs to
be pulled
function push_pull_remove_images::pull_ci_images_if_needed() {
if [[ "${DOCKER_CACHE}" == "pulled" ]]; then
+local python_image_hash
+python_image_hash=$(docker images -q "${AIRFLOW_CI_PYTHON_IMAGE}" 2>
/dev/null || true)
+if [[ -z "${python_image_hash=}" ]]; then
+FORCE_PULL_IMAGES="true"
+fi
if [[ "${FORCE_PULL_IMAGES}" == "true" ]]; then
echo
echo "Force pull base image ${PYTHON_BASE_IMAGE}"
@@ -122,6 +127,7 @@ Docker pulling ${PYTHON_BASE_IMAGE}.
push_pull_remove_images::pull_image_github_dockerhub
"${PYTHON_BASE_IMAGE}"
"${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
else
docker pull "${AIRFLOW_CI_PYTHON_IMAGE}"
+docker tag "${AIRFLOW_CI_PYTHON_IMAGE}" "${PYTHON_BASE_IMAGE}"
fi
echo
fi
[airflow] 39/44: Added k9s as integrated tool to help with kubernetes testing (#12163)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d8db36a1b1bee478af4b4ca3e8ac6471da557a96
Author: Jarek Potiuk
AuthorDate: Wed Nov 11 17:15:02 2020 +0100
Added k9s as integrated tool to help with kubernetes testing (#12163)
The K9s is fantastic tool that helps to debug a running k8s
instance. It is terminal-based windowed CLI that makes you
several times more productive comparing to using kubectl
commands. We've integrated k9s (it is run as a docker container
and downloaded on demand). We've also separated out KUBECONFIG
of the integrated kind cluster so that it does not mess with
kubernetes configuration you might already have.
Also - together with that the "surrounding" of the kubernetes
tests were simplified and improved so that the k9s integration
can be utilized well. Instead of kubectl port forwarding (which
caused multitude of problems) we are now utilizing kind's
portMapping feature + custom NodePort resource that maps
port 8080 to 30007 NodePort which in turn maps it to 8080
port of the Webserver. This way we do not have to establish
an external kubectl port forward which is prone to error and
management - everything is brought up when Airflow gets
deployed to the Kind Cluster and shuts down when the Kind
cluster is stopped.
Yet another problem fixed was killing of postgres by one of the
kubernetes tests ('test_integration_run_dag_with_scheduler_failure').
Instead of just killing the scheduler it killed all pods - including
the Postgres one (it was named 'airflow-postgres.*'). That caused
various problems, as the database could be left in a strange state.
I changed the tests to do what it claimed was doing - so killing only the
scheduler during the test. This seemed to improve the stability
of tests immensely in my local setup.
(cherry picked from commit 21999dd56e9dbe9f7f9e25961954c5677c3c7c58)
---
.github/workflows/ci.yml | 17 +-
BREEZE.rst | 11 +-
TESTING.rst| 151 -
breeze | 9 +
breeze-complete| 5 +-
chart/requirements.lock| 4 +-
images/testing/k9s.png | Bin 0 -> 238713 bytes
images/testing/kubeconfig-env.png | Bin 0 -> 231280 bytes
images/testing/kubernetes-virtualenv.png | Bin 0 -> 110011 bytes
images/testing/pytest-runner.png | Bin 0 -> 131589 bytes
images/testing/run-test.png| Bin 0 -> 140728 bytes
kubernetes_tests/test_kubernetes_executor.py | 7 +-
kubernetes_tests/test_kubernetes_pod_operator.py | 672 -
scripts/ci/kubernetes/ci_run_kubernetes_tests.sh | 7 +-
...up_cluster_and_deploy_airflow_to_kubernetes.sh} | 3 +-
scripts/ci/kubernetes/kind-cluster-conf.yaml | 5 +
.../{kind-cluster-conf.yaml => nodeport.yaml} | 30 +-
...oy_app_to_kubernetes.sh => redeploy_airflow.sh} | 6 +-
scripts/ci/libraries/_kind.sh | 126 ++--
19 files changed, 516 insertions(+), 537 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 6c854a1..81890a7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -636,23 +636,14 @@ jobs:
python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
- name: "Free space"
run: ./scripts/ci/tools/ci_free_space_on_ci.sh
- - name: "Setup Kind Cluster ${{ env.KIND_VERSION }}"
-uses: engineerd/setup-kind@v0.4.0
-with:
- version: "${{ env.KIND_VERSION }}"
- name:
airflow-python-${{matrix.python-version}}-${{matrix.kubernetes-version}}
- config: "scripts/ci/kubernetes/kind-cluster-conf.yaml"
- name: "Prepare PROD Image"
run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
- - name: "Deploy airflow to cluster"
-id: deploy-app
-run: ./scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh
+ - name: "Setup cluster and deploy Airflow"
+id: setp-cluster-deploy-app
+run:
./scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
env:
# We have the right image pulled already by the previous step
SKIP_BUILDING_PROD_IMAGE: "true"
- # due to some instabilities, in CI we try to increase port numbers
when trying to establish
- # port forwarding
- INCREASE_PORT_NUMBER_FOR_KUBERNETES: "true"
- name: "Cache
[airflow] 42/44: For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4ab76df0eb547fc2af670473d38536ab9b1774b1
Author: Jarek Potiuk
AuthorDate: Fri Nov 13 14:28:36 2020 +0100
For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
Previously, always master scripts were used to build images
for workflow_run, because workflow_run always runs from master
branch. However that causes some surprising effects becuase the
sripts from master had to support both master and 1.10.
This change utilises a new feature in the "get-workflow-origin"
action - to get the target branch of PR and uses ci scripts from that
target branch.
This is perfectly secure, because both v1-10-test, v1-10-stable
and future 2-0 branches can only be updated by committers,
either by direct push or by merge.
(cherry picked from commit 7c4fe19e41ae02a1df1c0a217501cae2e0e84819)
---
.github/workflows/build-images-workflow-run.yml| 12 +--
.github/workflows/ci.yml | 38 +++---
.../workflows/label_when_reviewed_workflow_run.yml | 2 +-
3 files changed, 22 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml
b/.github/workflows/build-images-workflow-run.yml
index a3277a7..af71710 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -53,13 +53,14 @@ jobs:
targetCommitSha: ${{ steps.source-run-info.outputs.targetCommitSha }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
+ targetBranch: ${{ steps.source-run-info.outputs.targetBranch }}
sourceEvent: ${{ steps.source-run-info.outputs.sourceEvent }}
cacheDirective: ${{ steps.cache-directive.outputs.docker-cache }}
buildImages: ${{ steps.build-images.outputs.buildImages }}
upgradeToLatestConstraints: ${{
steps.upgrade-constraints.outputs.upgradeToLatestConstraints }}
steps:
- name: "Get information about the original trigger of the run"
- uses:
potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+ uses:
potiuk/get-workflow-origin@588cc14f9f1cdf1b8be3db816855e96422204fec # v1_3
id: source-run-info
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -326,17 +327,22 @@ jobs:
[Image Build](https://github.com/${{ github.repository
}}/actions/runs/${{ github.run_id }})
for details" }
if: steps.defaults.outputs.proceed == 'true'
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} ) to
'main-airflow' to use main scripts"
+ - name: >
+ Checkout "${{ needs.cancel-workflow-runs.outputs.targetBranch }}"
branch to 'main-airflow' folder
+ to use ci/scripts from there.
uses: actions/checkout@v2
with:
path: "main-airflow"
+ ref: "${{ needs.cancel-workflow-runs.outputs.targetBranch }}"
if: steps.defaults.outputs.proceed == 'true'
- name: "Setup python"
uses: actions/setup-python@v2
with:
python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
if: steps.defaults.outputs.proceed == 'true'
- - name: "Override 'scripts/ci' with the ${{ github.ref }} version so
that the PR cannot override it."
+ - name: >
+ Override "scripts/ci" with the "${{
needs.cancel-workflow-runs.outputs.targetBranch }}" branch
+ so that the PR does not override it
# We should not override those scripts which become part of the image
as they will not be
# changed in the image built - we should only override those that are
executed to build
# the image.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 81890a7..df5f53e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -95,27 +95,15 @@ jobs:
image-build: ${{ steps.selective-checks.outputs.image-build }}
docs-build: ${{ steps.selective-checks.outputs.docs-build }}
needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
- needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
steps:
- name: "Get information about the PR"
-uses:
potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+uses:
potiuk/ge
[airflow] 40/44: Python base image is shared between CI and PROD image (#12280)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 76f51e15fd4afbaad77b22f01768cc1a795e988b
Author: Jarek Potiuk
AuthorDate: Thu Nov 12 12:31:14 2020 +0100
Python base image is shared between CI and PROD image (#12280)
When you are building CI images locally you use the CI
base images from apache:airflow/python* now to maintain
consistency and avoid often rebuilds. But when you build
prod images, you would accidentaly override it with the
python base image available in python repo which might be
different (newer and not yet tested in CI). This PR
changes it to use the same base image which is now
tagged in Apache Airflow's dockerhub repository.
(cherry picked from commit 3c2c29187afffb1a1dd1327803097651cdc079ee)
---
scripts/ci/libraries/_push_pull_remove_images.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh
b/scripts/ci/libraries/_push_pull_remove_images.sh
index 4c51f0a..7c65db1 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -154,7 +154,8 @@ function
push_pull_remove_images::pull_prod_images_if_needed() {
fi
push_pull_remove_images::pull_image_github_dockerhub
"${PYTHON_BASE_IMAGE}"
"${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
else
-docker pull "${PYTHON_BASE_IMAGE}"
+docker pull "${AIRFLOW_CI_PYTHON_IMAGE}"
+docker tag "${AIRFLOW_CI_PYTHON_IMAGE}" "${PYTHON_BASE_IMAGE}"
fi
echo
fi
[airflow] 26/44: Update to new helm stable repo (#12137)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2ddafd037a1f919aaabff4708c69b9766d2225b2
Author: Ash Berlin-Taylor
AuthorDate: Fri Nov 6 16:05:18 2020 +
Update to new helm stable repo (#12137)
Switch out deprecated helm repo for new stable repo.
-
https://www.cncf.io/blog/2020/11/05/helm-chart-repository-deprecation-update/
-
https://helm.sh/docs/faq/#i-am-getting-a-warning-about-unable-to-get-an-update-from-the-stable-chart-repository
(cherry picked from commit 128c9918b5f79cb46a563b77e803c29548c4319c)
---
chart/README.md | 2 +-
chart/requirements.lock | 6 +++---
chart/tests/conftest.py | 4 +---
scripts/ci/libraries/_kind.sh | 2 +-
4 files changed, 6 insertions(+), 8 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index e7ef973..7d23ce3 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -37,7 +37,7 @@ cluster using the [Helm](https://helm.sh) package manager.
To install this repository from source (using helm 3)
```bash
kubectl create namespace airflow
-helm repo add stable https://kubernetes-charts.storage.googleapis.com
+helm repo add stable https://charts.helm.sh/stable/
helm dep update
helm install airflow . --namespace airflow
```
diff --git a/chart/requirements.lock b/chart/requirements.lock
index 715458e..3f3c34a 100644
--- a/chart/requirements.lock
+++ b/chart/requirements.lock
@@ -1,6 +1,6 @@
dependencies:
- name: postgresql
- repository: https://kubernetes-charts.storage.googleapis.com/
+ repository: https://charts.helm.sh/stable/
version: 6.3.12
-digest: sha256:e8d53453861c590e6ae176331634c9268a11cf894be17ed580fa2b347101be97
-generated: "2020-10-27T21:16:13.0063538Z"
+digest: sha256:58d88cf56e78b2380091e9e16cc6ccf58b88b3abe4a1886dd47cd9faef5309af
+generated: "2020-11-04T15:59:36.967913-08:00"
diff --git a/chart/tests/conftest.py b/chart/tests/conftest.py
index db1f22f..789f311 100644
--- a/chart/tests/conftest.py
+++ b/chart/tests/conftest.py
@@ -26,7 +26,5 @@ def upgrade_helm():
"""
Upgrade Helm repo
"""
-subprocess.check_output(
-["helm", "repo", "add", "stable",
"https://kubernetes-charts.storage.googleapis.com/";]
-)
+subprocess.check_output(["helm", "repo", "add", "stable",
"https://charts.helm.sh/stable/";])
subprocess.check_output(["helm", "dep", "update", sys.path[0]])
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index 41048fc..6194742 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -336,7 +336,7 @@ function kind::deploy_airflow_with_helm() {
kubectl create namespace "${HELM_AIRFLOW_NAMESPACE}"
kubectl create namespace "test-namespace"
pushd "${AIRFLOW_SOURCES}/chart" || exit 1
-helm repo add stable https://kubernetes-charts.storage.googleapis.com
+helm repo add stable https://charts.helm.sh/stable/
helm dep update
helm install airflow . --namespace "${HELM_AIRFLOW_NAMESPACE}" \
--set "defaultAirflowRepository=${DOCKERHUB_USER}/${DOCKERHUB_REPO}" \
