[airflow] 05/09: Remove redundant airflowVersion from Helm Chart readme (#9592)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 93570f34ecd815ffadd1ecb01fbcd8b2f7505167 Author: Kaxil Naik AuthorDate: Tue Jun 30 17:02:56 2020 +0100 Remove redundant airflowVersion from Helm Chart readme (#9592) We no longer use `airflowVersion` , we instead use `defaultAirflowRepository` and `defaultAirflowTag` (cherry picked from commit d6b323b0cd9be2aa941cbb1e1e15d766b4d6539b) --- chart/README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/chart/README.md b/chart/README.md index 402a9d7..089ea22 100644 --- a/chart/README.md +++ b/chart/README.md @@ -91,7 +91,6 @@ The following tables lists the configurable parameters of the Airflow chart and | `networkPolicies.enabled` | Enable Network Policies to restrict traffic | `true`| | `airflowHome` | Location of airflow home directory | `/opt/airflow`| | `rbacEnabled` | Deploy pods with Kubernets RBAC enabled | `true`| -| `airflowVersion` | Default Airflow image version | `1.10.5` | | `executor`| Airflow executor (eg SequentialExecutor, LocalExecutor, CeleryExecutor, KubernetesExecutor) | `KubernetesExecutor` | | `allowPodLaunching` | Allow airflow pods to talk to Kubernetes API to launch more pods | `true`| | `defaultAirflowRepository`| Fallback docker repository to pull airflow image from | `apache/airflow` |
[airflow] 06/09: Fix broken link in chart/README.md (#9591)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit ad618a80178012cd5eb3995d2b0979717901d1b8 Author: Kaxil Naik AuthorDate: Tue Jun 30 17:03:11 2020 +0100 Fix broken link in chart/README.md (#9591) `CONTRIBUTING.md` -> `../CONTRIBUTING.rst` (cherry picked from commit bbfaafeb552b48560960ab4aba84723b7ccbf386) --- chart/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/chart/README.md b/chart/README.md index 089ea22..76d14b4 100644 --- a/chart/README.md +++ b/chart/README.md @@ -264,4 +264,4 @@ to port-forward the Airflow UI to http://localhost:8080/ to cofirm Airflow is wo ## Contributing -Check out [our contributing guide!](CONTRIBUTING.md) +Check out [our contributing guide!](../CONTRIBUTING.rst)
[airflow] 03/09: Remove non-existent chart value from readme (#9511)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c363c1b2f2de4e7906bf5807cae6d47b48d60eb2
Author: Ash Berlin-Taylor
AuthorDate: Thu Jun 25 12:19:26 2020 +0100
Remove non-existent chart value from readme (#9511)
This was accidentally left over when this was extracted from
Astronomer's chart.
(cherry picked from commit 561060aaa82ddb63fe2a38473bfd920a5aeff786)
---
chart/README.md | 1 -
1 file changed, 1 deletion(-)
diff --git a/chart/README.md b/chart/README.md
index 8657eee..d0366f6 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -157,7 +157,6 @@ The following tables lists the configurable parameters of
the Airflow chart and
| `webserver.resources.limits.memory` | Memory Limit of
webserver
| `~` |
| `webserver.resources.requests.cpu`| CPU Request of
webserver
| `~` |
| `webserver.resources.requests.memory` | Memory Request of
webserver
| `~` |
-| `webserver.jwtSigningCertificateSecretName` | Name of secret to
mount Airflow Webserver JWT singing certificate from
| `~` |
| `webserver.defaultUser` | Optional default
airflow user information
| `{}` |
[airflow] 07/09: Switches to Helm Chart for Kubernetes tests (#9468)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9253a8f65f6d25bed13c0f92c6c01d5b55950a06
Author: Jarek Potiuk
AuthorDate: Wed Jul 1 14:50:30 2020 +0200
Switches to Helm Chart for Kubernetes tests (#9468)
The Kubernetes tests are now run using Helm chart
rather than the custom templates we used to have.
The Helm Chart uses locally build production image
so the tests are testing not only Airflow but also
Helm Chart and a Production image - all at the
same time. Later on we will add more tests
covering more functionalities of both Helm Chart
and Production Image. This is the first step to
get all of those bundle together and become
testable.
This change introduces also 'shell' sub-command
for Breeze's kind-cluster command and
EMBEDDED_DAGS build args for production image -
both of them useful to run the Kubernetes tests
more easily - without building two images
and with an easy-to-iterate-over-tests
shell command - which works without any
other development environment.
Co-authored-by: Jarek Potiuk
Co-authored-by: Daniel Imberman
(cherry picked from commit 8bd15ef634cca40f3cf6ca3442262f3e05144512)
---
.github/workflows/ci.yml | 23 +-
BREEZE.rst | 81 +++--
CI.rst | 2 +-
Dockerfile | 4 +
IMAGES.rst | 3 +
TESTING.rst| 67 ++--
airflow/kubernetes/pod_launcher.py | 2 +-
breeze | 51 ++-
breeze-complete| 14 +-
chart/README.md| 5 +-
chart/charts/postgresql-6.3.12.tgz | Bin 22754 -> 0 bytes
chart/requirements.lock| 4 +-
chart/templates/configmap.yaml | 2 +
chart/templates/rbac/pod-launcher-role.yaml| 2 +-
chart/templates/rbac/pod-launcher-rolebinding.yaml | 4 +-
kubernetes_tests/test_kubernetes_executor.py | 40 ++-
scripts/ci/ci_build_production_images.sh | 25 --
scripts/ci/ci_count_changed_files.sh | 2 +-
scripts/ci/ci_deploy_app_to_kubernetes.sh | 16 +-
scripts/ci/ci_docs.sh | 2 +-
scripts/ci/ci_flake8.sh| 2 +-
scripts/ci/ci_generate_requirements.sh | 2 +-
scripts/ci/ci_load_image_to_kind.sh| 7 +-
scripts/ci/ci_mypy.sh | 2 +-
scripts/ci/ci_perform_kind_cluster_operation.sh| 6 +-
scripts/ci/ci_run_airflow_testing.sh | 2 +-
scripts/ci/ci_run_kubernetes_tests.sh | 6 +-
scripts/ci/ci_run_static_checks.sh | 2 +-
scripts/ci/kubernetes/app/postgres.yaml| 94 -
.../kubernetes/app/templates/airflow.template.yaml | 207 ---
.../app/templates/configmaps.template.yaml | 395 -
.../app/templates/init_git_sync.template.yaml | 36 --
scripts/ci/kubernetes/app/volumes.yaml | 87 -
.../docker/airflow-test-env-init-dags.sh | 36 --
.../kubernetes/docker/airflow-test-env-init-db.sh | 46 ---
scripts/ci/kubernetes/docker/bootstrap.sh | 74
scripts/ci/kubernetes/kind-cluster-conf.yaml | 3 -
.../kubernetes/{app/secrets.yaml => volumes.yaml} | 29 +-
scripts/ci/libraries/_build_images.sh | 11 +-
scripts/ci/libraries/_initialization.sh| 27 +-
scripts/ci/libraries/_kind.sh | 380 +++-
scripts/ci/libraries/_verbosity.sh | 31 ++
42 files changed, 424 insertions(+), 1410 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index d091d2e..195f7f7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -100,7 +100,7 @@ jobs:
steps:
- uses: actions/checkout@master
- name: "Build PROD image ${{ matrix.python-version }}"
-run: ./scripts/ci/ci_build_production_images.sh
+run: ./scripts/ci/ci_prepare_prod_image_on_ci.sh
tests-kubernetes:
timeout-minutes: 80
@@ -113,7 +113,11 @@ jobs:
kube-mode:
- image
kubernetes-version:
- - "v1.15.3"
+ - "v1.18.2"
+kind-version:
+ - "v0.8.0"
+helm-version:
+ - "v3.2.4"
fail-fast: false
env:
BACKEND: postgres
@@ -126,6 +130,8 @@ jobs:
PYTHON_MAJOR_MINOR_VERSION: "${{ matrix.python-version }}"
KUBERNETES_MODE: "${{ matrix.kube-mode }}"
KUBERNETES_VERSION: "${{
[airflow] 09/09: Update Breeze documentation (#9608)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4f40b889059b6eb8d3b710a3abca5c526659861c
Author: Jarek Potiuk
AuthorDate: Wed Jul 1 16:02:24 2020 +0200
Update Breeze documentation (#9608)
* Update Breeze documentation
(cherry picked from commit f3e1f9a313d8a6f841f6a5c9f2663518fee16b8f)
---
BREEZE.rst | 293
TESTING.rst | 2 +-
2 files changed, 198 insertions(+), 97 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 9b318e2..735286a 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -232,44 +232,6 @@ from your ``logs`` directory in the Airflow sources, so
all logs created in the
visible in the host as well. Every time you enter the container, the ``logs``
directory is
cleaned so that logs do not accumulate.
-CLIs for cloud providers
-
-
-For development convenience we installed simple wrappers for the most common
cloud providers CLIs. Those
-CLIs are not installed when you build or pull the image - they will be
downloaded as docker images
-the first time you attempt to use them. It is downloaded and executed in your
host's docker engine so once
-it is downloaded, it will stay until you remove the downloaded images from
your host container.
-
-For each of those CLI credentials are taken (automatically) from the
credentials you have defined in
-your ${HOME} directory on host.
-
-Those tools also have host Airflow source directory mounted in /opt/airflow
path
-so you can directly transfer files to/from your airflow host sources.
-
-Those are currently installed CLIs (they are available as aliases to the
docker commands):
-
-+---+--+-+---+
-| Cloud Provider| CLI tool | Docker image
| Configuration dir |
-+===+==+=+===+
-| Amazon Web Services | aws | amazon/aws-cli:latest
| .aws |
-+---+--+-+---+
-| Microsoft Azure | az | mcr.microsoft.com/azure-cli:latest
| .azure|
-+---+--+-+---+
-| Google Cloud Platform | bq |
gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud|
-|
+--+-+---+
-| | gcloud |
gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud|
-|
+--+-+---+
-| | gsutil |
gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud|
-+---+--+-+---+
-
-For each of the CLIs we have also an accompanying ``*-update`` alias (for
example ``aws-update``) which
-will pull the latest image for the tool. Note that all Google Cloud Platform
tools are served by one
-image and they are updated together.
-
-Also - in case you run several different Breeze containers in parallel (from
different directories,
-with different versions) - they docker images for CLI Cloud Providers tools
are shared so if you update it
-for one Breeze container, they will also get updated for all the other
containers.
-
Using the Airflow Breeze Environment
=
@@ -287,6 +249,7 @@ Managing CI environment:
* Stop running interactive environment with ``breeze stop`` command
* Restart running interactive environment with ``breeze restart`` command
* Run test specified with ``breeze tests`` command
+* Generate requirements with ``breeze generate-requirements`` command
* Execute arbitrary command in the test environment with ``breeze shell``
command
* Execute arbitrary docker-compose command with ``breeze docker-compose``
command
* Push docker images with ``breeze push-image`` command (require
committer's rights to push images)
@@ -319,7 +282,7 @@ Manage and Interact with Kubernetes tests environment:
Run static checks:
* Run static checks - either for currently staged change or for all files
with
- ``breeze static-check`` or ``breeze static-check-all-files`` command
+ ``breeze static-check`` command
Build documentation:
@@ -330,10 +293,12 @@ Set up local development environment:
* Setup local virtualenv with ``breeze setup-virtualenv`` command
* Setup autocomplete for itself with ``breeze setup-autocomplete`` command
-
[airflow] branch v1-10-test updated (836f717 -> 4f40b88)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. from 836f717 Fix task and dag stats on home page (#8865) new dc9c7fd Add Production Helm chart support (#8777) new d374278 Fix typo in helm chart upgrade command for 2.0 (#9484) new c363c1b Remove non-existent chart value from readme (#9511) new dd40f47 Fix typo of resultBackendConnection in chart README (#9537) new 93570f3 Remove redundant airflowVersion from Helm Chart readme (#9592) new ad618a8 Fix broken link in chart/README.md (#9591) new 9253a8f Switches to Helm Chart for Kubernetes tests (#9468) new ec3f4dc Removes importlib usage - it's not needed (fails on Airflow 1.10) (#9613) new 4f40b88 Update Breeze documentation (#9608) The 9 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/ci.yml | 23 +- .pre-commit-config.yaml| 2 +- BREEZE.rst | 374 -- CI.rst | 2 +- Dockerfile | 4 + IMAGES.rst | 3 + TESTING.rst| 67 ++-- airflow/kubernetes/pod_launcher.py | 2 +- breeze | 51 ++- breeze-complete| 14 +- chart/.gitignore | 9 + .../.helmignore| 33 +- .readthedocs.yml => chart/Chart.yaml | 20 +- chart/README.md| 270 + chart/charts/postgresql-6.3.12.tgz | Bin 22754 -> 0 bytes chart/requirements.lock| 6 + .../libs/helper.py => chart/requirements.yaml | 11 +- .../LICENSE.txt => chart/templates/NOTES.txt | 13 + chart/templates/_helpers.yaml | 260 chart/templates/cleanup/cleanup-cronjob.yaml | 67 .../templates/cleanup/cleanup-serviceaccount.yaml | 24 +- chart/templates/configmap.yaml | 119 ++ chart/templates/create-user-job.yaml | 87 chart/templates/flower/flower-deployment.yaml | 102 + chart/templates/flower/flower-networkpolicy.yaml | 51 +++ .../templates/flower/flower-service.yaml | 41 +- .../pod.yaml => chart/templates/limitrange.yaml| 33 +- .../templates/pgbouncer/pgbouncer-deployment.yaml | 128 ++ .../pgbouncer/pgbouncer-networkpolicy.yaml | 69 .../pgbouncer/pgbouncer-poddisruptionbudget.yaml | 56 ++- chart/templates/pgbouncer/pgbouncer-service.yaml | 56 +++ .../templates/rbac/pod-cleanup-role.yaml | 34 +- .../templates/rbac/pod-cleanup-rolebinding.yaml| 32 +- chart/templates/rbac/pod-launcher-role.yaml| 58 +++ chart/templates/rbac/pod-launcher-rolebinding.yaml | 51 +++ chart/templates/redis/redis-networkpolicy.yaml | 63 +++ .../templates/redis/redis-service.yaml | 41 +- chart/templates/redis/redis-statefulset.yaml | 99 + .../pod.yaml => chart/templates/resourcequota.yaml | 33 +- .../templates/scheduler/scheduler-deployment.yaml | 195 + .../scheduler/scheduler-networkpolicy.yaml | 55 +++ .../scheduler/scheduler-poddisruptionbudget.yaml | 39 +- .../templates/scheduler/scheduler-service.yaml | 41 +- .../scheduler/scheduler-serviceaccount.yaml| 24 +- .../templates/secrets/elasticsearch-secret.yaml| 22 +- .../templates/secrets/fernetkey-secret.yaml| 27 +- .../secrets/metadata-connection-secret.yaml| 42 ++ .../templates/secrets/pgbouncer-config-secret.yaml | 23 +- .../templates/secrets/pgbouncer-stats-secret.yaml | 22 +- chart/templates/secrets/redis-secrets.yaml | 61 +++ .../templates/secrets/registry-secret.yaml | 24 +- .../secrets/result-backend-connection-secret.yaml | 37 ++ chart/templates/statsd/statsd-deployment.yaml | 87 chart/templates/statsd/statsd-networkpolicy.yaml | 57 +++ chart/templates/statsd/statsd-service.yaml | 56 +++ .../templates/webserver/webserver-deployment.yaml | 139 +++ .../webserver/webserver-networkpolicy.yaml | 51 +++ .../templates/webserver/webserver-service.yaml | 39 +- chart/templates/workers/worker-deployment.yaml | 161 chart/templates/workers/worker-kedaautoscaler.yaml | 47 +++ chart/templates/workers/worker-networkpolicy.yaml | 53 +++ .../templates/workers/worker-service.yaml
[airflow] 02/09: Fix typo in helm chart upgrade command for 2.0 (#9484)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d37427839bcf75cb18190fb562d306a1bba12570
Author: Ash Berlin-Taylor
AuthorDate: Tue Jun 23 10:38:06 2020 +0100
Fix typo in helm chart upgrade command for 2.0 (#9484)
(cherry picked from commit b1cd382db9367ec828b8ee16899ecea9fcf824a7)
---
chart/templates/scheduler/scheduler-deployment.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml
b/chart/templates/scheduler/scheduler-deployment.yaml
index 1b46f6a..d5c3a06 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -96,7 +96,7 @@ spec:
image: {{ template "airflow_image" . }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
# Support running against 1.10.x and 2.0.0dev/master
- args: ["bash", "-c", "airflow upgradedb || airfow db upgrade"]
+ args: ["bash", "-c", "airflow upgradedb || airflow db upgrade"]
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
[airflow] 08/09: Removes importlib usage - it's not needed (fails on Airflow 1.10) (#9613)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ec3f4dc205e620a4e7bc563549624fdb9f59bf80
Author: Jarek Potiuk
AuthorDate: Wed Jul 1 18:07:12 2020 +0200
Removes importlib usage - it's not needed (fails on Airflow 1.10) (#9613)
(cherry picked from commit a3a52c78b274483f2035ad975fc218abd8ffdf8a)
---
chart/templates/_helpers.yaml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 66d1850..ac121a4 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -205,7 +205,7 @@ log_connections = {{ .Values.pgbouncer.logConnections }}
- python
- -c
- |
-import importlib
+import airflow
import os
import time
@@ -215,7 +215,7 @@ log_connections = {{ .Values.pgbouncer.logConnections }}
from airflow import settings
-package_dir =
os.path.dirname(importlib.util.find_spec('airflow').origin)
+package_dir = os.path.abspath(os.path.dirname(airflow.__file__))
directory = os.path.join(package_dir, 'migrations')
config = Config(os.path.join(package_dir, 'alembic.ini'))
config.set_main_option('script_location', directory)
[airflow] 04/09: Fix typo of resultBackendConnection in chart README (#9537)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit dd40f47c987a6df49e309cc00ac06fa33ec91506
Author: Vicken Simonian
AuthorDate: Fri Jun 26 11:40:30 2020 -0700
Fix typo of resultBackendConnection in chart README (#9537)
(cherry picked from commit 096f5c5cba963b364ee75f6686d128cd4d34d66e)
---
chart/README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index d0366f6..402a9d7 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -119,11 +119,11 @@ The following tables lists the configurable parameters of
the Airflow chart and
| `data.metadataSecretName` | Secret name to mount
Airflow connection string from
| `~` |
| `data.resultBackendSecretName`| Secret name to mount
Celery result backend connection string from
| `~` |
| `data.metadataConection` | Field separated
connection data (alternative to secret name)
| `{}` |
-| `data.resultBakcnedConnection`| Field separated
connection data (alternative to secret name)
| `{}` |
+| `data.resultBackendConnection`| Field separated
connection data (alternative to secret name)
| `{}` |
| `fernetKey` | String representing
an Airflow fernet key
| `~` |
| `fernetKeySecretName` | Secret name for
Airlow fernet key
| `~` |
| `workers.replicas`| Replica count for
Celery workers (if applicable)
| `1` |
-| `workers.keda.enabled` | Enable KEDA
autoscaling features
| `false` |
+| `workers.keda.enabled`| Enable KEDA
autoscaling features
| `false` |
| `workers.keda.pollingInverval`| How often KEDA
should poll the backend database for metrics in seconds
| `5` |
| `workers.keda.cooldownPeriod` | How often KEDA
should wait before scaling down in seconds
| `30` |
| `workers.keda.maxReplicaCount`| Maximum number of
Celery workers KEDA can scale to
| `10` |
[GitHub] [airflow] potiuk merged pull request #9613: Removes importlib usage - it's not needed (fails on Airflow 1.10)
potiuk merged pull request #9613: URL: https://github.com/apache/airflow/pull/9613 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch master updated (9e305d6 -> a3a52c7)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 9e305d6 Change default auth for experimental backend to deny_all (#9611) add a3a52c7 Removes importlib usage - it's not needed (fails on Airflow 1.10) (#9613) No new revisions were added by this update. Summary of changes: chart/templates/_helpers.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[GitHub] [airflow] j-y-matsubara commented on a change in pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on a change in pull request #9531:
URL: https://github.com/apache/airflow/pull/9531#discussion_r448466908
##
File path: airflow/utils/file.py
##
@@ -90,6 +90,48 @@ def open_maybe_zipped(fileloc, mode='r'):
return io.open(fileloc, mode=mode)
+def find_path_from_directory(
+base_dir_path: str,
+ignore_list_file: str) -> Generator[str, None, None]:
+"""
+Search the file and return the path of the file that should not be ignored.
+:param base_dir_path: the base path to be searched for.
+:param ignore_file_list_name: the file name in which specifies a regular
expression pattern is written.
+
+:return : file path not to be ignored
+"""
+
+patterns_by_dir: Dict[str, List[Pattern[str]]] = {}
+
+for root, dirs, files in os.walk(str(base_dir_path), followlinks=True):
+patterns: List[Pattern[str]] = patterns_by_dir.get(root, [])
+
+ignore_list_file_path = os.path.join(root, ignore_list_file)
+if os.path.isfile(ignore_list_file_path):
+with open(ignore_list_file_path, 'r') as file:
+lines_no_comments = [re.compile(r"\s*#.*").sub("", line) for
line in file.read().split("\n")]
+patterns += [re.compile(line) for line in lines_no_comments if
line]
+patterns = list(set(patterns))
+
+dirs[:] = [
+subdir
+for subdir in dirs
+if not any(p.search(
+os.path.join(os.path.relpath(root, str(base_dir_path)),
subdir)) for p in patterns)
+]
+
+for subdir in dirs:
+patterns_by_dir[os.path.join(root, subdir)] = patterns.copy()
+
+for file in files: # type: ignore
+if file == ignore_list_file:
+continue
+file_path = os.path.join(root, str(file))
+if any([re.findall(p, file_path) for p in patterns]):
Review comment:
Yes.
I fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch master updated (65855e5 -> 9e305d6)
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 65855e5 Add docs to change Colors on the Webserver (#9607) add 9e305d6 Change default auth for experimental backend to deny_all (#9611) No new revisions were added by this update. Summary of changes: UPDATING.md | 16 airflow/config_templates/config.yml | 6 -- airflow/config_templates/default_airflow.cfg | 6 -- 3 files changed, 24 insertions(+), 4 deletions(-)
[GitHub] [airflow] ashb merged pull request #9611: Change default auth for experimental backend to deny_all
ashb merged pull request #9611: URL: https://github.com/apache/airflow/pull/9611 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on a change in pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on a change in pull request #9531:
URL: https://github.com/apache/airflow/pull/9531#discussion_r448466461
##
File path: airflow/utils/file.py
##
@@ -90,6 +90,48 @@ def open_maybe_zipped(fileloc, mode='r'):
return io.open(fileloc, mode=mode)
+def find_path_from_directory(
+base_dir_path: str,
+ignore_list_file: str) -> Generator[str, None, None]:
+"""
+Search the file and return the path of the file that should not be ignored.
+:param base_dir_path: the base path to be searched for.
+:param ignore_file_list_name: the file name in which specifies a regular
expression pattern is written.
+
+:return : file path not to be ignored
+"""
+
+patterns_by_dir: Dict[str, List[Pattern[str]]] = {}
+
+for root, dirs, files in os.walk(str(base_dir_path), followlinks=True):
+patterns: List[Pattern[str]] = patterns_by_dir.get(root, [])
+
+ignore_list_file_path = os.path.join(root, ignore_list_file)
+if os.path.isfile(ignore_list_file_path):
+with open(ignore_list_file_path, 'r') as file:
+lines_no_comments = [re.compile(r"\s*#.*").sub("", line) for
line in file.read().split("\n")]
+patterns += [re.compile(line) for line in lines_no_comments if
line]
+patterns = list(set(patterns))
+
+dirs[:] = [
+subdir
+for subdir in dirs
+if not any(p.search(
+os.path.join(os.path.relpath(root, str(base_dir_path)),
subdir)) for p in patterns)
+]
+
+for subdir in dirs:
+patterns_by_dir[os.path.join(root, subdir)] = patterns.copy()
Review comment:
This is necessary.
A canonical pattern that is evaluated in a parent directory must also be
evaluated in its parent's child directories. At least that's how .airflowignore
(selection of dag) is currently specificated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on a change in pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on a change in pull request #9531:
URL: https://github.com/apache/airflow/pull/9531#discussion_r448466461
##
File path: airflow/utils/file.py
##
@@ -90,6 +90,48 @@ def open_maybe_zipped(fileloc, mode='r'):
return io.open(fileloc, mode=mode)
+def find_path_from_directory(
+base_dir_path: str,
+ignore_list_file: str) -> Generator[str, None, None]:
+"""
+Search the file and return the path of the file that should not be ignored.
+:param base_dir_path: the base path to be searched for.
+:param ignore_file_list_name: the file name in which specifies a regular
expression pattern is written.
+
+:return : file path not to be ignored
+"""
+
+patterns_by_dir: Dict[str, List[Pattern[str]]] = {}
+
+for root, dirs, files in os.walk(str(base_dir_path), followlinks=True):
+patterns: List[Pattern[str]] = patterns_by_dir.get(root, [])
+
+ignore_list_file_path = os.path.join(root, ignore_list_file)
+if os.path.isfile(ignore_list_file_path):
+with open(ignore_list_file_path, 'r') as file:
+lines_no_comments = [re.compile(r"\s*#.*").sub("", line) for
line in file.read().split("\n")]
+patterns += [re.compile(line) for line in lines_no_comments if
line]
+patterns = list(set(patterns))
+
+dirs[:] = [
+subdir
+for subdir in dirs
+if not any(p.search(
+os.path.join(os.path.relpath(root, str(base_dir_path)),
subdir)) for p in patterns)
+]
+
+for subdir in dirs:
+patterns_by_dir[os.path.join(root, subdir)] = patterns.copy()
Review comment:
This is necessary.
A canonical pattern that is evaluated in a parent directory must also be
evaluated in its parent's child directories.
At least that's how .airflowignore (selection of dag) is currently
specificated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on a change in pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on a change in pull request #9531:
URL: https://github.com/apache/airflow/pull/9531#discussion_r448459428
##
File path: airflow/utils/file.py
##
@@ -90,6 +90,48 @@ def open_maybe_zipped(fileloc, mode='r'):
return io.open(fileloc, mode=mode)
+def find_path_from_directory(
+base_dir_path: str,
+ignore_list_file: str) -> Generator[str, None, None]:
+"""
+Search the file and return the path of the file that should not be ignored.
+:param base_dir_path: the base path to be searched for.
+:param ignore_file_list_name: the file name in which specifies a regular
expression pattern is written.
+
+:return : file path not to be ignored
+"""
+
+patterns_by_dir: Dict[str, List[Pattern[str]]] = {}
+
+for root, dirs, files in os.walk(str(base_dir_path), followlinks=True):
+patterns: List[Pattern[str]] = patterns_by_dir.get(root, [])
+
+ignore_list_file_path = os.path.join(root, ignore_list_file)
+if os.path.isfile(ignore_list_file_path):
+with open(ignore_list_file_path, 'r') as file:
+lines_no_comments = [re.compile(r"\s*#.*").sub("", line) for
line in file.read().split("\n")]
Review comment:
It is not necessary.
And I have consolidated the wasteful looping process into one by making
modifications!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on a change in pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on a change in pull request #9531:
URL: https://github.com/apache/airflow/pull/9531#discussion_r448457433
##
File path: airflow/plugins_manager.py
##
@@ -164,34 +165,34 @@ def load_plugins_from_plugin_directory():
global plugins # pylint: disable=global-statement
log.debug("Loading plugins from directory: %s", settings.PLUGINS_FOLDER)
-# Crawl through the plugins folder to find AirflowPlugin derivatives
-for root, _, files in os.walk(settings.PLUGINS_FOLDER, followlinks=True):
# noqa # pylint: disable=too-many-nested-blocks
-for f in files:
-filepath = os.path.join(root, f)
-try:
-if not os.path.isfile(filepath):
-continue
-mod_name, file_ext = os.path.splitext(
-os.path.split(filepath)[-1])
-if file_ext != '.py':
-continue
-
-log.debug('Importing plugin module %s', filepath)
-
-loader = importlib.machinery.SourceFileLoader(mod_name,
filepath)
-spec = importlib.util.spec_from_loader(mod_name, loader)
-mod = importlib.util.module_from_spec(spec)
-sys.modules[spec.name] = mod
-loader.exec_module(mod)
-for mod_attr_value in list(mod.__dict__.values()):
-if is_valid_plugin(mod_attr_value):
-plugin_instance = mod_attr_value()
-plugins.append(plugin_instance)
-except Exception as e: # pylint: disable=broad-except
-log.exception(e)
-path = filepath or str(f)
-log.error('Failed to import plugin %s', path)
-import_errors[path] = str(e)
+ignore_list_file = ".airflowignore"
+
+for file_path in find_path_from_directory( # pylint:
disable=too-many-nested-blocks
+str(settings.PLUGINS_FOLDER), str(ignore_list_file)):
+
+try:
+if not os.path.isfile(file_path):
+continue
+mod_name, file_ext = os.path.splitext(
+os.path.split(file_path)[-1])
+if file_ext != '.py':
+continue
+
+log.info('Importing plugin module %s', file_path)
+
+loader = importlib.machinery.SourceFileLoader(mod_name, file_path)
+spec = importlib.util.spec_from_loader(mod_name, loader)
+mod = importlib.util.module_from_spec(spec)
+sys.modules[spec.name] = mod
+loader.exec_module(mod)
+for mod_attr_value in list(mod.__dict__.values()):
+if is_valid_plugin(mod_attr_value):
+plugin_instance = mod_attr_value()
+plugins.append(plugin_instance)
Review comment:
Beautiful code!
I reflect this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on a change in pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on a change in pull request #9531:
URL: https://github.com/apache/airflow/pull/9531#discussion_r448456168
##
File path: airflow/plugins_manager.py
##
@@ -164,34 +165,34 @@ def load_plugins_from_plugin_directory():
global plugins # pylint: disable=global-statement
log.debug("Loading plugins from directory: %s", settings.PLUGINS_FOLDER)
-# Crawl through the plugins folder to find AirflowPlugin derivatives
-for root, _, files in os.walk(settings.PLUGINS_FOLDER, followlinks=True):
# noqa # pylint: disable=too-many-nested-blocks
-for f in files:
-filepath = os.path.join(root, f)
-try:
-if not os.path.isfile(filepath):
-continue
-mod_name, file_ext = os.path.splitext(
-os.path.split(filepath)[-1])
-if file_ext != '.py':
-continue
-
-log.debug('Importing plugin module %s', filepath)
-
-loader = importlib.machinery.SourceFileLoader(mod_name,
filepath)
-spec = importlib.util.spec_from_loader(mod_name, loader)
-mod = importlib.util.module_from_spec(spec)
-sys.modules[spec.name] = mod
-loader.exec_module(mod)
-for mod_attr_value in list(mod.__dict__.values()):
-if is_valid_plugin(mod_attr_value):
-plugin_instance = mod_attr_value()
-plugins.append(plugin_instance)
-except Exception as e: # pylint: disable=broad-except
-log.exception(e)
-path = filepath or str(f)
-log.error('Failed to import plugin %s', path)
-import_errors[path] = str(e)
+ignore_list_file = ".airflowignore"
+
+for file_path in find_path_from_directory( # pylint:
disable=too-many-nested-blocks
+str(settings.PLUGINS_FOLDER), str(ignore_list_file)):
+
+try:
+if not os.path.isfile(file_path):
+continue
+mod_name, file_ext = os.path.splitext(
+os.path.split(file_path)[-1])
+if file_ext != '.py':
+continue
Review comment:
No
I fixed!
##
File path: airflow/plugins_manager.py
##
@@ -164,34 +165,34 @@ def load_plugins_from_plugin_directory():
global plugins # pylint: disable=global-statement
log.debug("Loading plugins from directory: %s", settings.PLUGINS_FOLDER)
-# Crawl through the plugins folder to find AirflowPlugin derivatives
-for root, _, files in os.walk(settings.PLUGINS_FOLDER, followlinks=True):
# noqa # pylint: disable=too-many-nested-blocks
-for f in files:
-filepath = os.path.join(root, f)
-try:
-if not os.path.isfile(filepath):
-continue
-mod_name, file_ext = os.path.splitext(
-os.path.split(filepath)[-1])
-if file_ext != '.py':
-continue
-
-log.debug('Importing plugin module %s', filepath)
-
-loader = importlib.machinery.SourceFileLoader(mod_name,
filepath)
-spec = importlib.util.spec_from_loader(mod_name, loader)
-mod = importlib.util.module_from_spec(spec)
-sys.modules[spec.name] = mod
-loader.exec_module(mod)
-for mod_attr_value in list(mod.__dict__.values()):
-if is_valid_plugin(mod_attr_value):
-plugin_instance = mod_attr_value()
-plugins.append(plugin_instance)
-except Exception as e: # pylint: disable=broad-except
-log.exception(e)
-path = filepath or str(f)
-log.error('Failed to import plugin %s', path)
-import_errors[path] = str(e)
+ignore_list_file = ".airflowignore"
+
+for file_path in find_path_from_directory( # pylint:
disable=too-many-nested-blocks
+str(settings.PLUGINS_FOLDER), str(ignore_list_file)):
+
+try:
+if not os.path.isfile(file_path):
+continue
+mod_name, file_ext = os.path.splitext(
+os.path.split(file_path)[-1])
+if file_ext != '.py':
+continue
Review comment:
No
I fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9596: Fix quarantined tests - TestCliWebServer (v1-10-test)
mik-laj commented on a change in pull request #9596: URL: https://github.com/apache/airflow/pull/9596#discussion_r448455220 ## File path: requirements/requirements-python3.8.txt ## @@ -50,7 +50,7 @@ aws-xray-sdk==2.6.0 azure-common==1.1.25 azure-cosmos==3.1.2 azure-datalake-store==0.0.48 -azure-mgmt-containerinstance==1.5.0 +azure-mgmt-containerinstance==2.0.0 Review comment: I just pushed a fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on a change in pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on a change in pull request #9531:
URL: https://github.com/apache/airflow/pull/9531#discussion_r448453473
##
File path: airflow/plugins_manager.py
##
@@ -164,34 +165,34 @@ def load_plugins_from_plugin_directory():
global plugins # pylint: disable=global-statement
log.debug("Loading plugins from directory: %s", settings.PLUGINS_FOLDER)
-# Crawl through the plugins folder to find AirflowPlugin derivatives
-for root, _, files in os.walk(settings.PLUGINS_FOLDER, followlinks=True):
# noqa # pylint: disable=too-many-nested-blocks
-for f in files:
-filepath = os.path.join(root, f)
-try:
-if not os.path.isfile(filepath):
-continue
-mod_name, file_ext = os.path.splitext(
-os.path.split(filepath)[-1])
-if file_ext != '.py':
-continue
-
-log.debug('Importing plugin module %s', filepath)
-
-loader = importlib.machinery.SourceFileLoader(mod_name,
filepath)
-spec = importlib.util.spec_from_loader(mod_name, loader)
-mod = importlib.util.module_from_spec(spec)
-sys.modules[spec.name] = mod
-loader.exec_module(mod)
-for mod_attr_value in list(mod.__dict__.values()):
-if is_valid_plugin(mod_attr_value):
-plugin_instance = mod_attr_value()
-plugins.append(plugin_instance)
-except Exception as e: # pylint: disable=broad-except
-log.exception(e)
-path = filepath or str(f)
-log.error('Failed to import plugin %s', path)
-import_errors[path] = str(e)
+ignore_list_file = ".airflowignore"
+
+for file_path in find_path_from_directory( # pylint:
disable=too-many-nested-blocks
+str(settings.PLUGINS_FOLDER), str(ignore_list_file)):
Review comment:
It is not necessary.
I fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on a change in pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on a change in pull request #9531:
URL: https://github.com/apache/airflow/pull/9531#discussion_r448452969
##
File path: airflow/plugins_manager.py
##
@@ -164,34 +165,34 @@ def load_plugins_from_plugin_directory():
global plugins # pylint: disable=global-statement
log.debug("Loading plugins from directory: %s", settings.PLUGINS_FOLDER)
-# Crawl through the plugins folder to find AirflowPlugin derivatives
-for root, _, files in os.walk(settings.PLUGINS_FOLDER, followlinks=True):
# noqa # pylint: disable=too-many-nested-blocks
-for f in files:
-filepath = os.path.join(root, f)
-try:
-if not os.path.isfile(filepath):
-continue
-mod_name, file_ext = os.path.splitext(
-os.path.split(filepath)[-1])
-if file_ext != '.py':
-continue
-
-log.debug('Importing plugin module %s', filepath)
-
-loader = importlib.machinery.SourceFileLoader(mod_name,
filepath)
-spec = importlib.util.spec_from_loader(mod_name, loader)
-mod = importlib.util.module_from_spec(spec)
-sys.modules[spec.name] = mod
-loader.exec_module(mod)
-for mod_attr_value in list(mod.__dict__.values()):
-if is_valid_plugin(mod_attr_value):
-plugin_instance = mod_attr_value()
-plugins.append(plugin_instance)
-except Exception as e: # pylint: disable=broad-except
-log.exception(e)
-path = filepath or str(f)
-log.error('Failed to import plugin %s', path)
-import_errors[path] = str(e)
+ignore_list_file = ".airflowignore"
Review comment:
It's not particularly necessary.
I fixed!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new pull request #9613: Removes importlib usage - it's not needed (fails on Airflow 1.10)
potiuk opened a new pull request #9613: URL: https://github.com/apache/airflow/pull/9613 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Target Github ISSUE in description if exists - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] OmairK commented on issue #8201: Create guide for Cloud Data Loss Prevention (DLP) operators
OmairK commented on issue #8201: URL: https://github.com/apache/airflow/issues/8201#issuecomment-652482106 @mik-laj I want to take up this task, can you please assign it to me ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch master updated: Add docs to change Colors on the Webserver (#9607)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new 65855e5 Add docs to change Colors on the Webserver (#9607)
65855e5 is described below
commit 65855e558f9b070004ba817e8cf94fd834d2a67a
Author: Kaxil Naik
AuthorDate: Wed Jul 1 16:17:29 2020 +0100
Add docs to change Colors on the Webserver (#9607)
Feature was added in https://github.com/apache/airflow/pull/9520
---
docs/howto/customize-state-colors-ui.rst | 70 +++
docs/howto/index.rst | 1 +
docs/img/change-ui-colors/dags-page-new.png | Bin 0 -> 483599 bytes
docs/img/change-ui-colors/dags-page-old.png | Bin 0 -> 493009 bytes
docs/img/change-ui-colors/graph-view-new.png | Bin 0 -> 56973 bytes
docs/img/change-ui-colors/graph-view-old.png | Bin 0 -> 54884 bytes
docs/img/change-ui-colors/tree-view-new.png | Bin 0 -> 36934 bytes
docs/img/change-ui-colors/tree-view-old.png | Bin 0 -> 21601 bytes
8 files changed, 71 insertions(+)
diff --git a/docs/howto/customize-state-colors-ui.rst
b/docs/howto/customize-state-colors-ui.rst
new file mode 100644
index 000..c856950
--- /dev/null
+++ b/docs/howto/customize-state-colors-ui.rst
@@ -0,0 +1,70 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+
+Customizing state colours in UI
+===
+
+.. versionadded:: 1.10.11
+
+To change the colors for TaskInstance/DagRun State in the Airflow Webserver,
perform the
+following steps:
+
+1. Create ``airflow_local_settings.py`` file and put in on ``$PYTHONPATH`` or
+to ``$AIRFLOW_HOME/config`` folder. (Airflow adds ``$AIRFLOW_HOME/config``
on ``PYTHONPATH`` when
+Airflow is initialized)
+
+2. Add the following contents to ``airflow_local_settings.py`` file. Change
the colors to whatever you
+would like.
+
+.. code-block:: python
+
+ STATE_COLORS = {
+"queued": 'darkgray',
+"running": '#01FF70',
+"success": '#2ECC40',
+"failed": 'firebrick',
+"up_for_retry": 'yellow',
+"up_for_reschedule": 'turquoise',
+"upstream_failed": 'orange',
+"skipped": 'darkorchid',
+"scheduled": 'tan',
+ }
+
+
+
+3. Restart Airflow Webserver.
+
+Screenshots
+---
+
+Before
+^^
+
+.. image:: ../img/change-ui-colors/dags-page-old.png
+
+.. image:: ../img/change-ui-colors/graph-view-old.png
+
+.. image:: ../img/change-ui-colors/tree-view-old.png
+
+After
+^^
+
+.. image:: ../img/change-ui-colors/dags-page-new.png
+
+.. image:: ../img/change-ui-colors/graph-view-new.png
+
+.. image:: ../img/change-ui-colors/tree-view-new.png
diff --git a/docs/howto/index.rst b/docs/howto/index.rst
index 387ba3b..837f0f3 100644
--- a/docs/howto/index.rst
+++ b/docs/howto/index.rst
@@ -34,6 +34,7 @@ configuring an Airflow environment.
set-config
initialize-database
operator/index
+customize-state-colors-ui
custom-operator
connection/index
write-logs
diff --git a/docs/img/change-ui-colors/dags-page-new.png
b/docs/img/change-ui-colors/dags-page-new.png
new file mode 100644
index 000..d2ffe1f
Binary files /dev/null and b/docs/img/change-ui-colors/dags-page-new.png differ
diff --git a/docs/img/change-ui-colors/dags-page-old.png
b/docs/img/change-ui-colors/dags-page-old.png
new file mode 100644
index 000..5078d01
Binary files /dev/null and b/docs/img/change-ui-colors/dags-page-old.png differ
diff --git a/docs/img/change-ui-colors/graph-view-new.png
b/docs/img/change-ui-colors/graph-view-new.png
new file mode 100644
index 000..b367461
Binary files /dev/null and b/docs/img/change-ui-colors/graph-view-new.png differ
diff --git a/docs/img/change-ui-colors/graph-view-old.png
b/docs/img/change-ui-colors/graph-view-old.png
new file mode 100644
index 000..ceaf8d4
Binary files /dev/null and b/docs/img/change-ui-colors/graph-view-old.png differ
diff --git a/docs/img/change-ui-colors/tree-view-new.png
b/docs/img/change-ui-colors/tree-view-new.png
new file mode 100644
index 000..6a5b2d7
Binary
[GitHub] [airflow] kaxil merged pull request #9607: Add docs to change Colors on the Webserver
kaxil merged pull request #9607: URL: https://github.com/apache/airflow/pull/9607 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9612: Restrict editing DagRun State in the old UI
kaxil commented on a change in pull request #9612: URL: https://github.com/apache/airflow/pull/9612#discussion_r448433288 ## File path: UPDATING.md ## @@ -66,6 +66,14 @@ https://developers.google.com/style/inclusive-documentation Now use NULL as default value for dag.description in dag table +### Restrict editing DagRun State in the old UI (Flask-admin based UI) + +Before 1.10.11 it was possible to edit DagRun State in the `/admin/dagrun/` page + to any text. + +From Airflow 1.10.11, you would still be able to set from dropdown to a pre-defined option + but would not allow setting it to any text. Review comment: I don't like the wording though, can you help me with that Ash? I can include screenshots if that is more helpful This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil opened a new pull request #9612: Restrict editing DagRun State in the old UI
kaxil opened a new pull request #9612: URL: https://github.com/apache/airflow/pull/9612 Restrict editing DagRun State in the old UI --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Target Github ISSUE in description if exists - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9596: Fix quarantined tests - TestCliWebServer (v1-10-test)
mik-laj commented on a change in pull request #9596: URL: https://github.com/apache/airflow/pull/9596#discussion_r448430604 ## File path: requirements/requirements-python3.8.txt ## @@ -50,7 +50,7 @@ aws-xray-sdk==2.6.0 azure-common==1.1.25 azure-cosmos==3.1.2 azure-datalake-store==0.0.48 -azure-mgmt-containerinstance==1.5.0 +azure-mgmt-containerinstance==2.0.0 Review comment: It broke my build. I'm working on a fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9611: Change default auth for experimental backend to deny_all
kaxil commented on a change in pull request #9611: URL: https://github.com/apache/airflow/pull/9611#discussion_r448424798 ## File path: UPDATING.md ## @@ -1425,6 +1425,22 @@ Now the `dag_id` will not appear repeated in the payload, and the response forma } ``` +### Experimental API will deny all request by default. + +The previous default setting was to allow all API requests without authentication, but this poses security +risks to users who miss this fact. This changes the default for new installs to deny all requests by default. + +**Note**: This will not change the behavior for existing installs, please update check your airflow.cfg + +If you wish to have the experimental API work, and aware of the risks of enabling this without authentication +(or if you have your own authentication layer in front of Airflow) you can get the old back on a new install +by setting this in your airflow.cfg: Review comment: ```suggestion (or if you have your own authentication layer in front of Airflow) you can get the previous behaviour back on a new install by setting this in your airflow.cfg: ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9607: Add docs to change Colors on the Webserver
kaxil commented on a change in pull request #9607: URL: https://github.com/apache/airflow/pull/9607#discussion_r448413280 ## File path: docs/howto/colorblind-friendly-ui.rst ## @@ -0,0 +1,70 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Colorblind friendly UI Review comment: Updated in https://github.com/apache/airflow/pull/9607/commits/5058e1fc24420c8edd383681ded8c7bff5f704a6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #9607: Add docs to change Colors on the Webserver
ashb commented on a change in pull request #9607: URL: https://github.com/apache/airflow/pull/9607#discussion_r448409664 ## File path: docs/howto/colorblind-friendly-ui.rst ## @@ -0,0 +1,70 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Colorblind friendly UI Review comment: ```suggestion Customizing state colours in UI ``` And rename file to match. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9596: Fix quarantined tests - TestCliWebServer (v1-10-test)
mik-laj commented on pull request #9596: URL: https://github.com/apache/airflow/pull/9596#issuecomment-652450871 I forgot to generate requirements for Python 3.5. I'm working on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch master updated (f3e1f9a -> 48a8316)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from f3e1f9a Update Breeze documentation (#9608) add 48a8316 Fix quarantined tests - TestCliWebServer (#9598) No new revisions were added by this update. Summary of changes: airflow/cli/commands/webserver_command.py | 26 +++-- requirements/requirements-python3.6.txt | 16 +-- requirements/requirements-python3.7.txt | 16 +-- requirements/requirements-python3.8.txt | 16 +-- requirements/setup-3.6.md5 | 2 +- requirements/setup-3.7.md5 | 2 +- requirements/setup-3.8.md5 | 2 +- setup.py| 2 +- tests/cli/commands/test_only_use_long_option.py | 48 tests/cli/commands/test_webserver_command.py| 147 10 files changed, 144 insertions(+), 133 deletions(-) delete mode 100644 tests/cli/commands/test_only_use_long_option.py
[GitHub] [airflow] kaxil merged pull request #9598: Fix quarantined tests - TestCliWebServer
kaxil merged pull request #9598: URL: https://github.com/apache/airflow/pull/9598 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #9596: Fix quarantined tests - TestCliWebServer (v1-10-test)
kaxil commented on pull request #9596: URL: https://github.com/apache/airflow/pull/9596#issuecomment-652443343 > @kaxil I made the necessary changes to make the tests stable and did rebase. Thanks, I will merge as soon as the CI is complete This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch master updated (8bd15ef -> f3e1f9a)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 8bd15ef Switches to Helm Chart for Kubernetes tests (#9468) add f3e1f9a Update Breeze documentation (#9608) No new revisions were added by this update. Summary of changes: BREEZE.rst | 320 TESTING.rst | 2 +- 2 files changed, 192 insertions(+), 130 deletions(-)
[GitHub] [airflow] potiuk merged pull request #9608: Update Breeze documentation
potiuk merged pull request #9608: URL: https://github.com/apache/airflow/pull/9608 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] chrismclennon opened a new issue #9610: Pod logs from KubernetesPodOperator occasionally get replaced with "Task is not able to run"
chrismclennon opened a new issue #9610:
URL: https://github.com/apache/airflow/issues/9610
**Apache Airflow version**: 1.10.10
**Kubernetes version (if you are using kubernetes)** (use `kubectl
version`): 1.17.2
**Environment**:
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): CentOS Linux
- **Kernel** (e.g. `uname -a`): `Linux airflow-worker-0
5.6.13-1.el7.elrepo.x86_64 #1 SMP Thu May 14 08:05:24 EDT 2020 x86_64 x86_64
x86_64 GNU/Linux`
- **Install tools**:
- **Others**:
**What happened**:
I run Airflow as a way to orchestrate jobs on Kubernetes using the
KubernetesPodOperator. While most of the time logs appear correctly in the
Airflow webserver, I do notice increasingly that the logs do not appear and
instead just show a message, "Task is not able to be run", such as in the
snippet below:
```
*** Reading remote log from
s3://*//**/2020-06-30T10:00:00+00:00/1.log.
[2020-06-30 23:07:40,362] {taskinstance.py:663} INFO - Dependencies not met
for , dependency
'Task Instance State' FAILED: Task is in the 'running' state which is not a
valid state for execution. The task must be cleared in order to be run.
[2020-06-30 23:07:40,363] {logging_mixin.py:112} INFO - [2020-06-30
23:07:40,363] {local_task_job.py:91} INFO - Task is not able to be run
```
Unusually, when I go check what is happening on the Kubernetes cluster, the
pod is actually running and emitting logs when I run a `kubectl logs` command.
When the pod is complete, Airflow will reflect that the task has completed as
well.
**What you expected to happen**: I expected pod logs to be printed out.
**How to reproduce it**: Very unfortunately, I am unsure what circumstances
cause this error and am currently trying to gather evidence to replicate.
**Anything else we need to know**:
* I have remote logging set to an S3 bucket.
* I've noticed this issue increasingly with the 1.10.10 update, and I find
this error daily.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #9610: Pod logs from KubernetesPodOperator occasionally get replaced with "Task is not able to run"
boring-cyborg[bot] commented on issue #9610: URL: https://github.com/apache/airflow/issues/9610#issuecomment-652435960 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9596: Fix quarantined tests - TestCliWebServer (v1-10-test)
mik-laj commented on pull request #9596: URL: https://github.com/apache/airflow/pull/9596#issuecomment-652431804 @kaxil I made the necessary changes to make the tests stable and did rebase. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9608: Update Breeze documentation
potiuk commented on a change in pull request #9608: URL: https://github.com/apache/airflow/pull/9608#discussion_r448369048 ## File path: BREEZE.rst ## @@ -330,13 +293,15 @@ Set up local development environment: * Setup local virtualenv with ``breeze setup-virtualenv`` command * Setup autocomplete for itself with ``breeze setup-autocomplete`` command - -Entering Breeze CI environment +Interactive Breeze Environment -- +Entering Breeze environment +... + You enter the Breeze test environment by running the ``./breeze`` script. You can run it with the ``help`` command to see the list of available options. See `Breeze Command-Line Interface Reference`_ -for details. +for details. By default you enter the `` Review comment: ```suggestion for details. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9531: Support .airflowignore for plugins
turbaszek commented on a change in pull request #9531:
URL: https://github.com/apache/airflow/pull/9531#discussion_r448366593
##
File path: airflow/utils/file.py
##
@@ -90,6 +90,48 @@ def open_maybe_zipped(fileloc, mode='r'):
return io.open(fileloc, mode=mode)
+def find_path_from_directory(
+base_dir_path: str,
+ignore_list_file: str) -> Generator[str, None, None]:
+"""
+Search the file and return the path of the file that should not be ignored.
+:param base_dir_path: the base path to be searched for.
+:param ignore_file_list_name: the file name in which specifies a regular
expression pattern is written.
+
+:return : file path not to be ignored
+"""
+
+patterns_by_dir: Dict[str, List[Pattern[str]]] = {}
+
+for root, dirs, files in os.walk(str(base_dir_path), followlinks=True):
+patterns: List[Pattern[str]] = patterns_by_dir.get(root, [])
+
+ignore_list_file_path = os.path.join(root, ignore_list_file)
+if os.path.isfile(ignore_list_file_path):
+with open(ignore_list_file_path, 'r') as file:
+lines_no_comments = [re.compile(r"\s*#.*").sub("", line) for
line in file.read().split("\n")]
+patterns += [re.compile(line) for line in lines_no_comments if
line]
+patterns = list(set(patterns))
+
+dirs[:] = [
+subdir
+for subdir in dirs
+if not any(p.search(
+os.path.join(os.path.relpath(root, str(base_dir_path)),
subdir)) for p in patterns)
+]
+
+for subdir in dirs:
+patterns_by_dir[os.path.join(root, subdir)] = patterns.copy()
Review comment:
```suggestion
patterns_by_dir = {os.path.join(root, sd): patterns.copy() for sd
in dirs}
```
WDYT? Also, do we have to create copy of `patterns` each time?
##
File path: airflow/plugins_manager.py
##
@@ -164,34 +165,34 @@ def load_plugins_from_plugin_directory():
global plugins # pylint: disable=global-statement
log.debug("Loading plugins from directory: %s", settings.PLUGINS_FOLDER)
-# Crawl through the plugins folder to find AirflowPlugin derivatives
-for root, _, files in os.walk(settings.PLUGINS_FOLDER, followlinks=True):
# noqa # pylint: disable=too-many-nested-blocks
-for f in files:
-filepath = os.path.join(root, f)
-try:
-if not os.path.isfile(filepath):
-continue
-mod_name, file_ext = os.path.splitext(
-os.path.split(filepath)[-1])
-if file_ext != '.py':
-continue
-
-log.debug('Importing plugin module %s', filepath)
-
-loader = importlib.machinery.SourceFileLoader(mod_name,
filepath)
-spec = importlib.util.spec_from_loader(mod_name, loader)
-mod = importlib.util.module_from_spec(spec)
-sys.modules[spec.name] = mod
-loader.exec_module(mod)
-for mod_attr_value in list(mod.__dict__.values()):
-if is_valid_plugin(mod_attr_value):
-plugin_instance = mod_attr_value()
-plugins.append(plugin_instance)
-except Exception as e: # pylint: disable=broad-except
-log.exception(e)
-path = filepath or str(f)
-log.error('Failed to import plugin %s', path)
-import_errors[path] = str(e)
+ignore_list_file = ".airflowignore"
Review comment:
Should we make it a constant?
##
File path: airflow/utils/file.py
##
@@ -90,6 +90,48 @@ def open_maybe_zipped(fileloc, mode='r'):
return io.open(fileloc, mode=mode)
+def find_path_from_directory(
+base_dir_path: str,
+ignore_list_file: str) -> Generator[str, None, None]:
+"""
+Search the file and return the path of the file that should not be ignored.
+:param base_dir_path: the base path to be searched for.
+:param ignore_file_list_name: the file name in which specifies a regular
expression pattern is written.
+
+:return : file path not to be ignored
+"""
+
+patterns_by_dir: Dict[str, List[Pattern[str]]] = {}
+
+for root, dirs, files in os.walk(str(base_dir_path), followlinks=True):
+patterns: List[Pattern[str]] = patterns_by_dir.get(root, [])
+
+ignore_list_file_path = os.path.join(root, ignore_list_file)
+if os.path.isfile(ignore_list_file_path):
+with open(ignore_list_file_path, 'r') as file:
+lines_no_comments = [re.compile(r"\s*#.*").sub("", line) for
line in file.read().split("\n")]
+patterns += [re.compile(line) for line in lines_no_comments if
line]
+patterns = list(set(patterns))
+
+dirs[:] = [
+subdir
+for subdir in dirs
+
[GitHub] [airflow] kaxil commented on a change in pull request #9608: Update Breeze documentation
kaxil commented on a change in pull request #9608: URL: https://github.com/apache/airflow/pull/9608#discussion_r448362144 ## File path: BREEZE.rst ## @@ -330,13 +293,15 @@ Set up local development environment: * Setup local virtualenv with ``breeze setup-virtualenv`` command * Setup autocomplete for itself with ``breeze setup-autocomplete`` command - -Entering Breeze CI environment +Interactive Breeze Environment -- +Entering Breeze environment +... + You enter the Breeze test environment by running the ``./breeze`` script. You can run it with the ``help`` command to see the list of available options. See `Breeze Command-Line Interface Reference`_ -for details. +for details. By default you enter the `` Review comment: incomplete sentence? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9598: Fix quarantined tests - TestCliWebServer
mik-laj commented on pull request #9598: URL: https://github.com/apache/airflow/pull/9598#issuecomment-652417156 All tests are green. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #9609: TimeSensor triggers immediately when used over midnight (UTC)
boring-cyborg[bot] commented on issue #9609: URL: https://github.com/apache/airflow/issues/9609#issuecomment-652408304 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] freget opened a new issue #9609: TimeSensor triggers immediately when used over midnight (UTC)
freget opened a new issue #9609:
URL: https://github.com/apache/airflow/issues/9609
**Apache Airflow version**: 1.10.10 (issue exists in current master as well)
**Environment**: does not seem relevant
**What happened**:
The TimeSensor does trigger if the current time is later than the defined
trigger time. Looking at the [source
code](https://github.com/apache/airflow/blob/master/airflow/sensors/time_sensor.py),
the trigger rule is defined as
```
return timezone.utcnow().time() > self.target_time
```
This leads to problems when the DAG runs over midnight UTC. For example,
suppose the following DAG:
```
with DAG('foo',
default_args={'start_date': datetime(2020, 7, 1,
tzinfo=pendulum.timezone("Europe/Berlin"))},
schedule_interval="0 0 * * *") as dag:
# in summer, Europe/Berlin is two hours after UTC, hence:
time_04h00_local = TimeSensor(task_id="time_01h30",
target_time=time(hour=2, minute=00))
```
This DAG will be triggered at 22:00 UTC. Then, according to the trigger rule:
```
22:00 UTC > 2:00 UTC
```
Hence, the TimeSensor will be triggered immediately.
**What you expected to happen**:
The TimeSensor should trigger at the following day if `target_time <
next_execution_date.time()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new pull request #9608: Update Breeze documentation
potiuk opened a new pull request #9608: URL: https://github.com/apache/airflow/pull/9608 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Target Github ISSUE in description if exists - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #8966: 2.0.0-dev - Webserver front page missing dag_id for dag runs and task instances
kaxil commented on issue #8966: URL: https://github.com/apache/airflow/issues/8966#issuecomment-652399344 Fixed by https://github.com/apache/airflow/commit/836f717d316d6b20bc2f2bc07d781dd332618471 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil closed issue #8966: 2.0.0-dev - Webserver front page missing dag_id for dag runs and task instances
kaxil closed issue #8966: URL: https://github.com/apache/airflow/issues/8966 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #9468: Switches to Helm Chart for Kubernetes tests
potiuk merged pull request #9468: URL: https://github.com/apache/airflow/pull/9468 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9531: Support .airflowignore for plugins
mik-laj commented on pull request #9531: URL: https://github.com/apache/airflow/pull/9531#issuecomment-652396856 @turbaszek Can I ask for review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9598: Fix quarantined tests - TestCliWebServer
mik-laj commented on a change in pull request #9598: URL: https://github.com/apache/airflow/pull/9598#discussion_r448293291 ## File path: tests/cli/commands/test_only_use_long_option.py ## @@ -1,48 +0,0 @@ -#!/usr/bin/env python -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -import os -import re -import unittest - - -class TestOnlyUseLongOption(unittest.TestCase): Review comment: Some commands do not have their long variant, e.g. `bash -c`, `ls -l`, etc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9598: Fix quarantined tests - TestCliWebServer
mik-laj commented on a change in pull request #9598: URL: https://github.com/apache/airflow/pull/9598#discussion_r448293291 ## File path: tests/cli/commands/test_only_use_long_option.py ## @@ -1,48 +0,0 @@ -#!/usr/bin/env python -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -import os -import re -import unittest - - -class TestOnlyUseLongOption(unittest.TestCase): Review comment: Some commands do not have their long variant, e.g. "bash -c", "ls -l", etc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9598: Fix quarantined tests - TestCliWebServer
mik-laj commented on pull request #9598: URL: https://github.com/apache/airflow/pull/9598#issuecomment-652357309 @ashb Here is the same change, but I had to make one more change. The CI command is not always successful because the server is terminated before it starts (-15). https://github.com/apache/airflow/pull/9598/checks?check_run_id=826014895 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch v1-10-test updated: fixup! Add Local and Sequential Executors to Doc (#8084)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v1-10-test by this push: new 99d8852 fixup! Add Local and Sequential Executors to Doc (#8084) 99d8852 is described below commit 99d885259c5802cd75169402a7e61fc76fafcb61 Author: Kaxil Naik AuthorDate: Wed Jul 1 12:11:04 2020 +0100 fixup! Add Local and Sequential Executors to Doc (#8084) --- docs/executor/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/executor/index.rst b/docs/executor/index.rst index 7994ac7..fc305ea 100644 --- a/docs/executor/index.rst +++ b/docs/executor/index.rst @@ -34,3 +34,4 @@ section of the configuration file. dask celery kubernetes +mesos
[GitHub] [airflow] kaxil opened a new pull request #9607: Add docs to change Colors on the Webserver
kaxil opened a new pull request #9607: URL: https://github.com/apache/airflow/pull/9607 Feature was added in https://github.com/apache/airflow/pull/9520 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Target Github ISSUE in description if exists - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #9596: Fix quarantined tests - TestCliWebServer (v1-10-test)
kaxil commented on pull request #9596: URL: https://github.com/apache/airflow/pull/9596#issuecomment-652353949 Sorry, it was because I force-pushed v1-10-test to remove all fixups as we are almost ready Can you please rebase on latest v1-10-test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #9596: Fix quarantined tests - TestCliWebServer (v1-10-test)
ashb edited a comment on pull request #9596: URL: https://github.com/apache/airflow/pull/9596#issuecomment-652353000 212 commits -- something has gone wrong with this PR (so we can't review it) Looking at last commit https://github.com/apache/airflow/commit/8d75bfec0d32c39d61ddded170e05ab20367a8ba this looks okay though. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #9596: Fix quarantined tests - TestCliWebServer (v1-10-test)
ashb commented on pull request #9596: URL: https://github.com/apache/airflow/pull/9596#issuecomment-652353000 212 commits -- something has gone wrong with this PR (so we can't review it) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9556: API Endpoint - DagRuns Batch
ephraimbuddy commented on a change in pull request #9556: URL: https://github.com/apache/airflow/pull/9556#discussion_r448256015 ## File path: airflow/api_connexion/schemas/dag_run_schema.py ## @@ -72,5 +72,25 @@ class DAGRunCollectionSchema(Schema): total_entries = fields.Int() +class DagRunsBatchFormSchema(Schema): +""" Schema to validate and deserialize the Form(request payload) submitted to DagRun Batch endpoint""" + +class Meta: +""" Meta """ +datetimeformat = 'iso' +strict = True + +page_offset = fields.Int(required=False, missing=0, min=0) +page_limit = fields.Int(required=False, missing=100, min=1) +dag_ids = fields.List(fields.Str(), required=False, missing=None) +execution_date_gte = fields.DateTime(required=False, missing=None) Review comment: ```suggestion execution_date_gte = fields.DateTime() ``` I suggest you remove the missing and required args. To only use them when the value should be something other than False and None respectively. This is already the default from marshmallow. https://marshmallow.readthedocs.io/en/2.x-line/api_reference.html#module-marshmallow.fields This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #9604: Move CloudwatchTaskHandler to the provider package
ephraimbuddy commented on pull request #9604: URL: https://github.com/apache/airflow/pull/9604#issuecomment-652347125 Hi @turbaszek , please take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara edited a comment on pull request #9531: Support .airflowignore for plugins
j-y-matsubara edited a comment on pull request #9531: URL: https://github.com/apache/airflow/pull/9531#issuecomment-651678532 I think the one failure of tests doesn't seem to have anything to do with this PR. The same error occurs in other PRs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek merged pull request #9605: Fix typo in tutorial.rst
turbaszek merged pull request #9605: URL: https://github.com/apache/airflow/pull/9605 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch master updated: Fix typo in tutorial.rst (#9605)
This is an automated email from the ASF dual-hosted git repository.
turbaszek pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new 2d3677f Fix typo in tutorial.rst (#9605)
2d3677f is described below
commit 2d3677fe7d164c782faf357d8335e3b25c83cafb
Author: Giancarlo Romeo
AuthorDate: Wed Jul 1 12:12:23 2020 +0200
Fix typo in tutorial.rst (#9605)
---
docs/tutorial.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/tutorial.rst b/docs/tutorial.rst
index ba63b95..2b79d33 100644
--- a/docs/tutorial.rst
+++ b/docs/tutorial.rst
@@ -166,7 +166,7 @@ Using that same DAG constructor call, it is possible to
define
``user_defined_macros`` which allow you to specify your own variables.
For example, passing ``dict(foo='bar')`` to this argument allows you
to use ``{{ foo }}`` in your templates. Moreover, specifying
-``user_defined_filters`` allow you to register you own filters. For example,
+``user_defined_filters`` allows you to register your own filters. For example,
passing ``dict(hello=lambda name: 'Hello %s' % name)`` to this argument allows
you to use ``{{ 'world' | hello }}`` in your templates. For more information
regarding custom filters have a look at the
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9605: Fix typo in tutorial.rst
boring-cyborg[bot] commented on pull request #9605: URL: https://github.com/apache/airflow/pull/9605#issuecomment-652328738 Awesome work, congrats on your first merged pull request! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch v1-10-test updated (e42a897 -> ec7ad93)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard e42a897 Bump version to 1.10.11 discard 73fefe6 Add docs on using DAGRun.conf (#9578) discard 57619cd Enforce code-block directives in doc (#9443) add 45a583f Enforce code-block directives in doc (#9443) add 641e110 Add docs on using DAGRun.conf (#9578) add ec7ad93 Bump version to 1.10.11 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (e42a897) \ N -- N -- N refs/heads/v1-10-test (ec7ad93) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: docs/concepts.rst | 4 ++-- docs/howto/secure-connections.rst | 2 +- docs/kubernetes.rst | 2 +- docs/lineage.rst | 2 +- docs/security.rst | 10 +- 5 files changed, 10 insertions(+), 10 deletions(-)
[airflow] 02/03: Add docs on using DAGRun.conf (#9578)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 641e1106495050f185265c6e3460758def80734a
Author: Kaxil Naik
AuthorDate: Tue Jun 30 00:59:40 2020 +0100
Add docs on using DAGRun.conf (#9578)
closes https://github.com/apache/airflow/issues/8900
(cherry picked from commit ce4c2297c6c30ccae4222633b923e1a1cda98ec6)
---
airflow/models/dag.py | 2 ++
docs/dag-run.rst | 37 +
docs/img/example_passing_conf.png | Bin 0 -> 97482 bytes
3 files changed, 39 insertions(+)
diff --git a/airflow/models/dag.py b/airflow/models/dag.py
index 54bc87e..e24c164 100644
--- a/airflow/models/dag.py
+++ b/airflow/models/dag.py
@@ -1463,6 +1463,8 @@ class DAG(BaseDag, LoggingMixin):
:type start_date: datetime
:param external_trigger: whether this dag run is externally triggered
:type external_trigger: bool
+:param conf: Dict containing configuration/parameters to pass to the
DAG
+:type conf: dict
:param session: database session
:type session: sqlalchemy.orm.session.Session
"""
diff --git a/docs/dag-run.rst b/docs/dag-run.rst
index 2477d6a..9ea79ea 100644
--- a/docs/dag-run.rst
+++ b/docs/dag-run.rst
@@ -189,6 +189,43 @@ The default is the current date in the UTC timezone.
In addition, you can also manually trigger a DAG Run using the web UI (tab
**DAGs** -> column **Links** -> button **Trigger Dag**)
+Passing Parameters when triggering dags
+--
+
+When triggering a DAG from the CLI, the REST API or the UI, it is possible to
pass configuration for a DAGRun as
+a JSON blob.
+
+Example of a parameterized DAG:
+
+.. code-block:: python
+
+from airflow import DAG
+from airflow.operators.bash_operator import BashOperator
+from airflow.utils.dates import days_ago
+
+dag = DAG("example_parametrized_dag", schedule_interval=None,
start_date=days_ago(2))
+
+parameterized_task = BashOperator(
+task_id='parameterized_task',
+bash_command="echo value: {{ dag_run.conf['conf1'] }}",
+dag=dag,
+)
+
+
+**Note**: The parameters from ``dag_run.conf`` can only be used in a template
field of an operator.
+
+Using CLI
+^^^
+
+.. code-block:: bash
+
+airflow dags trigger --conf '{"conf1": "value1"}' example_parametrized_dag
+
+Using UI
+^^
+
+.. image:: img/example_passing_conf.png
+
To Keep in Mind
* Marking task instances as failed can be done through the UI. This can be
used to stop running task instances.
diff --git a/docs/img/example_passing_conf.png
b/docs/img/example_passing_conf.png
new file mode 100644
index 000..411cae7
Binary files /dev/null and b/docs/img/example_passing_conf.png differ
[airflow] branch v1-10-test updated (e42a897 -> ec7ad93)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. omit e42a897 Bump version to 1.10.11 omit 73fefe6 Add docs on using DAGRun.conf (#9578) omit 57619cd Enforce code-block directives in doc (#9443) new 45a583f Enforce code-block directives in doc (#9443) new 641e110 Add docs on using DAGRun.conf (#9578) new ec7ad93 Bump version to 1.10.11 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (e42a897) \ N -- N -- N refs/heads/v1-10-test (ec7ad93) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/concepts.rst | 4 ++-- docs/howto/secure-connections.rst | 2 +- docs/kubernetes.rst | 2 +- docs/lineage.rst | 2 +- docs/security.rst | 10 +- 5 files changed, 10 insertions(+), 10 deletions(-)
[airflow] 01/03: Enforce code-block directives in doc (#9443)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 45a583f4d51d8bf7ca786a1a67b60ddf53e87c7c
Author: Kamil Breguła
AuthorDate: Sat Jun 20 22:02:13 2020 +0200
Enforce code-block directives in doc (#9443)
(cherry picked from commit 60d19dc2db79f6eb9be146b59e65a92323d25c08)
---
docs/best-practices.rst | 38 +++
docs/build| 25 +
docs/concepts.rst | 227 +++---
docs/dag-run.rst | 10 +-
docs/dag-serialization.rst| 2 +-
docs/faq.rst | 2 +-
docs/howto/custom-operator.rst| 14 +--
docs/howto/email-config.rst | 4 +-
docs/howto/secure-connections.rst | 2 +-
docs/kubernetes.rst | 4 +-
docs/lineage.rst | 4 +-
docs/plugins.rst | 6 +-
docs/scheduler.rst| 2 +-
docs/security.rst | 10 +-
docs/timezone.rst | 10 +-
docs/tutorial.rst | 2 +-
docs/ui.rst | 2 +-
17 files changed, 222 insertions(+), 142 deletions(-)
diff --git a/docs/best-practices.rst b/docs/best-practices.rst
index 1a27e73..fe25977 100644
--- a/docs/best-practices.rst
+++ b/docs/best-practices.rst
@@ -89,13 +89,13 @@ It can result in a lot of open connections.
The best way of using variables is via a Jinja template which will delay
reading the value until the task execution. The template synaxt to do this is:
-.. code::
+.. code-block::
{{ var.value. }}
or if you need to deserialize a json object from the variable :
-.. code::
+.. code-block::
{{ var.json. }}
@@ -117,7 +117,7 @@ DAG Loader Test
This test should ensure that your DAG does not contain a piece of code that
raises error while loading.
No additional code needs to be written by the user to run this test.
-.. code::
+.. code-block::
python your-dag-file.py
@@ -132,7 +132,7 @@ Unit tests ensure that there is no incorrect code in your
DAG. You can write a u
**Unit test for loading a DAG:**
-.. code::
+.. code-block::
from airflow.models import DagBag
import unittest
@@ -149,9 +149,9 @@ Unit tests ensure that there is no incorrect code in your
DAG. You can write a u
self.assertEqual(len(dag.tasks), 1)
**Unit test a DAG structure:**
-This is an example test want to verify the structure of a code-generated DAG
against a dict object
+This is an example test want to verify the structure of a code-generated DAG
against a dict object
-.. code::
+.. code-block::
import unittest
class testClass(unittest.TestCase):
@@ -163,16 +163,16 @@ This is an example test want to verify the structure of a
code-generated DAG aga
self.assertEqual(task.downstream_task_ids, set(downstream_list),
msg="unexpected downstream link in
{}".format(task_id))
def test_dag(self):
- self.assertDagDictEqual({
- "DummyInstruction_0": ["DummyInstruction_1"],
- "DummyInstruction_1": ["DummyInstruction_2"],
- "DummyInstruction_2": ["DummyInstruction_3"],
- "DummyInstruction_3": []
- },dag)
+ self.assertDagDictEqual({
+ "DummyInstruction_0": ["DummyInstruction_1"],
+ "DummyInstruction_1": ["DummyInstruction_2"],
+ "DummyInstruction_2": ["DummyInstruction_3"],
+ "DummyInstruction_3": []
+ },dag)
**Unit test for custom operator:**
-.. code::
+.. code-block::
import unittest
from airflow.utils.state import State
@@ -205,7 +205,7 @@ make sure that the partition is created in S3 and perform
some simple checks to
Similarly, if you have a task that starts a microservice in Kubernetes or
Mesos, you should check if the service has started or not using
:class:`airflow.sensors.http_sensor.HttpSensor`.
-.. code::
+.. code-block::
task = PushToS3(...)
check = S3KeySensor(
@@ -227,7 +227,7 @@ Do not hard code values inside the DAG and then change them
manually according t
You can use environment variables to parameterize the DAG.
-.. code::
+.. code-block::
import os
@@ -252,7 +252,7 @@ If you want to run Airflow in production, make sure you
:doc:`configure the back
You can change the backend using the following config
-.. code:: ini
+.. code-block:: ini
[core]
sql_alchemy_conn = my_conn_string
@@ -261,7 +261,7 @@ Once you have changed the backend, airflow needs to create
all the tables requir
Create an empty DB and give airflow's user the permission to ``CREATE/ALTER``
it.
Once that is done, you can run -
-.. code::
+.. code-block::
airflow upgradedb
@@ -305,14 +305,14 @@ Airflow comes bundles with a default ``airflow.cfg``
configuration file.
You should use environment variables
[airflow] 03/03: Bump version to 1.10.11
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit ec7ad934117be7651e5ac3624ae08baa7784953e Author: Kaxil Naik AuthorDate: Wed Jul 1 01:24:49 2020 +0100 Bump version to 1.10.11 --- airflow/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/airflow/version.py b/airflow/version.py index e6e92b1..a888b43 100644 --- a/airflow/version.py +++ b/airflow/version.py @@ -18,4 +18,4 @@ # under the License. # -version = '1.10.10' +version = '1.10.11'
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9556: API Endpoint - DagRuns Batch
ephraimbuddy commented on a change in pull request #9556: URL: https://github.com/apache/airflow/pull/9556#discussion_r448256015 ## File path: airflow/api_connexion/schemas/dag_run_schema.py ## @@ -72,5 +72,25 @@ class DAGRunCollectionSchema(Schema): total_entries = fields.Int() +class DagRunsBatchFormSchema(Schema): +""" Schema to validate and deserialize the Form(request payload) submitted to DagRun Batch endpoint""" + +class Meta: +""" Meta """ +datetimeformat = 'iso' +strict = True + +page_offset = fields.Int(required=False, missing=0, min=0) +page_limit = fields.Int(required=False, missing=100, min=1) +dag_ids = fields.List(fields.Str(), required=False, missing=None) +execution_date_gte = fields.DateTime(required=False, missing=None) Review comment: ```suggestion execution_date_gte = fields.DateTime() ``` I suggest you remove the missing and required field. To only use them when the value should be something other than False and None respectively. This is already the default from marshmallow. https://marshmallow.readthedocs.io/en/2.x-line/api_reference.html#module-marshmallow.fields This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on issue #9606: Turn off pickling of XCom by default in 2.0
ashb commented on issue #9606: URL: https://github.com/apache/airflow/issues/9606#issuecomment-652311248 /cc @turbaszek This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb opened a new issue #9606: Turn off pickling of XCom by default in 2.0
ashb opened a new issue #9606: URL: https://github.com/apache/airflow/issues/9606 The comment for airflow.cfg says this: ``` # Whether to enable pickling for xcom (note that this is insecure and allows for # RCE exploits). This will be deprecated in Airflow 2.0 (be forced to False). enable_xcom_pickling = True ``` With this false the kinds of values we can serialize is limited to JSON (dicts, lists, numbers, strings, None and boolean) there are valid reasons for wanting this on I don't think we should remove it entirely -- instead I think we should make JSON/not Pickled the default instead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9590: Improve idempotency of BigQueryInsertJobOperator
turbaszek commented on pull request #9590: URL: https://github.com/apache/airflow/pull/9590#issuecomment-652306141 @albertocalderari @edejong what do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9517: [AIRFLOW-XXXX] Remove now-inaccurate docstring
turbaszek commented on a change in pull request #9517: URL: https://github.com/apache/airflow/pull/9517#discussion_r448233835 ## File path: airflow/providers/amazon/aws/operators/athena.py ## @@ -28,9 +28,6 @@ class AWSAthenaOperator(BaseOperator): """ An operator that submit presto query to athena. -If ``do_xcom_push`` is True, the QueryExecutionID assigned to the -query will be pushed to an XCom when it successfuly completes. Review comment: If operator's `execute` method returns a `result` it doesn't mean that it will be always saved in XCom table. This will happen only if operator is used with `do_xcom_push=True` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9517: [AIRFLOW-XXXX] Remove now-inaccurate docstring
turbaszek commented on a change in pull request #9517: URL: https://github.com/apache/airflow/pull/9517#discussion_r448232645 ## File path: airflow/providers/amazon/aws/operators/athena.py ## @@ -28,9 +28,6 @@ class AWSAthenaOperator(BaseOperator): """ An operator that submit presto query to athena. -If ``do_xcom_push`` is True, the QueryExecutionID assigned to the -query will be pushed to an XCom when it successfuly completes. Review comment: I think this is still valid because: https://github.com/apache/airflow/blob/7a5441836bbc8bbcb0b02bf68dbd93d63fe4ec6a/airflow/models/taskinstance.py#L1035-L1037 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #9567: Production image is build on CI master from scratch
potiuk closed pull request #9567: URL: https://github.com/apache/airflow/pull/9567 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] giancarloromeo opened a new pull request #9605: Fix typo
giancarloromeo opened a new pull request #9605: URL: https://github.com/apache/airflow/pull/9605 --- Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Target Github ISSUE in description if exists - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9605: Fix typo
boring-cyborg[bot] commented on pull request #9605: URL: https://github.com/apache/airflow/pull/9605#issuecomment-652300786 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://apache-airflow-slack.herokuapp.com/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9567: Production image is build on CI master from scratch
potiuk commented on pull request #9567: URL: https://github.com/apache/airflow/pull/9567#issuecomment-652300456 Superseded by #9468 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-client-go] turbaszek commented on a change in pull request #1: Add generated go client
turbaszek commented on a change in pull request #1: URL: https://github.com/apache/airflow-client-go/pull/1#discussion_r448225350 ## File path: README.md ## @@ -1 +1,62 @@ -# Airflow API Client for Go + +Airflow Go client += + +Go Airflow OpenAPI client generated from [openapi spec](https://github.com/apache/airflow/tree/master/clients). Review comment: ```suggestion Go Airflow OpenAPI client generated from [openapi spec](https://github.com/apache/airflow/blob/master/airflow/api_connexion/openapi/v1.yaml). ``` Current link results in 404 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-client-go] turbaszek closed pull request #2: Test Github actions
turbaszek closed pull request #2: URL: https://github.com/apache/airflow-client-go/pull/2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-client-go] houqp commented on a change in pull request #1: Add generated go client
houqp commented on a change in pull request #1: URL: https://github.com/apache/airflow-client-go/pull/1#discussion_r448218603 ## File path: .github/workflows/ci.yml ## @@ -0,0 +1,25 @@ +name: CI + +on: + push: +branches: ['master'] +tags: + - '*' + pull_request: +branches: ['master'] + +jobs: + build: +runs-on: ubuntu-latest +steps: + - name: Checkout +uses: actions/checkout@master + +- name: Set up Go Review comment: good catch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-client-go] turbaszek commented on a change in pull request #1: Add generated go client
turbaszek commented on a change in pull request #1: URL: https://github.com/apache/airflow-client-go/pull/1#discussion_r448217033 ## File path: .github/workflows/ci.yml ## @@ -0,0 +1,25 @@ +name: CI + +on: + push: +branches: ['master'] +tags: + - '*' Review comment: > It doesn't work because we wouldn't contact to INFRA I think I can create an action in repo, take a look at #2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow-client-go] 01/01: Create blank.yml
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a commit to branch test-action in repository https://gitbox.apache.org/repos/asf/airflow-client-go.git commit 0d85382578757b78bff69f7ae146d7d43fe15406 Author: Tomek Urbaszek AuthorDate: Wed Jul 1 10:53:35 2020 +0200 Create blank.yml --- .github/workflows/blank.yml | 33 + 1 file changed, 33 insertions(+) diff --git a/.github/workflows/blank.yml b/.github/workflows/blank.yml new file mode 100644 index 000..d6783e2 --- /dev/null +++ b/.github/workflows/blank.yml @@ -0,0 +1,33 @@ +# This is a basic workflow to help you get started with Actions + +name: CI + +# Controls when the action will run. Triggers the workflow on push or pull request +# events but only for the master branch +on: + push: +branches: [ master ] + pull_request: +branches: [ master ] + +# A workflow run is made up of one or more jobs that can run sequentially or in parallel +jobs: + # This workflow contains a single job called "build" + build: +# The type of runner that the job will run on +runs-on: ubuntu-latest + +# Steps represent a sequence of tasks that will be executed as part of the job +steps: +# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it +- uses: actions/checkout@v2 + +# Runs a single command using the runners shell +- name: Run a one-line script + run: echo Hello, world! + +# Runs a set of commands using the runners shell +- name: Run a multi-line script + run: | +echo Add other actions to build, +echo test, and deploy your project.
[GitHub] [airflow-client-go] turbaszek opened a new pull request #2: Test Github actions
turbaszek opened a new pull request #2: URL: https://github.com/apache/airflow-client-go/pull/2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow-client-go] branch test-action created (now 0d85382)
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a change to branch test-action in repository https://gitbox.apache.org/repos/asf/airflow-client-go.git. at 0d85382 Create blank.yml This branch includes the following new commits: new 0d85382 Create blank.yml The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[GitHub] [airflow-client-go] turbaszek commented on a change in pull request #1: Add generated go client
turbaszek commented on a change in pull request #1: URL: https://github.com/apache/airflow-client-go/pull/1#discussion_r448217033 ## File path: .github/workflows/ci.yml ## @@ -0,0 +1,25 @@ +name: CI + +on: + push: +branches: ['master'] +tags: + - '*' Review comment: > It doesn't work because we wouldn't contact to INFRA I think I can create an action in repo This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-client-go] mik-laj commented on a change in pull request #1: Add generated go client
mik-laj commented on a change in pull request #1: URL: https://github.com/apache/airflow-client-go/pull/1#discussion_r448216216 ## File path: .github/workflows/ci.yml ## @@ -0,0 +1,25 @@ +name: CI + +on: + push: +branches: ['master'] +tags: + - '*' + pull_request: +branches: ['master'] + +jobs: + build: +runs-on: ubuntu-latest +steps: + - name: Checkout +uses: actions/checkout@master + +- name: Set up Go Review comment: Incorrect indentation https://github.com/mik-laj/airflow-client-go/actions/runs/153839158? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] christophlingg closed pull request #9535: boolean values in environment variables
christophlingg closed pull request #9535: URL: https://github.com/apache/airflow/pull/9535 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] christophlingg commented on a change in pull request #9535: boolean values in environment variables
christophlingg commented on a change in pull request #9535: URL: https://github.com/apache/airflow/pull/9535#discussion_r448213603 ## File path: docs/howto/set-config.rst ## @@ -37,6 +37,12 @@ or by creating a corresponding environment variable: .. code-block:: bash export AIRFLOW__CORE__SQL_ALCHEMY_CONN=my_conn_string + +For setting boolean configuration values use ``true``/``false``: + +.. code-block:: bask + +export AIRFLOW__CORE__REMOTE_LOGGING=true Review comment: Maybe I was thinking too much like a software developer here. And as the parsing is so clever, people won't have issues. I will close the PR, thanks for your feedback. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-client-go] mik-laj commented on a change in pull request #1: Add generated go client
mik-laj commented on a change in pull request #1: URL: https://github.com/apache/airflow-client-go/pull/1#discussion_r448212736 ## File path: .github/workflows/ci.yml ## @@ -0,0 +1,25 @@ +name: CI + +on: + push: +branches: ['master'] +tags: + - '*' Review comment: It doesn't work because we wouldn't contact to INFRA This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] chrismclennon commented on a change in pull request #8941: Tree view tooltip to show following execution date
chrismclennon commented on a change in pull request #8941:
URL: https://github.com/apache/airflow/pull/8941#discussion_r448211785
##
File path: airflow/www/static/js/task-instances.js
##
@@ -67,7 +67,8 @@ export default function tiTooltip(ti, {includeTryNumber =
false} = {}) {
if (ti.task_id !== undefined) {
tt += `Task_id: ${escapeHtml(ti.task_id)}`;
}
- tt += `Run: ${formatDateTime(ti.execution_date)}`;
+ tt += `Execution Date: ${formatDateTime(ti.execution_date)}`;
+ tt += `Following Execution Date:
${formatDateTime(ti.following_execution_date)}`;
Review comment:
That's a really great question.
For manual runs, it shows the following execution date, when it should be
showing the same execution date.
For schedule interval of type "None", it explodes in a traceback:
```
...
File "/opt/airflow/airflow/www/decorators.py", line 57, in wrapper
return f(*args, **kwargs)
File "/opt/airflow/airflow/www/views.py", line 1415, in tree
dag_run['following_execution_date'] =
dag.following_schedule(execution_date).isoformat()
AttributeError: 'NoneType' object has no attribute 'isoformat'
```
I'll make fixes to both and comment when done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow-client-go] turbaszek commented on a change in pull request #1: Add generated go client
turbaszek commented on a change in pull request #1: URL: https://github.com/apache/airflow-client-go/pull/1#discussion_r448192517 ## File path: .github/workflows/ci.yml ## @@ -0,0 +1,25 @@ +name: CI + +on: + push: +branches: ['master'] +tags: + - '*' Review comment: ```suggestion ``` The actions seem to not be running This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9593: Improve handling Dataproc cluster creation with ERROR state
turbaszek commented on a change in pull request #9593: URL: https://github.com/apache/airflow/pull/9593#discussion_r448189393 ## File path: airflow/providers/google/cloud/hooks/dataproc.py ## @@ -376,15 +376,25 @@ def diagnose_cluster( :type metadata: Sequence[Tuple[str, str]] """ client = self.get_cluster_client(location=region) -result = client.diagnose_cluster( +operation = client.diagnose_cluster( project_id=project_id, region=region, cluster_name=cluster_name, retry=retry, timeout=timeout, metadata=metadata, ) -return result +done = False +while not done: +time.sleep(3) +# The dataproc python client has some problems +# TypeError: Could not convert Any to Empty +try: +done = operation.done() +except TypeError: +pass Review comment: Due to error reported in https://github.com/googleapis/python-dataproc/issues/51 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-client-go] houqp commented on pull request #1: Add generated go client
houqp commented on pull request #1: URL: https://github.com/apache/airflow-client-go/pull/1#issuecomment-652254946 @mik-laj @turbaszek @potiuk added license header to go source and ci action, ready for another round of review. let's discuss how to do end to end test after we release a version of the client so it's easier to install the client from ci environment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jhtimmins commented on pull request #9517: [AIRFLOW-XXXX] Remove now-inaccurate docstring
jhtimmins commented on pull request #9517: URL: https://github.com/apache/airflow/pull/9517#issuecomment-652229569 This change looks good. The earliest history I see for this file is in November 2019, at which point the Athena operator did not make use of `do_xcom_push`. It's not immediately clear if it ever did use the variable, but it isn't now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9602: Move S3TaskHandler to the provider package
turbaszek commented on pull request #9602: URL: https://github.com/apache/airflow/pull/9602#issuecomment-652228295 > The `S3TaskHandler` class from `airflow.utils.log.s3_task_handler` was moved to > `airflow.providers.amazon.aws.log.s3_task_handler`. I'm ok with `.aws.log...` but what do @mik-laj think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] DanilBaibak commented on issue #9309: Bug / ThresholdCheckOperator Does Not Work
DanilBaibak commented on issue #9309: URL: https://github.com/apache/airflow/issues/9309#issuecomment-652225005 Fixed! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
