[jira] [Created] (AIRFLOW-6873) mypyd is failing again with new type errors
QP Hou created AIRFLOW-6873: --- Summary: mypyd is failing again with new type errors Key: AIRFLOW-6873 URL: https://issues.apache.org/jira/browse/AIRFLOW-6873 Project: Apache Airflow Issue Type: Improvement Components: build Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou New version of mypy is throwing new type errors, we should get that fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6871) Tree view unusable for large DAGs
QP Hou created AIRFLOW-6871: --- Summary: Tree view unusable for large DAGs Key: AIRFLOW-6871 URL: https://issues.apache.org/jira/browse/AIRFLOW-6871 Project: Apache Airflow Issue Type: Improvement Components: webserver Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou By default airflow loads 25 runs in the tree view. For our main DAG, it's just not usable because all we get is a 5xx error when hitting that page. Manually overriding number of runs to 15 makes the page to load again, but it's very slow and takes more than 1 minute to render. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6802) scheduler not honoring dag.max_active_run config
QP Hou created AIRFLOW-6802: --- Summary: scheduler not honoring dag.max_active_run config Key: AIRFLOW-6802 URL: https://issues.apache.org/jira/browse/AIRFLOW-6802 Project: Apache Airflow Issue Type: Bug Components: scheduler Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou commit https://github.com/apache/airflow/commit/50efda5c69c1ddfaa869b408540182fb19f1a286 introduced a bug that prevents the scheduler from enforcing max active run config for all DAGs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6779) Schedule stopped scheduling tasks with concurrency limit or depends on past
QP Hou created AIRFLOW-6779: --- Summary: Schedule stopped scheduling tasks with concurrency limit or depends on past Key: AIRFLOW-6779 URL: https://issues.apache.org/jira/browse/AIRFLOW-6779 Project: Apache Airflow Issue Type: Bug Components: scheduler Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou https://github.com/apache/airflow/pull/4751 introduced a bug that stops the scheduler from scheduling tasks with the following properties: * depends on past * has custom concurrency limit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6613) center dag on graph view load
QP Hou created AIRFLOW-6613: --- Summary: center dag on graph view load Key: AIRFLOW-6613 URL: https://issues.apache.org/jira/browse/AIRFLOW-6613 Project: Apache Airflow Issue Type: New Feature Components: ui Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou Centering DAG on center of the canvas with the right zoom level will make large DAG much easier to navigate form the UI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6605) code doesn't pass static check from latest mypy (0.761)
QP Hou created AIRFLOW-6605:
---
Summary: code doesn't pass static check from latest mypy (0.761)
Key: AIRFLOW-6605
URL: https://issues.apache.org/jira/browse/AIRFLOW-6605
Project: Apache Airflow
Issue Type: New Feature
Components: build
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou
Warnings from latest mypy:
{code:text}
tests/models/test_taskinstance.py:52: error: Need type annotation for 'task_id'
[var-annotated]
task_id = None
^
tests/models/test_taskinstance.py:53: error: Need type annotation for 'dag_id'
[var-annotated]
dag_id = None
^
tests/models/test_taskinstance.py:54: error: Need type annotation for
'execution_date' [var-annotated]
execution_date = None
^
tests/models/test_taskinstance.py:55: error: Need type annotation for
'task_state_in_callback' [var-annotated]
task_state_in_callback = None
^
Found 4 errors in 1 file (checked 1500 source files)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6576) Deleting a task with SLA crashes the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QP Hou updated AIRFLOW-6576: Description: When a task with SLA is deleted from a DAG after the SLA miss is logged but before the notification was sent, scheduler will crash with an AirflowException (was: When a task with SLA is deleted from a DAG after the SLA miss is logged by before the notification was sent, scheduler will crash with an AirflowException) > Deleting a task with SLA crashes the scheduler > -- > > Key: AIRFLOW-6576 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6576 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Major > > When a task with SLA is deleted from a DAG after the SLA miss is logged but > before the notification was sent, scheduler will crash with an > AirflowException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6576) Deleting a task with SLA crashes the scheduler
QP Hou created AIRFLOW-6576: --- Summary: Deleting a task with SLA crashes the scheduler Key: AIRFLOW-6576 URL: https://issues.apache.org/jira/browse/AIRFLOW-6576 Project: Apache Airflow Issue Type: New Feature Components: scheduler Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou When a task with SLA is deleted from a DAG after the SLA miss is logged by before the notification was sent, scheduler will crash with an AirflowException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6262) Add execute callback for operators
QP Hou created AIRFLOW-6262: --- Summary: Add execute callback for operators Key: AIRFLOW-6262 URL: https://issues.apache.org/jira/browse/AIRFLOW-6262 Project: Apache Airflow Issue Type: New Feature Components: operators Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou We are in the process of adding tracing events for various stages of task runs. Adding an execute callback to operator will help us emit task execution events to our monitoring system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6080) New version of mypy catches more type errors
QP Hou created AIRFLOW-6080: --- Summary: New version of mypy catches more type errors Key: AIRFLOW-6080 URL: https://issues.apache.org/jira/browse/AIRFLOW-6080 Project: Apache Airflow Issue Type: Improvement Components: ci Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou New version of mypy is reporting new list of errors. We should fix them so we can upgrade it in our CI/CD pipeline. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5902) local_task_job is sleeping more often than it needs and spamming task log
QP Hou created AIRFLOW-5902: --- Summary: local_task_job is sleeping more often than it needs and spamming task log Key: AIRFLOW-5902 URL: https://issues.apache.org/jira/browse/AIRFLOW-5902 Project: Apache Airflow Issue Type: Bug Components: core Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou local_task_log's _execute loop is calling extra unnecessary sleep after a heartbeat to keep up with the heart rate. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5900) heartbeat_callback for local_task_job can be more efficient
QP Hou created AIRFLOW-5900: --- Summary: heartbeat_callback for local_task_job can be more efficient Key: AIRFLOW-5900 URL: https://issues.apache.org/jira/browse/AIRFLOW-5900 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou local_task_job's heartbeat_callback function is issuing unnecessary system calls when they are not needed. We can move those calls into the if block to make it more efficient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5899) dmypy still reports error
QP Hou created AIRFLOW-5899: --- Summary: dmypy still reports error Key: AIRFLOW-5899 URL: https://issues.apache.org/jira/browse/AIRFLOW-5899 Project: Apache Airflow Issue Type: Improvement Components: ci Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou dmypy is still reporting error, would be nice to get those fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5898) alembic command crashes due to typing import path collision
QP Hou created AIRFLOW-5898:
---
Summary: alembic command crashes due to typing import path
collision
Key: AIRFLOW-5898
URL: https://issues.apache.org/jira/browse/AIRFLOW-5898
Project: Apache Airflow
Issue Type: New Feature
Components: database
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou
Running alemibc within airflow directory results in the following crash:
{code:java}
File
"/Users/qph/Envs/airflow/lib/python3.7/site-packages/typing_extensions.py",
line 17, in
from typing import Generic, Callable, TypeVar, Tuple
ImportError: cannot import name 'Generic' from 'typing'
{code}
This is due to typing.py within airlfow directory collides with the builtin
typing module when a command runs inside airflow directory. Renaming it to
`custom_typing.py` fixes the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-5832) pagerduty hook
QP Hou created AIRFLOW-5832: --- Summary: pagerduty hook Key: AIRFLOW-5832 URL: https://issues.apache.org/jira/browse/AIRFLOW-5832 Project: Apache Airflow Issue Type: New Feature Components: hooks Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou With pagerduty hook, users will be able to use pagerduty to notify critical DAG run issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5811) Add metric for externally killed task count
QP Hou created AIRFLOW-5811: --- Summary: Add metric for externally killed task count Key: AIRFLOW-5811 URL: https://issues.apache.org/jira/browse/AIRFLOW-5811 Project: Apache Airflow Issue Type: Improvement Components: scheduler Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou Number of externally killed task would be very useful for monitoring init container error when using k8s executor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5662) Scheduler incorrectly emits occupied slots count metric as used slot count
QP Hou created AIRFLOW-5662: --- Summary: Scheduler incorrectly emits occupied slots count metric as used slot count Key: AIRFLOW-5662 URL: https://issues.apache.org/jira/browse/AIRFLOW-5662 Project: Apache Airflow Issue Type: Bug Components: scheduler Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou Value of pool.used_slots metric is actually occupied slot count. There is also a redundant SQL query to the database that can be removed in that loop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5641) Not able to run git sync init container as root
QP Hou created AIRFLOW-5641: --- Summary: Not able to run git sync init container as root Key: AIRFLOW-5641 URL: https://issues.apache.org/jira/browse/AIRFLOW-5641 Project: Apache Airflow Issue Type: Bug Components: executor-kubernetes Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou We used to be able to run git sync init container as root user. A recent refactoring of the code base (AIRFLOW-4851) introduced a regression that makes it impossible to run git sync container as root. As a result, we are not able to start any task using kubernetes executor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5276) Remove unused helpers from airflow.utils.helpers
[
https://issues.apache.org/jira/browse/AIRFLOW-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
QP Hou reassigned AIRFLOW-5276:
---
Assignee: QP Hou
> Remove unused helpers from airflow.utils.helpers
>
>
> Key: AIRFLOW-5276
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5276
> Project: Apache Airflow
> Issue Type: Task
> Components: core
>Affects Versions: 1.10.4
>Reporter: Ash Berlin-Taylor
>Assignee: QP Hou
>Priority: Trivial
>
> At the very least {{is_in}} is not used in the code base any more, and it's
> behaviour has changed subtly in Python 3.7 so the tests fail there too.
> So lets remove that helper, and it's associated tests, plus anything else not
> used any more.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (AIRFLOW-5251) crash due to missing typing-extension dependency for python 3.7
QP Hou created AIRFLOW-5251: --- Summary: crash due to missing typing-extension dependency for python 3.7 Key: AIRFLOW-5251 URL: https://issues.apache.org/jira/browse/AIRFLOW-5251 Project: Apache Airflow Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou Latest master now crashes with python 3.7 due to missing typing-extension dependency. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AIRFLOW-5250) dmypy still reports error for gcp hooks
[
https://issues.apache.org/jira/browse/AIRFLOW-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
QP Hou updated AIRFLOW-5250:
Description:
dmypy still reports the following error:
{code:java}
airflow/contrib/hooks/gcp_cloud_build_hook.py:51: error: Need type annotation
for '_conn'
airflow/contrib/hooks/gcp_compute_hook.py:52: error: Need type annotation for
'_conn'
airflow/contrib/hooks/gcp_sql_hook.py:739: error: Need type annotation for
'_conn'
airflow/gcp/hooks/functions.py:43: error: Need type annotation for '_conn'{code}
> dmypy still reports error for gcp hooks
> ---
>
> Key: AIRFLOW-5250
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5250
> Project: Apache Airflow
> Issue Type: Bug
> Components: ci
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Trivial
>
> dmypy still reports the following error:
> {code:java}
> airflow/contrib/hooks/gcp_cloud_build_hook.py:51: error: Need type annotation
> for '_conn'
> airflow/contrib/hooks/gcp_compute_hook.py:52: error: Need type annotation for
> '_conn'
> airflow/contrib/hooks/gcp_sql_hook.py:739: error: Need type annotation for
> '_conn'
> airflow/gcp/hooks/functions.py:43: error: Need type annotation for
> '_conn'{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (AIRFLOW-5250) dmypy still reports error for gcp hooks
[ https://issues.apache.org/jira/browse/AIRFLOW-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QP Hou updated AIRFLOW-5250: Summary: dmypy still reports error for gcp hooks (was: fix missing dmypy error for gcp hooks) > dmypy still reports error for gcp hooks > --- > > Key: AIRFLOW-5250 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5250 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Trivial > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5250) fix missing dmypy error for gcp hooks
QP Hou created AIRFLOW-5250: --- Summary: fix missing dmypy error for gcp hooks Key: AIRFLOW-5250 URL: https://issues.apache.org/jira/browse/AIRFLOW-5250 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5140) missing type annotation errors reported by dmypy
QP Hou created AIRFLOW-5140:
---
Summary: missing type annotation errors reported by dmypy
Key: AIRFLOW-5140
URL: https://issues.apache.org/jira/browse/AIRFLOW-5140
Project: Apache Airflow
Issue Type: Bug
Components: ci
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou
dmypy is reporting the following error:
{code:bash}
$ dmypy run -- --follow-imports=error airflow tests
tests/core.py:2188: error: Need type annotation for 'HDFSHook'
tests/core.py:2189: error: Need type annotation for 'snakebite'
airflow/__init__.py:42: error: Need type annotation for 'login'
airflow/api/auth/backend/default.py:22: error: Need type annotation for
'CLIENT_AUTH'
airflow/api/auth/backend/deny_all.py:23: error: Need type annotation for
'CLIENT_AUTH'
airflow/contrib/example_dags/example_gcp_speech.py:60: error: Need type
annotation for 'SOURCE_LANGUAGE'
airflow/contrib/hooks/gcp_cloud_build_hook.py:50: error: Need type annotation
for '_conn'
airflow/contrib/hooks/gcp_compute_hook.py:50: error: Need type annotation for
'_conn'
airflow/contrib/hooks/gcp_function_hook.py:41: error: Need type annotation for
'_conn'
airflow/contrib/hooks/gcp_sql_hook.py:722: error: Need type annotation for
'_conn'
airflow/contrib/hooks/gcp_translate_hook.py:35: error: Need type annotation for
'_client'
airflow/contrib/hooks/gcs_hook.py:41: error: Need type annotation for '_conn'
airflow/contrib/operators/awsbatch_operator.py:62: error: Need type annotation
for 'client'
airflow/contrib/operators/awsbatch_operator.py:63: error: Need type annotation
for 'arn'
airflow/contrib/operators/ecs_operator.py:61: error: Need type annotation for
'client'
airflow/contrib/operators/ecs_operator.py:62: error: Need type annotation for
'arn'
airflow/contrib/operators/gcp_function_operator.py:229: error: Need type
annotation for 'upload_function'
airflow/executors/__init__.py:28: error: Need type annotation for
'DEFAULT_EXECUTOR'
airflow/hooks/dbapi_hook.py:41: error: Need type annotation for 'connector'
airflow/models/crypto.py:49: error: Need type annotation for '_fernet'
airflow/security/kerberos.py:25: error: Need type annotation for
'NEED_KRB181_WORKAROUND'
airflow/settings.py:66: error: Need type annotation for 'SQL_ALCHEMY_CONN'
airflow/settings.py:67: error: Need type annotation for 'DAGS_FOLDER'
airflow/settings.py:68: error: Need type annotation for 'PLUGINS_FOLDER'
airflow/settings.py:69: error: Need type annotation for 'LOGGING_CLASS_PATH'
airflow/settings.py:71: error: Need type annotation for 'engine'
airflow/settings.py:72: error: Need type annotation for 'Session'
airflow/settings.py:312: error: Need type annotation for 'CONTEXT_MANAGER_DAG'
airflow/utils/state.py:28: error: Need type annotation for 'NONE'
airflow/www/app.py:39: error: Need type annotation for 'appbuilder'
tests/contrib/hooks/test_databricks_hook.py:72: error: Need type annotation for
'RESULT_STATE'
tests/contrib/hooks/test_gcp_cloud_build_hook.py:49: error: Need type
annotation for 'hook'
tests/contrib/hooks/test_gcp_cloud_build_hook.py:122: error: Need type
annotation for 'hook'
tests/contrib/hooks/test_gcp_cloud_build_hook.py:191: error: Need type
annotation for 'hook'
tests/contrib/utils/gcp_authenticator.py:57: error: Need type annotation for
'original_account'
tests/test_utils/reset_warning_registry.py:45: error: Need type annotation for
'_pattern'
tests/test_utils/reset_warning_registry.py:48: error: Need type annotation for
'_backup'
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (AIRFLOW-5117) Airflow K8S not working with EKS when not deployed in cluster
QP Hou created AIRFLOW-5117: --- Summary: Airflow K8S not working with EKS when not deployed in cluster Key: AIRFLOW-5117 URL: https://issues.apache.org/jira/browse/AIRFLOW-5117 Project: Apache Airflow Issue Type: Bug Components: executors, hooks Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou When Airflow scheduler or worker is not deployed in K8S cluster, all K8S API call will fail with 401 error after running for about 14min. This is due to EKS enforces a 15min timeout for all API tokens but the official k8S python client doesn't support token refresh. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (AIRFLOW-4316) Keys/Variable names in the 'kubernetes_environment_variables' section are transformed to lower case
[ https://issues.apache.org/jira/browse/AIRFLOW-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QP Hou reassigned AIRFLOW-4316: --- Assignee: QP Hou > Keys/Variable names in the 'kubernetes_environment_variables' section are > transformed to lower case > --- > > Key: AIRFLOW-4316 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4316 > Project: Apache Airflow > Issue Type: Bug > Components: configuration, contrib >Affects Versions: 1.10.3 >Reporter: Vlad Precup >Assignee: QP Hou >Priority: Major > Labels: configuration, kubernetes > Fix For: 1.10.4 > > Original Estimate: 24h > Remaining Estimate: 24h > > All the environment variable names declared under the > kubernetes_environment_variables section are converted to lowercase in the > newly generated pods when running Airflow with the KubernetesExecutor in > version 1.10.3. > The behavior is not aligned with the comments in the documentation / code > comments in the default config file, airflow.cfg. > Investigation helper: > Since we are treating all the sections uniformly in the configuration > component, all the keys are lowered (i.e. `configuration.py`, line 342 ->`key > = env_var.replace(section_prefix, '').lower()`) > Please also write (a) unit test(s) for this special case. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AIRFLOW-5055) kubernetes_environment_variables section cannot be set using `AIRFLOW_` prefixed environment variables
[ https://issues.apache.org/jira/browse/AIRFLOW-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QP Hou updated AIRFLOW-5055: Description: When you set environment variables like `AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLESAIRFLOW__HOME`, the scheduler will pass `airflow_home` as environment variable key to workers instead, which gets ignored by the workers. (was: When you set environment variables like `AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__HOME`, the scheduler will pass `airflow_home` as environment variable key to workers instead, which gets ignored by the workers.) > kubernetes_environment_variables section cannot be set using `AIRFLOW_` > prefixed environment variables > -- > > Key: AIRFLOW-5055 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5055 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Minor > > When you set environment variables like > `AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLESAIRFLOW__HOME`, the scheduler > will pass `airflow_home` as environment variable key to workers instead, > which gets ignored by the workers. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AIRFLOW-5055) kubernetes_environment_variables section cannot be set using `AIRFLOW_` prefixed environment variables
[ https://issues.apache.org/jira/browse/AIRFLOW-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QP Hou updated AIRFLOW-5055: Summary: kubernetes_environment_variables section cannot be set using `AIRFLOW_` prefixed environment variables (was: kubernetes_environment_variables section cannot be set using `AIRFLOW_` environment variables) > kubernetes_environment_variables section cannot be set using `AIRFLOW_` > prefixed environment variables > -- > > Key: AIRFLOW-5055 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5055 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Minor > > When you set environment variables like > `AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__HOME`, the scheduler > will pass `airflow_home` as environment variable key to workers instead, > which gets ignored by the workers. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5055) kubernetes_environment_variables section cannot be set using `AIRFLOW_` environment variables
QP Hou created AIRFLOW-5055: --- Summary: kubernetes_environment_variables section cannot be set using `AIRFLOW_` environment variables Key: AIRFLOW-5055 URL: https://issues.apache.org/jira/browse/AIRFLOW-5055 Project: Apache Airflow Issue Type: Bug Components: configuration Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou When you set environment variables like `AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__HOME`, the scheduler will pass `airflow_home` as environment variable key to workers instead, which gets ignored by the workers. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5038) Pod deleted message is shown incorrectly when pod deletion is disabled
QP Hou created AIRFLOW-5038:
---
Summary: Pod deleted message is shown incorrectly when pod
deletion is disabled
Key: AIRFLOW-5038
URL: https://issues.apache.org/jira/browse/AIRFLOW-5038
Project: Apache Airflow
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou
When `delete_worker_pods` is set to False, scheduler still logs this line:
`self.log.info('Deleted pod: %s', str(key))`, which is incorrect and confusing.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Closed] (AIRFLOW-5032) build is broken due to version conflict between tzlocal and pendulum
[
https://issues.apache.org/jira/browse/AIRFLOW-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
QP Hou closed AIRFLOW-5032.
---
Resolution: Duplicate
> build is broken due to version conflict between tzlocal and pendulum
>
>
> Key: AIRFLOW-5032
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5032
> Project: Apache Airflow
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Major
>
> Lastest master build is broken due to dependency version conflict, here is
> the error:
> {code:java}
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 4, in
> __import__('pkg_resources').require('apache-airflow==2.0.0.dev0')
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
> 3241, in
> @_call_aside
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
> 3225, in _call_aside
> f(*args, **kwargs)
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
> 3254, in _initialize_master_working_set
> working_set = WorkingSet._build_master()
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
> 585, in _build_master
> return cls._build_from_requirements(__requires__)
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
> 598, in _build_from_requirements
> dists = ws.resolve(reqs, Environment())
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
> 791, in resolve
> raise VersionConflict(dist, req).with_context(dependent_req)
> pkg_resources.ContextualVersionConflict: (tzlocal 2.0.0
> (/usr/local/lib/python3.7/site-packages),
> Requirement.parse('tzlocal<2.0.0.0,>=1.5.0.0'), {'pendulum'})
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (AIRFLOW-5032) build is broken due to version conflict between tzlocal and pendulum
QP Hou created AIRFLOW-5032:
---
Summary: build is broken due to version conflict between tzlocal
and pendulum
Key: AIRFLOW-5032
URL: https://issues.apache.org/jira/browse/AIRFLOW-5032
Project: Apache Airflow
Issue Type: Bug
Components: build
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou
Lastest master build is broken due to dependency version conflict, here is the
error:
{code:java}
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 4, in
__import__('pkg_resources').require('apache-airflow==2.0.0.dev0')
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
3241, in
@_call_aside
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
3225, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
3254, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
585, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
598, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line
791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (tzlocal 2.0.0
(/usr/local/lib/python3.7/site-packages),
Requirement.parse('tzlocal<2.0.0.0,>=1.5.0.0'), {'pendulum'})
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (AIRFLOW-5030) config key name with __ in it is getting skipped in AirflowConfigParser.as_dict method
QP Hou created AIRFLOW-5030: --- Summary: config key name with __ in it is getting skipped in AirflowConfigParser.as_dict method Key: AIRFLOW-5030 URL: https://issues.apache.org/jira/browse/AIRFLOW-5030 Project: Apache Airflow Issue Type: Bug Components: configuration Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou When a key name contains one or more "__", it will get ignored by AirflowConfigParser.as_dict method. This makes it impossible to set worker environment variables using `KUBERNETES_ENVIRONMENT_VARIABLES` section from scheduler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5021) gitpyhon should not be included as runtime dependency
QP Hou created AIRFLOW-5021: --- Summary: gitpyhon should not be included as runtime dependency Key: AIRFLOW-5021 URL: https://issues.apache.org/jira/browse/AIRFLOW-5021 Project: Apache Airflow Issue Type: Improvement Components: build Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou Fix For: 2.0.0 gitpython is included as `install_requires` but is only used in setup.py. It's better to move it to `setup_requires` so we can reduce package size. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-4346) Kubernetes Executor Fails for Large Wide DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-4346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875295#comment-16875295 ] QP Hou commented on AIRFLOW-4346: - [~vcastane] could you share the persistent volume claim config as well? > Kubernetes Executor Fails for Large Wide DAGs > - > > Key: AIRFLOW-4346 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4346 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, executors >Affects Versions: 1.10.2, 1.10.3 >Reporter: Vincent Castaneda >Priority: Blocker > Labels: kubernetes > Attachments: configmap-airflow-share.yaml, sched_logs.txt, > wide_dag_bash_test.py, wide_dag_test_100_300.py, wide_dag_test_300_300.py > > > When running large DAGs–those with parallelism of over 100 task instances to > be running concurrently--several tasks fail on the executor and are reported > to the database, but the scheduler is never aware of them failing. > Attached are: > - A test DAG that we can use to replicate the issue. > - The configmap-airflow.yaml file > I will be available to answer any other questions that are raised about our > configuration. We are running this on GKE and giving the scheduler and web > pod a base 100m for execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
