[jira] [Created] (AIRFLOW-6873) mypyd is failing again with new type errors

2020-02-21 Thread QP Hou (Jira)
QP Hou created AIRFLOW-6873:
---

 Summary: mypyd is failing again with new type errors
 Key: AIRFLOW-6873
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6873
 Project: Apache Airflow
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


New version of mypy is throwing new type errors, we should get that fixed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-6871) Tree view unusable for large DAGs

2020-02-21 Thread QP Hou (Jira)
QP Hou created AIRFLOW-6871:
---

 Summary: Tree view unusable for large DAGs
 Key: AIRFLOW-6871
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6871
 Project: Apache Airflow
  Issue Type: Improvement
  Components: webserver
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


By default airflow loads 25 runs in the tree view. For our main DAG, it's just 
not usable because all we get is a 5xx error when hitting that page. Manually 
overriding number of runs to 15 makes the page to load again, but it's very 
slow and takes more than 1 minute to render.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-6802) scheduler not honoring dag.max_active_run config

2020-02-13 Thread QP Hou (Jira)
QP Hou created AIRFLOW-6802:
---

 Summary: scheduler not honoring dag.max_active_run config
 Key: AIRFLOW-6802
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6802
 Project: Apache Airflow
  Issue Type: Bug
  Components: scheduler
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


commit 
https://github.com/apache/airflow/commit/50efda5c69c1ddfaa869b408540182fb19f1a286
 introduced a bug that prevents the scheduler from enforcing max active run 
config for all DAGs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-6779) Schedule stopped scheduling tasks with concurrency limit or depends on past

2020-02-11 Thread QP Hou (Jira)
QP Hou created AIRFLOW-6779:
---

 Summary: Schedule stopped scheduling tasks with concurrency limit 
or depends on past
 Key: AIRFLOW-6779
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6779
 Project: Apache Airflow
  Issue Type: Bug
  Components: scheduler
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


https://github.com/apache/airflow/pull/4751 introduced a bug that stops the 
scheduler from scheduling tasks with the following properties:

* depends on past
* has custom concurrency limit



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-6613) center dag on graph view load

2020-01-21 Thread QP Hou (Jira)
QP Hou created AIRFLOW-6613:
---

 Summary: center dag on graph view load
 Key: AIRFLOW-6613
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6613
 Project: Apache Airflow
  Issue Type: New Feature
  Components: ui
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


Centering DAG on center of the canvas with the right zoom level will make large 
DAG much easier to navigate form the UI.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-6605) code doesn't pass static check from latest mypy (0.761)

2020-01-20 Thread QP Hou (Jira)
QP Hou created AIRFLOW-6605:
---

 Summary: code doesn't pass static check from latest mypy (0.761)
 Key: AIRFLOW-6605
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6605
 Project: Apache Airflow
  Issue Type: New Feature
  Components: build
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


Warnings from latest mypy:

{code:text}
tests/models/test_taskinstance.py:52: error: Need type annotation for 'task_id' 
 [var-annotated]
task_id = None
^
tests/models/test_taskinstance.py:53: error: Need type annotation for 'dag_id'  
[var-annotated]
dag_id = None
^
tests/models/test_taskinstance.py:54: error: Need type annotation for 
'execution_date'  [var-annotated]
execution_date = None
^
tests/models/test_taskinstance.py:55: error: Need type annotation for 
'task_state_in_callback'  [var-annotated]
task_state_in_callback = None
^
Found 4 errors in 1 file (checked 1500 source files)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (AIRFLOW-6576) Deleting a task with SLA crashes the scheduler

2020-01-15 Thread QP Hou (Jira)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

QP Hou updated AIRFLOW-6576:

Description: When a task with SLA is deleted from a DAG after the SLA miss 
is logged but before the notification was sent, scheduler will crash with an 
AirflowException  (was: When a task with SLA is deleted from a DAG after the 
SLA miss is logged by before the notification was sent, scheduler will crash 
with an AirflowException)

> Deleting a task with SLA crashes the scheduler
> --
>
> Key: AIRFLOW-6576
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6576
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: scheduler
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Major
>
> When a task with SLA is deleted from a DAG after the SLA miss is logged but 
> before the notification was sent, scheduler will crash with an 
> AirflowException



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-6576) Deleting a task with SLA crashes the scheduler

2020-01-15 Thread QP Hou (Jira)
QP Hou created AIRFLOW-6576:
---

 Summary: Deleting a task with SLA crashes the scheduler
 Key: AIRFLOW-6576
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6576
 Project: Apache Airflow
  Issue Type: New Feature
  Components: scheduler
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


When a task with SLA is deleted from a DAG after the SLA miss is logged by 
before the notification was sent, scheduler will crash with an AirflowException



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-6262) Add execute callback for operators

2019-12-16 Thread QP Hou (Jira)
QP Hou created AIRFLOW-6262:
---

 Summary: Add execute callback for operators
 Key: AIRFLOW-6262
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6262
 Project: Apache Airflow
  Issue Type: New Feature
  Components: operators
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


We are in the process of adding tracing events for various stages of task runs. 
Adding an execute callback to operator will help us emit task execution events 
to our monitoring system.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-6080) New version of mypy catches more type errors

2019-11-26 Thread QP Hou (Jira)
QP Hou created AIRFLOW-6080:
---

 Summary: New version of mypy catches more type errors
 Key: AIRFLOW-6080
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6080
 Project: Apache Airflow
  Issue Type: Improvement
  Components: ci
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


New version of mypy is reporting new list of errors. We should fix them so we 
can upgrade it in our CI/CD pipeline.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-5902) local_task_job is sleeping more often than it needs and spamming task log

2019-11-11 Thread QP Hou (Jira)
QP Hou created AIRFLOW-5902:
---

 Summary: local_task_job is sleeping more often than it needs and 
spamming task log
 Key: AIRFLOW-5902
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5902
 Project: Apache Airflow
  Issue Type: Bug
  Components: core
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


local_task_log's _execute loop is calling extra unnecessary sleep after a 
heartbeat to keep up with the heart rate.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-5900) heartbeat_callback for local_task_job can be more efficient

2019-11-11 Thread QP Hou (Jira)
QP Hou created AIRFLOW-5900:
---

 Summary: heartbeat_callback for local_task_job can be more 
efficient
 Key: AIRFLOW-5900
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5900
 Project: Apache Airflow
  Issue Type: Improvement
  Components: core
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


local_task_job's heartbeat_callback function is issuing unnecessary system 
calls when they are not needed. We can move those calls into the if block to 
make it more efficient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-5899) dmypy still reports error

2019-11-11 Thread QP Hou (Jira)
QP Hou created AIRFLOW-5899:
---

 Summary: dmypy still reports error
 Key: AIRFLOW-5899
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5899
 Project: Apache Airflow
  Issue Type: Improvement
  Components: ci
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


dmypy is still reporting error, would be nice to get those fixed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-5898) alembic command crashes due to typing import path collision

2019-11-11 Thread QP Hou (Jira)
QP Hou created AIRFLOW-5898:
---

 Summary: alembic command crashes due to typing import path 
collision
 Key: AIRFLOW-5898
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5898
 Project: Apache Airflow
  Issue Type: New Feature
  Components: database
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


Running alemibc within airflow directory results in the following crash:

 
{code:java}
  File 
"/Users/qph/Envs/airflow/lib/python3.7/site-packages/typing_extensions.py", 
line 17, in 
from typing import Generic, Callable, TypeVar, Tuple
ImportError: cannot import name 'Generic' from 'typing'
{code}
This is due to typing.py within airlfow directory collides with the builtin 
typing module when a command runs inside airflow directory. Renaming it to 
`custom_typing.py` fixes the issue.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-5832) pagerduty hook

2019-10-31 Thread QP Hou (Jira)
QP Hou created AIRFLOW-5832:
---

 Summary: pagerduty hook
 Key: AIRFLOW-5832
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5832
 Project: Apache Airflow
  Issue Type: New Feature
  Components: hooks
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


With pagerduty hook, users will be able to use pagerduty to notify critical DAG 
run issues.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-5811) Add metric for externally killed task count

2019-10-29 Thread QP Hou (Jira)
QP Hou created AIRFLOW-5811:
---

 Summary: Add metric for externally killed task count
 Key: AIRFLOW-5811
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5811
 Project: Apache Airflow
  Issue Type: Improvement
  Components: scheduler
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


Number of externally killed task would be very useful for monitoring init 
container error when using k8s executor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-5662) Scheduler incorrectly emits occupied slots count metric as used slot count

2019-10-15 Thread QP Hou (Jira)
QP Hou created AIRFLOW-5662:
---

 Summary: Scheduler incorrectly emits occupied slots count metric 
as used slot count
 Key: AIRFLOW-5662
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5662
 Project: Apache Airflow
  Issue Type: Bug
  Components: scheduler
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


Value of pool.used_slots metric is actually occupied slot count. There is also 
a redundant SQL query to the database that can be removed in that loop.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (AIRFLOW-5641) Not able to run git sync init container as root

2019-10-11 Thread QP Hou (Jira)
QP Hou created AIRFLOW-5641:
---

 Summary: Not able to run git sync init container as root
 Key: AIRFLOW-5641
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5641
 Project: Apache Airflow
  Issue Type: Bug
  Components: executor-kubernetes
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


We used to be able to run git sync init container as root user. A recent 
refactoring of the code base (AIRFLOW-4851) introduced a regression that makes 
it impossible to run git sync container as root. As a result, we are not able 
to start any task using kubernetes executor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Assigned] (AIRFLOW-5276) Remove unused helpers from airflow.utils.helpers

2019-08-21 Thread QP Hou (Jira)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

QP Hou reassigned AIRFLOW-5276:
---

Assignee: QP Hou

> Remove unused helpers from airflow.utils.helpers
> 
>
> Key: AIRFLOW-5276
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5276
> Project: Apache Airflow
>  Issue Type: Task
>  Components: core
>Affects Versions: 1.10.4
>Reporter: Ash Berlin-Taylor
>Assignee: QP Hou
>Priority: Trivial
>
> At the very least {{is_in}} is not used in the code base any more, and it's 
> behaviour has changed subtly in Python 3.7 so the tests fail there too.
> So lets remove that helper, and it's associated tests, plus anything else not 
> used any more.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Created] (AIRFLOW-5251) crash due to missing typing-extension dependency for python 3.7

2019-08-18 Thread QP Hou (JIRA)
QP Hou created AIRFLOW-5251:
---

 Summary: crash due to missing typing-extension dependency for 
python 3.7
 Key: AIRFLOW-5251
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5251
 Project: Apache Airflow
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


Latest master now crashes with python 3.7 due to missing typing-extension 
dependency.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (AIRFLOW-5250) dmypy still reports error for gcp hooks

2019-08-18 Thread QP Hou (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

QP Hou updated AIRFLOW-5250:

Description: 
dmypy still reports the following error:
{code:java}
airflow/contrib/hooks/gcp_cloud_build_hook.py:51: error: Need type annotation 
for '_conn'
airflow/contrib/hooks/gcp_compute_hook.py:52: error: Need type annotation for 
'_conn'
airflow/contrib/hooks/gcp_sql_hook.py:739: error: Need type annotation for 
'_conn'
airflow/gcp/hooks/functions.py:43: error: Need type annotation for '_conn'{code}

> dmypy still reports error for gcp hooks
> ---
>
> Key: AIRFLOW-5250
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5250
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Trivial
>
> dmypy still reports the following error:
> {code:java}
> airflow/contrib/hooks/gcp_cloud_build_hook.py:51: error: Need type annotation 
> for '_conn'
> airflow/contrib/hooks/gcp_compute_hook.py:52: error: Need type annotation for 
> '_conn'
> airflow/contrib/hooks/gcp_sql_hook.py:739: error: Need type annotation for 
> '_conn'
> airflow/gcp/hooks/functions.py:43: error: Need type annotation for 
> '_conn'{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (AIRFLOW-5250) dmypy still reports error for gcp hooks

2019-08-18 Thread QP Hou (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

QP Hou updated AIRFLOW-5250:

Summary: dmypy still reports error for gcp hooks  (was: fix missing dmypy 
error for gcp hooks)

> dmypy still reports error for gcp hooks
> ---
>
> Key: AIRFLOW-5250
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5250
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (AIRFLOW-5250) fix missing dmypy error for gcp hooks

2019-08-18 Thread QP Hou (JIRA)
QP Hou created AIRFLOW-5250:
---

 Summary: fix missing dmypy error for gcp hooks
 Key: AIRFLOW-5250
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5250
 Project: Apache Airflow
  Issue Type: Bug
  Components: ci
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (AIRFLOW-5140) missing type annotation errors reported by dmypy

2019-08-07 Thread QP Hou (JIRA)
QP Hou created AIRFLOW-5140:
---

 Summary: missing type annotation errors reported by dmypy
 Key: AIRFLOW-5140
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5140
 Project: Apache Airflow
  Issue Type: Bug
  Components: ci
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


dmypy is reporting the following error:

 
{code:bash}
$ dmypy run -- --follow-imports=error airflow tests
tests/core.py:2188: error: Need type annotation for 'HDFSHook'
tests/core.py:2189: error: Need type annotation for 'snakebite'
airflow/__init__.py:42: error: Need type annotation for 'login'
airflow/api/auth/backend/default.py:22: error: Need type annotation for 
'CLIENT_AUTH'
airflow/api/auth/backend/deny_all.py:23: error: Need type annotation for 
'CLIENT_AUTH'
airflow/contrib/example_dags/example_gcp_speech.py:60: error: Need type 
annotation for 'SOURCE_LANGUAGE'
airflow/contrib/hooks/gcp_cloud_build_hook.py:50: error: Need type annotation 
for '_conn'
airflow/contrib/hooks/gcp_compute_hook.py:50: error: Need type annotation for 
'_conn'
airflow/contrib/hooks/gcp_function_hook.py:41: error: Need type annotation for 
'_conn'
airflow/contrib/hooks/gcp_sql_hook.py:722: error: Need type annotation for 
'_conn'
airflow/contrib/hooks/gcp_translate_hook.py:35: error: Need type annotation for 
'_client'
airflow/contrib/hooks/gcs_hook.py:41: error: Need type annotation for '_conn'
airflow/contrib/operators/awsbatch_operator.py:62: error: Need type annotation 
for 'client'
airflow/contrib/operators/awsbatch_operator.py:63: error: Need type annotation 
for 'arn'
airflow/contrib/operators/ecs_operator.py:61: error: Need type annotation for 
'client'
airflow/contrib/operators/ecs_operator.py:62: error: Need type annotation for 
'arn'
airflow/contrib/operators/gcp_function_operator.py:229: error: Need type 
annotation for 'upload_function'
airflow/executors/__init__.py:28: error: Need type annotation for 
'DEFAULT_EXECUTOR'
airflow/hooks/dbapi_hook.py:41: error: Need type annotation for 'connector'
airflow/models/crypto.py:49: error: Need type annotation for '_fernet'
airflow/security/kerberos.py:25: error: Need type annotation for 
'NEED_KRB181_WORKAROUND'
airflow/settings.py:66: error: Need type annotation for 'SQL_ALCHEMY_CONN'
airflow/settings.py:67: error: Need type annotation for 'DAGS_FOLDER'
airflow/settings.py:68: error: Need type annotation for 'PLUGINS_FOLDER'
airflow/settings.py:69: error: Need type annotation for 'LOGGING_CLASS_PATH'
airflow/settings.py:71: error: Need type annotation for 'engine'
airflow/settings.py:72: error: Need type annotation for 'Session'
airflow/settings.py:312: error: Need type annotation for 'CONTEXT_MANAGER_DAG'
airflow/utils/state.py:28: error: Need type annotation for 'NONE'
airflow/www/app.py:39: error: Need type annotation for 'appbuilder'
tests/contrib/hooks/test_databricks_hook.py:72: error: Need type annotation for 
'RESULT_STATE'
tests/contrib/hooks/test_gcp_cloud_build_hook.py:49: error: Need type 
annotation for 'hook'
tests/contrib/hooks/test_gcp_cloud_build_hook.py:122: error: Need type 
annotation for 'hook'
tests/contrib/hooks/test_gcp_cloud_build_hook.py:191: error: Need type 
annotation for 'hook'
tests/contrib/utils/gcp_authenticator.py:57: error: Need type annotation for 
'original_account'
tests/test_utils/reset_warning_registry.py:45: error: Need type annotation for 
'_pattern'
tests/test_utils/reset_warning_registry.py:48: error: Need type annotation for 
'_backup'
{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (AIRFLOW-5117) Airflow K8S not working with EKS when not deployed in cluster

2019-08-05 Thread QP Hou (JIRA)
QP Hou created AIRFLOW-5117:
---

 Summary: Airflow K8S not working with EKS when not deployed in 
cluster
 Key: AIRFLOW-5117
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5117
 Project: Apache Airflow
  Issue Type: Bug
  Components: executors, hooks
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


When Airflow scheduler or worker is not deployed in K8S cluster, all K8S API 
call will fail with 401 error after running for about 14min.

This is due to EKS enforces a 15min timeout for all API tokens but the official 
k8S python client doesn't support token refresh.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Assigned] (AIRFLOW-4316) Keys/Variable names in the 'kubernetes_environment_variables' section are transformed to lower case

2019-08-02 Thread QP Hou (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

QP Hou reassigned AIRFLOW-4316:
---

Assignee: QP Hou

> Keys/Variable names in the 'kubernetes_environment_variables' section are 
> transformed to lower case
> ---
>
> Key: AIRFLOW-4316
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4316
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration, contrib
>Affects Versions: 1.10.3
>Reporter: Vlad Precup
>Assignee: QP Hou
>Priority: Major
>  Labels: configuration, kubernetes
> Fix For: 1.10.4
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> All the environment variable names declared under the 
> kubernetes_environment_variables section are converted to lowercase in the 
> newly generated pods when running Airflow with the KubernetesExecutor in 
> version 1.10.3.
> The behavior is not aligned with the comments in the documentation / code 
> comments in the default config file, airflow.cfg.
> Investigation helper:
> Since we are treating all the sections uniformly in the configuration 
> component, all the keys are lowered (i.e. `configuration.py`, line 342 ->`key 
> = env_var.replace(section_prefix, '').lower()`)
> Please also write (a) unit test(s) for this special case.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (AIRFLOW-5055) kubernetes_environment_variables section cannot be set using `AIRFLOW_` prefixed environment variables

2019-07-26 Thread QP Hou (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

QP Hou updated AIRFLOW-5055:

Description: When you set environment variables like 
`AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLESAIRFLOW__HOME`, the scheduler 
will pass `airflow_home` as environment variable key to workers instead, which 
gets ignored by the workers.  (was: When you set environment variables like 
`AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__HOME`, the scheduler will 
pass `airflow_home` as environment variable key to workers instead, which gets 
ignored by the workers.)

> kubernetes_environment_variables section cannot be set using `AIRFLOW_` 
> prefixed environment variables
> --
>
> Key: AIRFLOW-5055
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5055
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Minor
>
> When you set environment variables like 
> `AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLESAIRFLOW__HOME`, the scheduler 
> will pass `airflow_home` as environment variable key to workers instead, 
> which gets ignored by the workers.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (AIRFLOW-5055) kubernetes_environment_variables section cannot be set using `AIRFLOW_` prefixed environment variables

2019-07-26 Thread QP Hou (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

QP Hou updated AIRFLOW-5055:

Summary: kubernetes_environment_variables section cannot be set using 
`AIRFLOW_` prefixed environment variables  (was: 
kubernetes_environment_variables section cannot be set using `AIRFLOW_` 
environment variables)

> kubernetes_environment_variables section cannot be set using `AIRFLOW_` 
> prefixed environment variables
> --
>
> Key: AIRFLOW-5055
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5055
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Minor
>
> When you set environment variables like 
> `AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__HOME`, the scheduler 
> will pass `airflow_home` as environment variable key to workers instead, 
> which gets ignored by the workers.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (AIRFLOW-5055) kubernetes_environment_variables section cannot be set using `AIRFLOW_` environment variables

2019-07-26 Thread QP Hou (JIRA)
QP Hou created AIRFLOW-5055:
---

 Summary: kubernetes_environment_variables section cannot be set 
using `AIRFLOW_` environment variables
 Key: AIRFLOW-5055
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5055
 Project: Apache Airflow
  Issue Type: Bug
  Components: configuration
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


When you set environment variables like 
`AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__HOME`, the scheduler will 
pass `airflow_home` as environment variable key to workers instead, which gets 
ignored by the workers.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (AIRFLOW-5038) Pod deleted message is shown incorrectly when pod deletion is disabled

2019-07-24 Thread QP Hou (JIRA)
QP Hou created AIRFLOW-5038:
---

 Summary: Pod deleted message is shown incorrectly when pod 
deletion is disabled
 Key: AIRFLOW-5038
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5038
 Project: Apache Airflow
  Issue Type: Bug
  Components: scheduler
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


When `delete_worker_pods` is set to False, scheduler still logs this line: 
`self.log.info('Deleted pod: %s', str(key))`, which is incorrect and confusing.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Closed] (AIRFLOW-5032) build is broken due to version conflict between tzlocal and pendulum

2019-07-23 Thread QP Hou (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

QP Hou closed AIRFLOW-5032.
---
Resolution: Duplicate

> build is broken due to version conflict between tzlocal and pendulum
> 
>
> Key: AIRFLOW-5032
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5032
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: build
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Major
>
> Lastest master build is broken due to dependency version conflict, here is 
> the error: 
> {code:java}
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 4, in 
> __import__('pkg_resources').require('apache-airflow==2.0.0.dev0')
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
> 3241, in 
> @_call_aside
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
> 3225, in _call_aside
> f(*args, **kwargs)
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
> 3254, in _initialize_master_working_set
> working_set = WorkingSet._build_master()
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
> 585, in _build_master
> return cls._build_from_requirements(__requires__)
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
> 598, in _build_from_requirements
> dists = ws.resolve(reqs, Environment())
> File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
> 791, in resolve
> raise VersionConflict(dist, req).with_context(dependent_req)
> pkg_resources.ContextualVersionConflict: (tzlocal 2.0.0 
> (/usr/local/lib/python3.7/site-packages), 
> Requirement.parse('tzlocal<2.0.0.0,>=1.5.0.0'), {'pendulum'})
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (AIRFLOW-5032) build is broken due to version conflict between tzlocal and pendulum

2019-07-23 Thread QP Hou (JIRA)
QP Hou created AIRFLOW-5032:
---

 Summary: build is broken due to version conflict between tzlocal 
and pendulum
 Key: AIRFLOW-5032
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5032
 Project: Apache Airflow
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


Lastest master build is broken due to dependency version conflict, here is the 
error: 
{code:java}
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 4, in 
__import__('pkg_resources').require('apache-airflow==2.0.0.dev0')
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
3241, in 
@_call_aside
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
3225, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
3254, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
585, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
598, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 
791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (tzlocal 2.0.0 
(/usr/local/lib/python3.7/site-packages), 
Requirement.parse('tzlocal<2.0.0.0,>=1.5.0.0'), {'pendulum'})
{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (AIRFLOW-5030) config key name with __ in it is getting skipped in AirflowConfigParser.as_dict method

2019-07-23 Thread QP Hou (JIRA)
QP Hou created AIRFLOW-5030:
---

 Summary: config key name with __ in it is getting skipped in 
AirflowConfigParser.as_dict method
 Key: AIRFLOW-5030
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5030
 Project: Apache Airflow
  Issue Type: Bug
  Components: configuration
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou


When a key name contains one or more "__", it will get ignored by 
AirflowConfigParser.as_dict method. This makes it impossible to set worker 
environment variables using `KUBERNETES_ENVIRONMENT_VARIABLES` section from 
scheduler.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (AIRFLOW-5021) gitpyhon should not be included as runtime dependency

2019-07-22 Thread QP Hou (JIRA)
QP Hou created AIRFLOW-5021:
---

 Summary: gitpyhon should not be included as runtime dependency
 Key: AIRFLOW-5021
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5021
 Project: Apache Airflow
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.0
Reporter: QP Hou
Assignee: QP Hou
 Fix For: 2.0.0


gitpython is included as `install_requires` but is only used in setup.py. It's 
better to move it to `setup_requires` so we can reduce package size.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (AIRFLOW-4346) Kubernetes Executor Fails for Large Wide DAGs

2019-06-28 Thread QP Hou (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-4346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875295#comment-16875295
 ] 

QP Hou commented on AIRFLOW-4346:
-

[~vcastane] could you share the persistent volume claim config as well?

> Kubernetes Executor Fails for Large Wide DAGs
> -
>
> Key: AIRFLOW-4346
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4346
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, executors
>Affects Versions: 1.10.2, 1.10.3
>Reporter: Vincent Castaneda
>Priority: Blocker
>  Labels: kubernetes
> Attachments: configmap-airflow-share.yaml, sched_logs.txt, 
> wide_dag_bash_test.py, wide_dag_test_100_300.py, wide_dag_test_300_300.py
>
>
> When running large DAGs–those with parallelism of over 100 task instances to 
> be running concurrently--several tasks fail on the executor and are reported 
> to the database, but the scheduler is never aware of them failing.
> Attached are:
>  - A test DAG that we can use to replicate the issue.
>  - The configmap-airflow.yaml file
> I will be available to answer any other questions that are raised about our 
> configuration. We are running this on GKE and giving the scheduler and web 
> pod a base 100m for execution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)