[jira] [Created] (AIRFLOW-2512) GoogleCloudBaseHook using deprecated oauth2client

[Tim Swast (JIRA)]

Tim Swast created AIRFLOW-2512:
--

 Summary: GoogleCloudBaseHook using deprecated oauth2client
 Key: AIRFLOW-2512
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2512
 Project: Apache Airflow
  Issue Type: Improvement
  Components: contrib
Reporter: Tim Swast


oauth2client is deprecated in favor of the google-auth library. 
[https://google-auth.readthedocs.io/en/latest/oauth2client-deprecation.html]

The GoogleCloudBaseHook should be updated to use the google-auth library.

Note: to use the google-auth library with the google-api-python-client, the API 
client library may also require an update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-2385) Airflow task is not stopped when execution timeout gets triggered

[Yohei Onishi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yohei Onishi resolved AIRFLOW-2385.
---
Resolution: Fixed

> Airflow task is not stopped when execution timeout gets triggered
> -
>
> Key: AIRFLOW-2385
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2385
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.9.0
>Reporter: Yohei Onishi
>Priority: Major
>
> I have my own custom operator extends BaseOperator as follows. I tried to 
> kill a task if the task runs for more than 30 minutes. timeout seems to be 
> triggered according to a log but the task still continued.
> Am I missing something? I checked the official document but do not know what 
> is wrong.[https://airflow.apache.org/code.html#baseoperator]
> My operator is like as follows.
> {code:java}
> class MyOperator(BaseOperator):
>   @apply_defaults
>   def __init__(
> self,
> some_parameters_here,
> *args,
> **kwargs):
> super(MyOperator, self).__init__(*args, **kwargs)
> # some initialization here
>   def execute(self, context):
> # some code here
> {code}
>  
> {{}}My task is like as follows.
> {code:java}
> t = MyOperator(
>   task_id='task',
>   dag=scheduled_dag,
>   execution_timeout=timedelta(minutes=30)
> {code}
>  
> I found this error but the task continued.
> {code:java}
> [2018-04-12 03:30:28,353] {base_task_runner.py:98} INFO - Subtask: [Stage 
> 6:==(1380 + -160) / 
> 1224][2018-04- 12 03:30:28,353] {timeout.py:36} ERROR - Process timed out
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AIRFLOW-2385) Airflow task is not stopped when execution timeout gets triggered

[Yohei Onishi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yohei Onishi closed AIRFLOW-2385.
-

> Airflow task is not stopped when execution timeout gets triggered
> -
>
> Key: AIRFLOW-2385
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2385
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: 1.9.0
>Reporter: Yohei Onishi
>Priority: Major
>
> I have my own custom operator extends BaseOperator as follows. I tried to 
> kill a task if the task runs for more than 30 minutes. timeout seems to be 
> triggered according to a log but the task still continued.
> Am I missing something? I checked the official document but do not know what 
> is wrong.[https://airflow.apache.org/code.html#baseoperator]
> My operator is like as follows.
> {code:java}
> class MyOperator(BaseOperator):
>   @apply_defaults
>   def __init__(
> self,
> some_parameters_here,
> *args,
> **kwargs):
> super(MyOperator, self).__init__(*args, **kwargs)
> # some initialization here
>   def execute(self, context):
> # some code here
> {code}
>  
> {{}}My task is like as follows.
> {code:java}
> t = MyOperator(
>   task_id='task',
>   dag=scheduled_dag,
>   execution_timeout=timedelta(minutes=30)
> {code}
>  
> I found this error but the task continued.
> {code:java}
> [2018-04-12 03:30:28,353] {base_task_runner.py:98} INFO - Subtask: [Stage 
> 6:==(1380 + -160) / 
> 1224][2018-04- 12 03:30:28,353] {timeout.py:36} ERROR - Process timed out
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2511) Subdag failed by scheduler deadlock

[Yohei Onishi (JIRA)]

Yohei Onishi created AIRFLOW-2511:
-

 Summary: Subdag failed by scheduler deadlock
 Key: AIRFLOW-2511
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2511
 Project: Apache Airflow
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Yohei Onishi


I am using subdag and sometimes main dag marked failed because of the following 
error. In this case, tasks in the subdag stopped.
{code:java}
hourly_dag = DAG(
  hourly_dag_name,
  default_args=dag_default_args,
  params=dag_custom_params,
  schedule_interval=config_values.hourly_job_interval,
  max_active_runs=2)
hourly_subdag = SubDagOperator(
  task_id='s3_to_hive',
  subdag=LoadFromS3ToHive(
  hourly_dag,
  's3_to_hive'),
  dag=hourly_dag)
{code}
I got this error in main dag. bug in scheduler?
{code:java}
[2018-05-22 21:52:19,683] {models.py:1595} ERROR - This Session's transaction 
has been rolled back due to a previous exception during flush. To begin a new 
transaction with this Session, first issue Session.rollback(). Original 
exception was: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when 
trying to get lock; try restarting transaction') [SQL: 'UPDATE task_instance 
SET state=%s WHERE task_instance.task_id = %s AND task_instance.dag_id = %s AND 
task_instance.execution_date = %s'] [parameters: ('queued', 
'transfer_from_tmp_table_into_cleaned_table', 
'rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h.s3_to_hive', 
datetime.datetime(2018, 5, 7, 5, 2))] (Background on this error at: 
http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled 
back due to a previous exception during flush. To begin a new transaction with 
this Session, first issue Session.rollback(). Original exception was: 
(_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get 
lock; try restarting transaction') [SQL: 'UPDATE task_instance SET state=%s 
WHERE task_instance.task_id = %s AND task_instance.dag_id = %s AND 
task_instance.execution_date = %s'] [parameters: ('queued', 
'transfer_from_tmp_table_into_cleaned_table', 
'rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h.s3_to_hive', 
datetime.datetime(2018, 5, 7, 5, 2))] (Background on this error at: 
http://sqlalche.me/e/e3q8)
[2018-05-22 21:52:19,687] {models.py:1624} INFO - Marking task as FAILED.
[2018-05-22 21:52:19,688] {base_task_runner.py:98} INFO - Subtask: [2018-05-22 
21:52:19,688] {slack_hook.py:143} INFO - Message is prepared: 
[2018-05-22 21:52:19,688] {base_task_runner.py:98} INFO - Subtask: 
{"attachments": [{"color": "danger", "text": "", "fields": [{"title": "DAG", 
"value": 
"",
 "short": true}, {"title": "Owner", "value": "airflow", "short": true}, 
{"title": "Task", "value": "s3_to_hive", "short": false}, {"title": "Status", 
"value": "FAILED", "short": false}, {"title": "Execution Time", "value": 
"2018-05-07T05:02:00", "short": true}, {"title": "Duration", "value": 
"826.305929", "short": true}, {"value": 
"", "short": false}]}]}
[2018-05-22 21:52:19,688] {models.py:1638} ERROR - Failed at executing callback
[2018-05-22 21:52:19,688] {models.py:1639} ERROR - This Session's transaction 
has been rolled back due to a previous exception during flush. To begin a new 
transaction with this Session, first issue Session.rollback(). Original 
exception was: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when 
trying to get lock; try restarting transaction') [SQL: 'UPDATE task_instance 
SET state=%s WHERE task_instance.task_id = %s AND task_instance.dag_id = %s AND 
task_instance.execution_date = %s'] [parameters: ('queued', 
'transfer_from_tmp_table_into_cleaned_table', 
'rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h.s3_to_hive', 
datetime.datetime(2018, 5, 7, 5, 2))] (Background on this error at: 
http://sqlalche.me/e/e3q8)

{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-2429) Make Airflow Flake8 compliant

[Fokko Driesprong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AIRFLOW-2429.
---
   Resolution: Fixed
Fix Version/s: 2.0.0
   1.10.0

Issue resolved by pull request #3401
[https://github.com/apache/incubator-airflow/pull/3401]

> Make Airflow Flake8 compliant 
> --
>
> Key: AIRFLOW-2429
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2429
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Tao Feng
>Priority: Major
> Fix For: 1.10.0, 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2429) Make Airflow Flake8 compliant

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484671#comment-16484671
 ] 

ASF subversion and git services commented on AIRFLOW-2429:
--

Commit d52e9e6642d38fbaa4d4b7ffe49c62daa5a18ff8 in incubator-airflow's branch 
refs/heads/master from Tao feng
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d52e9e6 ]

[AIRFLOW-2429] Add BaseExecutor back

Closes #3401 from feng-tao/quick_fix_airflow_2429


> Make Airflow Flake8 compliant 
> --
>
> Key: AIRFLOW-2429
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2429
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Tao Feng
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-2429] Add BaseExecutor back

[fokko]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 272952a9d -> d52e9e664


[AIRFLOW-2429] Add BaseExecutor back

Closes #3401 from feng-tao/quick_fix_airflow_2429


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d52e9e66
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d52e9e66
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d52e9e66

Branch: refs/heads/master
Commit: d52e9e6642d38fbaa4d4b7ffe49c62daa5a18ff8
Parents: 272952a
Author: Tao feng 
Authored: Wed May 23 00:14:44 2018 +0200
Committer: Fokko Driesprong 
Committed: Wed May 23 00:14:44 2018 +0200

--
 airflow/executors/__init__.py | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/d52e9e66/airflow/executors/__init__.py
--
diff --git a/airflow/executors/__init__.py b/airflow/executors/__init__.py
index b91a24e..c33bbb6 100644
--- a/airflow/executors/__init__.py
+++ b/airflow/executors/__init__.py
@@ -21,6 +21,7 @@ import sys
 from airflow.utils.log.logging_mixin import LoggingMixin
 from airflow import configuration
 from airflow.exceptions import AirflowException
+from airflow.executors.base_executor import BaseExecutor # noqa
 from airflow.executors.local_executor import LocalExecutor
 from airflow.executors.sequential_executor import SequentialExecutor
 

[jira] [Comment Edited] (AIRFLOW-2510) Introduce new macros: prev_ds and next_ds

[Chao-Han Tsai (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484559#comment-16484559
 ] 

Chao-Han Tsai edited comment on AIRFLOW-2510 at 5/22/18 8:57 PM:
-

[~artwr]
Let me clarify a bit:
prev_ds = ds - schedule_interval in -MM-DD format
next_ds = ds + schedule_interval in -MM-DD format

whereas 
yesterday_ds = ds-1
tomorrow_ds = ds+1



was (Author: milton0825):
Let me clarify a bit:
prev_ds = ds - schedule_interval in -MM-DD format
next_ds = ds + schedule_interval in -MM-DD format

whereas 
yesterday_ds = ds-1
tomorrow_ds = ds+1


> Introduce new macros: prev_ds and next_ds
> -
>
> Key: AIRFLOW-2510
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2510
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Introduce new macros {{ prev_ds }} and {{ next_ds }}.
> {{ prev_ds }}: the previous execution date as {{ -MM-DD }}
> {{ next_ds }}: the next execution date as {{ -MM-DD }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2510) Introduce new macros: prev_ds and next_ds

[Arthur Wiedmer (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484511#comment-16484511
 ] 

Arthur Wiedmer commented on AIRFLOW-2510:
-

Have you tried using yesterday_ds and tomorrow_ds ?

 

https://github.com/apache/incubator-airflow/blob/1f0a717b65e0ea7e0127708b084baff0697f0946/airflow/models.py#L1755

> Introduce new macros: prev_ds and next_ds
> -
>
> Key: AIRFLOW-2510
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2510
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Introduce new macros {{ prev_ds }} and {{ next_ds }}.
> {{ prev_ds }}: the previous execution date as {{ -MM-DD }}
> {{ next_ds }}: the next execution date as {{ -MM-DD }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2510) Introduce new macros: prev_ds and next_ds

[Chao-Han Tsai (JIRA)]

Chao-Han Tsai created AIRFLOW-2510:
--

 Summary: Introduce new macros: prev_ds and next_ds
 Key: AIRFLOW-2510
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2510
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Chao-Han Tsai
Assignee: Chao-Han Tsai


Introduce new macros {{ prev_ds }} and {{ next_ds }}.

{{ prev_ds }}: the previous execution date as {{ -MM-DD }}
{{ next_ds }}: the next execution date as {{ -MM-DD }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-2509) Separate Configuration page into separate how-to guides

[Tim Swast (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Swast reassigned AIRFLOW-2509:
--

Assignee: Tim Swast

> Separate Configuration page into separate how-to guides
> ---
>
> Key: AIRFLOW-2509
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2509
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Tim Swast
>Assignee: Tim Swast
>Priority: Critical
>
> The existing "Configuration" page is attempting to be both a tutorial 
> (teaching the basics of Airflow configuration & a minimal production 
> deployment) as well as how-tos for specific tasks.
>  
> I propose we separate Configuration into separate how-to guides, keeping the 
> current sequence so that it can still maintain the tutorial properties (at 
> least until a "Deploying a Production Airflow Environment" tutorial is 
> written).
>  
> There's a principle that the [distinct kinds of 
> documentation|http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/]
>  should be organized separately. The Django project does this 
> [https://docs.djangoproject.com/en/2.0/] by splitting into
>  
>  * Tutorials
>  * Topic guides (what Airflow calls Concepts)
>  * Reference guides
>  * How-to guides
> I think the same could apply well here. (This issue covers only How-to for 
> Configuration. More work would be required to separate other docs into proper 
> document types.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2509) Separate Configuration page into separate how-to guides

[Tim Swast (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Swast updated AIRFLOW-2509:
---
External issue URL: https://github.com/apache/incubator-airflow/pull/3400

> Separate Configuration page into separate how-to guides
> ---
>
> Key: AIRFLOW-2509
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2509
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Tim Swast
>Priority: Critical
>
> The existing "Configuration" page is attempting to be both a tutorial 
> (teaching the basics of Airflow configuration & a minimal production 
> deployment) as well as how-tos for specific tasks.
>  
> I propose we separate Configuration into separate how-to guides, keeping the 
> current sequence so that it can still maintain the tutorial properties (at 
> least until a "Deploying a Production Airflow Environment" tutorial is 
> written).
>  
> There's a principle that the [distinct kinds of 
> documentation|http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/]
>  should be organized separately. The Django project does this 
> [https://docs.djangoproject.com/en/2.0/] by splitting into
>  
>  * Tutorials
>  * Topic guides (what Airflow calls Concepts)
>  * Reference guides
>  * How-to guides
> I think the same could apply well here. (This issue covers only How-to for 
> Configuration. More work would be required to separate other docs into proper 
> document types.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (AIRFLOW-2509) Separate Configuration page into separate how-to guides

[Tim Swast (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-2509 started by Tim Swast.
--
> Separate Configuration page into separate how-to guides
> ---
>
> Key: AIRFLOW-2509
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2509
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Tim Swast
>Assignee: Tim Swast
>Priority: Critical
>
> The existing "Configuration" page is attempting to be both a tutorial 
> (teaching the basics of Airflow configuration & a minimal production 
> deployment) as well as how-tos for specific tasks.
>  
> I propose we separate Configuration into separate how-to guides, keeping the 
> current sequence so that it can still maintain the tutorial properties (at 
> least until a "Deploying a Production Airflow Environment" tutorial is 
> written).
>  
> There's a principle that the [distinct kinds of 
> documentation|http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/]
>  should be organized separately. The Django project does this 
> [https://docs.djangoproject.com/en/2.0/] by splitting into
>  
>  * Tutorials
>  * Topic guides (what Airflow calls Concepts)
>  * Reference guides
>  * How-to guides
> I think the same could apply well here. (This issue covers only How-to for 
> Configuration. More work would be required to separate other docs into proper 
> document types.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2509) Separate Configuration page into separate how-to guides

[Tim Swast (JIRA)]

Tim Swast created AIRFLOW-2509:
--

 Summary: Separate Configuration page into separate how-to guides
 Key: AIRFLOW-2509
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2509
 Project: Apache Airflow
  Issue Type: Improvement
  Components: Documentation
Reporter: Tim Swast


The existing "Configuration" page is attempting to be both a tutorial (teaching 
the basics of Airflow configuration & a minimal production deployment) as well 
as how-tos for specific tasks.
 
I propose we separate Configuration into separate how-to guides, keeping the 
current sequence so that it can still maintain the tutorial properties (at 
least until a "Deploying a Production Airflow Environment" tutorial is written).
 
There's a principle that the [distinct kinds of 
documentation|http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/]
 should be organized separately. The Django project does this 
[https://docs.djangoproject.com/en/2.0/] by splitting into
 
 * Tutorials
 * Topic guides (what Airflow calls Concepts)
 * Reference guides
 * How-to guides

I think the same could apply well here. (This issue covers only How-to for 
Configuration. More work would be required to separate other docs into proper 
document types.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2508) Handle non string types in render_template_from_field

[Eugene Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Brown updated AIRFLOW-2508:
--
Description: 
The render_template_from_field method of the BaseOperator class raises an 
exception when it encounters content that is not a string_type, list, tuple or 
dict.

Example exception:
{noformat}
airflow.exceptions.AirflowException: Type '' used for parameter 
'job_flow_overrides[Instances][InstanceGroups][InstanceCount]' is not supported 
for templating{noformat}
I propose instead that when it encounters content of other types it returns the 
content unchanged, rather than raising an exception.

Consider this case: I extended the EmrCreateJobFlowOperator to make the 
job_flow_overrides argument a templatable field. job_flow_overrides is a 
dictionary with a mix of strings, integers and booleans for values.

When I extended the class as such:
{code:java}
class EmrCreateJobFlowOperatorTemplateOverrides(EmrCreateJobFlowOperator):
template_fields = ['job_flow_overrides']{code}
And added a task to my dag with this format:
{code:java}
step_create_cluster = EmrCreateJobFlowOperatorTemplateOverrides(
task_id="create_cluster",
job_flow_overrides={
"Name": "my-cluster {{ dag_run.conf['run_date'] }}",
"Instances": {
"InstanceGroups": [
{
"Name": "Master nodes",
"InstanceType": "c3.4xlarge",
"InstanceCount": 1
},
{
"Name": "Slave nodes",
"InstanceType": "c3.4xlarge",
"InstanceCount": 4
},
"TerminationProtected": False
]
},
"BootstrapActions": [{
 "Name": "Custom action",
 "ScriptBootstrapAction": {
 "Path": "s3://repo/{{ dag_run.conf['branch'] 
}}/requirements.txt"
 }
}],
   },
   aws_conn_id='aws_default',
   emr_conn_id='aws_default',
   dag=dag
)
{code}
The exception I gave above was raised and the step failed. I think it would be 
preferable for the method to instead pass over numeric and boolean values as 
users may want to use template_fields in the way I have to template string 
values in dictionaries or lists of mixed types.

Here is the render_template_from_field method from the BaseOperator:
{code:java}
def render_template_from_field(self, attr, content, context, jinja_env):
"""
Renders a template from a field. If the field is a string, it will
simply render the string and return the result. If it is a collection or
nested set of collections, it will traverse the structure and render
all strings in it.
"""
rt = self.render_template
if isinstance(content, six.string_types):
result = jinja_env.from_string(content).render(**context)
elif isinstance(content, (list, tuple)):
result = [rt(attr, e, context) for e in content]
elif isinstance(content, dict):
result = {
k: rt("{}[{}]".format(attr, k), v, context)
for k, v in list(content.items())}
else:
param_type = type(content)
msg = (
"Type '{param_type}' used for parameter '{attr}' is "
"not supported for templating").format(**locals())
raise AirflowException(msg)
return result{code}
 I propose that the method returns content unchanged if the content is of one 
of (int, float, complex, bool) types. So my solution would include an extra 
elif in the form:
{code}
elif isinstance(content, (int, float, complex, bool)):
result = content
{code}
 Are there any reasons this would be a bad idea?

  was:
The render_template_from_field method of the BaseOperator class raises an 
exception when it encounters content that is not a string_type, list, tuple or 
dict.

Example exception:
{noformat}
airflow.exceptions.AirflowException: Type '' used for parameter 
'job_flow_overrides[Instances][InstanceGroups][InstanceCount]' is not supported 
for templating{noformat}
I propose instead that when it encounters content of other types it returns the 
content unchanged, rather than raising an exception.

Consider this case: I extended the EmrCreateJobFlowOperator to make the 
job_flow_overrides argument a templatable field. job_flow_overrides is a 
dictionary with a mix of strings, integers and booleans for values.

When I extended the class as such:
{code:java}
class EmrCreateJobFlowOperatorTemplateOverrides(EmrCreateJobFlowOperator):
template_fields = ['job_flow_overrides']{code}
And added a task to my dag with this format:
{code:java}
step_create_cluster = EmrCreateJobFlowOperatorTemplateOverrides(
task_id="create_cluster",
job_flow_overrides={
"Name": "my-cluster {{ dag_run.conf['run_date'] }}",
"Instances": {
"InstanceGroups": [
{
 

[jira] [Created] (AIRFLOW-2508) Handle non string types in render_template_from_field

[Eugene Brown (JIRA)]

Eugene Brown created AIRFLOW-2508:
-

 Summary: Handle non string types in render_template_from_field
 Key: AIRFLOW-2508
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2508
 Project: Apache Airflow
  Issue Type: Bug
  Components: models
Affects Versions: Airflow 2.0
Reporter: Eugene Brown
Assignee: Eugene Brown


The render_template_from_field method of the BaseOperator class raises an 
exception when it encounters content that is not a string_type, list, tuple or 
dict.

Example exception:
{noformat}
airflow.exceptions.AirflowException: Type '' used for parameter 
'job_flow_overrides[Instances][InstanceGroups][InstanceCount]' is not supported 
for templating{noformat}
I propose instead that when it encounters content of other types it returns the 
content unchanged, rather than raising an exception.

Consider this case: I extended the EmrCreateJobFlowOperator to make the 
job_flow_overrides argument a templatable field. job_flow_overrides is a 
dictionary with a mix of strings, integers and booleans for values.

When I extended the class as such:
{code:java}
class EmrCreateJobFlowOperatorTemplateOverrides(EmrCreateJobFlowOperator):
template_fields = ['job_flow_overrides']{code}
And added a task to my dag with this format:
{code:java}
step_create_cluster = EmrCreateJobFlowOperatorTemplateOverrides(
task_id="create_cluster",
job_flow_overrides={
"Name": "my-cluster {{ dag_run.conf['run_date'] }}",
"Instances": {
"InstanceGroups": [
{
"Name": "Master nodes",
"InstanceType": "c3.4xlarge",
"InstanceCount": 1
},
{
"Name": "Slave nodes",
"InstanceType": "c3.4xlarge",
"InstanceCount": 4
},
"TerminationProtected": False
]
},
"BootstrapActions": [{
 "Name": "Custom action",
 "ScriptBootstrapAction": {
 "Path": "s3://repo/{{ dag_run.conf['branch'] 
}}/requirements.txt"
 }
}],
   },
   aws_conn_id='aws_default',
   emr_conn_id='aws_default',
   dag=dag
)
{code}
The exception I gave above was raised and the step failed. I think it would be 
preferable for the method to instead pass over numeric and boolean values as 
users may want to use template_fields in the way I have to template string 
values in dictionaries or lists of mixed types.

Here is the render_template_from_field method from the BaseOperator:
{code:java}
def render_template_from_field(self, attr, content, context, jinja_env):
"""
Renders a template from a field. If the field is a string, it will
simply render the string and return the result. If it is a collection or
nested set of collections, it will traverse the structure and render
all strings in it.
"""
rt = self.render_template
if isinstance(content, six.string_types):
result = jinja_env.from_string(content).render(**context)
elif isinstance(content, (list, tuple)):
result = [rt(attr, e, context) for e in content]
elif isinstance(content, dict):
result = {
k: rt("{}[{}]".format(attr, k), v, context)
for k, v in list(content.items())}
else:
param_type = type(content)
msg = (
"Type '{param_type}' used for parameter '{attr}' is "
"not supported for templating").format(**locals())
raise AirflowException(msg)
return result{code}
 I propose that the method returns content unchanged if the content is of one 
of (int, long, float, complex, bool) types. So my solution would include an 
extra elif in the form:
{code:python}
elif isinstance(content, (int, long, float, complex, bool)):
result = content
{code}
 Are there any reasons this would be a bad idea?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2507) No module named apiclient.discovery

[Omar Marzouk (JIRA)]

Omar Marzouk created AIRFLOW-2507:
-

 Summary: No module named apiclient.discovery
 Key: AIRFLOW-2507
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2507
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Omar Marzouk


Unable to import DataFlowPythonOperator:

 
{code:java}
webserver_1 | File "/usr/local/airflow/dags/train_classifier.py", line 4, in 

webserver_1 | from airflow.contrib.operators.dataflow_operator import 
DataFlowPythonOperator
webserver_1 | File 
"/usr/local/lib/python2.7/site-packages/airflow/contrib/operators/dataflow_operator.py",
 line 19, in 
webserver_1 | from airflow.contrib.hooks.gcs_hook import GoogleCloudStorageHook
webserver_1 | File 
"/usr/local/lib/python2.7/site-packages/airflow/contrib/hooks/gcs_hook.py", 
line 15, in 
webserver_1 | from apiclient.discovery import build
webserver_1 | ImportError: No module named apiclient.discovery
{code}
Environment:

python 2.7.15

project's requirements.txt:
{code:java}
sklearn
numpy
{code}
running using Dockerfile here: 
https://github.com/omarzouk/docker-airflow/blob/master/Dockerfile

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2429) Make Airflow Flake8 compliant

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484052#comment-16484052
 ] 

ASF subversion and git services commented on AIRFLOW-2429:
--

Commit 272952a9dce932cb2c648f82c9f9f2cafd572ff1 in incubator-airflow's branch 
refs/heads/master from Tao feng
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=272952a ]

[AIRFLOW-2429] Fix dag, example_dags, executors flake8 error

Closes #3398 from feng-tao/flake8_p3


> Make Airflow Flake8 compliant 
> --
>
> Key: AIRFLOW-2429
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2429
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Tao Feng
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-2429] Fix dag, example_dags, executors flake8 error

[kaxilnaik]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 1f0a717b6 -> 272952a9d


[AIRFLOW-2429] Fix dag, example_dags, executors flake8 error

Closes #3398 from feng-tao/flake8_p3


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/272952a9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/272952a9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/272952a9

Branch: refs/heads/master
Commit: 272952a9dce932cb2c648f82c9f9f2cafd572ff1
Parents: 1f0a717
Author: Tao feng 
Authored: Tue May 22 15:31:29 2018 +0100
Committer: Kaxil Naik 
Committed: Tue May 22 15:31:29 2018 +0100

--
 airflow/dag/__init__.py |  4 ++--
 airflow/dag/base_dag.py |  6 ++---
 airflow/example_dags/__init__.py|  5 ++---
 airflow/example_dags/docker_copy_data.py| 11 ++
 airflow/example_dags/entrypoint.sh  |  5 +++--
 airflow/example_dags/example_bash_operator.py   |  6 ++---
 airflow/example_dags/example_branch_operator.py |  4 ++--
 .../example_branch_python_dop_operator_3.py | 12 +-
 airflow/example_dags/example_docker_operator.py |  4 ++--
 airflow/example_dags/example_http_operator.py   |  4 ++--
 .../example_dags/example_kubernetes_executor.py | 23 
 .../example_dags/example_kubernetes_operator.py |  2 ++
 airflow/example_dags/example_latest_only.py |  5 ++---
 .../example_latest_only_with_trigger.py |  4 ++--
 .../example_passing_params_via_test_command.py  | 12 +-
 airflow/example_dags/example_python_operator.py |  2 ++
 .../example_short_circuit_operator.py   |  5 +++--
 airflow/example_dags/example_skip_dag.py|  4 ++--
 airflow/example_dags/example_subdag_operator.py |  5 +++--
 .../example_trigger_controller_dag.py   |  6 ++---
 .../example_dags/example_trigger_target_dag.py  | 13 +++
 airflow/example_dags/example_xcom.py|  4 ++--
 airflow/example_dags/subdags/__init__.py|  5 ++---
 airflow/example_dags/subdags/subdag.py  |  4 ++--
 airflow/example_dags/test_utils.py  |  5 ++---
 airflow/example_dags/tutorial.py|  4 ++--
 airflow/exceptions.py   |  4 ++--
 airflow/executors/__init__.py   | 17 ---
 airflow/executors/base_executor.py  |  3 +++
 airflow/executors/celery_executor.py|  8 +++
 airflow/executors/dask_executor.py  |  7 +++---
 airflow/executors/local_executor.py |  4 ++--
 airflow/executors/sequential_executor.py|  4 ++--
 33 files changed, 117 insertions(+), 94 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/272952a9/airflow/dag/__init__.py
--
diff --git a/airflow/dag/__init__.py b/airflow/dag/__init__.py
index 4067cc7..114d189 100644
--- a/airflow/dag/__init__.py
+++ b/airflow/dag/__init__.py
@@ -7,9 +7,9 @@
 # to you under the Apache License, Version 2.0 (the
 # "License"); you may not use this file except in compliance
 # with the License.  You may obtain a copy of the License at
-# 
+#
 #   http://www.apache.org/licenses/LICENSE-2.0
-# 
+#
 # Unless required by applicable law or agreed to in writing,
 # software distributed under the License is distributed on an
 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/272952a9/airflow/dag/base_dag.py
--
diff --git a/airflow/dag/base_dag.py b/airflow/dag/base_dag.py
index 43b2ec1..5719f57 100644
--- a/airflow/dag/base_dag.py
+++ b/airflow/dag/base_dag.py
@@ -7,16 +7,16 @@
 # to you under the Apache License, Version 2.0 (the
 # "License"); you may not use this file except in compliance
 # with the License.  You may obtain a copy of the License at
-# 
+#
 #   http://www.apache.org/licenses/LICENSE-2.0
-# 
+#
 # Unless required by applicable law or agreed to in writing,
 # software distributed under the License is distributed on an
 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-#
+
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/272952a9/airflow/example_dags/__init__.py
--
diff --git a/airflow/example_dags/__init__.py b/airflow/example_dags/__init__.py

[jira] [Commented] (AIRFLOW-2506) Dag Status evaluated incorrectly when last task is dummy operator and downstream of multiple tasks

[Ash Berlin-Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483917#comment-16483917
 ] 

Ash Berlin-Taylor commented on AIRFLOW-2506:


Can you provide a sample dag that illustrates this?

> Dag Status evaluated incorrectly when last task is dummy operator and 
> downstream of multiple tasks
> --
>
> Key: AIRFLOW-2506
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2506
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: models
>Affects Versions: Airflow 1.9.0
>Reporter: Debika Mukherjee
>Assignee: Debika Mukherjee
>Priority: Minor
> Fix For: Airflow 2.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2506) Dag Status evaluated incorrectly when last task is dummy operator and downstream of multiple tasks

[Debika Mukherjee (JIRA)]

Debika Mukherjee created AIRFLOW-2506:
-

 Summary: Dag Status evaluated incorrectly when last task is dummy 
operator and downstream of multiple tasks
 Key: AIRFLOW-2506
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2506
 Project: Apache Airflow
  Issue Type: Bug
  Components: models
Affects Versions: Airflow 1.9.0
Reporter: Debika Mukherjee
Assignee: Debika Mukherjee
 Fix For: Airflow 2.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend

[Bas Harenslak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bas Harenslak updated AIRFLOW-2505:
---
Attachment: (was: image-2018-05-22-08-57-35-362.png)

> Add state removed to the legend
> ---
>
> Key: AIRFLOW-2505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2505
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Bas Harenslak
>Priority: Minor
> Attachments: image-2018-05-22-09-09-04-754.png
>
>
> There's currently no legend item for removed tasks. I think we should add it.
> !image-2018-05-22-09-09-04-754.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend

[Bas Harenslak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bas Harenslak updated AIRFLOW-2505:
---
 Attachment: image-2018-05-22-09-09-04-754.png
Description: 
There's currently no legend item for removed tasks. I think we should add it.

!image-2018-05-22-09-09-04-754.png!

  was:
There's currently no legend item for removed tasks. I think we should add it.

!image-2018-05-22-08-57-35-362.png!


> Add state removed to the legend
> ---
>
> Key: AIRFLOW-2505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2505
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Bas Harenslak
>Priority: Minor
> Attachments: image-2018-05-22-09-09-04-754.png
>
>
> There's currently no legend item for removed tasks. I think we should add it.
> !image-2018-05-22-09-09-04-754.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend

[Bas Harenslak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bas Harenslak updated AIRFLOW-2505:
---
Attachment: (was: image-2018-05-22-08-57-05-525.png)

> Add state removed to the legend
> ---
>
> Key: AIRFLOW-2505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2505
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Bas Harenslak
>Priority: Minor
> Attachments: image-2018-05-22-08-57-35-362.png
>
>
> There's currently no legend item for removed tasks. I think we should add it.
> !image-2018-05-22-08-57-35-362.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend

[Bas Harenslak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bas Harenslak updated AIRFLOW-2505:
---
Attachment: (was: image-2018-05-22-08-54-48-137.png)

> Add state removed to the legend
> ---
>
> Key: AIRFLOW-2505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2505
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Bas Harenslak
>Priority: Minor
> Attachments: image-2018-05-22-08-57-05-525.png, 
> image-2018-05-22-08-57-35-362.png
>
>
> There's currently no legend item for removed tasks. I think we should add it.
> !image-2018-05-22-08-57-35-362.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend

[Bas Harenslak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bas Harenslak updated AIRFLOW-2505:
---
Description: 
There's currently no legend item for removed tasks. I think we should add it.

!image-2018-05-22-08-57-35-362.png!

  was:
There's currently no legend item for removed tasks. I think we should add it.

!image-2018-05-22-08-55-52-245.png!


> Add state removed to the legend
> ---
>
> Key: AIRFLOW-2505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2505
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Bas Harenslak
>Priority: Minor
> Attachments: image-2018-05-22-08-55-52-245.png, 
> image-2018-05-22-08-57-05-525.png, image-2018-05-22-08-57-35-362.png
>
>
> There's currently no legend item for removed tasks. I think we should add it.
> !image-2018-05-22-08-57-35-362.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend

[Bas Harenslak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bas Harenslak updated AIRFLOW-2505:
---
Attachment: image-2018-05-22-08-57-35-362.png

> Add state removed to the legend
> ---
>
> Key: AIRFLOW-2505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2505
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Bas Harenslak
>Priority: Minor
> Attachments: image-2018-05-22-08-55-52-245.png, 
> image-2018-05-22-08-57-05-525.png, image-2018-05-22-08-57-35-362.png
>
>
> There's currently no legend item for removed tasks. I think we should add it.
> !image-2018-05-22-08-55-52-245.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend

[Bas Harenslak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bas Harenslak updated AIRFLOW-2505:
---
Attachment: image-2018-05-22-08-57-05-525.png

> Add state removed to the legend
> ---
>
> Key: AIRFLOW-2505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2505
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Bas Harenslak
>Priority: Minor
> Attachments: image-2018-05-22-08-55-52-245.png, 
> image-2018-05-22-08-57-05-525.png, image-2018-05-22-08-57-35-362.png
>
>
> There's currently no legend item for removed tasks. I think we should add it.
> !image-2018-05-22-08-55-52-245.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend

[Bas Harenslak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bas Harenslak updated AIRFLOW-2505:
---
Attachment: (was: image-2018-05-22-08-55-52-245.png)

> Add state removed to the legend
> ---
>
> Key: AIRFLOW-2505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2505
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui
>Reporter: Bas Harenslak
>Priority: Minor
> Attachments: image-2018-05-22-08-57-05-525.png, 
> image-2018-05-22-08-57-35-362.png
>
>
> There's currently no legend item for removed tasks. I think we should add it.
> !image-2018-05-22-08-57-35-362.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2505) Add state removed to the legend

[Bas Harenslak (JIRA)]

Bas Harenslak created AIRFLOW-2505:
--

 Summary: Add state removed to the legend
 Key: AIRFLOW-2505
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2505
 Project: Apache Airflow
  Issue Type: Improvement
  Components: ui
Reporter: Bas Harenslak
 Attachments: image-2018-05-22-08-54-48-137.png, 
image-2018-05-22-08-55-52-245.png

There's currently no legend item for removed tasks. I think we should add it.

!image-2018-05-22-08-55-52-245.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AIRFLOW-2497) Cgroup task runner doesn't pass down correct env vars

[Kevin Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Yang closed AIRFLOW-2497.
---
Resolution: Fixed

> Cgroup task runner doesn't pass down correct env vars
> -
>
> Key: AIRFLOW-2497
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2497
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Kevin Yang
>Assignee: Kevin Yang
>Priority: Major
>
> From 
> [https://github.com/apache/incubator-airflow/blob/master/airflow/task/task_runner/base_task_runner.py#L79-L84,]
>  only PYTHONPATH is propagated to the child process, which make the behavior 
> of bash task runner and cgroup task runner different as bash task runner 
> would issue a `bash -c` command that automatically pass all the env var from 
> the parent process to the subprocess. Cgroup task runner should not behave 
> different.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2497) Cgroup task runner doesn't pass down correct env vars

[Kevin Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483547#comment-16483547
 ] 

Kevin Yang commented on AIRFLOW-2497:
-

This is actually resolved in https://issues.apache.org/jira/browse/AIRFLOW-2162

> Cgroup task runner doesn't pass down correct env vars
> -
>
> Key: AIRFLOW-2497
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2497
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Kevin Yang
>Assignee: Kevin Yang
>Priority: Major
>
> From 
> [https://github.com/apache/incubator-airflow/blob/master/airflow/task/task_runner/base_task_runner.py#L79-L84,]
>  only PYTHONPATH is propagated to the child process, which make the behavior 
> of bash task runner and cgroup task runner different as bash task runner 
> would issue a `bash -c` command that automatically pass all the env var from 
> the parent process to the subprocess. Cgroup task runner should not behave 
> different.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-2497) Cgroup task runner doesn't pass down correct env vars

[Kevin Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Yang reassigned AIRFLOW-2497:
---

Assignee: Kevin Yang

> Cgroup task runner doesn't pass down correct env vars
> -
>
> Key: AIRFLOW-2497
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2497
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Kevin Yang
>Assignee: Kevin Yang
>Priority: Major
>
> From 
> [https://github.com/apache/incubator-airflow/blob/master/airflow/task/task_runner/base_task_runner.py#L79-L84,]
>  only PYTHONPATH is propagated to the child process, which make the behavior 
> of bash task runner and cgroup task runner different as bash task runner 
> would issue a `bash -c` command that automatically pass all the env var from 
> the parent process to the subprocess. Cgroup task runner should not behave 
> different.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)