[jira] [Updated] (AIRFLOW-2343) pip gcp_api dependency conflicts

2018-04-18 Thread Simon Hellbe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Hellbe updated AIRFLOW-2343:
--
Description: 
In a clean virtualenv with python 2.7 on Ubuntu:
{code:java}
pip install apache-airflow[gcp_api]{code}
Fails with downstream dependency conflicts for the installed versions of 
google-cloud-bigquery and google-cloud-pubsub. Have tried installing various 
versions of the packages but can't find a compatible combination.
{code:java}
google-cloud-pubsub 0.26.0 has requirement google-cloud-core<0.26dev,>=0.25.0, 
but you'll have google-cloud-core 0.28.1 which is incompatible.
apache-beam 2.4.0 has requirement dill==0.2.6, but you'll have dill 0.2.7.1 
which is incompatible.apache-beam 2.4.0 has requirement httplib2<0.10,>=0.8, 
but you'll have httplib2 0.11.3 which is incompatible.
googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
httplib2 0.11.3 which is incompatible.{code}

  was:
In a clean virtualenv with python 2.7 on Ubuntu:
{code:java}
pip install apache-airflow[gcp_api]{code}
Fails with downstream dependency conflicts for the installed versions of 
google-cloud-bigquery and google-cloud-pubsub. Have tried installing various 
versions of the packages but can't find a compatible combination.
{code:java}
google-cloud-pubsub 0.26.0 has requirement google-cloud-core<0.26dev,>=0.25.0, 
but you'll have google-cloud-core 0.28.1 which is incompatible.apache-beam 
2.4.0 has requirement dill==0.2.6, but you'll have dill 0.2.7.1 which is 
incompatible.apache-beam 2.4.0 has requirement httplib2<0.10,>=0.8, but you'll 
have httplib2 0.11.3 which is incompatible.googledatastore 7.0.1 has 
requirement httplib2<0.10,>=0.9.1, but you'll have httplib2 0.11.3 which is 
incompatible.{code}


> pip gcp_api dependency conflicts
> 
>
> Key: AIRFLOW-2343
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2343
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: dependencies
>Affects Versions: Airflow 1.9.0
> Environment: Ubuntu Server 17.10
> Debian 9
>Reporter: Simon Hellbe
>Priority: Major
>
> In a clean virtualenv with python 2.7 on Ubuntu:
> {code:java}
> pip install apache-airflow[gcp_api]{code}
> Fails with downstream dependency conflicts for the installed versions of 
> google-cloud-bigquery and google-cloud-pubsub. Have tried installing various 
> versions of the packages but can't find a compatible combination.
> {code:java}
> google-cloud-pubsub 0.26.0 has requirement 
> google-cloud-core<0.26dev,>=0.25.0, but you'll have google-cloud-core 0.28.1 
> which is incompatible.
> apache-beam 2.4.0 has requirement dill==0.2.6, but you'll have dill 0.2.7.1 
> which is incompatible.apache-beam 2.4.0 has requirement httplib2<0.10,>=0.8, 
> but you'll have httplib2 0.11.3 which is incompatible.
> googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
> httplib2 0.11.3 which is incompatible.{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2343) pip gcp_api dependency conflicts

2018-04-18 Thread Simon Hellbe (JIRA) 
Simon Hellbe created AIRFLOW-2343:
-

 Summary: pip gcp_api dependency conflicts
 Key: AIRFLOW-2343
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2343
 Project: Apache Airflow
  Issue Type: Bug
  Components: dependencies
Affects Versions: Airflow 1.9.0
 Environment: Ubuntu Server 17.10
Debian 9
Reporter: Simon Hellbe


In a clean virtualenv with python 2.7 on Ubuntu:
{code:java}
pip install apache-airflow[gcp_api]{code}
Fails with downstream dependency conflicts for the installed versions of 
google-cloud-bigquery and google-cloud-pubsub. Have tried installing various 
versions of the packages but can't find a compatible combination.
{code:java}
google-cloud-pubsub 0.26.0 has requirement google-cloud-core<0.26dev,>=0.25.0, 
but you'll have google-cloud-core 0.28.1 which is incompatible.apache-beam 
2.4.0 has requirement dill==0.2.6, but you'll have dill 0.2.7.1 which is 
incompatible.apache-beam 2.4.0 has requirement httplib2<0.10,>=0.8, but you'll 
have httplib2 0.11.3 which is incompatible.googledatastore 7.0.1 has 
requirement httplib2<0.10,>=0.9.1, but you'll have httplib2 0.11.3 which is 
incompatible.{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2342) DAG in running state but tasks not running

2018-04-18 Thread chidrup jhanjhari (JIRA) 
chidrup jhanjhari created AIRFLOW-2342:
--

 Summary: DAG in running state but tasks not running
 Key: AIRFLOW-2342
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2342
 Project: Apache Airflow
  Issue Type: Bug
  Components: DagRun
Affects Versions: Airflow 1.8
 Environment: Redhat
Reporter: chidrup jhanjhari
 Attachments: job1.py.log

Hi, We are on Airflow 1.8.0. We have airflow production environment running 
well since 8 months. There has been no change on configuration etc. The issue 
is since 2 days DAGs are showing in running state but the tasks are not getting 
triggered. After the default start task, the DAG run is not moving to the next 
task. Attached is the scheduler throwing following error:

2018-04-19 01:32:22,586] \{jobs.py:354} DagFileProcessor17 ERROR - Got an 
exception! Propagating...
Traceback (most recent call last):

  Any help will be greatly appreciated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2339) setup.py version doesn't use git info

2018-04-18 Thread Siddharth Anand (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443385#comment-16443385
 ] 

Siddharth Anand commented on AIRFLOW-2339:
--

Feel free to implement a solution for us to review.

> setup.py version doesn't use git info
> -
>
> Key: AIRFLOW-2339
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2339
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration
>Affects Versions: 1.9.0
>Reporter: John Arnold
>Assignee: John Arnold
>Priority: Major
>
> In setup.py, the function git_version() is used to determine if the package 
> is being built from git, and if a release tag or commit hash should inform 
> the version information.
> This info is written to a file, but it's never used when building a package. 
> The package version is always 1.x.x.dev0+incubating when building a dev / 
> non-release package.
> This is not really PEP440 compliant, and is a sucky experience when trying to 
> generate packages in between major releases (for bugfixes, primarily).
> I can manually manipulate the version info in version.py to force unique 
> build numbers, but this makes for a lot of extra work.  It would be nice to 
> wire in either pbr or versioneer, or mayb ejust miniver/miniver2 to do the 
> work for us, and generate nice PEP440 versions, e.g.:
> 1.20.0.dev0+N.M    where N is the number of commits since the last 
> release, and M is the commit hash.   The incrementing of N makes for nice 
> version upgrades with pip.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2341) Trigger timing should follow cron style when we use cron style schedule_interval

2018-04-18 Thread Yu Ishikawa (JIRA) 
Yu Ishikawa created AIRFLOW-2341:


 Summary: Trigger timing should follow cron style when we use cron 
style schedule_interval
 Key: AIRFLOW-2341
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2341
 Project: Apache Airflow
  Issue Type: Wish
  Components: scheduler
Reporter: Yu Ishikawa


I understand "The Airflow scheduler triggers the task soon after the 
{{start_date + scheduler_interval}} is passed." in 
[FAQ|https://airflow.apache.org/faq.html#why-isn-t-my-task-getting-scheduled]. 
However, in my opinion, the trigger timing is confusing.

For example, when I set start_date to 2018-04-01 and set schedule_interval to 
'0 0 5 * *' in a DAG, the DAG will be triggered on 2018-05-05. In general, we 
expect the job should be trigger on 2018-04-05.

If we adapt the cron style scheduling, it should follow the trigger timing of 
the cron style as well. Otherwise, we should have another option for the way of 
cron style scheduling, instead of schedule_interval. What do you think?
{noformat}
default_args = {
'start_date': datetime(2018, 4, 1),
}
dag_id = get_file_name(__file__)
dag = DAG(
dag_id,
default_args=default_args,
catchup=True,
schedule_interval='0 0 5 * *'){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2340) SQLalchemy pessimistic connection handling not working

2018-04-18 Thread John Arnold (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443161#comment-16443161
 ] 

John Arnold commented on AIRFLOW-2340:
--

Now getting similar errors for the webserver too.. see attachment  webserver.txt

> SQLalchemy pessimistic connection handling not working
> --
>
> Key: AIRFLOW-2340
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2340
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.9.0
>Reporter: John Arnold
>Priority: Critical
> Attachments: airflow_traceback.txt, webserver.txt
>
>
> Our scheduler keeps crashing, about once a day.  It seems to be triggered by 
> a failure to connect to the postgresql database, but then it doesn't recover 
> and crashes the scheduler over and over.
> The scheduler runs in a container in our environment, so after several 
> container restarts, docker gives up and the container stays down.
> Airflow should be able to recover from a connection failure without blowing 
> up the container altogether.  Perhaps some exponential backoff is needed?
>  
> See attached log from the scheduler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2340) SQLalchemy pessimistic connection handling not working

2018-04-18 Thread John Arnold (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Arnold updated AIRFLOW-2340:
-
Attachment: webserver.txt

> SQLalchemy pessimistic connection handling not working
> --
>
> Key: AIRFLOW-2340
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2340
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.9.0
>Reporter: John Arnold
>Priority: Critical
> Attachments: airflow_traceback.txt, webserver.txt
>
>
> Our scheduler keeps crashing, about once a day.  It seems to be triggered by 
> a failure to connect to the postgresql database, but then it doesn't recover 
> and crashes the scheduler over and over.
> The scheduler runs in a container in our environment, so after several 
> container restarts, docker gives up and the container stays down.
> Airflow should be able to recover from a connection failure without blowing 
> up the container altogether.  Perhaps some exponential backoff is needed?
>  
> See attached log from the scheduler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2340) SQLalchemy pessimistic connection handling not working

2018-04-18 Thread John Arnold (JIRA) 
John Arnold created AIRFLOW-2340:


 Summary: SQLalchemy pessimistic connection handling not working
 Key: AIRFLOW-2340
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2340
 Project: Apache Airflow
  Issue Type: Bug
  Components: scheduler
Affects Versions: 1.9.0
Reporter: John Arnold
 Attachments: airflow_traceback.txt

Our scheduler keeps crashing, about once a day.  It seems to be triggered by a 
failure to connect to the postgresql database, but then it doesn't recover and 
crashes the scheduler over and over.

The scheduler runs in a container in our environment, so after several 
container restarts, docker gives up and the container stays down.

Airflow should be able to recover from a connection failure without blowing up 
the container altogether.  Perhaps some exponential backoff is needed?

 

See attached log from the scheduler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2339) setup.py version doesn't use git info

2018-04-18 Thread John Arnold (JIRA) 
John Arnold created AIRFLOW-2339:


 Summary: setup.py version doesn't use git info
 Key: AIRFLOW-2339
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2339
 Project: Apache Airflow
  Issue Type: Bug
  Components: configuration
Affects Versions: 1.9.0
Reporter: John Arnold
Assignee: John Arnold


In setup.py, the function git_version() is used to determine if the package is 
being built from git, and if a release tag or commit hash should inform the 
version information.

This info is written to a file, but it's never used when building a package. 
The package version is always 1.x.x.dev0+incubating when building a dev / 
non-release package.

This is not really PEP440 compliant, and is a sucky experience when trying to 
generate packages in between major releases (for bugfixes, primarily).

I can manually manipulate the version info in version.py to force unique build 
numbers, but this makes for a lot of extra work.  It would be nice to wire in 
either pbr or versioneer, or mayb ejust miniver/miniver2 to do the work for us, 
and generate nice PEP440 versions, e.g.:
1.20.0.dev0+N.M    where N is the number of commits since the last release, 
and M is the commit hash.   The incrementing of N makes for nice version 
upgrades with pip.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2338) Airflow requires Hive config

2018-04-18 Thread John Arnold (JIRA) 
John Arnold created AIRFLOW-2338:


 Summary: Airflow requires Hive config
 Key: AIRFLOW-2338
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2338
 Project: Apache Airflow
  Issue Type: Bug
  Components: configuration
Affects Versions: 1.9.0
Reporter: John Arnold


Had working airflow, upgraded to latest from master branch, started getting 
config exceptions:

[2018-04-16 23:52:11,732] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 [2018-04-16 23:52:11,731] \{__init__.py:50} INFO - Using executor 
CeleryExecutor
[2018-04-16 23:52:12,478] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 Traceback (most recent call last):
[2018-04-16 23:52:12,478] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 File "/var/lib/celery/venv/bin/airflow", line 32, in 
[2018-04-16 23:52:12,478] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 args.func(args)
[2018-04-16 23:52:12,478] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 File 
"/var/lib/celery/venv/lib/python3.6/site-packages/airflow/utils/cli.py", line 
77, in wrapper
[2018-04-16 23:52:12,478] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 raise e
[2018-04-16 23:52:12,479] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 File 
"/var/lib/celery/venv/lib/python3.6/site-packages/airflow/utils/cli.py", line 
74, in wrapper
[2018-04-16 23:52:12,479] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 return f(*args, **kwargs)
[2018-04-16 23:52:12,479] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 File 
"/var/lib/celery/venv/lib/python3.6/site-packages/airflow/bin/cli.py", line 
438, in run
[2018-04-16 23:52:12,479] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 conf.set(section, option, value)
[2018-04-16 23:52:12,479] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 File 
"/var/lib/celery/venv/lib/python3.6/site-packages/backports/configparser/__init__.py",
 line 1239, in set
[2018-04-16 23:52:12,479] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 super(ConfigParser, self).set(section, option, value)
[2018-04-16 23:52:12,479] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 File 
"/var/lib/celery/venv/lib/python3.6/site-packages/backports/configparser/__init__.py",
 line 921, in set
[2018-04-16 23:52:12,479] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 raise from_none(NoSectionError(section))
[2018-04-16 23:52:12,479] \{base_task_runner.py:106} INFO - Job 147: Subtask 
login_task0 backports.configparser.NoSectionError: No section: 'hive'
[2018-04-16 23:52:36,727] \{logging_mixin.py:95} INFO - [2018-04-16 
23:52:36,726] \{jobs.py:2548} INFO - Task exited with return code 1

It's looking for a hive section in the airflow.cfg but my config doesn't have 
one.

Not sure what changed to make this break, but the default values should be used 
/ we should not error out on a missing section that isn't even needed or used 
by current dags.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-2240) Add TLS/SSL to Dask Executor

2018-04-18 Thread Arthur Wiedmer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arthur Wiedmer resolved AIRFLOW-2240.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

Issue resolved by pull request #2683
[https://github.com/apache/incubator-airflow/pull/2683]

> Add TLS/SSL to Dask Executor
> 
>
> Key: AIRFLOW-2240
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2240
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: executor
>Affects Versions: Airflow 1.8
>Reporter: Marius Van Niekerk
>Assignee: Marius Van Niekerk
>Priority: Minor
> Fix For: 2.0.0
>
>
> As of distributed 0.17 dask distributed supports tls / ssl for communication.
>  
> We should allow this configuration to be used with airflow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-2240][DASK] Added TLS/SSL support for the dask-distributed scheduler.

2018-04-18 Thread arthur 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 3fa55db90 -> e95a1251b


[AIRFLOW-2240][DASK] Added TLS/SSL support for the dask-distributed scheduler.

As of 0.17.0 dask distributed has support for
TLS/SSL.

[dask] Added TLS/SSL support for the dask-
distributed scheduler.

As of 0.17.0 dask distributed has support for
TLS/SSL.

Add a test for tls under dask distributed

Closes #2683 from mariusvniekerk/dask-ssl


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e95a1251
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e95a1251
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e95a1251

Branch: refs/heads/master
Commit: e95a1251b746ac74a47b3ccae52d4abdc26add82
Parents: 3fa55db
Author: Marius van Niekerk 
Authored: Wed Apr 18 09:45:46 2018 -0700
Committer: Arthur Wiedmer 
Committed: Wed Apr 18 09:45:52 2018 -0700

--
 airflow/config_templates/default_airflow.cfg |  4 ++
 airflow/executors/dask_executor.py   | 16 +-
 setup.py |  2 +-
 tests/executors/dask_executor.py | 61 +++
 4 files changed, 72 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e95a1251/airflow/config_templates/default_airflow.cfg
--
diff --git a/airflow/config_templates/default_airflow.cfg 
b/airflow/config_templates/default_airflow.cfg
index bb3793b..400bcc0 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -364,6 +364,10 @@ ssl_cacert =
 
 # The IP address and port of the Dask cluster's scheduler.
 cluster_address = 127.0.0.1:8786
+# TLS/ SSL settings to access a secured Dask scheduler.
+tls_ca =
+tls_cert =
+tls_key =
 
 
 [scheduler]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e95a1251/airflow/executors/dask_executor.py
--
diff --git a/airflow/executors/dask_executor.py 
b/airflow/executors/dask_executor.py
index 0d914ba..17ace55 100644
--- a/airflow/executors/dask_executor.py
+++ b/airflow/executors/dask_executor.py
@@ -36,10 +36,24 @@ class DaskExecutor(BaseExecutor):
 raise ValueError(
 'Please provide a Dask cluster address in airflow.cfg')
 self.cluster_address = cluster_address
+# ssl / tls parameters
+self.tls_ca = configuration.get('dask', 'tls_ca')
+self.tls_key = configuration.get('dask', 'tls_key')
+self.tls_cert = configuration.get('dask', 'tls_cert')
 super(DaskExecutor, self).__init__(parallelism=0)
 
 def start(self):
-self.client = distributed.Client(self.cluster_address)
+if (self.tls_ca) or (self.tls_key) or (self.tls_cert):
+from distributed.security import Security
+security = Security(
+tls_client_key=self.tls_key,
+tls_client_cert=self.tls_cert,
+tls_ca_file=self.tls_ca,
+)
+else:
+security = None
+
+self.client = distributed.Client(self.cluster_address, 
security=security)
 self.futures = {}
 
 def execute_async(self, key, command, queue=None):

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e95a1251/setup.py
--
diff --git a/setup.py b/setup.py
index 700f3ae..9709ddb 100644
--- a/setup.py
+++ b/setup.py
@@ -112,7 +112,7 @@ cgroups = [
 ]
 crypto = ['cryptography>=0.9.3']
 dask = [
-'distributed>=1.15.2, <2'
+'distributed>=1.17.1, <2'
 ]
 databricks = ['requests>=2.5.1, <3']
 datadog = ['datadog>=0.14.0']

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e95a1251/tests/executors/dask_executor.py
--
diff --git a/tests/executors/dask_executor.py b/tests/executors/dask_executor.py
index 40796bc..7937b1b 100644
--- a/tests/executors/dask_executor.py
+++ b/tests/executors/dask_executor.py
@@ -29,6 +29,13 @@ from datetime import timedelta
 try:
 from airflow.executors.dask_executor import DaskExecutor
 from distributed import LocalCluster
+# utility functions imported from the dask testing suite to instantiate a 
test
+# cluster for tls tests
+from distributed.utils_test import (
+get_cert,
+cluster as dask_testing_cluster,
+tls_security,
+)
 SKIP_DASK = False
 except ImportError:
 SKIP_DASK = True
@@ -42,16 +49,9 @@ SKIP_DASK = True
 DEFAULT_DATE = timezone.datetime(2017, 1, 1)
 
 
-class

[jira] [Commented] (AIRFLOW-2240) Add TLS/SSL to Dask Executor

2018-04-18 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442827#comment-16442827
 ] 

ASF subversion and git services commented on AIRFLOW-2240:
--

Commit e95a1251b746ac74a47b3ccae52d4abdc26add82 in incubator-airflow's branch 
refs/heads/master from Marius van Niekerk
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=e95a125 ]

[AIRFLOW-2240][DASK] Added TLS/SSL support for the dask-distributed scheduler.

As of 0.17.0 dask distributed has support for
TLS/SSL.

[dask] Added TLS/SSL support for the dask-
distributed scheduler.

As of 0.17.0 dask distributed has support for
TLS/SSL.

Add a test for tls under dask distributed

Closes #2683 from mariusvniekerk/dask-ssl


> Add TLS/SSL to Dask Executor
> 
>
> Key: AIRFLOW-2240
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2240
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: executor
>Affects Versions: Airflow 1.8
>Reporter: Marius Van Niekerk
>Assignee: Marius Van Niekerk
>Priority: Minor
>
> As of distributed 0.17 dask distributed supports tls / ssl for communication.
>  
> We should allow this configuration to be used with airflow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-351) Failed to clear downstream tasks

2018-04-18 Thread Yannick Einsweiler (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442702#comment-16442702
 ] 

Yannick Einsweiler commented on AIRFLOW-351:


Just adding that 
{noformat}
google.cloud Client()
{noformat}
is not pickeable either, in case you want to use the google-cloud client 
libraries instead.

As these (storage, datastore and bigquery for us) were used in custom 
operators, we moved their declaration from *init* to *execute* so that clearing 
a task does not try to deepcopy.

> Failed to clear downstream tasks
> 
>
> Key: AIRFLOW-351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-351
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: models, subdag, webserver
>Affects Versions: Airflow 1.7.1.3
>Reporter: Adinata
>Priority: Major
> Attachments: dag_error.py, error.log, error_on_clear_dag.txt, 
> ubuntu-14-packages.log, ubuntu-16-oops.log, ubuntu-16-packages.log
>
>
> {code}
>   / (  ()   )  \___
>  /( (  (  )   _))  )   )\
>(( (   )()  )   (   )  )
>  ((/  ( _(   )   (   _) ) (  () )  )
> ( (  ( (_)   (((   )  .((_ ) .  )_
>( (  )(  (  ))   ) . ) (   )
>   (  (   (  (   ) (  _  ( _) ).  ) . ) ) ( )
>   ( (  (   ) (  )   (  )) ) _)(   )  )  )
>  ( (  ( \ ) ((_  ( ) ( )  )   ) )  )) ( )
>   (  (   (  (   (_ ( ) ( _)  ) (  )  )   )
>  ( (  ( (  (  ) (_  )  ) )  _)   ) _( ( )
>   ((  (   )(( _)   _) _(_ (  (_ )
>(_((__(_(__(( ( ( |  ) ) ) )_))__))_)___)
>((__)\\||lll|l||///  \_))
> (   /(/ (  )  ) )\   )
>   (( ( ( | | ) ) )\   )
>(   /(| / ( )) ) ) )) )
>  ( ( _(|)_) )
>   (  ||\(|(|)|/|| )
> (|(||(||))
>   ( //|/l|||)|\\ \ )
> (/ / //  /|//\\  \ \  \ _)
> ---
> Node: 9889a7c79e9b
> ---
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in 
> wsgi_app
> response = self.full_dispatch_request()
>   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in 
> full_dispatch_request
> rv = self.handle_user_exception(e)
>   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in 
> handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in 
> full_dispatch_request
> rv = self.dispatch_request()
>   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in 
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 68, 
> in inner
> return self._run_view(f, *args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 
> 367, in _run_view
> return fn(self, *args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flask_login.py", line 755, in 
> decorated_view
> return func(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line 
> 118, in wrapper
> return f(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line 
> 167, in wrapper
> return f(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 
> 1017, in clear
> include_upstream=upstream)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2870, 
> in sub_dag
> dag = copy.deepcopy(self)
>   File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2856, 
> in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
>   File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
> y = copier(x, memo)
>   File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
> y[deepcopy(key, memo)] = deepcopy(value, memo)
>   File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1974, 
> in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
>   File

[jira] [Updated] (AIRFLOW-2336) Update hive_hook dependencies so that it can work with Python 3

2018-04-18 Thread Giovanni Lanzani (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Giovanni Lanzani updated AIRFLOW-2336:
--
Labels: pull-request-available  (was: )

> Update hive_hook dependencies so that it can work with Python 3
> ---
>
> Key: AIRFLOW-2336
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2336
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: Airflow 1.9.0
>Reporter: Giovanni Lanzani
>Assignee: Giovanni Lanzani
>Priority: Major
>  Labels: pull-request-available
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> I already have a new version of the hive metastore thrift client out. I'm 
> updating it and I will update Airflow consequently (without changing the API)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2337) Broken Import Variables

2018-04-18 Thread Daniel Lamblin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Lamblin updated AIRFLOW-2337:

Description: 
Importing variables that were encrypted seems to have produced a far too long 
parameter value.
Using the UI in v1.8.2 I selected all variables, then with selected exported 
them.
This produced a json file. it is only 1617 bytes in size total with 26 
variables.
Using the UI in v1.9.0 I selected the file and clicked to import variables.

…/admin/airflow/varimport
gave me the error:
h2. Ooops.
{code:bash}

  / (  ()   )  \___
 /( (  (  )   _))  )   )\
Can this be briefer and cheerier?
   (_((__(_(__(( ( ( |  ) ) ) )_))__))_)___)
   ((__)\\||lll|l||///  \_))
(   /(/ (  )  ) )\   )
  (( ( ( | | ) ) )\   )
   (   /(| / ( )) ) ) )) )
 ( ( _(|)_) )
  (  ||\(|(|)|/|| )
(|(||(||))
  ( //|/l|||)|\\ \ )
(/ / //  /|//\\  \ \  \ _)
---
Node: 90f7f5d06c61
---
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1988, in 
wsgi_app
response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1641, in 
full_dispatch_request
rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1544, in 
handle_user_exception
reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1639, in 
full_dispatch_request
rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1625, in 
dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 69, 
in inner
return self._run_view(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 368, 
in _run_view
return fn(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_login.py", line 758, in 
decorated_view
return func(*args, **kwargs)
  File "/pang/service/airflow/airflow-src/airflow/www/utils.py", line 262, in 
wrapper
return f(*args, **kwargs)
  File "/pang/service/airflow/airflow-src/airflow/www/views.py", line 1787, in 
varimport
models.Variable.set(k, v, serialize_json=isinstance(v, dict))
  File "/pang/service/airflow/airflow-src/airflow/utils/db.py", line 55, in 
wrapper
result = func(*args, **kwargs)
  File "/pang/service/airflow/airflow-src/airflow/models.py", line 4031, in set
session.query(cls).filter(cls.key == key).delete()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 
3236, in delete
delete_op.exec_()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", 
line 1326, in exec_
self._do_exec()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", 
line 1518, in _do_exec
self._execute_stmt(delete_stmt)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", 
line 1333, in _execute_stmt
mapper=self.mapper)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 
1176, in execute
bind, close_with_result=True).execute(clause, params or {})
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 
1040, in _connection_for_bind
engine, execution_options)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 
388, in _connection_for_bind
self._assert_active()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 
276, in _assert_active
% self._rollback_exception
InvalidRequestError: This Session's transaction has been rolled back due to a 
previous exception during flush. To begin a new transaction with this Session, 
first issue Session.rollback(). Original exception was: 
(_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: 
u'INSERT INTO variable (`key`, val, is_encrypted) VALUES (%s, %s, %s)'] 
[parameters: (u'frontarch_testdate', 
u'gABa1wmXHtv6__5O-yvMn4PHiv_I9OPR4aLb0UKI9a-zS6SRJ3mNhR9gDC3_4siNZzUfTIwwt_FD-HASqtYVbU4rnlpfxhkoDC2g93bb73GUsgorKzfoEZFCOF_bDBB6HRTyUJ3y9rs2hyxM
 ... (26894735 characters truncated) ... 
ZRgN9HM4H7f6geAGdr20oR0fjVQyTy9-6S5eF7PEWN17Xa7CrPDob4NMvf0_AGylOUXc0xO9gmg2gbt-FWVh8QI9YK7J3N-CA9shHHX47iVegCq-4ofAmAgIHCBQ8exYS1CsM6LJn7orZAwedOG8=',
 1)] (Background on

[jira] [Created] (AIRFLOW-2337) Broken Import Variables

2018-04-18 Thread Daniel Lamblin (JIRA) 
Daniel Lamblin created AIRFLOW-2337:
---

 Summary: Broken Import Variables
 Key: AIRFLOW-2337
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2337
 Project: Apache Airflow
  Issue Type: Bug
  Components: db
Affects Versions: 1.9.0
Reporter: Daniel Lamblin


Importing variables that were encrypted seems to have produced a far too long 
parameter value.
Using the UI in v1.8.2 I selected all variables, then with selected exported 
them.
This produced a json file. it is only 1617 bytes in size total with 26 
variables.
Using the UI in v1.9.0 I selected the file and clicked to import variables.

…/admin/airflow/varimport
gave me the error:
(BTW the frontarch_testdate was supposed to be set to 20171201 (it was not 
quoted in the json).
h2. Ooops.
{code}

  / (  ()   )  \___
 /( (  (  )   _))  )   )\
Can this be briefer and cheerier?
   (_((__(_(__(( ( ( |  ) ) ) )_))__))_)___)
   ((__)\\||lll|l||///  \_))
(   /(/ (  )  ) )\   )
  (( ( ( | | ) ) )\   )
   (   /(| / ( )) ) ) )) )
 ( ( _(|)_) )
  (  ||\(|(|)|/|| )
(|(||(||))
  ( //|/l|||)|\\ \ )
(/ / //  /|//\\  \ \  \ _)
---
Node: 90f7f5d06c61
---
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1988, in 
wsgi_app
response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1641, in 
full_dispatch_request
rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1544, in 
handle_user_exception
reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1639, in 
full_dispatch_request
rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1625, in 
dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 69, 
in inner
return self._run_view(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 368, 
in _run_view
return fn(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask_login.py", line 758, in 
decorated_view
return func(*args, **kwargs)
  File "/pang/service/airflow/airflow-src/airflow/www/utils.py", line 262, in 
wrapper
return f(*args, **kwargs)
  File "/pang/service/airflow/airflow-src/airflow/www/views.py", line 1787, in 
varimport
models.Variable.set(k, v, serialize_json=isinstance(v, dict))
  File "/pang/service/airflow/airflow-src/airflow/utils/db.py", line 55, in 
wrapper
result = func(*args, **kwargs)
  File "/pang/service/airflow/airflow-src/airflow/models.py", line 4031, in set
session.query(cls).filter(cls.key == key).delete()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 
3236, in delete
delete_op.exec_()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", 
line 1326, in exec_
self._do_exec()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", 
line 1518, in _do_exec
self._execute_stmt(delete_stmt)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", 
line 1333, in _execute_stmt
mapper=self.mapper)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 
1176, in execute
bind, close_with_result=True).execute(clause, params or {})
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 
1040, in _connection_for_bind
engine, execution_options)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 
388, in _connection_for_bind
self._assert_active()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 
276, in _assert_active
% self._rollback_exception
InvalidRequestError: This Session's transaction has been rolled back due to a 
previous exception during flush. To begin a new transaction with this Session, 
first issue Session.rollback(). Original exception was: 
(_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: 
u'INSERT INTO variable (`key`, val, is_encrypted) VALUES (%s, %s, %s)'] 
[parameters: (u'frontarch_testdate',

[jira] [Commented] (AIRFLOW-363) Celery Queue becoming too huge when worker unavailable

2018-04-18 Thread Daniel Lamblin (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442170#comment-16442170
 ] 

Daniel Lamblin commented on AIRFLOW-363:


I'm not sure, but I think #3 was done, right [~criccomini]?

> Celery Queue becoming too huge when worker unavailable
> --
>
> Key: AIRFLOW-363
> URL: https://issues.apache.org/jira/browse/AIRFLOW-363
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
>Affects Versions: Airflow 1.7.1.3
>Reporter: Vineet Goel
>Priority: Major
>
> The scheduler currently queues executable tasks every run of the scheduler. I 
> ran into an issue where I was using a single worker and my dag was composed 
> of a bunch of tasks that could run simultaneously. I was setting a 
> concurrency on the worker to control how many of those tasks could run at the 
> same time. However, my scheduler kept queueing more and more tasks. This can 
> cause a HUGE back pressure on the celery queue, which becomes unmanageable at 
> times. 
> We should set the 
> http://docs.celeryproject.org/en/latest/configuration.html#celery-event-queue-ttl
> configuration to handle this. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-940) Failing to unencrypt one variable shouldn't break the admin/variable page

2018-04-18 Thread Daniel Lamblin (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442167#comment-16442167
 ] 

Daniel Lamblin commented on AIRFLOW-940:


Close; I can confirm that v1.9.0 release shows an "invalid" tag for any value 
that does not decrypt.

> Failing to unencrypt one variable shouldn't break the admin/variable page
> -
>
> Key: AIRFLOW-940
> URL: https://issues.apache.org/jira/browse/AIRFLOW-940
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Reporter: Dan Davydov
>Assignee: Edgar Rodriguez
>Priority: Major
>  Labels: beginner, starter
>
> At the moment failing to decrypt one of the airflow variables in the database 
> using the fernet key will cause the admin/variable page to crash. Instead 
> there should be an error message for only the broken variables at the top of 
> the page, and the other ones should still be loaded and displayed.
> This is what the trace on admin/variable currently looks like:
> {code}
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in 
> wsgi_app
> response = self.full_dispatch_request()
>   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in 
> full_dispatch_request
> rv = self.handle_user_exception(e)
>   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in 
> handle_user_exception
> reraise(exc_type, exc_value, tb)
>   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in 
> full_dispatch_request
> rv = self.dispatch_request()
>   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in 
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
>   File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 68, 
> in inner
> return self._run_view(f, *args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 
> 367, in _run_view
> return fn(self, *args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", 
> line 1814, in index_view
> return_url=self._get_list_url(view_args),
>   File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 
> 307, in render
> return render_template(template, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flask/templating.py", line 
> 128, in render_template
> context, ctx.app)
>   File "/usr/local/lib/python2.7/dist-packages/flask/templating.py", line 
> 110, in _render
> rv = template.render(context)
>   File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 
> 989, in render
> return self.environment.handle_exception(exc_info, True)
>   File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 
> 754, in handle_exception
> reraise(exc_type, exc_value, tb)
>   File 
> "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/airflow/variable_list.html",
>  line 18, in top-level template code
> {% extends 'admin/model/list.html' %}
>   File 
> "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html",
>  line 5, in top-level template code
> {% import 'admin/actions.html' as actionlib with context %}
>   File 
> "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/admin/master.html",
>  line 18, in top-level template code
> {% extends 'admin/base.html' %}
>   File 
> "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/base.html",
>  line 30, in top-level template code
> {% block page_body %}
>   File 
> "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/admin/master.html",
>  line 96, in block "page_body"
> {% block body %}
>   File 
> "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html",
>  line 62, in block "body"
> {% block model_list_table %}
>   File 
> "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html",
>  line 110, in block "model_list_table"
> {% block list_row scoped %}
>   File 
> "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html",
>  line 165, in block "list_row"
> {{ get_value(row, c) }}
>   File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", 
> line 1666, in get_list_value
> self.column_type_formatters,
>   File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", 
> line 1631, in _get_list_value
> value = column_fmt(self, context, model, name)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 
> 2146, in hidden_field_formatter
> return getattr(model, name)
>

[jira] [Resolved] (AIRFLOW-2309) TypeError exception when calculating duration

2018-04-18 Thread Fokko Driesprong (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AIRFLOW-2309.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

Issue resolved by pull request #3208
[https://github.com/apache/incubator-airflow/pull/3208]

> TypeError exception when calculating duration
> -
>
> Key: AIRFLOW-2309
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2309
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.9.0
>Reporter: John Arnold
>Assignee: John Arnold
>Priority: Major
> Fix For: 2.0.0
>
>
> TypeError can occur when setting TaskFail in certain scenarios, if start_date 
> or end_date are None.
> I lost the traceback for this error, but the change has been tested and 
> validated in our production environ.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2309) TypeError exception when calculating duration

2018-04-18 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442115#comment-16442115
 ] 

ASF subversion and git services commented on AIRFLOW-2309:
--

Commit 3fa55db90cbeee3746220d0ac0cdfa3db52c0753 in incubator-airflow's branch 
refs/heads/master from [~johnarnold]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3fa55db ]

[AIRFLOW-2309] Fix duration calculation on TaskFail

Closes #3208 from johnarnold/duration


> TypeError exception when calculating duration
> -
>
> Key: AIRFLOW-2309
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2309
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.9.0
>Reporter: John Arnold
>Assignee: John Arnold
>Priority: Major
> Fix For: 2.0.0
>
>
> TypeError can occur when setting TaskFail in certain scenarios, if start_date 
> or end_date are None.
> I lost the traceback for this error, but the change has been tested and 
> validated in our production environ.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-2309] Fix duration calculation on TaskFail

2018-04-18 Thread fokko 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 0f8507ae3 -> 3fa55db90


[AIRFLOW-2309] Fix duration calculation on TaskFail

Closes #3208 from johnarnold/duration


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3fa55db9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3fa55db9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3fa55db9

Branch: refs/heads/master
Commit: 3fa55db90cbeee3746220d0ac0cdfa3db52c0753
Parents: 0f8507a
Author: John Arnold (AZURE) 
Authored: Wed Apr 18 10:28:13 2018 +0200
Committer: Fokko Driesprong 
Committed: Wed Apr 18 10:28:13 2018 +0200

--
 airflow/models.py | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3fa55db9/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 9635460..ee92689 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -1988,7 +1988,10 @@ class TaskFail(Base):
 self.execution_date = execution_date
 self.start_date = start_date
 self.end_date = end_date
-self.duration = (self.end_date - self.start_date).total_seconds()
+if self.end_date and self.start_date:
+self.duration = (self.end_date - self.start_date).total_seconds()
+else:
+self.duration = None
 
 
 class Log(Base):

[jira] [Created] (AIRFLOW-2336) Update hive_hook dependencies so that it can work with Python 3

2018-04-18 Thread Giovanni Lanzani (JIRA) 
Giovanni Lanzani created AIRFLOW-2336:
-

 Summary: Update hive_hook dependencies so that it can work with 
Python 3
 Key: AIRFLOW-2336
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2336
 Project: Apache Airflow
  Issue Type: Bug
  Components: hooks
Affects Versions: Airflow 1.9.0
Reporter: Giovanni Lanzani
Assignee: Giovanni Lanzani


I already have a new version of the hive metastore thrift client out. I'm 
updating it and I will update Airflow consequently (without changing the API)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)