incubator-airflow git commit: [AIRFLOW-XXX] Add timeout units (seconds)

2018-02-22 Thread joygao 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 2143ccc67 -> ba126926d


[AIRFLOW-XXX] Add timeout units (seconds)

Timeout seconds according to paramiko SSHClient
docs: http://docs.paramiko.org/en/2.4/api/client.h
tml#paramiko.client.SSHClient.connect

Closes #3069 from miracodezu/master


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/ba126926
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/ba126926
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/ba126926

Branch: refs/heads/master
Commit: ba126926d7d8a3d8d5bae548c12b5ce4b2a9fbd8
Parents: 2143ccc
Author: miracodezu <30881196+miracod...@users.noreply.github.com>
Authored: Thu Feb 22 19:51:56 2018 -0800
Committer: Joy Gao 
Committed: Thu Feb 22 19:52:00 2018 -0800

--
 airflow/contrib/operators/ssh_operator.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ba126926/airflow/contrib/operators/ssh_operator.py
--
diff --git a/airflow/contrib/operators/ssh_operator.py 
b/airflow/contrib/operators/ssh_operator.py
index 9f2ca81..52eafc7 100644
--- a/airflow/contrib/operators/ssh_operator.py
+++ b/airflow/contrib/operators/ssh_operator.py
@@ -34,7 +34,7 @@ class SSHOperator(BaseOperator):
 :type remote_host: str
 :param command: command to execute on remote host
 :type command: str
-:param timeout: timeout for executing the command.
+:param timeout: timeout (in seconds) for executing the command.
 :type timeout: int
 :param do_xcom_push: return the stdout which also get set in xcom by 
airflow platform
 :type do_xcom_push: bool

[jira] [Resolved] (AIRFLOW-2131) API Reference includes confusing docs from airflow.utils.AirflowImporter

2018-02-22 Thread Joy Gao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joy Gao resolved AIRFLOW-2131.
--
   Resolution: Fixed
Fix Version/s: 1.10.0

Issue resolved by pull request #3062
[https://github.com/apache/incubator-airflow/pull/3062]

> API Reference includes confusing docs from airflow.utils.AirflowImporter
> 
>
> Key: AIRFLOW-2131
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2131
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Reid Beels
>Assignee: Reid Beels
>Priority: Critical
> Fix For: 1.10.0
>
> Attachments: image-2018-02-20-16-53-04-572.png
>
>
> The generated API documentation includes {{automodule}} declarations for 
> several modules (hooks and operators) that end up pulling in docs from 
> {{airflow.utils.helpers.AirflowImporter}}.
> This leads to the confusing situation for new users who think they're reading 
> docs about what Hooks are, but are instead reading unlabeled docs about the 
> seemingly-deprecated AirflowImporter.
> Like so:
> !image-2018-02-20-16-53-04-572.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[1/2] incubator-airflow git commit: [AIRFLOW-2131] Remove confusing AirflowImport docs

2018-02-22 Thread joygao 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master f30253cb3 -> 2143ccc67


[AIRFLOW-2131] Remove confusing AirflowImport docs


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/331a1dc0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/331a1dc0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/331a1dc0

Branch: refs/heads/master
Commit: 331a1dc0fcfc75ff5a132b1fb9a1d817b47de974
Parents: 6c93460
Author: Reid Beels 
Authored: Tue Feb 20 17:01:13 2018 -0800
Committer: Reid Beels 
Committed: Tue Feb 20 17:19:58 2018 -0800

--
 docs/code.rst | 22 +++---
 1 file changed, 3 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/331a1dc0/docs/code.rst
--
diff --git a/docs/code.rst b/docs/code.rst
index 597e35d..180e8da 100644
--- a/docs/code.rst
+++ b/docs/code.rst
@@ -43,11 +43,6 @@ attributes.
 Operator API
 
 
-.. automodule:: airflow.operators
-:no-members:
-.. deprecated:: 1.8
- Use :code:`from airflow.operators.bash_operator import BashOperator` instead.
-
 .. autoclass:: airflow.operators.bash_operator.BashOperator
 .. autoclass:: airflow.operators.python_operator.BranchPythonOperator
 .. autoclass:: airflow.operators.dagrun_operator.TriggerDagRunOperator
@@ -89,11 +84,6 @@ Operator API
 Community-contributed Operators
 '''
 
-.. automodule:: airflow.contrib.operators
-:no-members:
-.. deprecated:: 1.8
- Use :code:`from airflow.operators.bash_operator import BashOperator` instead.
-
 .. autoclass:: 
airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator
 .. autoclass:: 
airflow.contrib.operators.bigquery_check_operator.BigQueryValueCheckOperator
 .. autoclass:: 
airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator
@@ -263,10 +253,9 @@ persisted in the database.
 
 Hooks
 -
-.. automodule:: airflow.hooks
-:no-members:
-.. deprecated:: 1.8
- Use :code:`from airflow.operators.bash_operator import BashOperator` instead.
+
+Hooks are interfaces to external platforms and databases, implementing a common
+interface when possible and acting as building blocks for operators.
 
 .. autoclass:: airflow.hooks.dbapi_hook.DbApiHook
 .. autoclass:: airflow.hooks.docker_hook.DockerHook
@@ -288,11 +277,6 @@ Hooks
 Community contributed hooks
 '''
 
-.. automodule:: airflow.contrib.hooks
-:no-members:
-.. deprecated:: 1.8
- Use :code:`from airflow.operators.bash_operator import BashOperator` instead.
-
 .. autoclass:: airflow.contrib.hooks.redshift_hook.RedshiftHook
 .. autoclass:: airflow.contrib.hooks.bigquery_hook.BigQueryHook
 .. autoclass:: airflow.contrib.hooks.vertica_hook.VerticaHook

[jira] [Commented] (AIRFLOW-2131) API Reference includes confusing docs from airflow.utils.AirflowImporter

2018-02-22 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373869#comment-16373869
 ] 

ASF subversion and git services commented on AIRFLOW-2131:
--

Commit 331a1dc0fcfc75ff5a132b1fb9a1d817b47de974 in incubator-airflow's branch 
refs/heads/master from [~reidab]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=331a1dc ]

[AIRFLOW-2131] Remove confusing AirflowImport docs


> API Reference includes confusing docs from airflow.utils.AirflowImporter
> 
>
> Key: AIRFLOW-2131
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2131
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Reid Beels
>Assignee: Reid Beels
>Priority: Critical
> Attachments: image-2018-02-20-16-53-04-572.png
>
>
> The generated API documentation includes {{automodule}} declarations for 
> several modules (hooks and operators) that end up pulling in docs from 
> {{airflow.utils.helpers.AirflowImporter}}.
> This leads to the confusing situation for new users who think they're reading 
> docs about what Hooks are, but are instead reading unlabeled docs about the 
> seemingly-deprecated AirflowImporter.
> Like so:
> !image-2018-02-20-16-53-04-572.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2131) API Reference includes confusing docs from airflow.utils.AirflowImporter

2018-02-22 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373870#comment-16373870
 ] 

ASF subversion and git services commented on AIRFLOW-2131:
--

Commit 2143ccc670b574803b7692920c59877bea6a9663 in incubator-airflow's branch 
refs/heads/master from Joy Gao
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2143ccc ]

Merge pull request #3062 from reidab/AIRFLOW-2131


> API Reference includes confusing docs from airflow.utils.AirflowImporter
> 
>
> Key: AIRFLOW-2131
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2131
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs, Documentation
>Reporter: Reid Beels
>Assignee: Reid Beels
>Priority: Critical
> Attachments: image-2018-02-20-16-53-04-572.png
>
>
> The generated API documentation includes {{automodule}} declarations for 
> several modules (hooks and operators) that end up pulling in docs from 
> {{airflow.utils.helpers.AirflowImporter}}.
> This leads to the confusing situation for new users who think they're reading 
> docs about what Hooks are, but are instead reading unlabeled docs about the 
> seemingly-deprecated AirflowImporter.
> Like so:
> !image-2018-02-20-16-53-04-572.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[2/2] incubator-airflow git commit: Merge pull request #3062 from reidab/AIRFLOW-2131

2018-02-22 Thread joygao 
Merge pull request #3062 from reidab/AIRFLOW-2131


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2143ccc6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2143ccc6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2143ccc6

Branch: refs/heads/master
Commit: 2143ccc670b574803b7692920c59877bea6a9663
Parents: f30253c 331a1dc
Author: Joy Gao 
Authored: Thu Feb 22 19:39:36 2018 -0800
Committer: Joy Gao 
Committed: Thu Feb 22 19:39:36 2018 -0800

--
 docs/code.rst | 22 +++---
 1 file changed, 3 insertions(+), 19 deletions(-)
--

[jira] [Updated] (AIRFLOW-2144) Show warnings when ' or " are present in the value of a config key

2018-02-22 Thread Tim van Grieken (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim van Grieken updated AIRFLOW-2144:
-
Description: 
When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
values will not work.

Eg.

remote_log_conn_id = 'S3_MASTER' # will not work
remote_log_conn_id = S3_MASTER # will work


 

 

 

 

  was:
When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
values will not work.

Eg.

```python
remote_log_conn_id = 'S3_MASTER' # will not work
remote_log_conn_id = S3_MASTER # will work
```

 

 

 

 


> Show warnings when ' or " are present in the value of a config key
> --
>
> Key: AIRFLOW-2144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2144
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Tim van Grieken
>Priority: Trivial
>
> When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
> values will not work.
> Eg.
> remote_log_conn_id = 'S3_MASTER' # will not work
> remote_log_conn_id = S3_MASTER # will work
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2144) Show warnings when ' or " are present in the value of a config key

2018-02-22 Thread Tim van Grieken (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim van Grieken updated AIRFLOW-2144:
-
Description: 
When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
values will not work.

Eg.

```python
remote_log_conn_id = 'S3_MASTER' # will not work
remote_log_conn_id = S3_MASTER # will work
```

 

 

 

 

  was:
When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
values will not work.

Eg.

```
remote_log_conn_id = 'S3_MASTER' # will not work
remote_log_conn_id = S3_MASTER # will work
```

 

 

 

 


> Show warnings when ' or " are present in the value of a config key
> --
>
> Key: AIRFLOW-2144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2144
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Tim van Grieken
>Priority: Trivial
>
> When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
> values will not work.
> Eg.
> ```python
> remote_log_conn_id = 'S3_MASTER' # will not work
> remote_log_conn_id = S3_MASTER # will work
> ```
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2144) Show warnings when ' or " are present in the value of a config key

2018-02-22 Thread Tim van Grieken (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim van Grieken updated AIRFLOW-2144:
-
Description: 
When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
values will not work.

Eg.

```

remote_log_conn_id = 'S3_MASTER' # will not work

remote_log_conn_id = S3_MASTER # will work

```

 

 

 

 

> Show warnings when ' or " are present in the value of a config key
> --
>
> Key: AIRFLOW-2144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2144
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Tim van Grieken
>Priority: Trivial
>
> When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
> values will not work.
> Eg.
> ```
> remote_log_conn_id = 'S3_MASTER' # will not work
> remote_log_conn_id = S3_MASTER # will work
> ```
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2144) Show warnings when ' or " are present in the value of a config key

2018-02-22 Thread Tim van Grieken (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim van Grieken updated AIRFLOW-2144:
-
Description: 
When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
values will not work.

Eg.

```
remote_log_conn_id = 'S3_MASTER' # will not work
remote_log_conn_id = S3_MASTER # will work
```

 

 

 

 

  was:
When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
values will not work.

Eg.

```

remote_log_conn_id = 'S3_MASTER' # will not work

remote_log_conn_id = S3_MASTER # will work

```

 

 

 

 


> Show warnings when ' or " are present in the value of a config key
> --
>
> Key: AIRFLOW-2144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2144
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Tim van Grieken
>Priority: Trivial
>
> When using ' or " in config keys, either in airflow.cfg or ENV vars, these 
> values will not work.
> Eg.
> ```
> remote_log_conn_id = 'S3_MASTER' # will not work
> remote_log_conn_id = S3_MASTER # will work
> ```
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2144) Show warnings when ' or " are present in the value of a config key

2018-02-22 Thread Tim van Grieken (JIRA) 
Tim van Grieken created AIRFLOW-2144:


 Summary: Show warnings when ' or " are present in the value of a 
config key
 Key: AIRFLOW-2144
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2144
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Tim van Grieken






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (AIRFLOW-1003) DAG status flips erraticly

2018-02-22 Thread Luke Bodeen (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373383#comment-16373383
 ] 

Luke Bodeen edited comment on AIRFLOW-1003 at 2/22/18 9:18 PM:
---

how long is "awhile", 10 seconds or 2 minutes?

can turning max scheduler threads to 1 help this? - did not help


was (Author: lbodeen):
how long is "awhile", 10 seconds or 2 minutes?

can turning max scheduler threads to 1 help this?

> DAG status flips erraticly
> --
>
> Key: AIRFLOW-1003
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1003
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 1.8, 1.8.0rc5
>Reporter: Ruslan Dautkhanov
>Priority: Major
>
> Created a flow based on sample tutorial 
> https://airflow.incubator.apache.org/tutorial.html
> (just changed dag-id to 'turorial-RD').
> See a 10 seconds video on this behavior:
> http://screencast-o-matic.com/watch/cbebrn6kBw
> Notice last DAG 'turorial-RD' keeps changing state 
> (it loses link and all the buttons and icons to show status 
>  of the dag runs  / tasks etc), then links and icons show back up.
> In fact, during that short 10 seconds period it transitioned 
> that state 4 times (from "disabled" to a normal DAG).
> There were no changes happenning in the system - I was just clicking
> refresh in browser from time to time. All DAGs were disabled while
> this was happening.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-1003) DAG status flips erraticly

2018-02-22 Thread Luke Bodeen (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373463#comment-16373463
 ] 

Luke Bodeen commented on AIRFLOW-1003:
--

we turned down Gunicorn web server workers = 1

and saw this behavior diminish, long term if we need more workers we could also 
play with worker_refresh_interval to lower that and see it for less time

> DAG status flips erraticly
> --
>
> Key: AIRFLOW-1003
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1003
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 1.8, 1.8.0rc5
>Reporter: Ruslan Dautkhanov
>Priority: Major
>
> Created a flow based on sample tutorial 
> https://airflow.incubator.apache.org/tutorial.html
> (just changed dag-id to 'turorial-RD').
> See a 10 seconds video on this behavior:
> http://screencast-o-matic.com/watch/cbebrn6kBw
> Notice last DAG 'turorial-RD' keeps changing state 
> (it loses link and all the buttons and icons to show status 
>  of the dag runs  / tasks etc), then links and icons show back up.
> In fact, during that short 10 seconds period it transitioned 
> that state 4 times (from "disabled" to a normal DAG).
> There were no changes happenning in the system - I was just clicking
> refresh in browser from time to time. All DAGs were disabled while
> this was happening.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-659) Automatic Refresh on DAG Graph View

2018-02-22 Thread Luke Bodeen (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373460#comment-16373460
 ] 

Luke Bodeen commented on AIRFLOW-659:
-

agree on this, +1

> Automatic Refresh on DAG Graph View
> ---
>
> Key: AIRFLOW-659
> URL: https://issues.apache.org/jira/browse/AIRFLOW-659
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui, webserver
>Reporter: Robin Miller
>Assignee: Robin Miller
>Priority: Minor
>
> It would be nice not to have to press a button to refresh the graph view all 
> the time. There's no reason not to have the option to have this refresh 
> itself regularly so you can have it on a screen in the background and know 
> that it's reasonably up to date when you glance across at it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-1003) DAG status flips erraticly

2018-02-22 Thread Luke Bodeen (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373383#comment-16373383
 ] 

Luke Bodeen commented on AIRFLOW-1003:
--

how long is "awhile", 10 seconds or 2 minutes?

can turning max scheduler threads to 1 help this?

> DAG status flips erraticly
> --
>
> Key: AIRFLOW-1003
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1003
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 1.8, 1.8.0rc5
>Reporter: Ruslan Dautkhanov
>Priority: Major
>
> Created a flow based on sample tutorial 
> https://airflow.incubator.apache.org/tutorial.html
> (just changed dag-id to 'turorial-RD').
> See a 10 seconds video on this behavior:
> http://screencast-o-matic.com/watch/cbebrn6kBw
> Notice last DAG 'turorial-RD' keeps changing state 
> (it loses link and all the buttons and icons to show status 
>  of the dag runs  / tasks etc), then links and icons show back up.
> In fact, during that short 10 seconds period it transitioned 
> that state 4 times (from "disabled" to a normal DAG).
> There were no changes happenning in the system - I was just clicking
> refresh in browser from time to time. All DAGs were disabled while
> this was happening.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2143) Try number displays incorrect values in the web UI

2018-02-22 Thread James Davidheiser (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373277#comment-16373277
 ] 

James Davidheiser commented on AIRFLOW-2143:


I also confirmed that this affects error emails too - a task that fails on its 
first try starts the email out with `Try 2 out of 1`

> Try number displays incorrect values in the web UI
> --
>
> Key: AIRFLOW-2143
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2143
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: James Davidheiser
>Priority: Minor
> Attachments: adhoc_query.png, task_instance_page.png
>
>
> This was confusing us a lot in our task runs - in the database, a task that 
> ran is marked as 1 try.  However, when we view it in the UI, it shows at 2 
> tries in several places.  These include:
>  * Task Instance Details (ie 
> [https://airflow/task?execution_date=xxx_id=xxx_id=xxx 
> )|https://airflow/task?execution_date=xxx_id=xxx_id=xxx]
>  * Task instance browser (/admin/taskinstance/)
>  * Task Tries graph (/admin/airflow/tries)
> Notably, is is correctly shown as 1 try in the log filenames, on the log 
> viewer page (admin/airflow/log?execution_date=), and some other places.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (AIRFLOW-2143) Try number displays incorrect values in the web UI

2018-02-22 Thread James Davidheiser (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373271#comment-16373271
 ] 

James Davidheiser edited comment on AIRFLOW-2143 at 2/22/18 7:11 PM:
-

I am getting the correct log numbers, so 
https://issues.apache.org/jira/browse/AIRFLOW-1873 doesn't seem to be happening 
here.


was (Author: jdavidh):
I am getting the correct log numbers, so #1873 doesn't seem to be happening 
here.

> Try number displays incorrect values in the web UI
> --
>
> Key: AIRFLOW-2143
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2143
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: James Davidheiser
>Priority: Minor
> Attachments: adhoc_query.png, task_instance_page.png
>
>
> This was confusing us a lot in our task runs - in the database, a task that 
> ran is marked as 1 try.  However, when we view it in the UI, it shows at 2 
> tries in several places.  These include:
>  * Task Instance Details (ie 
> [https://airflow/task?execution_date=xxx_id=xxx_id=xxx 
> )|https://airflow/task?execution_date=xxx_id=xxx_id=xxx]
>  * Task instance browser (/admin/taskinstance/)
>  * Task Tries graph (/admin/airflow/tries)
> Notably, is is correctly shown as 1 try in the log filenames, on the log 
> viewer page (admin/airflow/log?execution_date=), and some other places.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2143) Try number displays incorrect values in the web UI

2018-02-22 Thread James Davidheiser (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373271#comment-16373271
 ] 

James Davidheiser commented on AIRFLOW-2143:


I am getting the correct log numbers, so #1873 doesn't seem to be happening 
here.

> Try number displays incorrect values in the web UI
> --
>
> Key: AIRFLOW-2143
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2143
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: James Davidheiser
>Priority: Minor
> Attachments: adhoc_query.png, task_instance_page.png
>
>
> This was confusing us a lot in our task runs - in the database, a task that 
> ran is marked as 1 try.  However, when we view it in the UI, it shows at 2 
> tries in several places.  These include:
>  * Task Instance Details (ie 
> [https://airflow/task?execution_date=xxx_id=xxx_id=xxx 
> )|https://airflow/task?execution_date=xxx_id=xxx_id=xxx]
>  * Task instance browser (/admin/taskinstance/)
>  * Task Tries graph (/admin/airflow/tries)
> Notably, is is correctly shown as 1 try in the log filenames, on the log 
> viewer page (admin/airflow/log?execution_date=), and some other places.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2143) Try number displays incorrect values in the web UI

2018-02-22 Thread James Davidheiser (JIRA) 
James Davidheiser created AIRFLOW-2143:
--

 Summary: Try number displays incorrect values in the web UI
 Key: AIRFLOW-2143
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2143
 Project: Apache Airflow
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: James Davidheiser
 Attachments: adhoc_query.png, task_instance_page.png

This was confusing us a lot in our task runs - in the database, a task that ran 
is marked as 1 try.  However, when we view it in the UI, it shows at 2 tries in 
several places.  These include:
 * Task Instance Details (ie 
[https://airflow/task?execution_date=xxx_id=xxx_id=xxx 
)|https://airflow/task?execution_date=xxx_id=xxx_id=xxx]
 * Task instance browser (/admin/taskinstance/)
 * Task Tries graph (/admin/airflow/tries)

Notably, is is correctly shown as 1 try in the log filenames, on the log viewer 
page (admin/airflow/log?execution_date=), and some other places.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2141) Cannot create airflow variables when there is a list of dictionary as a value

2018-02-22 Thread Soundar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Soundar updated AIRFLOW-2141:
-
Description: 
I'm trying to create Airflow variables using a json file. I am trying to import 
airflow variables using UI(webserver) when I upload the json file I get this 
error "Missing file or syntax error" and when I try to upload using airflow cli 
not all the variables gets uploaded properly. The catch is that I have a list 
of dictionary in my json file, say
 ex:
 {
 "demo_archivedir": "/home/ubuntu/folders/archive",
 "demo_filepattern": [

{ "id": "reference", "pattern": "Sample Data.xlsx" }

,

{ "id": "sale", "pattern": "Sales.xlsx" }

],
 "demo_sourcepath": "/home/ubuntu/folders/input",
 "demo_workdir": "/home/ubuntu/folders/working"
 }

I've attached two images

img1. Using airflow variables cli command I was able to create partial 
variables from my json file(airflow_cli.png)img2. After inserting logs in the 
"airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png)

The thing is I gave this value through the Admin UI one by one and it worked. 
Then I exported those same variable using "airflow variables" cli command and 
tried importing them, still it failed and the above mentioned error still 
occurs.

Note:
   I am using Python 3.5 with Airflow version 1.8

The stack trace is as follows

.compute-1.amazonaws.com:22] out: 0 of 4 variables successfully updated.
.compute-1.amazonaws.com:22] out: Traceback (most recent call last):
.compute-1.amazonaws.com:22] out:   File "/home/ubuntu/Env/bin/airflow", line 
28, in 
.compute-1.amazonaws.com:22] out: args.func(args)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 242, in 
variables
.compute-1.amazonaws.com:22] out: import_helper(imp)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 273, in 
import_helper
.compute-1.amazonaws.com:22] out: Variable.set(k, v)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/utils/db.py", line 53, in 
wrapper
.compute-1.amazonaws.com:22] out: result = func(*args, **kwargs)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3615, in 
set
.compute-1.amazonaws.com:22] out: session.add(Variable(key=key, 
val=stored_value))
.compute-1.amazonaws.com:22] out:   File "", line 4, in __init__
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 
417, in _initialize_instance
.compute-1.amazonaws.com:22] out: manager.dispatch.init_failure(self, args, 
kwargs)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", 
line 66, in __exit__
.compute-1.amazonaws.com:22] out: compat.reraise(exc_type, exc_value, 
exc_tb)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 
187, in reraise
.compute-1.amazonaws.com:22] out: raise value
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 
414, in _initialize_instance
.compute-1.amazonaws.com:22] out: return manager.original_init(*mixed[1:], 
**kwargs)
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/ext/declarative/base.py",
 line 700, in _declarative_constructor
.compute-1.amazonaws.com:22] out: setattr(self, k, kwargs[k])
compute-1.amazonaws.com:22] out:   File "", line 1, in __set__
.compute-1.amazonaws.com:22] out:   File 
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3550, in 
set_val
.compute-1.amazonaws.com:22] out: self._val = FERNET.encrypt(bytes(value, 
'utf-8')).decode()
.compute-1.amazonaws.com:22] out: TypeError: encoding without a string argument
.compute-1.amazonaws.com:22] out:

  was:
I'm trying to create Airflow variables using a json file. I am trying to import 
airflow variables using UI(webserver) when I upload the json file I get this 
error "Missing file or syntax error" and when I try to upload using airflow cli 
not all the variables gets uploaded properly. The catch is that I have a list 
of dictionary in my json file, say
 ex:
 {
 "demo_archivedir": "/home/ubuntu/folders/archive",
 "demo_filepattern": [

{ "id": "reference", "pattern": "Sample Data.xlsx" }

,

{ "id": "sale", "pattern": "Sales.xlsx" }

],
 "demo_sourcepath": "/home/ubuntu/folders/input",
 "demo_workdir": "/home/ubuntu/folders/working"
 }

I've attached two images 

img1. Using airflow variables cli command I was able to create partial 
variables from my json file(airflow_cli.png)img2. After inserting logs in the 
"airflow/bin/cli.py" file, I got this error.

[jira] [Created] (AIRFLOW-2141) Cannot create airflow variables when there is a list of dictionary as a value

2018-02-22 Thread Soundar (JIRA) 
Soundar created AIRFLOW-2141:


 Summary: Cannot create airflow variables when there is a list of 
dictionary as a value
 Key: AIRFLOW-2141
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2141
 Project: Apache Airflow
  Issue Type: Bug
  Components: aws
Affects Versions: Airflow 1.8
Reporter: Soundar
 Attachments: airflow_cli.png, airflow_cli2_crop.png

I'm trying to create Airflow variables using a json file. I am trying to import 
airflow variables using UI(webserver) when I upload the json file I get this 
error "Missing file or syntax error" and when I try to upload using airflow cli 
not all the variables gets uploaded properly. The catch is that I have a list 
of dictionary in my json file, say
ex:
{
 "demo_archivedir": "/home/ubuntu/folders/archive",
 "demo_filepattern": [
 {
 "id": "reference",
 "pattern": "Sample Data.xlsx"
 },
 {
 "id": "sale",
 "pattern": "Sales.xlsx"
 }
 ],
 "demo_sourcepath": "/home/ubuntu/folders/input",
 "demo_workdir": "/home/ubuntu/folders/working"
}

 

I've attached two images 
img1. Using airflow variables cli command I was able to create partial 
variables from my json file(airflow_cli.png)
img2. After inserting logs in the "airflow/bin/cli.py" file, I got this error. 
(airflow_cli2_crop.png)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2141) Cannot create airflow variables when there is a list of dictionary as a value

2018-02-22 Thread Soundar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Soundar updated AIRFLOW-2141:
-
Description: 
I'm trying to create Airflow variables using a json file. I am trying to import 
airflow variables using UI(webserver) when I upload the json file I get this 
error "Missing file or syntax error" and when I try to upload using airflow cli 
not all the variables gets uploaded properly. The catch is that I have a list 
of dictionary in my json file, say
 ex:
 {
 "demo_archivedir": "/home/ubuntu/folders/archive",
 "demo_filepattern": [

{ "id": "reference", "pattern": "Sample Data.xlsx" }

,

{ "id": "sale", "pattern": "Sales.xlsx" }

],
 "demo_sourcepath": "/home/ubuntu/folders/input",
 "demo_workdir": "/home/ubuntu/folders/working"
 }

I've attached two images 

img1. Using airflow variables cli command I was able to create partial 
variables from my json file(airflow_cli.png)img2. After inserting logs in the 
"airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png)

The thing is I gave this value through the Admin UI one by one and it worked. 
Then I exported those same variable using "airflow variables" cli command and 
tried importing them, still it failed and the above mentioned error still 
occurs.

Note:
  I am using Python 3.5 with Airflow version 1.8

  was:
I'm trying to create Airflow variables using a json file. I am trying to import 
airflow variables using UI(webserver) when I upload the json file I get this 
error "Missing file or syntax error" and when I try to upload using airflow cli 
not all the variables gets uploaded properly. The catch is that I have a list 
of dictionary in my json file, say
ex:
{
 "demo_archivedir": "/home/ubuntu/folders/archive",
 "demo_filepattern": [
 {
 "id": "reference",
 "pattern": "Sample Data.xlsx"
 },
 {
 "id": "sale",
 "pattern": "Sales.xlsx"
 }
 ],
 "demo_sourcepath": "/home/ubuntu/folders/input",
 "demo_workdir": "/home/ubuntu/folders/working"
}

 

I've attached two images 
img1. Using airflow variables cli command I was able to create partial 
variables from my json file(airflow_cli.png)
img2. After inserting logs in the "airflow/bin/cli.py" file, I got this error. 
(airflow_cli2_crop.png)


> Cannot create airflow variables when there is a list of dictionary as a value
> -
>
> Key: AIRFLOW-2141
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2141
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: aws
>Affects Versions: Airflow 1.8
>Reporter: Soundar
>Priority: Major
>  Labels: beginner, newbie
> Attachments: airflow_cli.png, airflow_cli2_crop.png
>
>
> I'm trying to create Airflow variables using a json file. I am trying to 
> import airflow variables using UI(webserver) when I upload the json file I 
> get this error "Missing file or syntax error" and when I try to upload using 
> airflow cli not all the variables gets uploaded properly. The catch is that I 
> have a list of dictionary in my json file, say
>  ex:
>  {
>  "demo_archivedir": "/home/ubuntu/folders/archive",
>  "demo_filepattern": [
> { "id": "reference", "pattern": "Sample Data.xlsx" }
> ,
> { "id": "sale", "pattern": "Sales.xlsx" }
> ],
>  "demo_sourcepath": "/home/ubuntu/folders/input",
>  "demo_workdir": "/home/ubuntu/folders/working"
>  }
> I've attached two images 
> img1. Using airflow variables cli command I was able to create partial 
> variables from my json file(airflow_cli.png)img2. After inserting logs in the 
> "airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png)
> The thing is I gave this value through the Admin UI one by one and it worked. 
> Then I exported those same variable using "airflow variables" cli command and 
> tried importing them, still it failed and the above mentioned error still 
> occurs.
> Note:
>   I am using Python 3.5 with Airflow version 1.8



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2142) Error messages are suppressed

2018-02-22 Thread William Pursell (JIRA) 
William Pursell created AIRFLOW-2142:


 Summary: Error messages are suppressed
 Key: AIRFLOW-2142
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2142
 Project: Apache Airflow
  Issue Type: Bug
Reporter: William Pursell
Assignee: William Pursell


Many fatal exceptions are caught and the text of the exception is replaced with 
less specific information, leading to increased difficulty in diagnosing 
errors.   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AIRFLOW-2137) Google Cloud Storage to BigQuery Operator to allow wildcards

2018-02-22 Thread Daniel (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel closed AIRFLOW-2137.
---
Resolution: Fixed

After careful investigation, the fix for this issue is already a feature. 
Closing.

> Google Cloud Storage to BigQuery Operator to allow wildcards
> 
>
> Key: AIRFLOW-2137
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2137
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, gcp, operators
>Affects Versions: 1.9.1
>Reporter: Daniel
>Assignee: Daniel
>Priority: Minor
>   Original Estimate: 5h
>  Remaining Estimate: 5h
>
> Currently, the GoogleCloudStorageToBigQuery Operator requires exact matches 
> for files that are to be uploaded to BigQuery. 
> This Issue proposes that, if a wildcard is provided, this is used to load all 
> files that match that wildcard expansion. The hook and cursor that this 
> Operator is based off already supports wildcards in the argument source_uris, 
> so the update is minor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2140) Add Kubernetes Scheduler to Spark Submit Operator

2018-02-22 Thread Rob Keevil (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rob Keevil updated AIRFLOW-2140:

Summary: Add Kubernetes Scheduler to Spark Submit Operator  (was: Add 
Kubernetes to Spark Submit Operator)

> Add Kubernetes Scheduler to Spark Submit Operator
> -
>
> Key: AIRFLOW-2140
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2140
> Project: Apache Airflow
>  Issue Type: New Feature
>Affects Versions: 1.9.0
>Reporter: Rob Keevil
>Assignee: Rob Keevil
>Priority: Major
>
> Spark 2.3 adds the Kubernetes resource manager to Spark, alongside the 
> existing Standalone, Yarn and Mesos resource managers. 
> https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md
> We should extend the spark submit operator to enable the new K8s spark submit 
> options, and to be able to monitor Spark jobs running within Kubernetes.
> I already have working code for this, I need to test the monitoring/log 
> parsing code and make sure that Airflow is able to terminate Kubernetes pods 
> when jobs are cancelled etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2140) Add Kubernetes to Spark Submit Operator

2018-02-22 Thread Rob Keevil (JIRA) 
Rob Keevil created AIRFLOW-2140:
---

 Summary: Add Kubernetes to Spark Submit Operator
 Key: AIRFLOW-2140
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2140
 Project: Apache Airflow
  Issue Type: New Feature
Affects Versions: 1.9.0
Reporter: Rob Keevil
Assignee: Rob Keevil


Spark 2.3 adds the Kubernetes resource manager to Spark, alongside the existing 
Standalone, Yarn and Mesos resource managers. 

https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md

We should extend the spark submit operator to enable the new K8s spark submit 
options, and to be able to monitor Spark jobs running within Kubernetes.

I already have working code for this, I need to test the monitoring/log parsing 
code and make sure that Airflow is able to terminate Kubernetes pods when jobs 
are cancelled etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2139) BigQueryHook: Don't use private methods of a third-party package

2018-02-22 Thread Matti Remes (JIRA) 
Matti Remes created AIRFLOW-2139:


 Summary: BigQueryHook: Don't use private methods of a third-party 
package
 Key: AIRFLOW-2139
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2139
 Project: Apache Airflow
  Issue Type: Improvement
  Components: hooks
Reporter: Matti Remes
Assignee: Matti Remes


Don't use private methods of pandas-gbq package as it brakes things when 
upgrading package version



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)