[jira] [Commented] (AIRFLOW-885) Add Change.org to the list of Airflow users

2017-02-17 Thread Vijay Krishna Ramesh (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872991#comment-15872991
 ] 

Vijay Krishna Ramesh commented on AIRFLOW-885:
--

https://github.com/apache/incubator-airflow/pull/2089

> Add Change.org to the list of Airflow users
> ---
>
> Key: AIRFLOW-885
> URL: https://issues.apache.org/jira/browse/AIRFLOW-885
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Arthur Wiedmer
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (AIRFLOW-885) Add Change.org to the list of Airflow users

2017-02-17 Thread Arthur Wiedmer (JIRA) 
Arthur Wiedmer created AIRFLOW-885:
--

 Summary: Add Change.org to the list of Airflow users
 Key: AIRFLOW-885
 URL: https://issues.apache.org/jira/browse/AIRFLOW-885
 Project: Apache Airflow
  Issue Type: Task
Reporter: Arthur Wiedmer






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-884) SlackAPIPostOperator works inconsistently across two DAGs / Python scripts.

2017-02-17 Thread Matt Inwood (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872499#comment-15872499
 ] 

Matt Inwood commented on AIRFLOW-884:
-

Full code of a test file where it's not completing downstream slack tasks. 

from airflow import DAG
from airflow.utils.trigger_rule import TriggerRule
from airflow.operators import PythonOperator
from airflow.operators.slack_operator import SlackAPIPostOperator
from datetime import datetime, timedelta
import pyodbc
from datetime import datetime
import os
import urllib.parse


default_args = {
'owner': 'analytics',
'depends_on_past': False,
#'start_date': sixty_days_ago,
'start_date': datetime(2017, 2, 16),
'retries': 0,
'retry_delay': timedelta(minutes=5),
}

dag = DAG(
'Valet_Data',
default_args=default_args,
schedule_interval='05 * * * *',
dagrun_timeout=timedelta(seconds=60))

def valet_function(locdata, ds, **kwargs):
print(locdata)

slack_success = SlackAPIPostOperator(
task_id='slack_success',

token='xoxp-59581512119-59573138118-125681286114-e24e7f4d73c8efefa50bdfcbd4b9c6f5',
channel='#airflow',
username='airflow',
text="job {{ dag }} finished at {{ ts }} successfully 8)",
dag=dag,
trigger_rule=TriggerRule.ALL_SUCCESS
)

slack_fail = SlackAPIPostOperator(
task_id='slack_fail',

token='xoxp-59581512119-59573138118-125681286114-e24e7f4d73c8efefa50bdfcbd4b9c6f5',
channel='#airflow',
username='airflow',
text="job '{{ dag }}' finished at {{ ts }} in disgrace >:(",
dag=dag,
trigger_rule=TriggerRule.ONE_FAILED
)

fac = {
'LAX_S': [123, 0, 'LOREM'],
'LAX_C': [234, 11, 'IPSUM'],
'IAH_JFK': [345, 22, 'LOREM'],
'STL1': [456, 33, 'IPSUM'],
'EWR_H': [678, 44, 'LOREM'],
'EWR_M': [789, 55, 'IPSUM'],
'JFK': [890, 66, 'LOREM'],
'LGA': [901, 77, 'IPSUM'],
'PHL1': [12, 88, 'LOREM'],
'BNA': [23, 99, 'IPSUM']
}

for loc in fac:
task = PythonOperator(
task_id='valet_{0}'.format(loc),
provide_context=True,
python_callable=valet_function,
op_kwargs={'locdata': fac[loc]},
dag=dag)
slack_success.set_upstream(task)
slack_fail.set_upstream(task)


> SlackAPIPostOperator works inconsistently across two DAGs / Python scripts.
> ---
>
> Key: AIRFLOW-884
> URL: https://issues.apache.org/jira/browse/AIRFLOW-884
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: api
>Affects Versions: Airflow 1.7.1.3
>Reporter: Matt Inwood
> Attachments: airflow_error.PNG
>
>
> Wrote a DAG file where the script executes and then sends a message to slack. 
> The code at the bottom is used to implement that.
> I used the same code in another DAG file. The upstream task completes 
> successfully, but the slack_success task does not. If I force an error into 
> my script, it does return the slack_fail task. 
> On success, both Slack operators indicate Operater Undefined, per attached 
> screenshot
> slack_success = SlackAPIPostOperator(
> task_id='slack_success',
> token=TOKEN,
> channel='#airflow',
> username='airflow',
> text="job {{ dag }} finished at {{ ts }} successfully 8)",
> dag=dag
> )
> slack_fail = SlackAPIPostOperator(
> task_id='slack_fail',
> token=TOKEN,
> channel='#airflow',
> username='airflow',
> text="job '{{ dag }}' finished at {{ ts }} in disgrace >:(",
> dag=dag,
> trigger_rule=TriggerRule.ONE_FAILED
> )
> slack_success.set_upstream(sample_task)
> slack_fail.set_upstream(sample_task)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-861) Pickle_info endpoint is unauthenticated

2017-02-17 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872403#comment-15872403
 ] 

ASF subversion and git services commented on AIRFLOW-861:
-

Commit 8d93c6b012114d534e9924d61ab926c2a691938e in incubator-airflow's branch 
refs/heads/master from [~saguziel]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8d93c6b ]

[AIRFLOW-861] make pickle_info endpoint be login_required

Testing Done:
- Unittests pass

Closes #2077 from saguziel/aguziel-fix-login-
required


> Pickle_info endpoint is unauthenticated
> ---
>
> Key: AIRFLOW-861
> URL: https://issues.apache.org/jira/browse/AIRFLOW-861
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Alex Guziel
>Assignee: Alex Guziel
>
> Right now the admin/airflow/pickle_info is unauthenticated, allowing anyone 
> to see the list of dags



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-861] make pickle_info endpoint be login_required

2017-02-17 Thread davydov 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 28cfd2c54 -> 8d93c6b01


[AIRFLOW-861] make pickle_info endpoint be login_required

Testing Done:
- Unittests pass

Closes #2077 from saguziel/aguziel-fix-login-
required


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8d93c6b0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8d93c6b0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8d93c6b0

Branch: refs/heads/master
Commit: 8d93c6b012114d534e9924d61ab926c2a691938e
Parents: 28cfd2c
Author: Alex Guziel 
Authored: Fri Feb 17 11:45:45 2017 -0800
Committer: Dan Davydov 
Committed: Fri Feb 17 11:45:48 2017 -0800

--
 airflow/www/views.py | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8d93c6b0/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 0391775..bda4921 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -640,6 +640,7 @@ class Airflow(BaseView):
 return wwwutils.json_response(d)
 
 @expose('/pickle_info')
+@login_required
 def pickle_info(self):
 d = {}
 dag_id = request.args.get('dag_id')

[jira] [Work started] (AIRFLOW-882) Code example in docs has unnecessary DAG>>Operator assignment

2017-02-17 Thread Daniel Huang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-882 started by Daniel Huang.

> Code example in docs has unnecessary DAG>>Operator assignment
> -
>
> Key: AIRFLOW-882
> URL: https://issues.apache.org/jira/browse/AIRFLOW-882
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: docs
>Reporter: Daniel Huang
>Assignee: Daniel Huang
>Priority: Trivial
>
> The docs currently say:
> {code}
> We can put this all together to build a simple pipeline:
> with DAG('my_dag', start_date=datetime(2016, 1, 1)) as dag:
> (
> dag
> >> DummyOperator(task_id='dummy_1')
> >> BashOperator(
> task_id='bash_1',
> bash_command='echo "HELLO!"')
> >> PythonOperator(
> task_id='python_1',
> python_callable=lambda: print("GOODBYE!"))
> )
> {code}
> But the {{dag >> ...}} is unnecessary because the operators are already 
> initialized with the proper DAG 
> (https://github.com/apache/incubator-airflow/blob/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6/airflow/models.py#L1699).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (AIRFLOW-884) SlackAPIPostOperator works inconsistently across two DAGs / Python scripts.

2017-02-17 Thread Matt Inwood (JIRA) 
Matt Inwood created AIRFLOW-884:
---

 Summary: SlackAPIPostOperator works inconsistently across two DAGs 
/ Python scripts.
 Key: AIRFLOW-884
 URL: https://issues.apache.org/jira/browse/AIRFLOW-884
 Project: Apache Airflow
  Issue Type: Bug
  Components: api
Affects Versions: Airflow 1.7.1.3
Reporter: Matt Inwood
 Attachments: airflow_error.PNG

Wrote a DAG file where the script executes and then sends a message to slack. 
The code at the bottom is used to implement that.

I used the same code in another DAG file. The upstream task completes 
successfully, but the slack_success task does not. If I force an error into my 
script, it does return the slack_fail task. 

On success, both Slack operators indicate Operater Undefined, per attached 
screenshot

slack_success = SlackAPIPostOperator(
task_id='slack_success',
token=TOKEN,
channel='#airflow',
username='airflow',
text="job {{ dag }} finished at {{ ts }} successfully 8)",
dag=dag
)

slack_fail = SlackAPIPostOperator(
task_id='slack_fail',
token=TOKEN,
channel='#airflow',
username='airflow',
text="job '{{ dag }}' finished at {{ ts }} in disgrace >:(",
dag=dag,
trigger_rule=TriggerRule.ONE_FAILED
)

slack_success.set_upstream(sample_task)
slack_fail.set_upstream(sample_task)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-847) Xcoms are not passed into SubDAG

2017-02-17 Thread Robin B (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871591#comment-15871591
 ] 

Robin B commented on AIRFLOW-847:
-

Many thanks [~dxhuang]!

> Xcoms are not passed into SubDAG
> 
>
> Key: AIRFLOW-847
> URL: https://issues.apache.org/jira/browse/AIRFLOW-847
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: subdag, xcom
>Reporter: Robin B
>Priority: Blocker
>
> It's not possible to do a xcom_pull within a subdag
> None of the following seems to be working:
> * As templated var in SubDagoperator
> * As var in SubDagoperator
> * From within Subdag-factory



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (AIRFLOW-847) Xcoms are not passed into SubDAG

2017-02-17 Thread Robin B (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robin B closed AIRFLOW-847.
---
Resolution: Not A Problem

> Xcoms are not passed into SubDAG
> 
>
> Key: AIRFLOW-847
> URL: https://issues.apache.org/jira/browse/AIRFLOW-847
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: subdag, xcom
>Reporter: Robin B
>Priority: Blocker
>
> It's not possible to do a xcom_pull within a subdag
> None of the following seems to be working:
> * As templated var in SubDagoperator
> * As var in SubDagoperator
> * From within Subdag-factory



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

svn commit: r18377 - in /dev/incubator/airflow: airflow-1.8.0rc4+apache.incubating.tar.gz airflow-1.8.0rc4+apache.incubating.tar.gz.asc airflow-1.8.0rc4+apache.incubating.tar.gz.md5 airflow-1.8.0rc4+a

2017-02-17 Thread bolke 
Author: bolke
Date: Fri Feb 17 08:19:59 2017
New Revision: 18377

Log:
Add rc4 of Airflow 1.8.0

Added:
dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz   (with 
props)
dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.md5
dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.sha

Added: dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz
==
Binary file - no diff available.

Propchange: dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.asc
==
--- dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.asc (added)
+++ dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.asc Fri Feb 
17 08:19:59 2017
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIcBAABCgAGBQJYprHtAAoJEDUZC4PZBaC62i8P/3fRGjaFv7kHTWinrHfNq07s
+cf73SVzvDsRcK8VuYeo33/CuaLGg8DzRBpKE/rUQ6egg0Q6kUyE0B4Vrvha4Uonx
+cqE+SIXvW51yZ7drhN8SgjZswvVz0ai1GeeSvqbs7lyw30WwHt9gs06/ST65NdZC
+eUGwKo/beLDiFCfCYG/2g5+RijGsfiNpY/NHjqPGupUl0n8Bp71ZbXkR5wIrvpm4
+pXshXeK8Rgbu02KbtbqipT6bwiUFf2NEPpApPlgBfOO3EkXD2nnXCweK6dRR3zq/
+mPNSRLPz51JelMsMof7ENZYdc5e7nSvTAPURdxMre/o7wa7EFcorPBULl2Nf0Pzo
++the88bx1yZryK9dAmhJcH1rHcblGDYLsz5XvmF41nQdLrhL1lOc/1E6LFozY0cB
+Oig356rswNS4tnHXCNuH2akfEbg9oRcRfOGlht8Rux6FCyjqp7LUTBLsdXJZo4wl
+rBGF29pkrjpxpVsrgA3mBEco33HjeOc7HJ1nujkeTfXJDRAvRs1k3yx2I0gERin2
+acBF5SAvuv4CX1jeQnxgwuKkL4H1CGckWVjalhKAuGX/778IJMh1MrdsmEQYHwDP
+d+OJYkZcsGqDbMdhx/yFVEP3fvaJbs6DV89K+sIvgyjJow8LdyyvnzC8gJ/nxsuF
+uMsp+w4KnIbqunRvX5v3
+=0tul
+-END PGP SIGNATURE-

Added: dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.md5
==
--- dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.md5 (added)
+++ dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.md5 Fri Feb 
17 08:19:59 2017
@@ -0,0 +1,2 @@
+airflow-1.8.0rc4+apache.incubating.tar.gz: 
+FA BD 34 18 47 AF 4F 93  0D 91 2F 1D D7 29 63 CD

Added: dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.sha
==
--- dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.sha (added)
+++ dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.sha Fri Feb 
17 08:19:59 2017
@@ -0,0 +1,3 @@
+airflow-1.8.0rc4+apache.incubating.tar.gz: 
+3CB93385 B480F8D1 91564EE7 5CD446B3 A3D36CBA EDC1F37E BFF04B90 B6D7854C 
D1213887
+ 571D40AB 4A6069F4 BE0835A9 9A71EBA8 57C4ADDA 013512E2 3CB95798

incubator-airflow git commit: Bump version to 1.8.0rc4

2017-02-17 Thread bolke 
Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-8-stable d31d2dbff -> 310fb589a


Bump version to 1.8.0rc4


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/310fb589
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/310fb589
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/310fb589

Branch: refs/heads/v1-8-stable
Commit: 310fb589ae867ff2ec8b7ce3cc5b1659db4dad49
Parents: d31d2db
Author: Bolke de Bruin 
Authored: Fri Feb 17 09:17:47 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Feb 17 09:17:47 2017 +0100

--
 airflow/version.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/310fb589/airflow/version.py
--
diff --git a/airflow/version.py b/airflow/version.py
index 376f182..c280ed0 100644
--- a/airflow/version.py
+++ b/airflow/version.py
@@ -13,4 +13,4 @@
 # limitations under the License.
 #
 
-version = '1.8.0rc3+apache.incubating'
+version = '1.8.0rc4+apache.incubating'

[jira] [Commented] (AIRFLOW-869) Web UI Mark Success Upstream Option Bug

2017-02-17 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871392#comment-15871392
 ] 

ASF subversion and git services commented on AIRFLOW-869:
-

Commit d31d2dbff505b12446f4645f1a17cc140bef7ab9 in incubator-airflow's branch 
refs/heads/v1-8-stable from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d31d2db ]

[AIRFLOW-869] Refactor mark success functionality

This refactors the mark success functionality in a
more generic function that can set multiple states
and properly drills down on SubDags.

Closes #2085 from bolkedebruin/AIRFLOW-869

(cherry picked from commit 28cfd2c541c12468b3e4f634545dfa31a77b0091)
Signed-off-by: Bolke de Bruin 
(cherry picked from commit 563cc9a3c8414725a615a93d3910e7a2dbb94999)
Signed-off-by: Bolke de Bruin 


> Web UI Mark Success Upstream Option Bug
> ---
>
> Key: AIRFLOW-869
> URL: https://issues.apache.org/jira/browse/AIRFLOW-869
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.8
>Reporter: Yi Chen
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> A simple bug report: I tracked down to the source code of Airflow Web UI, 
> look at this line, 
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/www/views.py#L1127
>  .  It should be `relatives = task.get_flat_relatives(upstream=True)`. But 
> even with this fix, there are still issues about the "Mark Success" 
> functionality. I hope we ship this bug fix along with v1.8. And I will open 
> another ticket discussing the functionality of "Mark Success".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-869) Web UI Mark Success Upstream Option Bug

2017-02-17 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871393#comment-15871393
 ] 

ASF subversion and git services commented on AIRFLOW-869:
-

Commit d31d2dbff505b12446f4645f1a17cc140bef7ab9 in incubator-airflow's branch 
refs/heads/v1-8-stable from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d31d2db ]

[AIRFLOW-869] Refactor mark success functionality

This refactors the mark success functionality in a
more generic function that can set multiple states
and properly drills down on SubDags.

Closes #2085 from bolkedebruin/AIRFLOW-869

(cherry picked from commit 28cfd2c541c12468b3e4f634545dfa31a77b0091)
Signed-off-by: Bolke de Bruin 
(cherry picked from commit 563cc9a3c8414725a615a93d3910e7a2dbb94999)
Signed-off-by: Bolke de Bruin 


> Web UI Mark Success Upstream Option Bug
> ---
>
> Key: AIRFLOW-869
> URL: https://issues.apache.org/jira/browse/AIRFLOW-869
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.8
>Reporter: Yi Chen
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> A simple bug report: I tracked down to the source code of Airflow Web UI, 
> look at this line, 
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/www/views.py#L1127
>  .  It should be `relatives = task.get_flat_relatives(upstream=True)`. But 
> even with this fix, there are still issues about the "Mark Success" 
> functionality. I hope we ship this bug fix along with v1.8. And I will open 
> another ticket discussing the functionality of "Mark Success".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-869) Web UI Mark Success Upstream Option Bug

2017-02-17 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871389#comment-15871389
 ] 

ASF subversion and git services commented on AIRFLOW-869:
-

Commit 563cc9a3c8414725a615a93d3910e7a2dbb94999 in incubator-airflow's branch 
refs/heads/v1-8-test from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=563cc9a ]

[AIRFLOW-869] Refactor mark success functionality

This refactors the mark success functionality in a
more generic function that can set multiple states
and properly drills down on SubDags.

Closes #2085 from bolkedebruin/AIRFLOW-869

(cherry picked from commit 28cfd2c541c12468b3e4f634545dfa31a77b0091)
Signed-off-by: Bolke de Bruin 


> Web UI Mark Success Upstream Option Bug
> ---
>
> Key: AIRFLOW-869
> URL: https://issues.apache.org/jira/browse/AIRFLOW-869
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.8
>Reporter: Yi Chen
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> A simple bug report: I tracked down to the source code of Airflow Web UI, 
> look at this line, 
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/www/views.py#L1127
>  .  It should be `relatives = task.get_flat_relatives(upstream=True)`. But 
> even with this fix, there are still issues about the "Mark Success" 
> functionality. I hope we ship this bug fix along with v1.8. And I will open 
> another ticket discussing the functionality of "Mark Success".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-869] Refactor mark success functionality

2017-02-17 Thread bolke 
Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-8-stable e0ec5a303 -> d31d2dbff


[AIRFLOW-869] Refactor mark success functionality

This refactors the mark success functionality in a
more generic function that can set multiple states
and properly drills down on SubDags.

Closes #2085 from bolkedebruin/AIRFLOW-869

(cherry picked from commit 28cfd2c541c12468b3e4f634545dfa31a77b0091)
Signed-off-by: Bolke de Bruin 
(cherry picked from commit 563cc9a3c8414725a615a93d3910e7a2dbb94999)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d31d2dbf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d31d2dbf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d31d2dbf

Branch: refs/heads/v1-8-stable
Commit: d31d2dbff505b12446f4645f1a17cc140bef7ab9
Parents: e0ec5a3
Author: Bolke de Bruin 
Authored: Fri Feb 17 09:05:41 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Feb 17 09:12:37 2017 +0100

--
 airflow/api/common/experimental/mark_tasks.py | 187 ++
 airflow/jobs.py   |   4 +-
 airflow/models.py |  18 +-
 airflow/www/templates/airflow/dag.html|   5 -
 airflow/www/views.py  | 119 +++-
 tests/api/__init__.py |   2 +
 tests/api/common/__init__.py  |  13 ++
 tests/api/common/mark_tasks.py| 211 +
 tests/core.py |  46 +++--
 tests/dags/test_example_bash_operator.py  |  55 ++
 tests/models.py   |   2 +-
 11 files changed, 536 insertions(+), 126 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/d31d2dbf/airflow/api/common/experimental/mark_tasks.py
--
diff --git a/airflow/api/common/experimental/mark_tasks.py 
b/airflow/api/common/experimental/mark_tasks.py
new file mode 100644
index 000..0ddbf98
--- /dev/null
+++ b/airflow/api/common/experimental/mark_tasks.py
@@ -0,0 +1,187 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+
+from airflow.jobs import BackfillJob
+from airflow.models import DagRun, TaskInstance
+from airflow.operators.subdag_operator import SubDagOperator
+from airflow.settings import Session
+from airflow.utils.state import State
+
+from sqlalchemy import or_
+
+
+def _create_dagruns(dag, execution_dates, state, run_id_template):
+"""
+Infers from the dates which dag runs need to be created and does so.
+:param dag: the dag to create dag runs for
+:param execution_dates: list of execution dates to evaluate
+:param state: the state to set the dag run to
+:param run_id_template:the template for run id to be with the execution 
date
+:return: newly created and existing dag runs for the execution dates 
supplied
+"""
+# find out if we need to create any dag runs
+drs = DagRun.find(dag_id=dag.dag_id, execution_date=execution_dates)
+dates_to_create = list(set(execution_dates) - set([dr.execution_date for 
dr in drs]))
+
+for date in dates_to_create:
+dr = dag.create_dagrun(
+run_id=run_id_template.format(date.isoformat()),
+execution_date=date,
+start_date=datetime.datetime.now(),
+external_trigger=False,
+state=state,
+)
+drs.append(dr)
+
+return drs
+
+
+def set_state(task, execution_date, upstream=False, downstream=False,
+  future=False, past=False, state=State.SUCCESS, commit=False):
+"""
+Set the state of a task instance and if needed its relatives. Can set state
+for future tasks (calculated from execution_date) and retroactively
+for past tasks. Will verify integrity of past dag runs in order to create
+tasks that did not exist. It will not create dag runs that are missing
+on the schedule (but it will as for subdag dag runs if needed).
+:param task: the task from which to work. task.task.dag needs to be set
+:param execution_date: the execution date from

[jira] [Commented] (AIRFLOW-869) Web UI Mark Success Upstream Option Bug

2017-02-17 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871390#comment-15871390
 ] 

ASF subversion and git services commented on AIRFLOW-869:
-

Commit 563cc9a3c8414725a615a93d3910e7a2dbb94999 in incubator-airflow's branch 
refs/heads/v1-8-test from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=563cc9a ]

[AIRFLOW-869] Refactor mark success functionality

This refactors the mark success functionality in a
more generic function that can set multiple states
and properly drills down on SubDags.

Closes #2085 from bolkedebruin/AIRFLOW-869

(cherry picked from commit 28cfd2c541c12468b3e4f634545dfa31a77b0091)
Signed-off-by: Bolke de Bruin 


> Web UI Mark Success Upstream Option Bug
> ---
>
> Key: AIRFLOW-869
> URL: https://issues.apache.org/jira/browse/AIRFLOW-869
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.8
>Reporter: Yi Chen
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> A simple bug report: I tracked down to the source code of Airflow Web UI, 
> look at this line, 
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/www/views.py#L1127
>  .  It should be `relatives = task.get_flat_relatives(upstream=True)`. But 
> even with this fix, there are still issues about the "Mark Success" 
> functionality. I hope we ship this bug fix along with v1.8. And I will open 
> another ticket discussing the functionality of "Mark Success".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-869] Refactor mark success functionality

2017-02-17 Thread bolke 
Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-8-test eddecd59d -> 563cc9a3c


[AIRFLOW-869] Refactor mark success functionality

This refactors the mark success functionality in a
more generic function that can set multiple states
and properly drills down on SubDags.

Closes #2085 from bolkedebruin/AIRFLOW-869

(cherry picked from commit 28cfd2c541c12468b3e4f634545dfa31a77b0091)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/563cc9a3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/563cc9a3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/563cc9a3

Branch: refs/heads/v1-8-test
Commit: 563cc9a3c8414725a615a93d3910e7a2dbb94999
Parents: eddecd5
Author: Bolke de Bruin 
Authored: Fri Feb 17 09:05:41 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Feb 17 09:11:41 2017 +0100

--
 airflow/api/common/experimental/mark_tasks.py | 187 ++
 airflow/jobs.py   |   4 +-
 airflow/models.py |  18 +-
 airflow/www/templates/airflow/dag.html|   5 -
 airflow/www/views.py  | 119 +++-
 tests/api/__init__.py |   2 +
 tests/api/common/__init__.py  |  13 ++
 tests/api/common/mark_tasks.py| 211 +
 tests/core.py |  46 +++--
 tests/dags/test_example_bash_operator.py  |  55 ++
 tests/models.py   |   2 +-
 11 files changed, 536 insertions(+), 126 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/563cc9a3/airflow/api/common/experimental/mark_tasks.py
--
diff --git a/airflow/api/common/experimental/mark_tasks.py 
b/airflow/api/common/experimental/mark_tasks.py
new file mode 100644
index 000..0ddbf98
--- /dev/null
+++ b/airflow/api/common/experimental/mark_tasks.py
@@ -0,0 +1,187 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+
+from airflow.jobs import BackfillJob
+from airflow.models import DagRun, TaskInstance
+from airflow.operators.subdag_operator import SubDagOperator
+from airflow.settings import Session
+from airflow.utils.state import State
+
+from sqlalchemy import or_
+
+
+def _create_dagruns(dag, execution_dates, state, run_id_template):
+"""
+Infers from the dates which dag runs need to be created and does so.
+:param dag: the dag to create dag runs for
+:param execution_dates: list of execution dates to evaluate
+:param state: the state to set the dag run to
+:param run_id_template:the template for run id to be with the execution 
date
+:return: newly created and existing dag runs for the execution dates 
supplied
+"""
+# find out if we need to create any dag runs
+drs = DagRun.find(dag_id=dag.dag_id, execution_date=execution_dates)
+dates_to_create = list(set(execution_dates) - set([dr.execution_date for 
dr in drs]))
+
+for date in dates_to_create:
+dr = dag.create_dagrun(
+run_id=run_id_template.format(date.isoformat()),
+execution_date=date,
+start_date=datetime.datetime.now(),
+external_trigger=False,
+state=state,
+)
+drs.append(dr)
+
+return drs
+
+
+def set_state(task, execution_date, upstream=False, downstream=False,
+  future=False, past=False, state=State.SUCCESS, commit=False):
+"""
+Set the state of a task instance and if needed its relatives. Can set state
+for future tasks (calculated from execution_date) and retroactively
+for past tasks. Will verify integrity of past dag runs in order to create
+tasks that did not exist. It will not create dag runs that are missing
+on the schedule (but it will as for subdag dag runs if needed).
+:param task: the task from which to work. task.task.dag needs to be set
+:param execution_date: the execution date from which to start looking
+:param upstream: Mark all parents (upstream tasks)
+:param downstream: Mark all siblings

[jira] [Commented] (AIRFLOW-869) Web UI Mark Success Upstream Option Bug

2017-02-17 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871383#comment-15871383
 ] 

ASF subversion and git services commented on AIRFLOW-869:
-

Commit 28cfd2c541c12468b3e4f634545dfa31a77b0091 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=28cfd2c ]

[AIRFLOW-869] Refactor mark success functionality

This refactors the mark success functionality in a
more generic function that can set multiple states
and
properly drills down on SubDags.

Closes #2085 from bolkedebruin/AIRFLOW-869


> Web UI Mark Success Upstream Option Bug
> ---
>
> Key: AIRFLOW-869
> URL: https://issues.apache.org/jira/browse/AIRFLOW-869
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.8
>Reporter: Yi Chen
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> A simple bug report: I tracked down to the source code of Airflow Web UI, 
> look at this line, 
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/www/views.py#L1127
>  .  It should be `relatives = task.get_flat_relatives(upstream=True)`. But 
> even with this fix, there are still issues about the "Mark Success" 
> functionality. I hope we ship this bug fix along with v1.8. And I will open 
> another ticket discussing the functionality of "Mark Success".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-869) Web UI Mark Success Upstream Option Bug

2017-02-17 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871382#comment-15871382
 ] 

ASF subversion and git services commented on AIRFLOW-869:
-

Commit 28cfd2c541c12468b3e4f634545dfa31a77b0091 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=28cfd2c ]

[AIRFLOW-869] Refactor mark success functionality

This refactors the mark success functionality in a
more generic function that can set multiple states
and
properly drills down on SubDags.

Closes #2085 from bolkedebruin/AIRFLOW-869


> Web UI Mark Success Upstream Option Bug
> ---
>
> Key: AIRFLOW-869
> URL: https://issues.apache.org/jira/browse/AIRFLOW-869
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.8
>Reporter: Yi Chen
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> A simple bug report: I tracked down to the source code of Airflow Web UI, 
> look at this line, 
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/www/views.py#L1127
>  .  It should be `relatives = task.get_flat_relatives(upstream=True)`. But 
> even with this fix, there are still issues about the "Mark Success" 
> functionality. I hope we ship this bug fix along with v1.8. And I will open 
> another ticket discussing the functionality of "Mark Success".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-869] Refactor mark success functionality

2017-02-17 Thread bolke 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master debc69e27 -> 28cfd2c54


[AIRFLOW-869] Refactor mark success functionality

This refactors the mark success functionality in a
more generic function that can set multiple states
and
properly drills down on SubDags.

Closes #2085 from bolkedebruin/AIRFLOW-869


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/28cfd2c5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/28cfd2c5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/28cfd2c5

Branch: refs/heads/master
Commit: 28cfd2c541c12468b3e4f634545dfa31a77b0091
Parents: debc69e
Author: Bolke de Bruin 
Authored: Fri Feb 17 09:05:41 2017 +0100
Committer: Bolke de Bruin 
Committed: Fri Feb 17 09:05:41 2017 +0100

--
 airflow/api/common/experimental/mark_tasks.py | 187 ++
 airflow/jobs.py   |   4 +-
 airflow/models.py |  18 +-
 airflow/www/templates/airflow/dag.html|   5 -
 airflow/www/views.py  | 119 +++-
 tests/api/__init__.py |   2 +
 tests/api/common/__init__.py  |  13 ++
 tests/api/common/mark_tasks.py| 211 +
 tests/core.py |  46 +++--
 tests/dags/test_example_bash_operator.py  |  55 ++
 tests/models.py   |   2 +-
 11 files changed, 536 insertions(+), 126 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/28cfd2c5/airflow/api/common/experimental/mark_tasks.py
--
diff --git a/airflow/api/common/experimental/mark_tasks.py 
b/airflow/api/common/experimental/mark_tasks.py
new file mode 100644
index 000..0ddbf98
--- /dev/null
+++ b/airflow/api/common/experimental/mark_tasks.py
@@ -0,0 +1,187 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+
+from airflow.jobs import BackfillJob
+from airflow.models import DagRun, TaskInstance
+from airflow.operators.subdag_operator import SubDagOperator
+from airflow.settings import Session
+from airflow.utils.state import State
+
+from sqlalchemy import or_
+
+
+def _create_dagruns(dag, execution_dates, state, run_id_template):
+"""
+Infers from the dates which dag runs need to be created and does so.
+:param dag: the dag to create dag runs for
+:param execution_dates: list of execution dates to evaluate
+:param state: the state to set the dag run to
+:param run_id_template:the template for run id to be with the execution 
date
+:return: newly created and existing dag runs for the execution dates 
supplied
+"""
+# find out if we need to create any dag runs
+drs = DagRun.find(dag_id=dag.dag_id, execution_date=execution_dates)
+dates_to_create = list(set(execution_dates) - set([dr.execution_date for 
dr in drs]))
+
+for date in dates_to_create:
+dr = dag.create_dagrun(
+run_id=run_id_template.format(date.isoformat()),
+execution_date=date,
+start_date=datetime.datetime.now(),
+external_trigger=False,
+state=state,
+)
+drs.append(dr)
+
+return drs
+
+
+def set_state(task, execution_date, upstream=False, downstream=False,
+  future=False, past=False, state=State.SUCCESS, commit=False):
+"""
+Set the state of a task instance and if needed its relatives. Can set state
+for future tasks (calculated from execution_date) and retroactively
+for past tasks. Will verify integrity of past dag runs in order to create
+tasks that did not exist. It will not create dag runs that are missing
+on the schedule (but it will as for subdag dag runs if needed).
+:param task: the task from which to work. task.task.dag needs to be set
+:param execution_date: the execution date from which to start looking
+:param upstream: Mark all parents (upstream tasks)
+:param downstream: Mark all siblings (downstream tasks) of task_id, 
including SubDags
+:param future: Mark all future tasks on the interval of the dag up