[jira] [Updated] (AIRFLOW-1844) task would not be executed when celery broker recovery

[hujiahua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hujiahua updated AIRFLOW-1844:
--
Description: When the scheduler fail to send task during celery broker not 
working, then the task will never send again and never be executed when celery 
broker recovery.  (was: when the scheduler failed to send task  during celery 
broker not working, then the task will never send again and never be executed.)

> task would not be executed when celery broker recovery
> --
>
> Key: AIRFLOW-1844
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1844
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery, executor, scheduler
>Affects Versions: 1.8.0, 1.9.0
>Reporter: hujiahua
>
> When the scheduler fail to send task during celery broker not working, then 
> the task will never send again and never be executed when celery broker 
> recovery.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1844) task would not be executed when celery broker recovery

[hujiahua (JIRA)]

hujiahua created AIRFLOW-1844:
-

 Summary: task would not be executed when celery broker recovery
 Key: AIRFLOW-1844
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1844
 Project: Apache Airflow
  Issue Type: Bug
  Components: celery, executor, scheduler
Affects Versions: 1.8.0, 1.9.0
Reporter: hujiahua


when the scheduler failed to send task  during celery broker not working, then 
the task will never send again and never be executed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (AIRFLOW-1455) Move logging related configs out of airflow.cfg

[Igors Vaitkus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Igors Vaitkus reassigned AIRFLOW-1455:
--

Assignee: Igors Vaitkus

> Move logging related configs out of airflow.cfg
> ---
>
> Key: AIRFLOW-1455
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1455
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Allison Wang
>Assignee: Igors Vaitkus
>
> All logging related configurations including {{LOG_BASE_FOLDER}}, 
> {{REMOTE_LOG_BASE_FOLDER}}, {{LOG_LEVEL}} and {{LOG_FORMAT}} should be placed 
> inside {{default_airflow_logging}}. This task also includes refactoring all 
> occurrence of those variables and make them handler specific rather than 
> global. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (AIRFLOW-1842) Create gcs to gcs operator to copy, rename and both

[Igors Vaitkus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Igors Vaitkus closed AIRFLOW-1842.
--

> Create gcs to gcs operator to copy, rename and both
> ---
>
> Key: AIRFLOW-1842
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1842
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, gcp, operators
>Reporter: Igors Vaitkus
>Assignee: Igors Vaitkus
>Priority: Minor
>  Labels: features
>
> Copies an object from a Google Cloud Storage bucket to another, with renaming 
> if requested.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1842) Create gcs to gcs operator to copy, rename and both

[Igors Vaitkus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Igors Vaitkus resolved AIRFLOW-1842.

Resolution: Resolved

> Create gcs to gcs operator to copy, rename and both
> ---
>
> Key: AIRFLOW-1842
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1842
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, gcp, operators
>Reporter: Igors Vaitkus
>Assignee: Igors Vaitkus
>Priority: Minor
>  Labels: features
>
> Copies an object from a Google Cloud Storage bucket to another, with renaming 
> if requested.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (AIRFLOW-1843) Add Google Cloud Storage Sensor with prefix

[Igors Vaitkus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-1843 started by Igors Vaitkus.
--
> Add Google Cloud Storage Sensor with prefix
> ---
>
> Key: AIRFLOW-1843
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1843
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, gcp
>Affects Versions: 1.8.1, 1.9.1
>Reporter: Igors Vaitkus
>Assignee: Igors Vaitkus
>Priority: Minor
>
> Hook can do list objects in bucket with prefix so I need sensor which will 
> check bucket with prefix if there any incoming files.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (AIRFLOW-1841) GoogleCloudStorageDownloadOperator filename parameter

[Igors Vaitkus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Igors Vaitkus closed AIRFLOW-1841.
--

> GoogleCloudStorageDownloadOperator filename parameter
> -
>
> Key: AIRFLOW-1841
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1841
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, gcp, operators
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Igors Vaitkus
>Assignee: Igors Vaitkus
>Priority: Trivial
>  Labels: documentation
>
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/contrib/operators/gcs_download_operator.py
> Documentation - "... If false, the downloaded data will not be stored on the 
> local file system."
> But filename type is a string in the documentation, it should be a union of 
> String and Bool, but better change it to None. And change Hook as well to 
> None because all it does just check the presence of a value in the parameter.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1841) GoogleCloudStorageDownloadOperator filename parameter

[Igors Vaitkus (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Igors Vaitkus resolved AIRFLOW-1841.

Resolution: Resolved

> GoogleCloudStorageDownloadOperator filename parameter
> -
>
> Key: AIRFLOW-1841
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1841
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, gcp, operators
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Igors Vaitkus
>Assignee: Igors Vaitkus
>Priority: Trivial
>  Labels: documentation
>
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/contrib/operators/gcs_download_operator.py
> Documentation - "... If false, the downloaded data will not be stored on the 
> local file system."
> But filename type is a string in the documentation, it should be a union of 
> String and Bool, but better change it to None. And change Hook as well to 
> None because all it does just check the presence of a value in the parameter.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1843) Add Google Cloud Storage Sensor with prefix

[Igors Vaitkus (JIRA)]

Igors Vaitkus created AIRFLOW-1843:
--

 Summary: Add Google Cloud Storage Sensor with prefix
 Key: AIRFLOW-1843
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1843
 Project: Apache Airflow
  Issue Type: Improvement
  Components: contrib, gcp
Affects Versions: 1.8.1, 1.9.1
Reporter: Igors Vaitkus
Assignee: Igors Vaitkus
Priority: Minor


Hook can do list objects in bucket with prefix so I need sensor which will 
check bucket with prefix if there any incoming files.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1229) Make "Run Id" column clickable in Browse -> DAG Runs

[Erik Cederstrand (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264101#comment-16264101
 ] 

Erik Cederstrand commented on AIRFLOW-1229:
---

Thanks for taking the time to get this in, [~abij]!

> Make "Run Id" column clickable in Browse -> DAG Runs
> 
>
> Key: AIRFLOW-1229
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1229
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webapp
>Affects Versions: Airflow 1.8
> Environment: Python3.4
>Reporter: Erik Cederstrand
>Assignee: Alexander Bij
>  Labels: patch
> Attachments: dag_run_link.patch
>
>
> I'm triggering a lot of DAGs manually using "airflow trigger_dag my_dag 
> --run_id=some_unique_id". I would like to be able in the UI to browse easily 
> to this specific DAG run using the "some_unique_id" label. In the graph page 
> of the DAG, I need to know the exact execution date, which is inconvenient, 
> and in the Browse -> DAG Runs page I can search by "some_unique_id", but the 
> "Run Ids" column is not clickable.
> The attached patch makes the aforementioned column clickable, so I'm sent 
> directly to the graph view for that specific DAG run, not the DAG in general.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1229) Make "Run Id" column clickable in Browse -> DAG Runs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264069#comment-16264069
 ] 

ASF subversion and git services commented on AIRFLOW-1229:
--

Commit d76bf76de2d2d4d3041e4d1bf9304dce8a1bcefb in incubator-airflow's branch 
refs/heads/master from [~abij]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d76bf76 ]

[AIRFLOW-1229] Add link to Run Id, incl execution_date

Add two UI improvement. 1: the links from "DAG
runs" to DAG graph view
include the execution_date. So you land on the
expected DAG, instead of
the last DAG run. 2: A new link is added for the
column "Run Id" with the
same behaviour.

Closes #2801 from abij/AIRFLOW-1229


> Make "Run Id" column clickable in Browse -> DAG Runs
> 
>
> Key: AIRFLOW-1229
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1229
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webapp
>Affects Versions: Airflow 1.8
> Environment: Python3.4
>Reporter: Erik Cederstrand
>Assignee: Alexander Bij
>  Labels: patch
> Attachments: dag_run_link.patch
>
>
> I'm triggering a lot of DAGs manually using "airflow trigger_dag my_dag 
> --run_id=some_unique_id". I would like to be able in the UI to browse easily 
> to this specific DAG run using the "some_unique_id" label. In the graph page 
> of the DAG, I need to know the exact execution date, which is inconvenient, 
> and in the Browse -> DAG Runs page I can search by "some_unique_id", but the 
> "Run Ids" column is not clickable.
> The attached patch makes the aforementioned column clickable, so I'm sent 
> directly to the graph view for that specific DAG run, not the DAG in general.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1229) Make "Run Id" column clickable in Browse -> DAG Runs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264068#comment-16264068
 ] 

ASF subversion and git services commented on AIRFLOW-1229:
--

Commit d76bf76de2d2d4d3041e4d1bf9304dce8a1bcefb in incubator-airflow's branch 
refs/heads/master from [~abij]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d76bf76 ]

[AIRFLOW-1229] Add link to Run Id, incl execution_date

Add two UI improvement. 1: the links from "DAG
runs" to DAG graph view
include the execution_date. So you land on the
expected DAG, instead of
the last DAG run. 2: A new link is added for the
column "Run Id" with the
same behaviour.

Closes #2801 from abij/AIRFLOW-1229


> Make "Run Id" column clickable in Browse -> DAG Runs
> 
>
> Key: AIRFLOW-1229
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1229
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webapp
>Affects Versions: Airflow 1.8
> Environment: Python3.4
>Reporter: Erik Cederstrand
>Assignee: Alexander Bij
>  Labels: patch
> Attachments: dag_run_link.patch
>
>
> I'm triggering a lot of DAGs manually using "airflow trigger_dag my_dag 
> --run_id=some_unique_id". I would like to be able in the UI to browse easily 
> to this specific DAG run using the "some_unique_id" label. In the graph page 
> of the DAG, I need to know the exact execution date, which is inconvenient, 
> and in the Browse -> DAG Runs page I can search by "some_unique_id", but the 
> "Run Ids" column is not clickable.
> The attached patch makes the aforementioned column clickable, so I'm sent 
> directly to the graph view for that specific DAG run, not the DAG in general.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1229] Add link to Run Id, incl execution_date

[fokko]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 149195845 -> d76bf76de


[AIRFLOW-1229] Add link to Run Id, incl execution_date

Add two UI improvement. 1: the links from "DAG
runs" to DAG graph view
include the execution_date. So you land on the
expected DAG, instead of
the last DAG run. 2: A new link is added for the
column "Run Id" with the
same behaviour.

Closes #2801 from abij/AIRFLOW-1229


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d76bf76d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d76bf76d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d76bf76d

Branch: refs/heads/master
Commit: d76bf76de2d2d4d3041e4d1bf9304dce8a1bcefb
Parents: 1491958
Author: Alexander Bij 
Authored: Thu Nov 23 10:46:21 2017 +0100
Committer: Fokko Driesprong 
Committed: Thu Nov 23 10:46:21 2017 +0100

--
 airflow/www/views.py | 21 ++---
 1 file changed, 18 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/d76bf76d/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index c7869c5..1191bde 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -103,7 +103,8 @@ def dag_link(v, c, m, p):
 dag_id = bleach.clean(m.dag_id)
 url = url_for(
 'airflow.graph',
-dag_id=dag_id)
+dag_id=dag_id,
+execution_date=m.execution_date)
 return Markup(
 '{}'.format(url, dag_id))
 
@@ -115,6 +116,16 @@ def log_url_formatter(v, c, m, p):
 '').format(**locals())
 
 
+def dag_run_link(v, c, m, p):
+dag_id = bleach.clean(m.dag_id)
+url = url_for(
+'airflow.graph',
+dag_id=m.dag_id,
+run_id=m.run_id,
+execution_date=m.execution_date)
+return Markup('{m.run_id}'.format(**locals()))
+
+
 def task_instance_link(v, c, m, p):
 dag_id = bleach.clean(m.dag_id)
 task_id = bleach.clean(m.task_id)
@@ -2380,7 +2391,9 @@ class DagRunModelView(ModelViewOnly):
 execution_date=datetime_f,
 state=state_f,
 start_date=datetime_f,
-dag_id=dag_link)
+dag_id=dag_link,
+run_id=dag_run_link
+)
 
 @action('new_delete', "Delete", "Are you sure you want to delete selected 
records?")
 @provide_session
@@ -2459,7 +2472,9 @@ class TaskInstanceModelView(ModelViewOnly):
 start_date=datetime_f,
 end_date=datetime_f,
 queued_dttm=datetime_f,
-dag_id=dag_link, duration=duration_f)
+dag_id=dag_link,
+run_id=dag_run_link,
+duration=duration_f)
 column_searchable_list = ('dag_id', 'task_id', 'state')
 column_default_sort = ('job_id', True)
 form_choices = {