[jira] [Commented] (AIRFLOW-2565) Make cluster_label in QuboleOperator templated_field
[ https://issues.apache.org/jira/browse/AIRFLOW-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502837#comment-16502837 ] ASF subversion and git services commented on AIRFLOW-2565: -- Commit 0bc6042ce83d3f7303c5bc1e7988e5a4daa44631 in incubator-airflow's branch refs/heads/master from milton0825 [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0bc6042 ] [AIRFLOW-2565] templatize cluster_label Make cluster_label in QuboleOperator a templated field. Closes #3463 from milton0825/make-cluster-label- template-field > Make cluster_label in QuboleOperator templated_field > > > Key: AIRFLOW-2565 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2565 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2565) Make cluster_label in QuboleOperator templated_field
[ https://issues.apache.org/jira/browse/AIRFLOW-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-2565. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3463 [https://github.com/apache/incubator-airflow/pull/3463] > Make cluster_label in QuboleOperator templated_field > > > Key: AIRFLOW-2565 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2565 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2565] templatize cluster_label
Repository: incubator-airflow
Updated Branches:
refs/heads/master b77e0dc69 -> 0bc6042ce
[AIRFLOW-2565] templatize cluster_label
Make cluster_label in QuboleOperator a templated
field.
Closes #3463 from milton0825/make-cluster-label-
template-field
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/0bc6042c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/0bc6042c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/0bc6042c
Branch: refs/heads/master
Commit: 0bc6042ce83d3f7303c5bc1e7988e5a4daa44631
Parents: b77e0dc
Author: milton0825
Authored: Wed Jun 6 11:09:54 2018 +0530
Committer: Sumit Maheshwari
Committed: Wed Jun 6 11:09:54 2018 +0530
--
.../example_dags/example_qubole_operator.py | 16 ++
airflow/contrib/operators/qubole_operator.py| 2 +-
tests/contrib/operators/test_qubole_operator.py | 22 +---
3 files changed, 32 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0bc6042c/airflow/contrib/example_dags/example_qubole_operator.py
--
diff --git a/airflow/contrib/example_dags/example_qubole_operator.py
b/airflow/contrib/example_dags/example_qubole_operator.py
index bfa8355..826a50a 100644
--- a/airflow/contrib/example_dags/example_qubole_operator.py
+++ b/airflow/contrib/example_dags/example_qubole_operator.py
@@ -61,7 +61,7 @@ t1 = QuboleOperator(
task_id='hive_show_table',
command_type='hivecmd',
query='show tables',
-cluster_label='default',
+cluster_label='{{ params.cluster_label }}',
fetch_logs=True,
# If `fetch_logs`=true, will fetch qubole command logs and concatenate
# them into corresponding airflow task logs
@@ -69,7 +69,11 @@ t1 = QuboleOperator(
# To attach tags to qubole command, auto attach 3 tags - dag_id, task_id,
run_id
qubole_conn_id='qubole_default',
# Connection id to submit commands inside QDS, if not set "qubole_default"
is used
-dag=dag)
+dag=dag,
+params={
+'cluster_label': 'default',
+}
+)
t2 = QuboleOperator(
task_id='hive_s3_location',
@@ -115,9 +119,13 @@ t4 = QuboleOperator(
'-numReduceTasks 0 -input s3://paid-qubole/HadoopAPITests/'
'data/3.tsv -output '
's3://paid-qubole/HadoopAPITests/data/3_wc',
-cluster_label='default',
+cluster_label='{{ params.cluster_label }}',
fetch_logs=True,
-dag=dag)
+dag=dag,
+params={
+'cluster_label': 'default',
+}
+)
t5 = QuboleOperator(
task_id='pig_cmd',
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0bc6042c/airflow/contrib/operators/qubole_operator.py
--
diff --git a/airflow/contrib/operators/qubole_operator.py
b/airflow/contrib/operators/qubole_operator.py
index 88e8bdf..82ee293 100755
--- a/airflow/contrib/operators/qubole_operator.py
+++ b/airflow/contrib/operators/qubole_operator.py
@@ -122,7 +122,7 @@ class QuboleOperator(BaseOperator):
'extract_query', 'boundary_query', 'macros', 'name',
'parameters',
'dbtap_id', 'hive_table', 'db_table', 'split_column',
'note_id',
'db_update_keys', 'export_dir', 'partition_spec',
'qubole_conn_id',
- 'arguments', 'user_program_arguments')
+ 'arguments', 'user_program_arguments', 'cluster_label')
template_ext = ('.txt',)
ui_color = '#3064A1'
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0bc6042c/tests/contrib/operators/test_qubole_operator.py
--
diff --git a/tests/contrib/operators/test_qubole_operator.py
b/tests/contrib/operators/test_qubole_operator.py
index ad66f75..bf61262 100644
--- a/tests/contrib/operators/test_qubole_operator.py
+++ b/tests/contrib/operators/test_qubole_operator.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -21,7 +21,7 @@
import unittest
from datetime import datetime
-from airflow.models import DAG, Connection
+from airflow.models import DAG, Connection, TaskInstance
from airflow.utils import db
from airflow.contrib.hooks.qubole_hook import QuboleHook
@@ -64,6 +64,22 @@ class
incubator-airflow git commit: closes apache/incubator-airflow#1699 *Closed for inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master a8953f65c -> b77e0dc69 closes apache/incubator-airflow#1699 *Closed for inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b77e0dc6 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b77e0dc6 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b77e0dc6 Branch: refs/heads/master Commit: b77e0dc69867ed1789d6dc5fb7100661421edf98 Parents: a8953f6 Author: Kaxil Naik Authored: Wed Jun 6 01:11:56 2018 +0100 Committer: Kaxil Naik Committed: Wed Jun 6 01:11:56 2018 +0100 -- --
incubator-airflow git commit: closes apache/incubator-airflow#1477 *Obsolete and Outdated PR*
Repository: incubator-airflow Updated Branches: refs/heads/master 230c8c5cf -> a8953f65c closes apache/incubator-airflow#1477 *Obsolete and Outdated PR* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a8953f65 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a8953f65 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a8953f65 Branch: refs/heads/master Commit: a8953f65ce0dab586e8d598d8ce6351bc44aeb60 Parents: 230c8c5 Author: Kaxil Naik Authored: Wed Jun 6 01:07:38 2018 +0100 Committer: Kaxil Naik Committed: Wed Jun 6 01:07:38 2018 +0100 -- --
incubator-airflow git commit: closes apache/incubator-airflow#3399 *Obsolete PR*
Repository: incubator-airflow Updated Branches: refs/heads/master 9e1d8ee83 -> 230c8c5cf closes apache/incubator-airflow#3399 *Obsolete PR* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/230c8c5c Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/230c8c5c Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/230c8c5c Branch: refs/heads/master Commit: 230c8c5cf99e75d5ce92cdc61994c4a6d2b3fcdb Parents: 9e1d8ee Author: Kaxil Naik Authored: Wed Jun 6 00:28:46 2018 +0100 Committer: Kaxil Naik Committed: Wed Jun 6 00:28:46 2018 +0100 -- --
[jira] [Resolved] (AIRFLOW-83) Add MongoDB hook and operator
[ https://issues.apache.org/jira/browse/AIRFLOW-83?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-83. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3454 https://github.com/apache/incubator-airflow/pull/3440 > Add MongoDB hook and operator > - > > Key: AIRFLOW-83 > URL: https://issues.apache.org/jira/browse/AIRFLOW-83 > Project: Apache Airflow > Issue Type: Wish > Components: contrib, hooks, operators >Reporter: vishal srivastava >Assignee: Ajay Yadava >Priority: Minor > Fix For: 2.0.0 > > > A mongodb hook and operator will be really useful for people who use airflow > with the mongo database. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-83) Add MongoDB hook and operator
[ https://issues.apache.org/jira/browse/AIRFLOW-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502593#comment-16502593 ] ASF subversion and git services commented on AIRFLOW-83: Commit 9e1d8ee837ea2c23e828d070b6a72a6331d98602 in incubator-airflow's branch refs/heads/master from [~ajc] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=9e1d8ee ] [AIRFLOW-83] add mongo hook and operator Closes #3440 from andscoop/AIRFLOW_83_add_mongo_hooks_and_operators > Add MongoDB hook and operator > - > > Key: AIRFLOW-83 > URL: https://issues.apache.org/jira/browse/AIRFLOW-83 > Project: Apache Airflow > Issue Type: Wish > Components: contrib, hooks, operators >Reporter: vishal srivastava >Assignee: Ajay Yadava >Priority: Minor > > A mongodb hook and operator will be really useful for people who use airflow > with the mongo database. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-83] add mongo hook and operator
Repository: incubator-airflow
Updated Branches:
refs/heads/master c2b962ca9 -> 9e1d8ee83
[AIRFLOW-83] add mongo hook and operator
Closes #3440 from
andscoop/AIRFLOW_83_add_mongo_hooks_and_operators
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/9e1d8ee8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/9e1d8ee8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/9e1d8ee8
Branch: refs/heads/master
Commit: 9e1d8ee837ea2c23e828d070b6a72a6331d98602
Parents: c2b962c
Author: Andy Cooper
Authored: Tue Jun 5 23:30:02 2018 +0100
Committer: Kaxil Naik
Committed: Tue Jun 5 23:30:02 2018 +0100
--
airflow/contrib/hooks/mongo_hook.py | 119 ++
airflow/contrib/operators/mongo_to_s3.py| 119 ++
airflow/utils/db.py | 4 +
docs/code.rst | 2 +
scripts/ci/requirements.txt | 1 +
setup.py| 6 +-
tests/contrib/hooks/test_mongo_hook.py | 123 +++
.../operators/test_mongo_to_s3_operator.py | 118 ++
8 files changed, 491 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9e1d8ee8/airflow/contrib/hooks/mongo_hook.py
--
diff --git a/airflow/contrib/hooks/mongo_hook.py
b/airflow/contrib/hooks/mongo_hook.py
new file mode 100644
index 000..6ae71a8
--- /dev/null
+++ b/airflow/contrib/hooks/mongo_hook.py
@@ -0,0 +1,119 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from ssl import CERT_NONE
+
+from airflow.hooks.base_hook import BaseHook
+from pymongo import MongoClient
+
+
+class MongoHook(BaseHook):
+"""
+PyMongo Wrapper to Interact With Mongo Database
+Mongo Connection Documentation
+https://docs.mongodb.com/manual/reference/connection-string/index.html
+You can specify connection string options in extra field of your connection
+
https://docs.mongodb.com/manual/reference/connection-string/index.html#connection-string-options
+ex.
+{replicaSet: test, ssl: True, connectTimeoutMS: 3}
+"""
+conn_type = 'MongoDb'
+
+def __init__(self, conn_id='mongo_default', *args, **kwargs):
+super(MongoHook, self).__init__(source='mongo')
+
+self.mongo_conn_id = conn_id
+self.connection = self.get_connection(conn_id)
+self.extras = self.connection.extra_dejson
+self.client = None
+
+def get_conn(self):
+"""
+Fetches PyMongo Client
+"""
+if self.client is not None:
+return self.client
+
+conn = self.connection
+
+uri = 'mongodb://{creds}{host}{port}/{database}'.format(
+creds='{}:{}@'.format(
+conn.login, conn.password
+) if conn.login is not None else '',
+
+host=conn.host,
+port='' if conn.port is None else ':{}'.format(conn.port),
+database='' if conn.schema is None else conn.schema
+)
+
+# Mongo Connection Options dict that is unpacked when passed to
MongoClient
+options = self.extras
+
+# If we are using SSL disable requiring certs from specific hostname
+if options.get('ssl', False):
+options.update({'ssl_cert_reqs': CERT_NONE})
+
+self.client = MongoClient(uri, **options)
+
+return self.client
+
+def get_collection(self, mongo_collection, mongo_db=None):
+"""
+Fetches a mongo collection object for querying.
+
+Uses connection schema as DB unless specified.
+"""
+mongo_db = mongo_db if mongo_db is not None else self.connection.schema
+mongo_conn = self.get_conn()
+
+return
mongo_conn.get_database(mongo_db).get_collection(mongo_collection)
+
+def aggregate(self, mongo_collection, aggregate_query, mongo_db=None,
**kwargs):
+"""
+Runs an aggregation pipeline and returns the results
+
https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.aggregate
+
[jira] [Resolved] (AIRFLOW-2558) Clear TASK/DAG is clearing all executions
[
https://issues.apache.org/jira/browse/AIRFLOW-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin resolved AIRFLOW-2558.
Resolution: Fixed
Fix Version/s: 1.10.0
Issue resolved by pull request #3465
[https://github.com/apache/incubator-airflow/pull/3465]
> Clear TASK/DAG is clearing all executions
> -
>
> Key: AIRFLOW-2558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2558
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: Airflow 2.0
>Reporter: Marcos Bernardelli
>Assignee: Tao Feng
>Priority: Major
> Fix For: 1.10.0
>
>
> When I try to clear a DAG/TASK specific execution, the Airflow try to execute
> all the past executions:
> [Animeted
> GIF|https://gist.githubusercontent.com/bern4rdelli/34c1e57acd53c8c67417604202f3e0e6/raw/4bcb3d3c23f2a3bb7f7bfb3e977d935e5bb9f0ee/clear.gif]
> (I failed miserable trying to attache the animated GIF :()
>
> This behavior was changed here:
> [https://github.com/apache/incubator-airflow/pull/3444]
> The old version looks like this:
> {code:python}
> drs = session.query(DagModel).filter_by(dag_id=self.dag_id).all()
> {code}
> Then it's changed to:
> {code:python}
> drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
> {code}
> This new query (using DagRun) get all the past executions, even when the
> "Past" button is not checked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2558) Clear TASK/DAG is clearing all executions
[
https://issues.apache.org/jira/browse/AIRFLOW-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502570#comment-16502570
]
ASF subversion and git services commented on AIRFLOW-2558:
--
Commit c2b962ca9831cbf56f6dbb0147c97563b7ebbd8f in incubator-airflow's branch
refs/heads/master from Tao feng
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=c2b962c ]
[AIRFLOW-2558] Clear task/dag is clearing all executions
Closes #3465 from feng-tao/airflow_2588_new
> Clear TASK/DAG is clearing all executions
> -
>
> Key: AIRFLOW-2558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2558
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: Airflow 2.0
>Reporter: Marcos Bernardelli
>Assignee: Tao Feng
>Priority: Major
>
> When I try to clear a DAG/TASK specific execution, the Airflow try to execute
> all the past executions:
> [Animeted
> GIF|https://gist.githubusercontent.com/bern4rdelli/34c1e57acd53c8c67417604202f3e0e6/raw/4bcb3d3c23f2a3bb7f7bfb3e977d935e5bb9f0ee/clear.gif]
> (I failed miserable trying to attache the animated GIF :()
>
> This behavior was changed here:
> [https://github.com/apache/incubator-airflow/pull/3444]
> The old version looks like this:
> {code:python}
> drs = session.query(DagModel).filter_by(dag_id=self.dag_id).all()
> {code}
> Then it's changed to:
> {code:python}
> drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
> {code}
> This new query (using DagRun) get all the past executions, even when the
> "Past" button is not checked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2558] Clear task/dag is clearing all executions
Repository: incubator-airflow
Updated Branches:
refs/heads/master 817296a7b -> c2b962ca9
[AIRFLOW-2558] Clear task/dag is clearing all executions
Closes #3465 from feng-tao/airflow_2588_new
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c2b962ca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c2b962ca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c2b962ca
Branch: refs/heads/master
Commit: c2b962ca9831cbf56f6dbb0147c97563b7ebbd8f
Parents: 817296a
Author: Tao feng
Authored: Tue Jun 5 15:14:43 2018 -0700
Committer: Maxime Beauchemin
Committed: Tue Jun 5 15:14:43 2018 -0700
--
airflow/bin/cli.py | 1 -
airflow/models.py | 43 ++-
tests/models.py| 4 ++--
3 files changed, 20 insertions(+), 28 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c2b962ca/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 3bff685..2742df5 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -199,7 +199,6 @@ def backfill(args, dag=None):
end_date=args.end_date,
confirm_prompt=True,
include_subdags=False,
-only_backfill_dagruns=True,
)
dag.run(
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c2b962ca/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index eda4808..c26ec01 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -132,17 +132,15 @@ def clear_task_instances(tis,
session,
activate_dag_runs=True,
dag=None,
- only_backfill_dagruns=False,
):
"""
Clears a set of task instances, but makes sure the running ones
-get killed. Reset backfill dag run state to removed if
only_backfill_dagruns is set
+get killed.
:param tis: a list of task instances
:param session: current session
:param activate_dag_runs: flag to check for active dag run
:param dag: DAG object
-:param only_backfill_dagruns: flag for setting backfill state
"""
job_ids = []
for ti in tis:
@@ -176,13 +174,8 @@ def clear_task_instances(tis,
DagRun.execution_date.in_({ti.execution_date for ti in tis}),
).all()
for dr in drs:
-if only_backfill_dagruns and dr.is_backfill:
-# If the flag is set, we reset backfill dag run for retry.
-# dont reset start date
-dr.state = State.REMOVED
-else:
-dr.state = State.RUNNING
-dr.start_date = timezone.utcnow()
+dr.state = State.RUNNING
+dr.start_date = timezone.utcnow()
class DagBag(BaseDagBag, LoggingMixin):
@@ -3697,18 +3690,20 @@ class DAG(BaseDag, LoggingMixin):
self,
state=State.RUNNING,
session=None,
-only_backfill_dagruns=False):
-drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
+start_date=None,
+end_date=None,
+):
+query = session.query(DagRun).filter_by(dag_id=self.dag_id)
+if start_date:
+query = query.filter(DagRun.execution_date >= start_date)
+if end_date:
+query = query.filter(DagRun.execution_date <= end_date)
+drs = query.all()
+
dirty_ids = []
for dr in drs:
-if only_backfill_dagruns:
-if dr.is_backfill:
-dr.state = state
-dirty_ids.append(dr.dag_id)
-else:
-if not dr.is_backfill:
-dr.state = state
-dirty_ids.append(dr.dag_id)
+dr.state = state
+dirty_ids.append(dr.dag_id)
DagStat.update(dirty_ids, session=session)
@provide_session
@@ -3721,7 +3716,6 @@ class DAG(BaseDag, LoggingMixin):
reset_dag_runs=True,
dry_run=False,
session=None,
-only_backfill_dagruns=False,
):
"""
Clears a set of task instances associated with the current dag for
@@ -3772,11 +3766,12 @@ class DAG(BaseDag, LoggingMixin):
clear_task_instances(tis.all(),
session,
dag=self,
- only_backfill_dagruns=only_backfill_dagruns,
)
if reset_dag_runs:
[jira] [Updated] (AIRFLOW-2570) Nested/organized DAG view based on subfolders
[ https://issues.apache.org/jira/browse/AIRFLOW-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Kontra updated AIRFLOW-2570: -- Description: As Airflow has become increasingly integrated with our ETL workflows it has also become a highly utilized shared service: multiple teams have adopted it for disparate projects. The primary concern now is the cluttering of the web UI with the addition of *many* dags, from all different projects. I am seeking a feature that would change the web view for the DAG list to nest dags based on their subfolders in the main airflow dags folder. Collapsing the nested sections would be ideal. The `admin/dagmodel/edit` route has a Fileloc field, would this be the best place to hook into DAG file locations and render the html accordingly? Would this be done in a javascript component or a python component? was: As Airflow has become increasingly integrated with our ETL workflows it has become a highly utilized shared service: multiple teams have adopted it for disparate projects. The primary concern now is the cluttering of the web UI with the addition of *many* dags, from all different projects. I am seeking a feature that would change the web view for the DAG list to nest dags based on their subfolders in the main airflow dags folder. Collapsing the nested sections would be ideal. The `admin/dagmodel/edit` route has a Fileloc field, would this be the best place to hook into DAG file locations and render the html accordingly? Would this be done in a javascript component or a python component? > Nested/organized DAG view based on subfolders > - > > Key: AIRFLOW-2570 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2570 > Project: Apache Airflow > Issue Type: Improvement > Components: ui, webserver >Reporter: Tyler Kontra >Priority: Trivial > > As Airflow has become increasingly integrated with our ETL workflows it has > also become a highly utilized shared service: multiple teams have adopted it > for disparate projects. The primary concern now is the cluttering of the web > UI with the addition of *many* dags, from all different projects. > I am seeking a feature that would change the web view for the DAG list to > nest dags based on their subfolders in the main airflow dags folder. > Collapsing the nested sections would be ideal. > The `admin/dagmodel/edit` route has a Fileloc field, would this be the best > place to hook into DAG file locations and render the html accordingly? Would > this be done in a javascript component or a python component? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2570) Nested/organized DAG view based on subfolders
[ https://issues.apache.org/jira/browse/AIRFLOW-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Kontra updated AIRFLOW-2570: -- Description: As Airflow has become increasingly integrated with our ETL workflows it has become a highly utilized shared service: multiple teams have adopted it for disparate projects. The primary concern now is the cluttering of the web UI with the addition of *many* dags, from all different projects. I am seeking a feature that would change the web view for the DAG list to nest dags based on their subfolders in the main airflow dags folder. Collapsing the nested sections would be ideal. The `admin/dagmodel/edit` route has a Fileloc field, would this be the best place to hook into DAG file locations and render the html accordingly? Would this be done in a javascript component or a python component? was: As Airflow has become increasingly integrated with our ETL workflows, multiple teams have adopted it for disparate projects. The primary concern now is the cluttering of the web UI with the addition of *many* dags, from all different projects. I am seeking a feature that would change the web view for the DAG list to nest dags based on their subfolders in the main airflow dags folder. Collapsing the nested sections would be ideal. The `admin/dagmodel/edit` route has a Fileloc field, would this be the best place to hook into DAG file locations and render the html accordingly? Would this be done in a javascript component or a python component? > Nested/organized DAG view based on subfolders > - > > Key: AIRFLOW-2570 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2570 > Project: Apache Airflow > Issue Type: Improvement > Components: ui, webserver >Reporter: Tyler Kontra >Priority: Trivial > > As Airflow has become increasingly integrated with our ETL workflows it has > become a highly utilized shared service: multiple teams have adopted it for > disparate projects. The primary concern now is the cluttering of the web UI > with the addition of *many* dags, from all different projects. > I am seeking a feature that would change the web view for the DAG list to > nest dags based on their subfolders in the main airflow dags folder. > Collapsing the nested sections would be ideal. > The `admin/dagmodel/edit` route has a Fileloc field, would this be the best > place to hook into DAG file locations and render the html accordingly? Would > this be done in a javascript component or a python component? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2570) Nested/organized DAG view based on subfolders
Tyler Kontra created AIRFLOW-2570: - Summary: Nested/organized DAG view based on subfolders Key: AIRFLOW-2570 URL: https://issues.apache.org/jira/browse/AIRFLOW-2570 Project: Apache Airflow Issue Type: Improvement Components: ui, webserver Reporter: Tyler Kontra As Airflow has become increasingly integrated with our ETL workflows, multiple teams have adopted it for disparate projects. The primary concern now is the cluttering of the web UI with the addition of *many* dags, from all different projects. I am seeking a feature that would change the web view for the DAG list to nest dags based on their subfolders in the main airflow dags folder. Collapsing the nested sections would be ideal. The `admin/dagmodel/edit` route has a Fileloc field, would this be the best place to hook into DAG file locations and render the html accordingly? Would this be done in a javascript component or a python component? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2569) test_scheduler_run_duration is flaky
Chao-Han Tsai created AIRFLOW-2569: -- Summary: test_scheduler_run_duration is flaky Key: AIRFLOW-2569 URL: https://issues.apache.org/jira/browse/AIRFLOW-2569 Project: Apache Airflow Issue Type: Improvement Reporter: Chao-Han Tsai Assignee: Chao-Han Tsai test_scheduler_run_duration is a flaky test that need to be fixed. logs: [https://api.travis-ci.org/v3/job/388141915/log.txt] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2569) test_scheduler_run_duration is flaky
[ https://issues.apache.org/jira/browse/AIRFLOW-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao-Han Tsai updated AIRFLOW-2569: --- Issue Type: Bug (was: Improvement) > test_scheduler_run_duration is flaky > > > Key: AIRFLOW-2569 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2569 > Project: Apache Airflow > Issue Type: Bug >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > > test_scheduler_run_duration is a flaky test that need to be fixed. > logs: [https://api.travis-ci.org/v3/job/388141915/log.txt] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2561) Typo in Email operator call to send_email
[ https://issues.apache.org/jira/browse/AIRFLOW-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Ma updated AIRFLOW-2561: - External issue URL: https://github.com/apache/incubator-airflow/pull/3468 > Typo in Email operator call to send_email > - > > Key: AIRFLOW-2561 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2561 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alan Ma >Assignee: Alan Ma >Priority: Critical > > There's a typo in the params in send_email. It should be mime_charset instead > of mine_charset. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2568) Implement a Azure Container Instances operator
Niels Zeilemaker created AIRFLOW-2568: - Summary: Implement a Azure Container Instances operator Key: AIRFLOW-2568 URL: https://issues.apache.org/jira/browse/AIRFLOW-2568 Project: Apache Airflow Issue Type: New Feature Reporter: Niels Zeilemaker Assignee: Niels Zeilemaker -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2567) Allow Kubernetes Pod Operator to extract result from the container as Xcom
Shintaro Murakami created AIRFLOW-2567: -- Summary: Allow Kubernetes Pod Operator to extract result from the container as Xcom Key: AIRFLOW-2567 URL: https://issues.apache.org/jira/browse/AIRFLOW-2567 Project: Apache Airflow Issue Type: Improvement Components: operators Affects Versions: 1.10.0 Reporter: Shintaro Murakami In order to simplify the construction of DAG, it is useful to be able to extract the execution result of the container as XCom. Users will be able to do this by writing the result as the `result.json` in `/airflow/xcom` directory mounted on the container automatically. This function is realized by Pod's sidecar container. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2566) Change backfill to rerun failed tasks instead of throwing exception
Tao Feng created AIRFLOW-2566: - Summary: Change backfill to rerun failed tasks instead of throwing exception Key: AIRFLOW-2566 URL: https://issues.apache.org/jira/browse/AIRFLOW-2566 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
