[jira] [Updated] (AIRFLOW-685) Add unit test for MySqlHook.bulk_load()
[ https://issues.apache.org/jira/browse/AIRFLOW-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kengo Seki updated AIRFLOW-685: --- External issue URL: https://github.com/apache/incubator-airflow/pull/1929 > Add unit test for MySqlHook.bulk_load() > --- > > Key: AIRFLOW-685 > URL: https://issues.apache.org/jira/browse/AIRFLOW-685 > Project: Apache Airflow > Issue Type: Test > Components: tests >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Minor > > This method doesn't seem to be covered by unit test, so let's add it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-685) Add unit test for MySqlHook.bulk_load()
Kengo Seki created AIRFLOW-685: -- Summary: Add unit test for MySqlHook.bulk_load() Key: AIRFLOW-685 URL: https://issues.apache.org/jira/browse/AIRFLOW-685 Project: Apache Airflow Issue Type: Test Components: tests Reporter: Kengo Seki Assignee: Kengo Seki Priority: Minor This method doesn't seem to be covered by unit test, so let's add it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-684) Add task log pagination to webservers
Dan Davydov created AIRFLOW-684: --- Summary: Add task log pagination to webservers Key: AIRFLOW-684 URL: https://issues.apache.org/jira/browse/AIRFLOW-684 Project: Apache Airflow Issue Type: Improvement Components: webserver Reporter: Dan Davydov At the moment very large task instances logs do not load in the browser, they should be paginated in an efficient manner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-674) Ability to add descriptions for DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729990#comment-15729990 ] ASF subversion and git services commented on AIRFLOW-674: - Commit b674656312d3ed785d86472a956bb477b5977f97 in incubator-airflow's branch refs/heads/master from [~aoen] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b674656 ] [AIRFLOW-674] Ability to add descriptions for DAGs Please accept this PR that addresses the following issues: - https://issues.apache.org/jira/browse/AIRFLOW-674 Descriptions are rendered in two places right now, as a tooltip on the DAGs page for each DAG, and after the DAG name in the various individual DAG views (see screenshots below). Testing Done: - Spun up local webserver, tried empty description, normal description, very long description. Screenshots: https://cloud.githubusercontent.com/asset s/1592778/20906424/dbf2885e-bafc-11e6-8eeb- 78302e87d25a.png https://cloud.githubusercontent.com/asset s/1592778/20906420/d651ccfc-bafc- 11e6-9893-ef677be50bf0.png Closes #1920 from aoen/ddavydov/dag_desc > Ability to add descriptions for DAGs > > > Key: AIRFLOW-674 > URL: https://issues.apache.org/jira/browse/AIRFLOW-674 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Minor > > Add a "description" property to DAGs that will be shown in various places in > the webserver. This is useful to e.g. see what a DAG does at a high level > without having to resort to looking at code comments/git blame. Kind of like > doc_md but more lightweight and visible without having to drill into task > details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-674) Ability to add descriptions for DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729989#comment-15729989 ] ASF subversion and git services commented on AIRFLOW-674: - Commit b674656312d3ed785d86472a956bb477b5977f97 in incubator-airflow's branch refs/heads/master from [~aoen] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b674656 ] [AIRFLOW-674] Ability to add descriptions for DAGs Please accept this PR that addresses the following issues: - https://issues.apache.org/jira/browse/AIRFLOW-674 Descriptions are rendered in two places right now, as a tooltip on the DAGs page for each DAG, and after the DAG name in the various individual DAG views (see screenshots below). Testing Done: - Spun up local webserver, tried empty description, normal description, very long description. Screenshots: https://cloud.githubusercontent.com/asset s/1592778/20906424/dbf2885e-bafc-11e6-8eeb- 78302e87d25a.png https://cloud.githubusercontent.com/asset s/1592778/20906420/d651ccfc-bafc- 11e6-9893-ef677be50bf0.png Closes #1920 from aoen/ddavydov/dag_desc > Ability to add descriptions for DAGs > > > Key: AIRFLOW-674 > URL: https://issues.apache.org/jira/browse/AIRFLOW-674 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Minor > > Add a "description" property to DAGs that will be shown in various places in > the webserver. This is useful to e.g. see what a DAG does at a high level > without having to resort to looking at code comments/git blame. Kind of like > doc_md but more lightweight and visible without having to drill into task > details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
incubator-airflow git commit: [AIRFLOW-674] Ability to add descriptions for DAGs
Repository: incubator-airflow
Updated Branches:
refs/heads/master 37a324ae4 -> b67465631
[AIRFLOW-674] Ability to add descriptions for DAGs
Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-674
Descriptions are rendered in two places right now,
as a tooltip on the DAGs page for each DAG, and
after the DAG name in the various individual DAG
views (see screenshots below).
Testing Done:
- Spun up local webserver, tried empty
description, normal description, very long
description.
Screenshots:
https://cloud.githubusercontent.com/asset
s/1592778/20906424/dbf2885e-bafc-11e6-8eeb-
78302e87d25a.png
https://cloud.githubusercontent.com/asset
s/1592778/20906420/d651ccfc-bafc-
11e6-9893-ef677be50bf0.png
Closes #1920 from aoen/ddavydov/dag_desc
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b6746563
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b6746563
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b6746563
Branch: refs/heads/master
Commit: b674656312d3ed785d86472a956bb477b5977f97
Parents: 37a324a
Author: Dan Davydov
Authored: Wed Dec 7 13:36:52 2016 -0800
Committer: Dan Davydov
Committed: Wed Dec 7 13:36:57 2016 -0800
--
airflow/example_dags/tutorial.py| 6 +-
airflow/models.py | 8
airflow/www/templates/airflow/dag.html | 2 +-
airflow/www/templates/airflow/dags.html | 2 +-
4 files changed, 15 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b6746563/airflow/example_dags/tutorial.py
--
diff --git a/airflow/example_dags/tutorial.py b/airflow/example_dags/tutorial.py
index 5621a7c..c7b2e0f 100644
--- a/airflow/example_dags/tutorial.py
+++ b/airflow/example_dags/tutorial.py
@@ -50,7 +50,11 @@ default_args = {
# 'trigger_rule': u'all_success'
}
-dag = DAG('tutorial', default_args=default_args,
schedule_interval=timedelta(days=1))
+dag = DAG(
+'tutorial',
+default_args=default_args,
+description='A simple tutorial DAG',
+schedule_interval=timedelta(days=1))
# t1, t2 and t3 are examples of tasks created by instantiating operators
t1 = BashOperator(
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b6746563/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 02e4046..5763b96 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -2483,6 +2483,8 @@ class DAG(BaseDag, LoggingMixin):
:param dag_id: The id of the DAG
:type dag_id: string
+:param description: The description for the DAG to e.g. be shown on the
webserver
+:type description: string
:param schedule_interval: Defines how often that DAG runs, this
timedelta object gets added to your latest task instance's
execution_date to figure out the next schedule
@@ -2536,6 +2538,7 @@ class DAG(BaseDag, LoggingMixin):
def __init__(
self, dag_id,
+description='',
schedule_interval=timedelta(days=1),
start_date=None, end_date=None,
full_filepath=None,
@@ -2567,6 +2570,7 @@ class DAG(BaseDag, LoggingMixin):
self._concurrency = concurrency
self._pickle_id = None
+self._description = description
self.task_dict = dict()
self.start_date = start_date
self.end_date = end_date
@@ -2726,6 +2730,10 @@ class DAG(BaseDag, LoggingMixin):
self._concurrency = value
@property
+def description(self):
+return self._description
+
+@property
def pickle_id(self):
return self._pickle_id
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b6746563/airflow/www/templates/airflow/dag.html
--
diff --git a/airflow/www/templates/airflow/dag.html
b/airflow/www/templates/airflow/dag.html
index e9c1940..b9b1afa 100644
--- a/airflow/www/templates/airflow/dag.html
+++ b/airflow/www/templates/airflow/dag.html
@@ -33,7 +33,7 @@
SUBDAG: {{ dag.dag_id
}}
{% else %}
-DAG: {{ dag.dag_id }}
+DAG: {{ dag.dag_id }}
{{ dag.description }}
{% endif %}
{% if root %}
ROOT: {{ root }}
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b6746563/airflow/www/templates/airflow/dags.html
--
diff --git a/airflow/www/templates/airflow/dags.html
b/airflow/www/templates/airflow/dags.html
index 347015d..2cbd12e 100644
--- a/airflow/www/templates/airflow/d
[jira] [Created] (AIRFLOW-683) provide Jira Hook, Operator & Sensor
Jayesh created AIRFLOW-683: -- Summary: provide Jira Hook, Operator & Sensor Key: AIRFLOW-683 URL: https://issues.apache.org/jira/browse/AIRFLOW-683 Project: Apache Airflow Issue Type: New Feature Components: hooks, operators Reporter: Jayesh Assignee: Jayesh Task to add Jira Hook, Operator & Sensor so that user can interact with Jira system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (AIRFLOW-682) Bump MAX_PERIODS
[ https://issues.apache.org/jira/browse/AIRFLOW-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Davydov resolved AIRFLOW-682. - Resolution: Fixed > Bump MAX_PERIODS > > > Key: AIRFLOW-682 > URL: https://issues.apache.org/jira/browse/AIRFLOW-682 > Project: Apache Airflow > Issue Type: Bug >Reporter: Dan Davydov >Assignee: Dan Davydov > > It is not possible to mark success on some large DAGs due to the MAX_PERIODS > being set to 1000. We should temporarily bump it up until work can be done to > scale the mark success endpoint much higher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-682) Bump MAX_PERIODS
[ https://issues.apache.org/jira/browse/AIRFLOW-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729685#comment-15729685 ] ASF subversion and git services commented on AIRFLOW-682: - Commit 37a324ae4ffeee6b4a460cebe3c1cfec8f6ef91a in incubator-airflow's branch refs/heads/master from [~aoen] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=37a324a ] [AIRFLOW-682] Bump MAX_PERIODS to make mark_success work for large DAGs It is not possible to mark success on some large DAGs due to the MAX_PERIODS being set to 1000. We should temporarily bump it up until work can be done to scale the mark success endpoint much higher. Testing Done: - Has been running in Airbnb prod for many months now Closes #1928 from aoen/ddavydov/bump_max_periods > Bump MAX_PERIODS > > > Key: AIRFLOW-682 > URL: https://issues.apache.org/jira/browse/AIRFLOW-682 > Project: Apache Airflow > Issue Type: Bug >Reporter: Dan Davydov >Assignee: Dan Davydov > > It is not possible to mark success on some large DAGs due to the MAX_PERIODS > being set to 1000. We should temporarily bump it up until work can be done to > scale the mark success endpoint much higher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
incubator-airflow git commit: [AIRFLOW-682] Bump MAX_PERIODS to make mark_success work for large DAGs
Repository: incubator-airflow
Updated Branches:
refs/heads/master 64a1b19a3 -> 37a324ae4
[AIRFLOW-682] Bump MAX_PERIODS to make mark_success work for large DAGs
It is not possible to mark success on some large
DAGs due to the MAX_PERIODS being set to 1000. We
should temporarily bump it up until work can be
done to scale the mark success endpoint much
higher.
Testing Done:
- Has been running in Airbnb prod for many months
now
Closes #1928 from aoen/ddavydov/bump_max_periods
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/37a324ae
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/37a324ae
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/37a324ae
Branch: refs/heads/master
Commit: 37a324ae4ffeee6b4a460cebe3c1cfec8f6ef91a
Parents: 64a1b19
Author: Dan Davydov
Authored: Wed Dec 7 11:31:02 2016 -0800
Committer: Dan Davydov
Committed: Wed Dec 7 11:31:02 2016 -0800
--
airflow/www/views.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/37a324ae/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index d22e8e4..7134264 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -1095,7 +1095,7 @@ class Airflow(BaseView):
future = request.args.get('future') == "true"
past = request.args.get('past') == "true"
recursive = request.args.get('recursive') == "true"
-MAX_PERIODS = 1000
+MAX_PERIODS = 5000
# Flagging tasks as successful
session = settings.Session()
[jira] [Commented] (AIRFLOW-682) Bump MAX_PERIODS
[ https://issues.apache.org/jira/browse/AIRFLOW-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729673#comment-15729673 ] Arthur Wiedmer commented on AIRFLOW-682: +1. Very useful for large-ish DAGs > 1k tasks as this limit applies also for the max number of tasks when marking upstream or downstream success. > Bump MAX_PERIODS > > > Key: AIRFLOW-682 > URL: https://issues.apache.org/jira/browse/AIRFLOW-682 > Project: Apache Airflow > Issue Type: Bug >Reporter: Dan Davydov >Assignee: Dan Davydov > > It is not possible to mark success on some large DAGs due to the MAX_PERIODS > being set to 1000. We should temporarily bump it up until work can be done to > scale the mark success endpoint much higher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-682) Bump MAX_PERIODS
Dan Davydov created AIRFLOW-682: --- Summary: Bump MAX_PERIODS Key: AIRFLOW-682 URL: https://issues.apache.org/jira/browse/AIRFLOW-682 Project: Apache Airflow Issue Type: Bug Reporter: Dan Davydov Assignee: Dan Davydov It is not possible to mark success on some large DAGs due to the MAX_PERIODS being set to 1000. We should temporarily bump it up until work can be done to scale the mark success endpoint much higher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
