[jira] [Closed] (AIRFLOW-2527) Be able to clear Dag Runs + Tasks from command line
[ https://issues.apache.org/jira/browse/AIRFLOW-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-2527. - Resolution: Not A Problem > Be able to clear Dag Runs + Tasks from command line > --- > > Key: AIRFLOW-2527 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2527 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > Make sure airflow clear could clear both dag runs and tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2093) Feature DagBag loading from S3 (and others)
[ https://issues.apache.org/jira/browse/AIRFLOW-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494304#comment-16494304 ] Aditi Verma commented on AIRFLOW-2093: -- We have a PR for periodically syncing DAGs from S3 on the scheduler and web-server: https://github.com/apache/incubator-airflow/pull/3318 > Feature DagBag loading from S3 (and others) > --- > > Key: AIRFLOW-2093 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2093 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, boto3, DAG, scheduler, worker >Reporter: Bruno Bonagura >Priority: Major > Labels: features, usability > > Deploying DAGs is a pain point for CeleryExecutor Airflow infrastructure. > Allowing it to be configured to read them directly from remote sources (S3, > FTP, HTTP...) not just local FS would enhance a lot this aspect. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2402) Airflow 1.10 Logs UI throws oops error
[
https://issues.apache.org/jira/browse/AIRFLOW-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-2402.
-
Resolution: Fixed
Fix Version/s: (was: 1.10.0)
2.0.0
Issue resolved by pull request #3319
[https://github.com/apache/incubator-airflow/pull/3319]
> Airflow 1.10 Logs UI throws oops error
> --
>
> Key: AIRFLOW-2402
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2402
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication, ui
>Affects Versions: 1.10.0
>Reporter: Ramki Subramanian
>Assignee: Kevin Yang
>Priority: Major
> Fix For: 2.0.0
>
>
> Hi,
> I am getting an error at
> [incubator-airflow/airflow/www_rbac/views.py|https://github.com/apache/incubator-airflow/blob/4d64ad4928f0188f7532936e8da6612f5ec7170d/airflow/www_rbac/views.py#L454]
> Line 454 in
> [4d64ad4|https://github.com/apache/incubator-airflow/commit/4d64ad4928f0188f7532936e8da6612f5ec7170d]
> | |logs[i] = log.decode('utf-8')|
>
> {{/home/user/lib/python2.7/site-packages/airflow/www_rbac/views.py", line
> 454, in log logs[i] = log.decode('utf-8') AttributeError: 'list' object has
> no attribute 'decode' }}
> Not sure if someone is already looking into this, or I am missing some config?
> Branch : 1.10_test
> More Info here:
> [https://github.com/apache/incubator-airflow/commit/05e1861e24de42f9a2c649cd93041c5c744504e1#diff-77df5adb32d964f37748c4557ffb3c4c]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2402] Fix RBAC task log
Repository: incubator-airflow
Updated Branches:
refs/heads/master 9b661fa61 -> 7c3435442
[AIRFLOW-2402] Fix RBAC task log
Closes #3319 from yrqls21/kevin_yang_fix_rbac_view
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7c343544
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7c343544
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7c343544
Branch: refs/heads/master
Commit: 7c34354427f8c047b2cfb3f472a38bc50fe91d45
Parents: 9b661fa
Author: Kevin Yang
Authored: Tue May 29 20:46:01 2018 +0100
Committer: Kaxil Naik
Committed: Tue May 29 20:46:01 2018 +0100
--
airflow/configuration.py| 4 +-
airflow/www/templates/airflow/ti_log.html | 1 -
airflow/www/views.py| 11 +-
airflow/www_rbac/templates/airflow/ti_log.html | 158 +++
airflow/www_rbac/views.py | 92 ---
.../2017-09-01T00.00.00+00.00/1.log | 1 +
.../2017-09-01T00.00.00/1.log | 1 -
tests/www/test_views.py | 21 ++-
.../2017-09-01T00.00.00+00.00/1.log | 1 +
.../2017-09-01T00.00.00/1.log | 1 -
tests/www_rbac/test_views.py| 117 +-
11 files changed, 337 insertions(+), 71 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7c343544/airflow/configuration.py
--
diff --git a/airflow/configuration.py b/airflow/configuration.py
index e19a8b1..2ee453f 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -346,7 +346,7 @@ class AirflowConfigParser(ConfigParser):
Note: this is not reversible.
"""
# override any custom settings with defaults
-self.defaults.read_string(parameterized_config(DEFAULT_CONFIG))
+self.read_string(parameterized_config(DEFAULT_CONFIG))
# then read test config
self.read_string(parameterized_config(TEST_CONFIG))
# then read any "custom" test settings
@@ -446,8 +446,10 @@ if not os.path.isfile(AIRFLOW_CONFIG):
log.info("Reading the config from %s", AIRFLOW_CONFIG)
conf = AirflowConfigParser(default_config=parameterized_config(DEFAULT_CONFIG))
+
conf.read(AIRFLOW_CONFIG)
+
if conf.getboolean('webserver', 'rbac'):
with open(os.path.join(_templates_dir, 'default_webserver_config.py')) as
f:
DEFAULT_WEBSERVER_CONFIG = f.read()
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7c343544/airflow/www/templates/airflow/ti_log.html
--
diff --git a/airflow/www/templates/airflow/ti_log.html
b/airflow/www/templates/airflow/ti_log.html
index 2615c45..2b50061 100644
--- a/airflow/www/templates/airflow/ti_log.html
+++ b/airflow/www/templates/airflow/ti_log.html
@@ -41,7 +41,6 @@ limitations under the License.
{% endblock %}
{% block tail %}
-{{ lib.form_js() }}
{{ super() }}
// TODO: make those constants configurable.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7c343544/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 4860dbd..eb29b89 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -733,10 +733,12 @@ class Airflow(BaseView):
execution_date = request.args.get('execution_date')
dttm = pendulum.parse(execution_date)
try_number = int(request.args.get('try_number'))
-# metadata may be None
metadata = request.args.get('metadata')
-if metadata:
-metadata = json.loads(metadata)
+metadata = json.loads(metadata)
+
+# metadata may be null
+if not metadata:
+metadata = {}
# Convert string datetime into actual datetime
try:
@@ -779,9 +781,6 @@ class Airflow(BaseView):
.format(task_log_reader, str(e))]
metadata['end_of_log'] = True
return jsonify(message=error_message, error=True,
metadata=metadata)
-except AirflowException as e:
-metadata['end_of_log'] = True
-return jsonify(message=str(e), error=True, metadata=metadata)
@expose('/log')
@login_required
[jira] [Commented] (AIRFLOW-2402) Airflow 1.10 Logs UI throws oops error
[
https://issues.apache.org/jira/browse/AIRFLOW-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494137#comment-16494137
]
ASF subversion and git services commented on AIRFLOW-2402:
--
Commit 7c34354427f8c047b2cfb3f472a38bc50fe91d45 in incubator-airflow's branch
refs/heads/master from [~kevinyang]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7c34354 ]
[AIRFLOW-2402] Fix RBAC task log
Closes #3319 from yrqls21/kevin_yang_fix_rbac_view
> Airflow 1.10 Logs UI throws oops error
> --
>
> Key: AIRFLOW-2402
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2402
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication, ui
>Affects Versions: 1.10.0
>Reporter: Ramki Subramanian
>Assignee: Kevin Yang
>Priority: Major
> Fix For: 2.0.0
>
>
> Hi,
> I am getting an error at
> [incubator-airflow/airflow/www_rbac/views.py|https://github.com/apache/incubator-airflow/blob/4d64ad4928f0188f7532936e8da6612f5ec7170d/airflow/www_rbac/views.py#L454]
> Line 454 in
> [4d64ad4|https://github.com/apache/incubator-airflow/commit/4d64ad4928f0188f7532936e8da6612f5ec7170d]
> | |logs[i] = log.decode('utf-8')|
>
> {{/home/user/lib/python2.7/site-packages/airflow/www_rbac/views.py", line
> 454, in log logs[i] = log.decode('utf-8') AttributeError: 'list' object has
> no attribute 'decode' }}
> Not sure if someone is already looking into this, or I am missing some config?
> Branch : 1.10_test
> More Info here:
> [https://github.com/apache/incubator-airflow/commit/05e1861e24de42f9a2c649cd93041c5c744504e1#diff-77df5adb32d964f37748c4557ffb3c4c]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2531) Add M4U user to README
[ https://issues.apache.org/jira/browse/AIRFLOW-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2531. - Resolution: Fixed Fix Version/s: (was: Airflow 2.0) 2.0.0 Resolved by https://github.com/apache/incubator-airflow/pull/3426 > Add M4U user to README > -- > > Key: AIRFLOW-2531 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2531 > Project: Apache Airflow > Issue Type: Wish >Reporter: Marcelo Santino >Assignee: Marcelo Santino >Priority: Trivial > Fix For: 2.0.0 > > > Add M4U to current list of Airflow users on README. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2437) Add PubNub to README
[ https://issues.apache.org/jira/browse/AIRFLOW-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494125#comment-16494125 ] ASF subversion and git services commented on AIRFLOW-2437: -- Commit 9b661fa61343def72b47195d0421a4224ecceedf in incubator-airflow's branch refs/heads/master from [~msantino] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=9b661fa ] [AIRFLOW-XXX] Add M4U to user list Closes #3426 from msantino/AIRFLOW-2437-add_m4u_to_users_list > Add PubNub to README > > > Key: AIRFLOW-2437 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2437 > Project: Apache Airflow > Issue Type: Wish >Reporter: Jordan Zucker >Assignee: Jordan Zucker >Priority: Trivial > Fix For: 2.0.0 > > > Add PubNub to current list of Airflow users -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-XXX] Add M4U to user list
Repository: incubator-airflow Updated Branches: refs/heads/master d5d97dc97 -> 9b661fa61 [AIRFLOW-XXX] Add M4U to user list Closes #3426 from msantino/AIRFLOW-2437-add_m4u_to_users_list Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/9b661fa6 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/9b661fa6 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/9b661fa6 Branch: refs/heads/master Commit: 9b661fa61343def72b47195d0421a4224ecceedf Parents: d5d97dc Author: Marcelo Santino Authored: Tue May 29 20:38:08 2018 +0100 Committer: Kaxil Naik Committed: Tue May 29 20:38:11 2018 +0100 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9b661fa6/README.md -- diff --git a/README.md b/README.md index f3fe985..0f94de0 100644 --- a/README.md +++ b/README.md @@ -178,6 +178,7 @@ Currently **officially** using Airflow: 1. [Lucid](http://luc.id) [[@jbrownlucid](https://github.com/jbrownlucid) & [@kkourtchikov](https://github.com/kkourtchikov)] 1. [Lumos Labs](https://www.lumosity.com/) [[@rfroetscher](https://github.com/rfroetscher/) & [@zzztimbo](https://github.com/zzztimbo/)] 1. [Lyft](https://www.lyft.com/)[[@SaurabhBajaj](https://github.com/SaurabhBajaj)] +1. [M4U](https://www.m4u.com.br/) [[@msantino](https://github.com/msantino)] 1. [Madrone](http://madroneco.com/) [[@mbreining](https://github.com/mbreining) & [@scotthb](https://github.com/scotthb)] 1. [Markovian](https://markovian.com/) [[@al-xv](https://github.com/al-xv), [@skogsbaeck](https://github.com/skogsbaeck), [@waltherg](https://github.com/waltherg)] 1. [Mercadoni](https://www.mercadoni.com.co) [[@demorenoc](https://github.com/demorenoc)]
[jira] [Resolved] (AIRFLOW-2536) Adding docs about how to deal with Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql
[
https://issues.apache.org/jira/browse/AIRFLOW-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-2536.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Resolved by https://github.com/apache/incubator-airflow/pull/3429
> Adding docs about how to deal with Exception: Global variable
> explicit_defaults_for_timestamp needs to be on (1) for mysql
> --
>
> Key: AIRFLOW-2536
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2536
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 2.0.0
>
>
> Add docs to faq.rst to talk about how to deal with
> {code}
> Exception: Global variable explicit_defaults_for_timestamp needs to be on (1)
> for mysql
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2536) Adding docs about how to deal with Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql
[
https://issues.apache.org/jira/browse/AIRFLOW-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494106#comment-16494106
]
ASF subversion and git services commented on AIRFLOW-2536:
--
Commit d5d97dc9719fd71625d8d252315ec01c7415c11a in incubator-airflow's branch
refs/heads/master from [~milton0825]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d5d97dc ]
[AIRFLOW-2536] docs about how to deal with airflow initdb failure
Add docs to faq.rst to talk about how to deal with
Exception: Global variable
explicit_defaults_for_timestamp needs to be on (1)
for mysql
Closes #3429 from milton0825/fix-docs
> Adding docs about how to deal with Exception: Global variable
> explicit_defaults_for_timestamp needs to be on (1) for mysql
> --
>
> Key: AIRFLOW-2536
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2536
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Add docs to faq.rst to talk about how to deal with
> {code}
> Exception: Global variable explicit_defaults_for_timestamp needs to be on (1)
> for mysql
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2536] docs about how to deal with airflow initdb failure
Repository: incubator-airflow Updated Branches: refs/heads/master 11e670ddb -> d5d97dc97 [AIRFLOW-2536] docs about how to deal with airflow initdb failure Add docs to faq.rst to talk about how to deal with Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql Closes #3429 from milton0825/fix-docs Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d5d97dc9 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d5d97dc9 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d5d97dc9 Branch: refs/heads/master Commit: d5d97dc9719fd71625d8d252315ec01c7415c11a Parents: 11e670d Author: Chao-Han Tsai Authored: Tue May 29 20:27:28 2018 +0100 Committer: Kaxil Naik Committed: Tue May 29 20:29:27 2018 +0100 -- docs/faq.rst | 8 1 file changed, 8 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/d5d97dc9/docs/faq.rst -- diff --git a/docs/faq.rst b/docs/faq.rst index 53242e4..d2c6188 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -161,3 +161,11 @@ How can we reduce the airflow UI page load time? If your dag takes long time to load, you could reduce the value of ``default_dag_run_display_number`` configuration in ``airflow.cfg`` to a smaller value. This configurable controls the number of dag run to show in UI with default value 25. + +How to fix Exception: Global variable explicit_defaults_for_timestamp needs to be on (1)? +- + +This means ``explicit_defaults_for_timestamp`` is disabled in your mysql server and you need to enable it by: + +#. Set ``explicit_defaults_for_timestamp = 1`` under the mysqld section in your my.cnf file. +#. Restart the Mysql server.
[jira] [Commented] (AIRFLOW-2513) Change `bql` to `sql` for BigQuery Hooks & Operators for consistency
[ https://issues.apache.org/jira/browse/AIRFLOW-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494011#comment-16494011 ] Tim Swast commented on AIRFLOW-2513: Thanks! The *bql* parameter bugged me, too. As I understand it, the contrib packages have no guarantees for changes, so I think changing it and adding note to release notes makes sense. > Change `bql` to `sql` for BigQuery Hooks & Operators for consistency > > > Key: AIRFLOW-2513 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2513 > Project: Apache Airflow > Issue Type: Task >Affects Versions: 1.9.0, 1.8.2 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Major > Fix For: 2.0.0 > > > After discussing this yesterday with Google data engineers I can confirm that > there is nothing called *bql* as used as parameter to pass SQL queries by > some of the BigQuery operators (*BigQueryCheckOperator* for example uses > *sql* and *BigQueryOperator* uses *bql* ) and hooks while some of the other > Bigquery operators and Hooks have *sql* instead. It makes more sense to stick > with *sql* instead of *bql*. > However as most of the previous stable versions used it, we should first show > a deprecation warning for a stable release or we can change this in *Airflow > 2.0* and mention it in the release doc that regarding the change of parameter > name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2536) Adding docs about how to deal with Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql
[
https://issues.apache.org/jira/browse/AIRFLOW-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao-Han Tsai updated AIRFLOW-2536:
---
Description:
Add docs to faq.rst to talk about how to deal with
{code}
Exception: Global variable explicit_defaults_for_timestamp needs to be on (1)
for mysql
{code}
was:
adding docs about how to deal with
{code}
Exception: Global variable explicit_defaults_for_timestamp needs to be on (1)
for mysql
{code}
> Adding docs about how to deal with Exception: Global variable
> explicit_defaults_for_timestamp needs to be on (1) for mysql
> --
>
> Key: AIRFLOW-2536
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2536
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Add docs to faq.rst to talk about how to deal with
> {code}
> Exception: Global variable explicit_defaults_for_timestamp needs to be on (1)
> for mysql
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2536) Adding docs about how to deal with Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql
[
https://issues.apache.org/jira/browse/AIRFLOW-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao-Han Tsai updated AIRFLOW-2536:
---
Summary: Adding docs about how to deal with Exception: Global variable
explicit_defaults_for_timestamp needs to be on (1) for mysql (was: Adding docs
about how to deal with {code} Exception: Global variable
explicit_defaults_for_timestamp needs to be on (1) for mysql {code})
> Adding docs about how to deal with Exception: Global variable
> explicit_defaults_for_timestamp needs to be on (1) for mysql
> --
>
> Key: AIRFLOW-2536
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2536
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> adding docs about how to deal with
> {code}
> Exception: Global variable explicit_defaults_for_timestamp needs to be on (1)
> for mysql
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2536) Adding docs about how to deal with {code} Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql {code}
Chao-Han Tsai created AIRFLOW-2536:
--
Summary: Adding docs about how to deal with {code} Exception:
Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql
{code}
Key: AIRFLOW-2536
URL: https://issues.apache.org/jira/browse/AIRFLOW-2536
Project: Apache Airflow
Issue Type: Improvement
Reporter: Chao-Han Tsai
Assignee: Chao-Han Tsai
adding docs about how to deal with
{code}
Exception: Global variable explicit_defaults_for_timestamp needs to be on (1)
for mysql
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2535) TriggerDagRunOperator ignores max_active_runs, leading to scheduler breakdown
[ https://issues.apache.org/jira/browse/AIRFLOW-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tylar Murray updated AIRFLOW-2535: -- Description: `TriggerDagRunOperator` does not respect the `max_active_runs` setting. This in itself is not a huge issue, but the scheduler breaks down if `max_active_runs` is exceeded. Task scheduling becomes absurdly slow, quickly leading to perpetual DAG buildup. `TriggerDagRunOperator` could throw exception if `max_active_runs` will be exceeded, or maybe dags could be put into a `queued` state rather than directly into `running`? Alternatively: the scheduler probably shouldn't slow task scheduling when `max_active_runs` is exceeded. was: `TriggerDagRunOperator` does not respect the `max_active_runs` setting. This in itself is not a huge issue, but the scheduler breaks down if `max_active_runs` is exceeded. Task scheduling becomes absurdly slow quickly leading to perpetual DAG buildup. `TriggerDagRunOperator` could throw exception if `max_active_runs` will be exceeded, or maybe dags could be put into a `queued` state rather than directly into `running`? Alternatively: the scheduler probably shouldn't slow task scheduling when `max_active_runs` is exceeded. > TriggerDagRunOperator ignores max_active_runs, leading to scheduler breakdown > - > > Key: AIRFLOW-2535 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2535 > Project: Apache Airflow > Issue Type: Bug > Components: operators, scheduler >Affects Versions: 1.9.0 > Environment: CeleryExec, Rabbitmq, mysql >Reporter: Tylar Murray >Priority: Minor > > `TriggerDagRunOperator` does not respect the `max_active_runs` setting. This > in itself is not a huge issue, but the scheduler breaks down if > `max_active_runs` is exceeded. Task scheduling becomes absurdly slow, quickly > leading to perpetual DAG buildup. > `TriggerDagRunOperator` could throw exception if `max_active_runs` will be > exceeded, or maybe dags could be put into a `queued` state rather than > directly into `running`? > Alternatively: the scheduler probably shouldn't slow task scheduling when > `max_active_runs` is exceeded. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2535) TriggerDagRunOperator ignores max_active_runs, leading to scheduler breakdown
Tylar Murray created AIRFLOW-2535: - Summary: TriggerDagRunOperator ignores max_active_runs, leading to scheduler breakdown Key: AIRFLOW-2535 URL: https://issues.apache.org/jira/browse/AIRFLOW-2535 Project: Apache Airflow Issue Type: Bug Components: operators, scheduler Affects Versions: 1.9.0 Environment: CeleryExec, Rabbitmq, mysql Reporter: Tylar Murray `TriggerDagRunOperator` does not respect the `max_active_runs` setting. This in itself is not a huge issue, but the scheduler breaks down if `max_active_runs` is exceeded. Task scheduling becomes absurdly slow quickly leading to perpetual DAG buildup. `TriggerDagRunOperator` could throw exception if `max_active_runs` will be exceeded, or maybe dags could be put into a `queued` state rather than directly into `running`? Alternatively: the scheduler probably shouldn't slow task scheduling when `max_active_runs` is exceeded. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2534) HiveServer2Hook is overwriting results when passing multiple sql statements
Giovanni Lanzani created AIRFLOW-2534: - Summary: HiveServer2Hook is overwriting results when passing multiple sql statements Key: AIRFLOW-2534 URL: https://issues.apache.org/jira/browse/AIRFLOW-2534 Project: Apache Airflow Issue Type: Test Components: hooks Affects Versions: Airflow 1.9.0 Reporter: Giovanni Lanzani Assignee: Giovanni Lanzani `HiveServer2Hook.get_results` is overwriting records when multiple sql statements are passed. This might be superflous when the first statements are just `SET`, but in principle we want to check if that's also the case -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2533) Kubernetes worker configuration improperly sets path to DAG on worker node
Brian Nutt created AIRFLOW-2533: --- Summary: Kubernetes worker configuration improperly sets path to DAG on worker node Key: AIRFLOW-2533 URL: https://issues.apache.org/jira/browse/AIRFLOW-2533 Project: Apache Airflow Issue Type: Bug Components: executor, worker Affects Versions: 1.10.0 Reporter: Brian Nutt Fix For: 1.10 Attachments: Screen Shot 2018-05-24 at 5.06.25 PM.png When triggering a DAG using the kubernetes executor, the path to the DAG on the worker node is not properly set due to the modification of the command that is sent to the pod for the worker node to perform. See !Screen Shot 2018-05-24 at 5.06.25 PM.png! This means that the only DAG's that are working at the example DAGs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2532) Support logs_volume_subpath for KubernetesExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roc chan updated AIRFLOW-2532: -- Description: The kubernetes section in the configuration file supports the *dags_volume_subpath* parameter, but *logs_volume_subpath* does not support. If both are supported, the dags and the logs volume can use the same pvc, sharing the same backend storage that support mounted by multiple pod (e.g. NFS), only different subpaths. So, I added *logs_volume_subpath* parameter support. (was: The kubernetes section in the configuration file supports the *dags_volume_subpath* parameter, but *logs_volume_subpath* does not support. If both are supported, the dags and the logs volume can use the same pvc, only different subpaths) > Support logs_volume_subpath for KubernetesExecutor > -- > > Key: AIRFLOW-2532 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2532 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0, 1.10 >Reporter: roc chan >Assignee: roc chan >Priority: Major > > The kubernetes section in the configuration file supports the > *dags_volume_subpath* parameter, but *logs_volume_subpath* does not support. > If both are supported, the dags and the logs volume can use the same pvc, > sharing the same backend storage that support mounted by multiple pod (e.g. > NFS), only different subpaths. So, I added *logs_volume_subpath* parameter > support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2532) Support logs_volume_subpath for KubernetesExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roc chan updated AIRFLOW-2532: -- Description: The kubernetes section in the configuration file supports the *dags_volume_subpath* parameter, but *logs_volume_subpath* does not support. If both are supported, the dags and the logs volume can use the same pvc, only different subpaths (was: The kubernetes section in the configuration file supports the *dags_volume_subpath* parameter, but *logs_volume_subpath* does not support.) > Support logs_volume_subpath for KubernetesExecutor > -- > > Key: AIRFLOW-2532 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2532 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0, 1.10 >Reporter: roc chan >Assignee: roc chan >Priority: Major > > The kubernetes section in the configuration file supports the > *dags_volume_subpath* parameter, but *logs_volume_subpath* does not support. > If both are supported, the dags and the logs volume can use the same pvc, > only different subpaths -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (AIRFLOW-2532) Support logs_volume_subpath for KubernetesExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-2532 started by roc chan. - > Support logs_volume_subpath for KubernetesExecutor > -- > > Key: AIRFLOW-2532 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2532 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0, 1.10 >Reporter: roc chan >Assignee: roc chan >Priority: Major > > The kubernetes section in the configuration file supports the > *dags_volume_subpath* parameter, but *logs_volume_subpath* does not support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2532) Support logs_volume_subpath for KubernetesExecutor
roc chan created AIRFLOW-2532: - Summary: Support logs_volume_subpath for KubernetesExecutor Key: AIRFLOW-2532 URL: https://issues.apache.org/jira/browse/AIRFLOW-2532 Project: Apache Airflow Issue Type: Improvement Components: contrib Affects Versions: 2.0.0, 1.10 Reporter: roc chan Assignee: roc chan The kubernetes section in the configuration file supports the *dags_volume_subpath* parameter, but *logs_volume_subpath* does not support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
