[jira] [Updated] (AIRFLOW-940) Failing to unencrypt one variable shouldn't break the admin/variable page
[ https://issues.apache.org/jira/browse/AIRFLOW-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Davydov updated AIRFLOW-940: Labels: beginner starter (was: ) > Failing to unencrypt one variable shouldn't break the admin/variable page > - > > Key: AIRFLOW-940 > URL: https://issues.apache.org/jira/browse/AIRFLOW-940 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Dan Davydov > Labels: beginner, starter > > At the moment failing to decrypt one of the airflow variables in the database > using the fernet key will cause the admin/variable page to crash. Instead > there should be an error message for only the broken variables at the top of > the page, and the other ones should still be loaded and displayed. > This is what the trace on admin/variable currently looks like: > {code} > Traceback (most recent call last): > File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in > wsgi_app > response = self.full_dispatch_request() > File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in > full_dispatch_request > rv = self.handle_user_exception(e) > File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in > handle_user_exception > reraise(exc_type, exc_value, tb) > File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in > full_dispatch_request > rv = self.dispatch_request() > File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in > dispatch_request > return self.view_functions[rule.endpoint](**req.view_args) > File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 68, > in inner > return self._run_view(f, *args, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line > 367, in _run_view > return fn(self, *args, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", > line 1814, in index_view > return_url=self._get_list_url(view_args), > File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line > 307, in render > return render_template(template, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/flask/templating.py", line > 128, in render_template > context, ctx.app) > File "/usr/local/lib/python2.7/dist-packages/flask/templating.py", line > 110, in _render > rv = template.render(context) > File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line > 989, in render > return self.environment.handle_exception(exc_info, True) > File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line > 754, in handle_exception > reraise(exc_type, exc_value, tb) > File > "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/airflow/variable_list.html", > line 18, in top-level template code > {% extends 'admin/model/list.html' %} > File > "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", > line 5, in top-level template code > {% import 'admin/actions.html' as actionlib with context %} > File > "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/admin/master.html", > line 18, in top-level template code > {% extends 'admin/base.html' %} > File > "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/base.html", > line 30, in top-level template code > {% block page_body %} > File > "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/admin/master.html", > line 96, in block "page_body" > {% block body %} > File > "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", > line 62, in block "body" > {% block model_list_table %} > File > "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", > line 110, in block "model_list_table" > {% block list_row scoped %} > File > "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", > line 165, in block "list_row" > {{ get_value(row, c) }} > File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", > line 1666, in get_list_value > self.column_type_formatters, > File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", > line 1631, in _get_list_value > value = column_fmt(self, context, model, name) > File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line > 2146, in hidden_field_formatter > return getattr(model, name) > File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", > line 293, in __get__ > return self.descriptor.__get__(instance, owner) > File
[jira] [Created] (AIRFLOW-940) Failing to unencrypt one variable shouldn't break the admin/variable page
Dan Davydov created AIRFLOW-940: --- Summary: Failing to unencrypt one variable shouldn't break the admin/variable page Key: AIRFLOW-940 URL: https://issues.apache.org/jira/browse/AIRFLOW-940 Project: Apache Airflow Issue Type: Bug Components: webserver Reporter: Dan Davydov At the moment failing to decrypt one of the airflow variables in the database using the fernet key will cause the admin/variable page to crash. Instead there should be an error message for only the broken variables at the top of the page, and the other ones should still be loaded and displayed. This is what the trace on admin/variable currently looks like: {code} Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 68, in inner return self._run_view(f, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 367, in _run_view return fn(self, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", line 1814, in index_view return_url=self._get_list_url(view_args), File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 307, in render return render_template(template, **kwargs) File "/usr/local/lib/python2.7/dist-packages/flask/templating.py", line 128, in render_template context, ctx.app) File "/usr/local/lib/python2.7/dist-packages/flask/templating.py", line 110, in _render rv = template.render(context) File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 989, in render return self.environment.handle_exception(exc_info, True) File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 754, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/airflow/variable_list.html", line 18, in top-level template code {% extends 'admin/model/list.html' %} File "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", line 5, in top-level template code {% import 'admin/actions.html' as actionlib with context %} File "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/admin/master.html", line 18, in top-level template code {% extends 'admin/base.html' %} File "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/base.html", line 30, in top-level template code {% block page_body %} File "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/admin/master.html", line 96, in block "page_body" {% block body %} File "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", line 62, in block "body" {% block model_list_table %} File "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", line 110, in block "model_list_table" {% block list_row scoped %} File "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", line 165, in block "list_row" {{ get_value(row, c) }} File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", line 1666, in get_list_value self.column_type_formatters, File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", line 1631, in _get_list_value value = column_fmt(self, context, model, name) File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 2146, in hidden_field_formatter return getattr(model, name) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 293, in __get__ return self.descriptor.__get__(instance, owner) File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 3487, in get_val return FERNET.decrypt(bytes(self._val, 'utf-8')).decode() File "/usr/local/lib/python2.7/dist-packages/cryptography/fernet.py", line 82, in decrypt raise InvalidToken InvalidToken {quote} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-940) Failing to unencrypt one variable shouldn't break the admin/variable page
[ https://issues.apache.org/jira/browse/AIRFLOW-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Davydov updated AIRFLOW-940: Description: At the moment failing to decrypt one of the airflow variables in the database using the fernet key will cause the admin/variable page to crash. Instead there should be an error message for only the broken variables at the top of the page, and the other ones should still be loaded and displayed. This is what the trace on admin/variable currently looks like: {code} Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 68, in inner return self._run_view(f, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 367, in _run_view return fn(self, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", line 1814, in index_view return_url=self._get_list_url(view_args), File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 307, in render return render_template(template, **kwargs) File "/usr/local/lib/python2.7/dist-packages/flask/templating.py", line 128, in render_template context, ctx.app) File "/usr/local/lib/python2.7/dist-packages/flask/templating.py", line 110, in _render rv = template.render(context) File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 989, in render return self.environment.handle_exception(exc_info, True) File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 754, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/airflow/variable_list.html", line 18, in top-level template code {% extends 'admin/model/list.html' %} File "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", line 5, in top-level template code {% import 'admin/actions.html' as actionlib with context %} File "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/admin/master.html", line 18, in top-level template code {% extends 'admin/base.html' %} File "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/base.html", line 30, in top-level template code {% block page_body %} File "/usr/local/lib/python2.7/dist-packages/airflow/www/templates/admin/master.html", line 96, in block "page_body" {% block body %} File "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", line 62, in block "body" {% block model_list_table %} File "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", line 110, in block "model_list_table" {% block list_row scoped %} File "/usr/local/lib/python2.7/dist-packages/flask_admin/templates/bootstrap3/admin/model/list.html", line 165, in block "list_row" {{ get_value(row, c) }} File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", line 1666, in get_list_value self.column_type_formatters, File "/usr/local/lib/python2.7/dist-packages/flask_admin/model/base.py", line 1631, in _get_list_value value = column_fmt(self, context, model, name) File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 2146, in hidden_field_formatter return getattr(model, name) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 293, in __get__ return self.descriptor.__get__(instance, owner) File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 3487, in get_val return FERNET.decrypt(bytes(self._val, 'utf-8')).decode() File "/usr/local/lib/python2.7/dist-packages/cryptography/fernet.py", line 82, in decrypt raise InvalidToken InvalidToken {code} was: At the moment failing to decrypt one of the airflow variables in the database using the fernet key will cause the admin/variable page to crash. Instead there should be an error message for only the broken variables at the top of the page, and the other ones should still be loaded and displayed. This is what the trace on admin/variable currently looks like:
[jira] [Commented] (AIRFLOW-938) SQLAlchemy query in task_stats should be compatible with Postgres
[ https://issues.apache.org/jira/browse/AIRFLOW-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895088#comment-15895088 ] ASF subversion and git services commented on AIRFLOW-938: - Commit 7764c75a7c8d1cd3d4a3f9ce021d92988bb45698 in incubator-airflow's branch refs/heads/master from [~saguziel] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7764c75 ] [AIRFLOW-938] Use test for True in task_stats queries Fix a bug with the task_stats query on postgres which doesn't support == 1. https://issues.apache.org/jira/browse/AIRFLOW-938 I've seen the other PR but I'll try to see if this method works because I believe `__eq__(True)` is just `== True`, and it is how it is down here http ://docs.sqlalchemy.org/en/latest/core/sqlelement.h tml#sqlalchemy.sql.expression.and_ (underscore is part of link) Closes #2123 from saguziel/aguziel-fix-task- stats-2 > SQLAlchemy query in task_stats should be compatible with Postgres > - > > Key: AIRFLOW-938 > URL: https://issues.apache.org/jira/browse/AIRFLOW-938 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > > Right now, we check for truthiness by comparing to 1, which is not portable > and does not work on pgsql -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-938) SQLAlchemy query in task_stats should be compatible with Postgres
[ https://issues.apache.org/jira/browse/AIRFLOW-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895087#comment-15895087 ] ASF subversion and git services commented on AIRFLOW-938: - Commit 7764c75a7c8d1cd3d4a3f9ce021d92988bb45698 in incubator-airflow's branch refs/heads/master from [~saguziel] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7764c75 ] [AIRFLOW-938] Use test for True in task_stats queries Fix a bug with the task_stats query on postgres which doesn't support == 1. https://issues.apache.org/jira/browse/AIRFLOW-938 I've seen the other PR but I'll try to see if this method works because I believe `__eq__(True)` is just `== True`, and it is how it is down here http ://docs.sqlalchemy.org/en/latest/core/sqlelement.h tml#sqlalchemy.sql.expression.and_ (underscore is part of link) Closes #2123 from saguziel/aguziel-fix-task- stats-2 > SQLAlchemy query in task_stats should be compatible with Postgres > - > > Key: AIRFLOW-938 > URL: https://issues.apache.org/jira/browse/AIRFLOW-938 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > > Right now, we check for truthiness by comparing to 1, which is not portable > and does not work on pgsql -- This message was sent by Atlassian JIRA (v6.3.15#6346)
incubator-airflow git commit: [AIRFLOW-938] Use test for True in task_stats queries
Repository: incubator-airflow Updated Branches: refs/heads/master ae44dddcb -> 7764c75a7 [AIRFLOW-938] Use test for True in task_stats queries Fix a bug with the task_stats query on postgres which doesn't support == 1. https://issues.apache.org/jira/browse/AIRFLOW-938 I've seen the other PR but I'll try to see if this method works because I believe `__eq__(True)` is just `== True`, and it is how it is down here http ://docs.sqlalchemy.org/en/latest/core/sqlelement.h tml#sqlalchemy.sql.expression.and_ (underscore is part of link) Closes #2123 from saguziel/aguziel-fix-task- stats-2 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7764c75a Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7764c75a Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7764c75a Branch: refs/heads/master Commit: 7764c75a7c8d1cd3d4a3f9ce021d92988bb45698 Parents: ae44ddd Author: Alex GuzielAuthored: Fri Mar 3 13:52:03 2017 -0800 Committer: Dan Davydov Committed: Fri Mar 3 13:52:06 2017 -0800 -- airflow/www/views.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7764c75a/airflow/www/views.py -- diff --git a/airflow/www/views.py b/airflow/www/views.py index f7d4584..de33843 100644 --- a/airflow/www/views.py +++ b/airflow/www/views.py @@ -506,7 +506,7 @@ class Airflow(BaseView): session.query(DagRun.dag_id, sqla.func.max(DagRun.execution_date).label('execution_date')) .join(Dag, Dag.dag_id == DagRun.dag_id) .filter(DagRun.state != State.RUNNING) -.filter(Dag.is_active == 1) +.filter(Dag.is_active == True) .group_by(DagRun.dag_id) .subquery('last_dag_run') ) @@ -514,7 +514,7 @@ class Airflow(BaseView): session.query(DagRun.dag_id, DagRun.execution_date) .join(Dag, Dag.dag_id == DagRun.dag_id) .filter(DagRun.state == State.RUNNING) -.filter(Dag.is_active == 1) +.filter(Dag.is_active == True) .subquery('running_dag_run') )
[jira] [Commented] (AIRFLOW-932) Backfills delete existing task instances and mark them as removed
[ https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894640#comment-15894640 ] Bolke de Bruin commented on AIRFLOW-932: PR is available with unit tests [~aoen]. Please review > Backfills delete existing task instances and mark them as removed > - > > Key: AIRFLOW-932 > URL: https://issues.apache.org/jira/browse/AIRFLOW-932 > Project: Apache Airflow > Issue Type: Sub-task > Components: backfill >Reporter: Dan Davydov >Assignee: Bolke de Bruin >Priority: Blocker > > I'm still investigating. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-846) Release schedule, latest tag is too old
[ https://issues.apache.org/jira/browse/AIRFLOW-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894565#comment-15894565 ] Ultrabug commented on AIRFLOW-846: -- BTW [~artwr], would you please be kind enough to point me where this vote / release tracking is taking place ? Looks like I'm lost on JIRA... > Release schedule, latest tag is too old > --- > > Key: AIRFLOW-846 > URL: https://issues.apache.org/jira/browse/AIRFLOW-846 > Project: Apache Airflow > Issue Type: Task >Reporter: Ultrabug >Priority: Blocker > Labels: release, tagging > > To my understanding, there is no clear point about the release schedule of > the project. > The latest tag is 1.7.1.3 from June 2016, which is not well suited for > production now days. > For example, the latest available release is still affected by AIRFLOW-178 > which means that we have to patch the sources on production to work with ZIP > files. > Could you please share your thoughts and position on the release planning of > the project ? > Would it be possible to get a newer tag sometimes soon ? > Thank you -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-846) Release schedule, latest tag is too old
[ https://issues.apache.org/jira/browse/AIRFLOW-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894564#comment-15894564 ] Ultrabug commented on AIRFLOW-846: -- Thanks [~artwr], there are quite a bunch of people happy to hear this :) > Release schedule, latest tag is too old > --- > > Key: AIRFLOW-846 > URL: https://issues.apache.org/jira/browse/AIRFLOW-846 > Project: Apache Airflow > Issue Type: Task >Reporter: Ultrabug >Priority: Blocker > Labels: release, tagging > > To my understanding, there is no clear point about the release schedule of > the project. > The latest tag is 1.7.1.3 from June 2016, which is not well suited for > production now days. > For example, the latest available release is still affected by AIRFLOW-178 > which means that we have to patch the sources on production to work with ZIP > files. > Could you please share your thoughts and position on the release planning of > the project ? > Would it be possible to get a newer tag sometimes soon ? > Thank you -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-846) Release schedule, latest tag is too old
[ https://issues.apache.org/jira/browse/AIRFLOW-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894536#comment-15894536 ] Arthur Wiedmer commented on AIRFLOW-846: Hi [~ultrabug], We are on RC5 now, and will release to PyPI once the current blockers are cleared, and a new vote on the release is taken. All of this combined might take nother couple of weeks. Best, Arthur > Release schedule, latest tag is too old > --- > > Key: AIRFLOW-846 > URL: https://issues.apache.org/jira/browse/AIRFLOW-846 > Project: Apache Airflow > Issue Type: Task >Reporter: Ultrabug >Priority: Blocker > Labels: release, tagging > > To my understanding, there is no clear point about the release schedule of > the project. > The latest tag is 1.7.1.3 from June 2016, which is not well suited for > production now days. > For example, the latest available release is still affected by AIRFLOW-178 > which means that we have to patch the sources on production to work with ZIP > files. > Could you please share your thoughts and position on the release planning of > the project ? > Would it be possible to get a newer tag sometimes soon ? > Thank you -- This message was sent by Atlassian JIRA (v6.3.15#6346)