[jira] [Created] (AIRFLOW-2606) Test needed to ensure database schema always match SQLAlchemy models
Joy Gao created AIRFLOW-2606: Summary: Test needed to ensure database schema always match SQLAlchemy models Key: AIRFLOW-2606 URL: https://issues.apache.org/jira/browse/AIRFLOW-2606 Project: Apache Airflow Issue Type: Improvement Reporter: Joy Gao An issue was discovered by [this PR|https://github.com/apache/incubator-airflow/pull/3492#issuecomment-396815203] where database schema does not match its corresponding SQLAlchemy model declaration. We should add generic unit test for this to prevent similar bugs from occurring in the future. (Alternatively, we can add the policing logic to `airflow upgradedb` command so each migrations can do the check) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2605) MySqlHook().run() will not commit if autocommit is set to True.
Kevin Yang created AIRFLOW-2605: --- Summary: MySqlHook().run() will not commit if autocommit is set to True. Key: AIRFLOW-2605 URL: https://issues.apache.org/jira/browse/AIRFLOW-2605 Project: Apache Airflow Issue Type: Bug Reporter: Kevin Yang Assignee: Kevin Yang MySql [set autocommit in a different way|https://github.com/PyMySQL/mysqlclient-python/blob/master/MySQLdb/connections.py#L249-L256]. Thus setting it by doing `conn.autocommit = True` as we currently do will not set autocommit correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2604) dag_id, task_id, execution_date in dag_fail should be indexed
Joy Gao created AIRFLOW-2604: Summary: dag_id, task_id, execution_date in dag_fail should be indexed Key: AIRFLOW-2604 URL: https://issues.apache.org/jira/browse/AIRFLOW-2604 Project: Apache Airflow Issue Type: Improvement Affects Versions: 1.10 Reporter: Joy Gao As a follow-up to AIRFLOW-2602, we should index dag_id, task_id and execution_date to make sure the /gantt page (and any other future UIs replying on task_fail) can still be rendered quickly as the table grows in size. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2414) Fix RBAC log display
[
https://issues.apache.org/jira/browse/AIRFLOW-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joy Gao closed AIRFLOW-2414.
Resolution: Fixed
> Fix RBAC log display
>
>
> Key: AIRFLOW-2414
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2414
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
>Affects Versions: 1.10.0
>Reporter: Oleg Yamin
>Assignee: Oleg Yamin
>Priority: Major
> Fix For: 1.10.0
>
>
> Getting the following error when trying to view the log file in new RBAC UI.
> {code:java}
> [2018-05-02 17:49:47,716] ERROR in app: Exception on /log [GET]
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1982, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1614, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1517, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1612, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1598, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/usr/lib/python2.7/site-packages/flask_appbuilder/security/decorators.py",
> line 26, in wraps
> return f(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/www_rbac/decorators.py", line
> 55, in wrapper
> return f(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in
> wrapper
> return func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/www_rbac/views.py", line 456,
> in log
> logs = log.decode('utf-8')
> AttributeError: 'list' object has no attribute 'decode'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2414) Fix RBAC log display
[
https://issues.apache.org/jira/browse/AIRFLOW-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510358#comment-16510358
]
Joy Gao commented on AIRFLOW-2414:
--
A fix for this was merged recently:
[https://github.com/apache/incubator-airflow/pull/3310]
[~rushtokunal] let me know if you are still seeing issues with this. Going to
close the ticket for now.
> Fix RBAC log display
>
>
> Key: AIRFLOW-2414
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2414
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
>Affects Versions: 1.10.0
>Reporter: Oleg Yamin
>Assignee: Oleg Yamin
>Priority: Major
> Fix For: 1.10.0
>
>
> Getting the following error when trying to view the log file in new RBAC UI.
> {code:java}
> [2018-05-02 17:49:47,716] ERROR in app: Exception on /log [GET]
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1982, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1614, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1517, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1612, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1598, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/usr/lib/python2.7/site-packages/flask_appbuilder/security/decorators.py",
> line 26, in wraps
> return f(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/www_rbac/decorators.py", line
> 55, in wrapper
> return f(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in
> wrapper
> return func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/www_rbac/views.py", line 456,
> in log
> logs = log.decode('utf-8')
> AttributeError: 'list' object has no attribute 'decode'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2512) GoogleCloudBaseHook using deprecated oauth2client
[ https://issues.apache.org/jira/browse/AIRFLOW-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2512. - Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3488 [https://github.com/apache/incubator-airflow/pull/3488] > GoogleCloudBaseHook using deprecated oauth2client > - > > Key: AIRFLOW-2512 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2512 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Reporter: Tim Swast >Assignee: Tim Swast >Priority: Minor > Fix For: 2.0.0 > > > oauth2client is deprecated in favor of the google-auth library. > [https://google-auth.readthedocs.io/en/latest/oauth2client-deprecation.html] > The GoogleCloudBaseHook should be updated to use the google-auth library. > Note: to use the google-auth library with the google-api-python-client, the > API client library may also require an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2512) GoogleCloudBaseHook using deprecated oauth2client
[ https://issues.apache.org/jira/browse/AIRFLOW-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510350#comment-16510350 ] ASF subversion and git services commented on AIRFLOW-2512: -- Commit 0f4d681f6f6e15acd1399dede146e75cb688d536 in incubator-airflow's branch refs/heads/master from Tim Swast [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0f4d681 ] [AIRFLOW-2512][AIRFLOW-2522] Use google-auth instead of oauth2client * Updates the GCP hooks to use the google-auth library and removes dependencies on the deprecated oauth2client package. * Removes inconsistent handling of the scope parameter for different auth methods. Note: using google-auth for credentials requires a newer version of the google-api-python-client package, so this commit also updates the minimum version for that. To avoid some annoying warnings about the discovery cache not being supported, so disable the discovery cache explicitly as recommend here: https://stackoverflow.com/a/44518587/101923 Tested by running: nosetests tests/contrib/operators/test_dataflow_operator.py \ tests/contrib/operators/test_gcs*.py \ tests/contrib/operators/test_mlengine_*.py \ tests/contrib/operators/test_pubsub_operator.py \ tests/contrib/hooks/test_gcp*.py \ tests/contrib/hooks/test_gcs_hook.py \ tests/contrib/hooks/test_bigquery_hook.py and also tested by running some GCP-related DAGs locally, such as the Dataproc DAG example at https://cloud.google.com/composer/docs/quickstart Closes #3488 from tswast/google-auth > GoogleCloudBaseHook using deprecated oauth2client > - > > Key: AIRFLOW-2512 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2512 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Reporter: Tim Swast >Assignee: Tim Swast >Priority: Minor > > oauth2client is deprecated in favor of the google-auth library. > [https://google-auth.readthedocs.io/en/latest/oauth2client-deprecation.html] > The GoogleCloudBaseHook should be updated to use the google-auth library. > Note: to use the google-auth library with the google-api-python-client, the > API client library may also require an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2522) Cannot use GOOGLE_APPLICATION_CREDENTIALS to authenticate for GCP connections
[ https://issues.apache.org/jira/browse/AIRFLOW-2522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2522. - Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3488 [https://github.com/apache/incubator-airflow/pull/3488] > Cannot use GOOGLE_APPLICATION_CREDENTIALS to authenticate for GCP connections > - > > Key: AIRFLOW-2522 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2522 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Reporter: Tim Swast >Assignee: Tim Swast >Priority: Major > Fix For: 2.0.0 > > > If you try to use the GOOGLE_APPLICATION_CREDENTIALS environment variable > with a service account key to authenticate to Google Cloud, as described at > [https://cloud.google.com/docs/authentication/production] you get an error > "HttpAccessTokenRefreshError: invalid_scope: Empty or missing scope not > allowed." > This error occurs even if you fill in the scope field of the GCP connection. > The root cause is that scopes are ignored by the GCP hook when using > application default credentials. They should not be ignored when the default > credentials are using a service account. (And probably shouldn't be ignored > at all, preferring an error when scopes are filled in but don't apply to the > credential type) > I'll try to fix this while I'm working on > https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-2512. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2522) Cannot use GOOGLE_APPLICATION_CREDENTIALS to authenticate for GCP connections
[ https://issues.apache.org/jira/browse/AIRFLOW-2522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510351#comment-16510351 ] ASF subversion and git services commented on AIRFLOW-2522: -- Commit 0f4d681f6f6e15acd1399dede146e75cb688d536 in incubator-airflow's branch refs/heads/master from Tim Swast [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0f4d681 ] [AIRFLOW-2512][AIRFLOW-2522] Use google-auth instead of oauth2client * Updates the GCP hooks to use the google-auth library and removes dependencies on the deprecated oauth2client package. * Removes inconsistent handling of the scope parameter for different auth methods. Note: using google-auth for credentials requires a newer version of the google-api-python-client package, so this commit also updates the minimum version for that. To avoid some annoying warnings about the discovery cache not being supported, so disable the discovery cache explicitly as recommend here: https://stackoverflow.com/a/44518587/101923 Tested by running: nosetests tests/contrib/operators/test_dataflow_operator.py \ tests/contrib/operators/test_gcs*.py \ tests/contrib/operators/test_mlengine_*.py \ tests/contrib/operators/test_pubsub_operator.py \ tests/contrib/hooks/test_gcp*.py \ tests/contrib/hooks/test_gcs_hook.py \ tests/contrib/hooks/test_bigquery_hook.py and also tested by running some GCP-related DAGs locally, such as the Dataproc DAG example at https://cloud.google.com/composer/docs/quickstart Closes #3488 from tswast/google-auth > Cannot use GOOGLE_APPLICATION_CREDENTIALS to authenticate for GCP connections > - > > Key: AIRFLOW-2522 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2522 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Reporter: Tim Swast >Assignee: Tim Swast >Priority: Major > Fix For: 2.0.0 > > > If you try to use the GOOGLE_APPLICATION_CREDENTIALS environment variable > with a service account key to authenticate to Google Cloud, as described at > [https://cloud.google.com/docs/authentication/production] you get an error > "HttpAccessTokenRefreshError: invalid_scope: Empty or missing scope not > allowed." > This error occurs even if you fill in the scope field of the GCP connection. > The root cause is that scopes are ignored by the GCP hook when using > application default credentials. They should not be ignored when the default > credentials are using a service account. (And probably shouldn't be ignored > at all, preferring an error when scopes are filled in but don't apply to the > credential type) > I'll try to fix this while I'm working on > https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-2512. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2512][AIRFLOW-2522] Use google-auth instead of oauth2client
Repository: incubator-airflow
Updated Branches:
refs/heads/master cff8318b9 -> 0f4d681f6
[AIRFLOW-2512][AIRFLOW-2522] Use google-auth instead of oauth2client
* Updates the GCP hooks to use the google-auth
library and removes
dependencies on the deprecated oauth2client
package.
* Removes inconsistent handling of the scope
parameter for different
auth methods.
Note: using google-auth for credentials requires a
newer version of the
google-api-python-client package, so this commit
also updates the
minimum version for that.
To avoid some annoying warnings about the
discovery cache not being
supported, so disable the discovery cache
explicitly as recommend here:
https://stackoverflow.com/a/44518587/101923
Tested by running:
nosetests
tests/contrib/operators/test_dataflow_operator.py
\
tests/contrib/operators/test_gcs*.py \
tests/contrib/operators/test_mlengine_*.py \
tests/contrib/operators/test_pubsub_operator.py \
tests/contrib/hooks/test_gcp*.py \
tests/contrib/hooks/test_gcs_hook.py \
tests/contrib/hooks/test_bigquery_hook.py
and also tested by running some GCP-related DAGs
locally, such as the
Dataproc DAG example at
https://cloud.google.com/composer/docs/quickstart
Closes #3488 from tswast/google-auth
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/0f4d681f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/0f4d681f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/0f4d681f
Branch: refs/heads/master
Commit: 0f4d681f6f6e15acd1399dede146e75cb688d536
Parents: cff8318
Author: Tim Swast
Authored: Tue Jun 12 23:53:21 2018 +0100
Committer: Kaxil Naik
Committed: Tue Jun 12 23:53:21 2018 +0100
--
airflow/contrib/hooks/bigquery_hook.py| 3 +-
airflow/contrib/hooks/datastore_hook.py | 3 +-
airflow/contrib/hooks/gcp_api_base_hook.py| 52 --
airflow/contrib/hooks/gcp_dataflow_hook.py| 3 +-
airflow/contrib/hooks/gcp_dataproc_hook.py| 4 +-
airflow/contrib/hooks/gcp_mlengine_hook.py| 5 +-
airflow/contrib/hooks/gcp_pubsub_hook.py | 3 +-
airflow/contrib/hooks/gcs_hook.py | 3 +-
docs/conf.py | 4 +-
scripts/ci/requirements.txt | 5 +-
setup.py | 7 +-
tests/contrib/hooks/test_bigquery_hook.py | 14 ++--
tests/contrib/hooks/test_gcp_api_base_hook.py | 83 ++
tests/contrib/hooks/test_gcp_mlengine_hook.py | 17 +++--
14 files changed, 154 insertions(+), 52 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0f4d681f/airflow/contrib/hooks/bigquery_hook.py
--
diff --git a/airflow/contrib/hooks/bigquery_hook.py
b/airflow/contrib/hooks/bigquery_hook.py
index 3fc0b37..b452f51 100644
--- a/airflow/contrib/hooks/bigquery_hook.py
+++ b/airflow/contrib/hooks/bigquery_hook.py
@@ -72,7 +72,8 @@ class BigQueryHook(GoogleCloudBaseHook, DbApiHook,
LoggingMixin):
Returns a BigQuery service object.
"""
http_authorized = self._authorize()
-return build('bigquery', 'v2', http=http_authorized)
+return build(
+'bigquery', 'v2', http=http_authorized, cache_discovery=False)
def insert_rows(self, table, rows, target_fields=None, commit_every=1000):
"""
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0f4d681f/airflow/contrib/hooks/datastore_hook.py
--
diff --git a/airflow/contrib/hooks/datastore_hook.py
b/airflow/contrib/hooks/datastore_hook.py
index bbd1199..5e54cf2 100644
--- a/airflow/contrib/hooks/datastore_hook.py
+++ b/airflow/contrib/hooks/datastore_hook.py
@@ -44,7 +44,8 @@ class DatastoreHook(GoogleCloudBaseHook):
Returns a Google Cloud Storage service object.
"""
http_authorized = self._authorize()
-return build('datastore', version, http=http_authorized)
+return build(
+'datastore', version, http=http_authorized, cache_discovery=False)
def allocate_ids(self, partialKeys):
"""
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0f4d681f/airflow/contrib/hooks/gcp_api_base_hook.py
--
diff --git a/airflow/contrib/hooks/gcp_api_base_hook.py
b/airflow/contrib/hooks/gcp_api_base_hook.py
index 852854b..0534947 100644
--- a/airflow/contrib/hooks/gcp_api_base_hook.py
+++ b/airflow/contrib/hooks/gcp_api_base_hook.py
@@ -20,14 +20,18 @@
import json
import httplib2
-from oauth2client.client import GoogleCredentials
-from oauth2client.se
[jira] [Commented] (AIRFLOW-2602) Show failed attempts in Gantt view
[ https://issues.apache.org/jira/browse/AIRFLOW-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510307#comment-16510307 ] Stefan Seelmann commented on AIRFLOW-2602: -- Screenshot of Gantt view with failed attempts: !Screenshot_2018-06-13_00-13-21.png! > Show failed attempts in Gantt view > -- > > Key: AIRFLOW-2602 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2602 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.9.0 >Reporter: Stefan Seelmann >Assignee: Stefan Seelmann >Priority: Major > Fix For: Airflow 2.0 > > Attachments: Screenshot_2018-06-13_00-13-21.png > > > The Gantt view only shows the last attempt (successful or failed). It would > be nice to also visualize failed attempts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2602) Show failed attempts in Gantt view
[ https://issues.apache.org/jira/browse/AIRFLOW-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Seelmann updated AIRFLOW-2602: - Description: The Gantt view only shows the last attempt (successful or failed). It would be nice to also visualize failed attempts. (was: The Gantt view only shows the last attempt (successful or failed). It would be nice to also visulize failed attempts.) > Show failed attempts in Gantt view > -- > > Key: AIRFLOW-2602 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2602 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.9.0 >Reporter: Stefan Seelmann >Assignee: Stefan Seelmann >Priority: Major > Fix For: Airflow 2.0 > > Attachments: Screenshot_2018-06-13_00-13-21.png > > > The Gantt view only shows the last attempt (successful or failed). It would > be nice to also visualize failed attempts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2414) Fix RBAC log display
[
https://issues.apache.org/jira/browse/AIRFLOW-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510306#comment-16510306
]
kunal gupta commented on AIRFLOW-2414:
--
I don't think it is related to subdags, i created a simple DAG and im seeing
the same error while trying to see the logs from the UI
> Fix RBAC log display
>
>
> Key: AIRFLOW-2414
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2414
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
>Affects Versions: 1.10.0
>Reporter: Oleg Yamin
>Assignee: Oleg Yamin
>Priority: Major
> Fix For: 1.10.0
>
>
> Getting the following error when trying to view the log file in new RBAC UI.
> {code:java}
> [2018-05-02 17:49:47,716] ERROR in app: Exception on /log [GET]
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1982, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1614, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1517, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1612, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1598, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/usr/lib/python2.7/site-packages/flask_appbuilder/security/decorators.py",
> line 26, in wraps
> return f(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/www_rbac/decorators.py", line
> 55, in wrapper
> return f(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in
> wrapper
> return func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/airflow/www_rbac/views.py", line 456,
> in log
> logs = log.decode('utf-8')
> AttributeError: 'list' object has no attribute 'decode'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2602) Show failed attempts in Gantt view
[ https://issues.apache.org/jira/browse/AIRFLOW-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Seelmann updated AIRFLOW-2602: - Attachment: Screenshot_2018-06-13_00-13-21.png > Show failed attempts in Gantt view > -- > > Key: AIRFLOW-2602 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2602 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.9.0 >Reporter: Stefan Seelmann >Assignee: Stefan Seelmann >Priority: Major > Fix For: Airflow 2.0 > > Attachments: Screenshot_2018-06-13_00-13-21.png > > > The Gantt view only shows the last attempt (successful or failed). It would > be nice to also visualize failed attempts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1104) Concurrency check in scheduler should count queued tasks as well as running
[ https://issues.apache.org/jira/browse/AIRFLOW-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510304#comment-16510304 ] Dan Fowler commented on AIRFLOW-1104: - Thanks for taking a look, I agree, my fix for this was a small change. I just want to make sure that change wouldn't cause other issues. > Concurrency check in scheduler should count queued tasks as well as running > --- > > Key: AIRFLOW-1104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1104 > Project: Apache Airflow > Issue Type: Bug > Environment: see https://github.com/apache/incubator-airflow/pull/2221 > "Tasks with the QUEUED state should also be counted below, but for now we > cannot count them. This is because there is no guarantee that queued tasks in > failed dagruns will or will not eventually run and queued tasks that will > never run will consume slots and can stall a DAG. Once we can guarantee that > all queued tasks in failed dagruns will never run (e.g. make sure that all > running/newly queued TIs have running dagruns), then we can include QUEUED > tasks here, with the constraint that they are in running dagruns." >Reporter: Alex Guziel >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1104) Concurrency check in scheduler should count queued tasks as well as running
[ https://issues.apache.org/jira/browse/AIRFLOW-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510224#comment-16510224 ] Tao Feng commented on AIRFLOW-1104: --- hey [~dfowler] , looking at the history of the change. I think the change is easy, but it is hard to ignore or reproduce the issue(task instance could associated with a fail dag run when origin pr is created). I think it would be good if [~saguziel] could comment in current state of art whether this is still an issue. > Concurrency check in scheduler should count queued tasks as well as running > --- > > Key: AIRFLOW-1104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1104 > Project: Apache Airflow > Issue Type: Bug > Environment: see https://github.com/apache/incubator-airflow/pull/2221 > "Tasks with the QUEUED state should also be counted below, but for now we > cannot count them. This is because there is no guarantee that queued tasks in > failed dagruns will or will not eventually run and queued tasks that will > never run will consume slots and can stall a DAG. Once we can guarantee that > all queued tasks in failed dagruns will never run (e.g. make sure that all > running/newly queued TIs have running dagruns), then we can include QUEUED > tasks here, with the constraint that they are in running dagruns." >Reporter: Alex Guziel >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1104) Concurrency check in scheduler should count queued tasks as well as running
[ https://issues.apache.org/jira/browse/AIRFLOW-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anonymous reassigned AIRFLOW-1104: -- Assignee: (was: Tao Feng) > Concurrency check in scheduler should count queued tasks as well as running > --- > > Key: AIRFLOW-1104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1104 > Project: Apache Airflow > Issue Type: Bug > Environment: see https://github.com/apache/incubator-airflow/pull/2221 > "Tasks with the QUEUED state should also be counted below, but for now we > cannot count them. This is because there is no guarantee that queued tasks in > failed dagruns will or will not eventually run and queued tasks that will > never run will consume slots and can stall a DAG. Once we can guarantee that > all queued tasks in failed dagruns will never run (e.g. make sure that all > running/newly queued TIs have running dagruns), then we can include QUEUED > tasks here, with the constraint that they are in running dagruns." >Reporter: Alex Guziel >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2603) Set up Intersphinx for cross-project references in the docs
Tim Swast created AIRFLOW-2603: -- Summary: Set up Intersphinx for cross-project references in the docs Key: AIRFLOW-2603 URL: https://issues.apache.org/jira/browse/AIRFLOW-2603 Project: Apache Airflow Issue Type: Improvement Components: Documentation Reporter: Tim Swast Currently there are many modules mocked out in docs/conf.py These modules could be unmocked and linked to from the API reference documentation by using Intersphinx. For example, see how this is configured in Pandas at https://github.com/pandas-dev/pandas/blob/master/doc/source/conf.py#L366-L374 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2603) Set up Intersphinx for cross-project references in the docs
[ https://issues.apache.org/jira/browse/AIRFLOW-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Swast reassigned AIRFLOW-2603: -- Assignee: Tim Swast > Set up Intersphinx for cross-project references in the docs > --- > > Key: AIRFLOW-2603 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2603 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Tim Swast >Assignee: Tim Swast >Priority: Minor > > Currently there are many modules mocked out in docs/conf.py These modules > could be unmocked and linked to from the API reference documentation by using > Intersphinx. > > For example, see how this is configured in Pandas at > https://github.com/pandas-dev/pandas/blob/master/doc/source/conf.py#L366-L374 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2429) Make Airflow Flake8 compliant
[ https://issues.apache.org/jira/browse/AIRFLOW-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510184#comment-16510184 ] ASF subversion and git services commented on AIRFLOW-2429: -- Commit cff8318b9907416545c7bda517eb69ad3ccb6f4d in incubator-airflow's branch refs/heads/master from Tao feng [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=cff8318 ] [AIRFLOW-2429] Fix operators folder flake8 error Closes #3481 from feng-tao/flake8_p6 > Make Airflow Flake8 compliant > -- > > Key: AIRFLOW-2429 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2429 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Fokko Driesprong >Assignee: Tao Feng >Priority: Major > Fix For: 1.10.0, 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2429] Fix operators folder flake8 error
Repository: incubator-airflow
Updated Branches:
refs/heads/master 48fccefab -> cff8318b9
[AIRFLOW-2429] Fix operators folder flake8 error
Closes #3481 from feng-tao/flake8_p6
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/cff8318b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/cff8318b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/cff8318b
Branch: refs/heads/master
Commit: cff8318b9907416545c7bda517eb69ad3ccb6f4d
Parents: 48fccef
Author: Tao feng
Authored: Tue Jun 12 22:47:31 2018 +0200
Committer: Fokko Driesprong
Committed: Tue Jun 12 22:47:31 2018 +0200
--
airflow/operators/__init__.py | 5 +-
airflow/operators/check_operator.py | 2 +-
airflow/operators/docker_operator.py| 44 ++-
airflow/operators/dummy_operator.py | 4 +-
airflow/operators/generic_transfer.py | 5 +-
airflow/operators/hive_operator.py | 5 +-
airflow/operators/hive_stats_operator.py| 4 +-
airflow/operators/hive_to_druid.py | 18 +++--
airflow/operators/hive_to_mysql.py | 9 ++-
airflow/operators/hive_to_samba_operator.py | 5 +-
airflow/operators/http_operator.py | 4 +-
airflow/operators/jdbc_operator.py | 5 +-
airflow/operators/latest_only_operator.py | 5 +-
airflow/operators/mssql_operator.py | 5 +-
airflow/operators/mssql_to_hive.py | 7 +-
airflow/operators/mysql_operator.py | 5 +-
airflow/operators/mysql_to_hive.py | 4 +-
airflow/operators/oracle_operator.py| 5 +-
airflow/operators/pig_operator.py | 5 +-
airflow/operators/postgres_operator.py | 4 +-
airflow/operators/presto_check_operator.py | 7 +-
airflow/operators/presto_to_mysql.py| 5 +-
airflow/operators/python_operator.py| 77
airflow/operators/redshift_to_s3_operator.py| 5 +-
airflow/operators/s3_file_transform_operator.py | 4 +-
airflow/operators/s3_to_hive_operator.py| 16 ++--
airflow/operators/s3_to_redshift_operator.py| 5 +-
airflow/operators/sensors.py| 6 +-
airflow/operators/slack_operator.py | 19 +++--
airflow/operators/sqlite_operator.py| 5 +-
30 files changed, 173 insertions(+), 126 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/cff8318b/airflow/operators/__init__.py
--
diff --git a/airflow/operators/__init__.py b/airflow/operators/__init__.py
index 003b855..efdfd3e 100644
--- a/airflow/operators/__init__.py
+++ b/airflow/operators/__init__.py
@@ -7,16 +7,15 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-#
import sys
import os
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/cff8318b/airflow/operators/check_operator.py
--
diff --git a/airflow/operators/check_operator.py
b/airflow/operators/check_operator.py
index 6a2cbb0..5a31737 100644
--- a/airflow/operators/check_operator.py
+++ b/airflow/operators/check_operator.py
@@ -217,7 +217,7 @@ class IntervalCheckOperator(BaseOperator):
sqlt = ("SELECT {sqlexp} FROM {table}"
" WHERE {date_filter_column}=").format(**locals())
self.sql1 = sqlt + "'{{ ds }}'"
-self.sql2 = sqlt + "'{{ macros.ds_add(ds, "+str(self.days_back)+") }}'"
+self.sql2 = sqlt + "'{{ macros.ds_add(ds, " + str(self.days_back) + ")
}}'"
def execute(self, context=None):
hook = self.get_db_hook()
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/cff8318b/airflow/operators/docker_operator.py
--
diff --git a/airflow/operators/docker_operator.py
b/airflow/operators/docker_operator.py
index 421ce49..69dc1eb 100644
--- a/airflow/operators/docker_operator.py
+++ b/airflow/operators/docker_operator.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. Y
[jira] [Created] (AIRFLOW-2602) Show failed attempts in Gantt view
Stefan Seelmann created AIRFLOW-2602: Summary: Show failed attempts in Gantt view Key: AIRFLOW-2602 URL: https://issues.apache.org/jira/browse/AIRFLOW-2602 Project: Apache Airflow Issue Type: Improvement Components: webapp Affects Versions: 1.9.0 Reporter: Stefan Seelmann Assignee: Stefan Seelmann Fix For: Airflow 2.0 The Gantt view only shows the last attempt (successful or failed). It would be nice to also visulize failed attempts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2585) Bug fix in CassandraHook and CassandraToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-2585. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3483 [https://github.com/apache/incubator-airflow/pull/3483] > Bug fix in CassandraHook and CassandraToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-2585 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2585 > Project: Apache Airflow > Issue Type: Bug >Reporter: Joy Gao >Assignee: Joy Gao >Priority: Major > Fix For: 2.0.0 > > > * Issue with UUID type conversion: currently UUID is converted to hex string, > but should be converted to base64-encoded as that is the required format in > BigQuery for uploading. > * Issue with configuring load balancing policy in CassandraHook: currently > the hook only successfully instantiate with the default LB policy, but throw > an exception if attempts to pass in a custom LB policy in the extra field. > * Issue with connections not closed properly after use: should always shut > down the cluster in the operator to close all sessions/connections associated > with the cluster instance. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2585) Bug fix in CassandraHook and CassandraToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510171#comment-16510171 ] ASF subversion and git services commented on AIRFLOW-2585: -- Commit 48fccefaba35f16dfff10b7629f64844bb850553 in incubator-airflow's branch refs/heads/master from Joy Gao [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=48fccef ] [AIRFLOW-2585] Fix several bugs in CassandraHook and CassandraToGCSOperator Closes #3483 from jgao54/fix-cassandra-hook-bug > Bug fix in CassandraHook and CassandraToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-2585 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2585 > Project: Apache Airflow > Issue Type: Bug >Reporter: Joy Gao >Assignee: Joy Gao >Priority: Major > > * Issue with UUID type conversion: currently UUID is converted to hex string, > but should be converted to base64-encoded as that is the required format in > BigQuery for uploading. > * Issue with configuring load balancing policy in CassandraHook: currently > the hook only successfully instantiate with the default LB policy, but throw > an exception if attempts to pass in a custom LB policy in the extra field. > * Issue with connections not closed properly after use: should always shut > down the cluster in the operator to close all sessions/connections associated > with the cluster instance. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2585] Fix several bugs in CassandraHook and CassandraToGCSOperator
Repository: incubator-airflow
Updated Branches:
refs/heads/master 5bb547a85 -> 48fccefab
[AIRFLOW-2585] Fix several bugs in CassandraHook and CassandraToGCSOperator
Closes #3483 from jgao54/fix-cassandra-hook-bug
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/48fccefa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/48fccefa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/48fccefa
Branch: refs/heads/master
Commit: 48fccefaba35f16dfff10b7629f64844bb850553
Parents: 5bb547a
Author: Joy Gao
Authored: Tue Jun 12 22:39:26 2018 +0200
Committer: Fokko Driesprong
Committed: Tue Jun 12 22:39:26 2018 +0200
--
airflow/contrib/hooks/cassandra_hook.py | 100 ---
airflow/contrib/operators/cassandra_to_gcs.py | 20 ++--
tests/contrib/hooks/test_cassandra_hook.py | 70 -
.../operators/test_cassandra_to_gcs_operator.py | 13 +--
4 files changed, 172 insertions(+), 31 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/48fccefa/airflow/contrib/hooks/cassandra_hook.py
--
diff --git a/airflow/contrib/hooks/cassandra_hook.py
b/airflow/contrib/hooks/cassandra_hook.py
index 90046a8..6fce98b 100644
--- a/airflow/contrib/hooks/cassandra_hook.py
+++ b/airflow/contrib/hooks/cassandra_hook.py
@@ -19,8 +19,7 @@
from cassandra.cluster import Cluster
from cassandra.policies import (RoundRobinPolicy, DCAwareRoundRobinPolicy,
-TokenAwarePolicy, HostFilterPolicy,
-WhiteListRoundRobinPolicy)
+TokenAwarePolicy, WhiteListRoundRobinPolicy)
from cassandra.auth import PlainTextAuthProvider
from airflow.hooks.base_hook import BaseHook
@@ -31,12 +30,45 @@ class CassandraHook(BaseHook, LoggingMixin):
"""
Hook used to interact with Cassandra
-Contact_points can be specified as a comma-separated string in the 'hosts'
-field of the connection. Port can be specified in the port field of the
-connection. Load_alancing_policy, ssl_options, cql_version can be specified
-in the extra field of the connection.
-
-For details of the Cluster config, see cassandra.cluster for more details.
+Contact points can be specified as a comma-separated string in the 'hosts'
+field of the connection.
+
+Port can be specified in the port field of the connection.
+
+If SSL is enabled in Cassandra, pass in a dict in the extra field as
kwargs for
+``ssl.wrap_socket()``. For example:
+{
+'ssl_options' : {
+'ca_certs' : PATH_TO_CA_CERTS
+}
+}
+
+Default load balancing policy is RoundRobinPolicy. To specify a different
LB policy:
+- DCAwareRoundRobinPolicy
+{
+'load_balancing_policy': 'DCAwareRoundRobinPolicy',
+'load_balancing_policy_args': {
+'local_dc': LOCAL_DC_NAME, // optional
+'used_hosts_per_remote_dc': SOME_INT_VALUE, // optional
+}
+ }
+- WhiteListRoundRobinPolicy
+{
+'load_balancing_policy': 'WhiteListRoundRobinPolicy',
+'load_balancing_policy_args': {
+'hosts': ['HOST1', 'HOST2', 'HOST3']
+}
+}
+- TokenAwarePolicy
+{
+'load_balancing_policy': 'TokenAwarePolicy',
+'load_balancing_policy_args': {
+'child_load_balancing_policy': CHILD_POLICY_NAME, //
optional
+'child_load_balancing_policy_args': { ... } //
optional
+}
+}
+
+For details of the Cluster config, see cassandra.cluster.
"""
def __init__(self, cassandra_conn_id='cassandra_default'):
conn = self.get_connection(cassandra_conn_id)
@@ -52,7 +84,9 @@ class CassandraHook(BaseHook, LoggingMixin):
conn_config['auth_provider'] = PlainTextAuthProvider(
username=conn.login, password=conn.password)
-lb_policy =
self.get_policy(conn.extra_dejson.get('load_balancing_policy', None))
+policy_name = conn.extra_dejson.get('load_balancing_policy', None)
+policy_args = conn.extra_dejson.get('load_balancing_policy_args', {})
+lb_policy = self.get_lb_policy(policy_name, policy_args)
if lb_policy:
conn_config['load_balancing_policy'] = lb_policy
@@ -66,23 +100,59 @@ class CassandraHook(BaseHook, LoggingMixin):
self.cluster = Cluster(**conn_config)
self.keyspace = conn.schema
+sel
[jira] [Created] (AIRFLOW-2601) Allow user to specify config file location for Kubernetes operator
Cameron Moberg created AIRFLOW-2601: --- Summary: Allow user to specify config file location for Kubernetes operator Key: AIRFLOW-2601 URL: https://issues.apache.org/jira/browse/AIRFLOW-2601 Project: Apache Airflow Issue Type: Improvement Components: operators Reporter: Cameron Moberg Assignee: Cameron Moberg Currently the config file for kubernetes defaults to ~/.kube/config -- allow users to change that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2599) Sort for task instance and dagrun list broken
[ https://issues.apache.org/jira/browse/AIRFLOW-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] andy dreyfuss closed AIRFLOW-2599. -- Resolution: Fixed I no longer see this after fetching the new tip of master. > Sort for task instance and dagrun list broken > - > > Key: AIRFLOW-2599 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2599 > Project: Apache Airflow > Issue Type: Bug > Components: webapp >Affects Versions: Airflow 2.0 > Environment: ubuntu >Reporter: andy dreyfuss >Priority: Major > Fix For: Airflow 2.0 > > > On the current tip of the master branch I'm seeing no change to the table > ordering when clicking sort of the dag run and task instance tables. Sorry if > this is a known issue but my Jira search fu is weak. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2600) airflow excessively logging about libs in a dag directory not containing dags
Javier L. Velasquez created AIRFLOW-2600:
Summary: airflow excessively logging about libs in a dag directory
not containing dags
Key: AIRFLOW-2600
URL: https://issues.apache.org/jira/browse/AIRFLOW-2600
Project: Apache Airflow
Issue Type: Bug
Reporter: Javier L. Velasquez
Running a single master + single node. Seeing roughly 1500 messages every hour
in the logs:
{code:java}
[2018-06-12 15:39:50,217] {jobs.py:1756} WARNING - No viable dags retrieved
from /home/airflow/airflow/dags/libs/common.py{code}
This seems a little excessive. I was wondering if this is actually a feature
request in disguise? I was thinking we could even exclusion rules around
directories within a dag directory?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2600) airflow excessively logging about libs in a dag directory not containing dags
[
https://issues.apache.org/jira/browse/AIRFLOW-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Javier L. Velasquez updated AIRFLOW-2600:
-
Affects Version/s: 1.9.0
> airflow excessively logging about libs in a dag directory not containing dags
> -
>
> Key: AIRFLOW-2600
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2600
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Javier L. Velasquez
>Priority: Major
>
> Running a single master + single node. Seeing roughly 1500 messages every
> hour in the logs:
> {code:java}
> [2018-06-12 15:39:50,217] {jobs.py:1756} WARNING - No viable dags retrieved
> from /home/airflow/airflow/dags/libs/common.py{code}
> This seems a little excessive. I was wondering if this is actually a feature
> request in disguise? I was thinking we could even exclusion rules around
> directories within a dag directory?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2599) Sort for task instance and dagrun list broken
andy dreyfuss created AIRFLOW-2599: -- Summary: Sort for task instance and dagrun list broken Key: AIRFLOW-2599 URL: https://issues.apache.org/jira/browse/AIRFLOW-2599 Project: Apache Airflow Issue Type: Bug Components: webapp Affects Versions: Airflow 2.0 Environment: ubuntu Reporter: andy dreyfuss Fix For: Airflow 2.0 On the current tip of the master branch I'm seeing no change to the table ordering when clicking sort of the dag run and task instance tables. Sorry if this is a known issue but my Jira search fu is weak. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2598) Migrate Airflow to ReactJS
[ https://issues.apache.org/jira/browse/AIRFLOW-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509450#comment-16509450 ] Verdan Mahmood edited comment on AIRFLOW-2598 at 6/12/18 10:00 AM: --- Facebook actually removed all the BSD + Patent-licensed code from ReactJS last year with the launch of React 16. (which was the reason ASF took the drastic step of banning the library) React is now MIT licensed. ref: [https://github.com/facebook/react/blob/master/LICENSE] was (Author: verdan): Facebook actually removed all the BSD + Patent-licensed code from ReactJS (which was the reason ASF took the drastic step of banning the library) last year with the launch of React 16. React is now MIT licensed. ref: https://github.com/facebook/react/blob/master/LICENSE > Migrate Airflow to ReactJS > -- > > Key: AIRFLOW-2598 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2598 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Reporter: Verdan Mahmood >Priority: Major > Labels: frontend > > Hi guys, > In an effort to make the frontend of Apache Airflow more interactive and > feature-rich, I am planning to migrate the frontend to ReactJS. > The whole idea involves adding/updating the backend APIs and redesigning the > frontend in ReactJS, along with fixing bugs and improving layouts. This issue > is the starting point to discuss the need, process and timeline to make it > happen collectively. > For the next couple of weeks I would be working on backend in order to add > remaining APIs and make them ready for the frontend use. Question: Do we have > any kind of documentation or list of existing and outstanding APIs (roadmap)? > I am hoping to get your input on how to structure the frontend code, so that > we can deliver more frequently and incrementally in the best possible way. It > could be a separate repository or a directory within the same repository. > Thoughts? It would be great if I can also get some feedback on UI/UX, as > whether to keep it same or do some user research to improve that? > Note: I'll make sure to keep the backward compatibility in mind all the time > while doing that :) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2598) Migrate Airflow to ReactJS
[ https://issues.apache.org/jira/browse/AIRFLOW-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509450#comment-16509450 ] Verdan Mahmood commented on AIRFLOW-2598: - Facebook actually removed all the BSD + Patent-licensed code from ReactJS (which was the reason ASF took the drastic step of banning the library) last year with the launch of React 16. React is now MIT licensed. ref: https://github.com/facebook/react/blob/master/LICENSE > Migrate Airflow to ReactJS > -- > > Key: AIRFLOW-2598 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2598 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Reporter: Verdan Mahmood >Priority: Major > Labels: frontend > > Hi guys, > In an effort to make the frontend of Apache Airflow more interactive and > feature-rich, I am planning to migrate the frontend to ReactJS. > The whole idea involves adding/updating the backend APIs and redesigning the > frontend in ReactJS, along with fixing bugs and improving layouts. This issue > is the starting point to discuss the need, process and timeline to make it > happen collectively. > For the next couple of weeks I would be working on backend in order to add > remaining APIs and make them ready for the frontend use. Question: Do we have > any kind of documentation or list of existing and outstanding APIs (roadmap)? > I am hoping to get your input on how to structure the frontend code, so that > we can deliver more frequently and incrementally in the best possible way. It > could be a separate repository or a directory within the same repository. > Thoughts? It would be great if I can also get some feedback on UI/UX, as > whether to keep it same or do some user research to improve that? > Note: I'll make sure to keep the backward compatibility in mind all the time > while doing that :) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2598) Migrate Airflow to ReactJS
[ https://issues.apache.org/jira/browse/AIRFLOW-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509388#comment-16509388 ] Michal Niemiec commented on AIRFLOW-2598: - Is it actually a good idea to try ReactJS considering its licensing model vs airflow being apache project? > Migrate Airflow to ReactJS > -- > > Key: AIRFLOW-2598 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2598 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Reporter: Verdan Mahmood >Priority: Major > Labels: frontend > > Hi guys, > In an effort to make the frontend of Apache Airflow more interactive and > feature-rich, I am planning to migrate the frontend to ReactJS. > The whole idea involves adding/updating the backend APIs and redesigning the > frontend in ReactJS, along with fixing bugs and improving layouts. This issue > is the starting point to discuss the need, process and timeline to make it > happen collectively. > For the next couple of weeks I would be working on backend in order to add > remaining APIs and make them ready for the frontend use. Question: Do we have > any kind of documentation or list of existing and outstanding APIs (roadmap)? > I am hoping to get your input on how to structure the frontend code, so that > we can deliver more frequently and incrementally in the best possible way. It > could be a separate repository or a directory within the same repository. > Thoughts? It would be great if I can also get some feedback on UI/UX, as > whether to keep it same or do some user research to improve that? > Note: I'll make sure to keep the backward compatibility in mind all the time > while doing that :) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2598) Migrate Airflow to ReactJS
Verdan Mahmood created AIRFLOW-2598: --- Summary: Migrate Airflow to ReactJS Key: AIRFLOW-2598 URL: https://issues.apache.org/jira/browse/AIRFLOW-2598 Project: Apache Airflow Issue Type: Improvement Components: webapp Reporter: Verdan Mahmood Hi guys, In an effort to make the frontend of Apache Airflow more interactive and feature-rich, I am planning to migrate the frontend to ReactJS. The whole idea involves adding/updating the backend APIs and redesigning the frontend in ReactJS, along with fixing bugs and improving layouts. This issue is the starting point to discuss the need, process and timeline to make it happen collectively. For the next couple of weeks I would be working on backend in order to add remaining APIs and make them ready for the frontend use. Question: Do we have any kind of documentation or list of existing and outstanding APIs (roadmap)? I am hoping to get your input on how to structure the frontend code, so that we can deliver more frequently and incrementally in the best possible way. It could be a separate repository or a directory within the same repository. Thoughts? It would be great if I can also get some feedback on UI/UX, as whether to keep it same or do some user research to improve that? Note: I'll make sure to keep the backward compatibility in mind all the time while doing that :) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2597) dbapi hook not committing when autocommit is set to false
[ https://issues.apache.org/jira/browse/AIRFLOW-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2597. - Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3490 [https://github.com/apache/incubator-airflow/pull/3490] > dbapi hook not committing when autocommit is set to false > - > > Key: AIRFLOW-2597 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2597 > Project: Apache Airflow > Issue Type: Bug >Reporter: Kevin Yang >Assignee: Kevin Yang >Priority: Major > Fix For: 2.0.0 > > > dbapi.run() right now commits only when autocommit is set to true or db does > not support autocommit. > This is breaking CI now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2597) dbapi hook not committing when autocommit is set to false
[ https://issues.apache.org/jira/browse/AIRFLOW-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509270#comment-16509270 ] ASF subversion and git services commented on AIRFLOW-2597: -- Commit 5bb547a85e4064dba64980fcdab14e1191ce88c0 in incubator-airflow's branch refs/heads/master from [~kevinyang] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5bb547a ] [AIRFLOW-2597] Restore original dbapi.run() behavior Closes #3490 from yrqls21/kevin_yang_fix_ci > dbapi hook not committing when autocommit is set to false > - > > Key: AIRFLOW-2597 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2597 > Project: Apache Airflow > Issue Type: Bug >Reporter: Kevin Yang >Assignee: Kevin Yang >Priority: Major > Fix For: 2.0.0 > > > dbapi.run() right now commits only when autocommit is set to true or db does > not support autocommit. > This is breaking CI now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2597] Restore original dbapi.run() behavior
Repository: incubator-airflow Updated Branches: refs/heads/master dae8ac3d6 -> 5bb547a85 [AIRFLOW-2597] Restore original dbapi.run() behavior Closes #3490 from yrqls21/kevin_yang_fix_ci Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/5bb547a8 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/5bb547a8 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/5bb547a8 Branch: refs/heads/master Commit: 5bb547a85e4064dba64980fcdab14e1191ce88c0 Parents: dae8ac3 Author: Kevin Yang Authored: Tue Jun 12 08:02:16 2018 +0100 Committer: Kaxil Naik Committed: Tue Jun 12 08:02:16 2018 +0100 -- airflow/hooks/dbapi_hook.py | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/5bb547a8/airflow/hooks/dbapi_hook.py -- diff --git a/airflow/hooks/dbapi_hook.py b/airflow/hooks/dbapi_hook.py index 09dbb83..05d2084 100644 --- a/airflow/hooks/dbapi_hook.py +++ b/airflow/hooks/dbapi_hook.py @@ -169,10 +169,9 @@ class DbApiHook(BaseHook): else: cur.execute(s) -should_commit = getattr(conn, 'autocommit', -False) or not self.supports_autocommit - -if should_commit: +# If autocommit was set to False for db that supports autocommit, +# or if db does not supports autocommit, we do a manual commit. +if not getattr(conn, 'autocommit', False): conn.commit() def set_autocommit(self, conn, autocommit):
