[jira] [Created] (AIRFLOW-3112) Aline SFTP hook with SSH hook
John Cheng created AIRFLOW-3112: --- Summary: Aline SFTP hook with SSH hook Key: AIRFLOW-3112 URL: https://issues.apache.org/jira/browse/AIRFLOW-3112 Project: Apache Airflow Issue Type: Improvement Components: hooks Affects Versions: 1.10.0 Reporter: John Cheng Assignee: John Cheng Fix For: 1.10.1 SFTP hook input arguments should be alined with SSH hook so that it can use SSH connections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3940: [AIRFLOW-3054] Add show statements to hql filtering
feng-tao commented on issue #3940: [AIRFLOW-3054] Add show statements to hql filtering URL: https://github.com/apache/incubator-airflow/pull/3940#issuecomment-424209171 JIRA ticket needs to append to pr title. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko closed pull request #2: Install some Python requirements for supporting local testing.
Fokko closed pull request #2: Install some Python requirements for supporting local testing. URL: https://github.com/apache/incubator-airflow-ci/pull/2 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/Dockerfile b/Dockerfile index e13c185..af724d9 100644 --- a/Dockerfile +++ b/Dockerfile @@ -114,10 +114,17 @@ RUN adduser airflow && \ echo "airflow ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/airflow && \ chmod 0440 /etc/sudoers.d/airflow +# Install Python requirements +RUN sudo -H pip install --upgrade pip && \ +sudo -H pip install wheel tox && \ +sudo -H pip3 install --upgrade pip && \ +sudo -H pip3 install wheel tox && \ +rm -rf ~/.cache + EXPOSE 8080 WORKDIR /home/airflow -ENV PATH "$PATH:/tmp/hive/bin" +ENV PATH "$PATH:/tmp/hive/bin:$ADDITIONAL_PATH" USER airflow This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-airflow-ci] 01/01: Merge pull request #2 from holdenk/install-some-requirements-for-testing
This is an automated email from the ASF dual-hosted git repository. fokko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-airflow-ci.git commit 19e3c591c7c9bf0f346e9c532e58528daabefcff Merge: 182cfa4 914d17a Author: Fokko Driesprong AuthorDate: Tue Sep 25 07:01:54 2018 +0200 Merge pull request #2 from holdenk/install-some-requirements-for-testing Install some Python requirements for supporting local testing. Dockerfile | 9 - 1 file changed, 8 insertions(+), 1 deletion(-)
[incubator-airflow-ci] branch master updated (182cfa4 -> 19e3c59)
This is an automated email from the ASF dual-hosted git repository. fokko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-airflow-ci.git. from 182cfa4 Merge pull request #1 from seelmann/improve-for-local-develompent add 29a21b7 Install some Python requirements for supporting local testing. add 56097d9 Use -H for sudo add 5564cd0 Allow ADDITIONAL_PATH to be set add 914d17a Also install with pip3 new 19e3c59 Merge pull request #2 from holdenk/install-some-requirements-for-testing The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: Dockerfile | 9 - 1 file changed, 8 insertions(+), 1 deletion(-)
[jira] [Closed] (AIRFLOW-3051) The cli tools for manipulating users should be symmetric with connections
[ https://issues.apache.org/jira/browse/AIRFLOW-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand closed AIRFLOW-3051. Resolution: Fixed > The cli tools for manipulating users should be symmetric with connections > - > > Key: AIRFLOW-3051 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3051 > Project: Apache Airflow > Issue Type: Wish >Affects Versions: 1.10.0 >Reporter: William Pursell >Assignee: Kaxil Naik >Priority: Major > > The ability to manipulate users from the command line is a bit clunky. > Currently 'airflow create_user' and 'airflow delete_user' and 'airflow > list_users'. It seems that these ought to be made more like connections, so > that it becomes 'airflow users list ...', 'airflow users delete ...' and > 'airflow users create ...' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3051) The cli tools for manipulating users should be symmetric with connections
[
https://issues.apache.org/jira/browse/AIRFLOW-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626766#comment-16626766
]
ASF GitHub Bot commented on AIRFLOW-3051:
-
r39132 closed pull request #3936: [AIRFLOW-3051] Change CLI to make users ops
similar to connections
URL: https://github.com/apache/incubator-airflow/pull/3936
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index dbab8b38d8..1c5494ead1 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -1347,74 +1347,75 @@ def kerberos(args): # noqa
@cli_utils.action_logging
-def create_user(args):
-fields = {
-'role': args.role,
-'username': args.username,
-'email': args.email,
-'firstname': args.firstname,
-'lastname': args.lastname,
-}
-empty_fields = [k for k, v in fields.items() if not v]
-if empty_fields:
-raise SystemExit('Required arguments are missing: {}.'.format(
-', '.join(empty_fields)))
-
-appbuilder = cached_appbuilder()
-role = appbuilder.sm.find_role(args.role)
-if not role:
-raise SystemExit('{} is not a valid role.'.format(args.role))
-
-if args.use_random_password:
-password = ''.join(random.choice(string.printable) for _ in range(16))
-elif args.password:
-password = args.password
-else:
-password = getpass.getpass('Password:')
-password_confirmation = getpass.getpass('Repeat for confirmation:')
-if password != password_confirmation:
-raise SystemExit('Passwords did not match!')
+def users(args):
+if args.list:
+
+appbuilder = cached_appbuilder()
+users = appbuilder.sm.get_all_users()
+fields = ['id', 'username', 'email', 'first_name', 'last_name',
'roles']
+users = [[user.__getattribute__(field) for field in fields] for user
in users]
+msg = tabulate(users, [field.capitalize().replace('_', ' ') for field
in fields],
+ tablefmt="fancy_grid")
+if sys.version_info[0] < 3:
+msg = msg.encode('utf-8')
+print(msg)
-if appbuilder.sm.find_user(args.username):
-print('{} already exist in the db'.format(args.username))
return
-user = appbuilder.sm.add_user(args.username, args.firstname, args.lastname,
- args.email, role, password)
-if user:
-print('{} user {} created.'.format(args.role, args.username))
-else:
-raise SystemExit('Failed to create user.')
+if args.create:
+fields = {
+'role': args.role,
+'username': args.username,
+'email': args.email,
+'firstname': args.firstname,
+'lastname': args.lastname,
+}
+empty_fields = [k for k, v in fields.items() if not v]
+if empty_fields:
+raise SystemExit('Required arguments are missing: {}.'.format(
+', '.join(empty_fields)))
-@cli_utils.action_logging
-def delete_user(args):
-if not args.username:
-raise SystemExit('Required arguments are missing: username')
+appbuilder = cached_appbuilder()
+role = appbuilder.sm.find_role(args.role)
+if not role:
+raise SystemExit('{} is not a valid role.'.format(args.role))
+
+if args.use_random_password:
+password = ''.join(random.choice(string.printable) for _ in
range(16))
+elif args.password:
+password = args.password
+else:
+password = getpass.getpass('Password:')
+password_confirmation = getpass.getpass('Repeat for confirmation:')
+if password != password_confirmation:
+raise SystemExit('Passwords did not match!')
-appbuilder = cached_appbuilder()
+if appbuilder.sm.find_user(args.username):
+print('{} already exist in the db'.format(args.username))
+return
+user = appbuilder.sm.add_user(args.username, args.firstname,
args.lastname,
+ args.email, role, password)
+if user:
+print('{} user {} created.'.format(args.role, args.username))
+else:
+raise SystemExit('Failed to create user.')
-try:
-u = next(u for u in appbuilder.sm.get_all_users() if u.username ==
args.username)
-except StopIteration:
-raise SystemExit('{} is not a valid user.'.format(args.username))
+if args.delete:
+if not args.username:
+raise SystemExit('Required arguments are missing: username')
-if appbuilder.sm.del_register_user(u):
-
[GitHub] r39132 closed pull request #3936: [AIRFLOW-3051] Change CLI to make users ops similar to connections
r39132 closed pull request #3936: [AIRFLOW-3051] Change CLI to make users ops
similar to connections
URL: https://github.com/apache/incubator-airflow/pull/3936
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index dbab8b38d8..1c5494ead1 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -1347,74 +1347,75 @@ def kerberos(args): # noqa
@cli_utils.action_logging
-def create_user(args):
-fields = {
-'role': args.role,
-'username': args.username,
-'email': args.email,
-'firstname': args.firstname,
-'lastname': args.lastname,
-}
-empty_fields = [k for k, v in fields.items() if not v]
-if empty_fields:
-raise SystemExit('Required arguments are missing: {}.'.format(
-', '.join(empty_fields)))
-
-appbuilder = cached_appbuilder()
-role = appbuilder.sm.find_role(args.role)
-if not role:
-raise SystemExit('{} is not a valid role.'.format(args.role))
-
-if args.use_random_password:
-password = ''.join(random.choice(string.printable) for _ in range(16))
-elif args.password:
-password = args.password
-else:
-password = getpass.getpass('Password:')
-password_confirmation = getpass.getpass('Repeat for confirmation:')
-if password != password_confirmation:
-raise SystemExit('Passwords did not match!')
+def users(args):
+if args.list:
+
+appbuilder = cached_appbuilder()
+users = appbuilder.sm.get_all_users()
+fields = ['id', 'username', 'email', 'first_name', 'last_name',
'roles']
+users = [[user.__getattribute__(field) for field in fields] for user
in users]
+msg = tabulate(users, [field.capitalize().replace('_', ' ') for field
in fields],
+ tablefmt="fancy_grid")
+if sys.version_info[0] < 3:
+msg = msg.encode('utf-8')
+print(msg)
-if appbuilder.sm.find_user(args.username):
-print('{} already exist in the db'.format(args.username))
return
-user = appbuilder.sm.add_user(args.username, args.firstname, args.lastname,
- args.email, role, password)
-if user:
-print('{} user {} created.'.format(args.role, args.username))
-else:
-raise SystemExit('Failed to create user.')
+if args.create:
+fields = {
+'role': args.role,
+'username': args.username,
+'email': args.email,
+'firstname': args.firstname,
+'lastname': args.lastname,
+}
+empty_fields = [k for k, v in fields.items() if not v]
+if empty_fields:
+raise SystemExit('Required arguments are missing: {}.'.format(
+', '.join(empty_fields)))
-@cli_utils.action_logging
-def delete_user(args):
-if not args.username:
-raise SystemExit('Required arguments are missing: username')
+appbuilder = cached_appbuilder()
+role = appbuilder.sm.find_role(args.role)
+if not role:
+raise SystemExit('{} is not a valid role.'.format(args.role))
+
+if args.use_random_password:
+password = ''.join(random.choice(string.printable) for _ in
range(16))
+elif args.password:
+password = args.password
+else:
+password = getpass.getpass('Password:')
+password_confirmation = getpass.getpass('Repeat for confirmation:')
+if password != password_confirmation:
+raise SystemExit('Passwords did not match!')
-appbuilder = cached_appbuilder()
+if appbuilder.sm.find_user(args.username):
+print('{} already exist in the db'.format(args.username))
+return
+user = appbuilder.sm.add_user(args.username, args.firstname,
args.lastname,
+ args.email, role, password)
+if user:
+print('{} user {} created.'.format(args.role, args.username))
+else:
+raise SystemExit('Failed to create user.')
-try:
-u = next(u for u in appbuilder.sm.get_all_users() if u.username ==
args.username)
-except StopIteration:
-raise SystemExit('{} is not a valid user.'.format(args.username))
+if args.delete:
+if not args.username:
+raise SystemExit('Required arguments are missing: username')
-if appbuilder.sm.del_register_user(u):
-print('User {} deleted.'.format(args.username))
-else:
-raise SystemExit('Failed to delete user.')
+appbuilder = cached_appbuilder()
+try:
+u = next(u for u in appbuilder.sm.get_all_users()
+
[GitHub] r39132 commented on issue #3936: [AIRFLOW-3051] Change CLI to make users ops similar to connections
r39132 commented on issue #3936: [AIRFLOW-3051] Change CLI to make users ops similar to connections URL: https://github.com/apache/incubator-airflow/pull/3936#issuecomment-424206648 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3054) Show statements don't work in Hive hook
[
https://issues.apache.org/jira/browse/AIRFLOW-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626762#comment-16626762
]
ASF GitHub Bot commented on AIRFLOW-3054:
-
r39132 closed pull request #3940: [AIRFLOW-3054] Add show statements to hql
filtering
URL: https://github.com/apache/incubator-airflow/pull/3940
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/hooks/hive_hooks.py b/airflow/hooks/hive_hooks.py
index ffa1c863a2..178606aa4e 100644
--- a/airflow/hooks/hive_hooks.py
+++ b/airflow/hooks/hive_hooks.py
@@ -810,6 +810,7 @@ def _get_results(self, hql, schema='default',
fetch_size=None, hive_conf=None):
lowered_statement = statement.lower().strip()
if (lowered_statement.startswith('select') or
lowered_statement.startswith('with') or
+lowered_statement.startswith('show') or
(lowered_statement.startswith('set') and
'=' not in lowered_statement)):
description = [c for c in cur.description]
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Show statements don't work in Hive hook
>
>
> Key: AIRFLOW-3054
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3054
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks
>Reporter: Anton Protopopov
>Assignee: Anton Protopopov
>Priority: Minor
>
> Currently in airflow.hooks.hive_hooks.py in HiveServer2Hook class in
> _get_results method there are filter for incoming hql:
> source code:
> https://github.com/apache/incubator-airflow/blob/a79b92ae40c005d8239dfc7711ff44ae69b9c65b/airflow/hooks/hive_hooks.py#L809-L814
> {code:java}
> # we only get results of statements that returns
> lowered_statement = statement.lower().strip()
> if (lowered_statement.startswith('select') or
> lowered_statement.startswith('with') or
> (lowered_statement.startswith('set') and
> '=' not in lowered_statement)):
> {code}
> Queries which starts with show also returning results (like "show tables in
> default", or "show partitions in default.table_name").
> So I'm thinking it worth to add another statement which starts with "show":
> {code:java}
> lowered_statement.startswith('show')
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3054) Show statements don't work in Hive hook
[
https://issues.apache.org/jira/browse/AIRFLOW-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Anand closed AIRFLOW-3054.
> Show statements don't work in Hive hook
>
>
> Key: AIRFLOW-3054
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3054
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks
>Reporter: Anton Protopopov
>Assignee: Anton Protopopov
>Priority: Minor
>
> Currently in airflow.hooks.hive_hooks.py in HiveServer2Hook class in
> _get_results method there are filter for incoming hql:
> source code:
> https://github.com/apache/incubator-airflow/blob/a79b92ae40c005d8239dfc7711ff44ae69b9c65b/airflow/hooks/hive_hooks.py#L809-L814
> {code:java}
> # we only get results of statements that returns
> lowered_statement = statement.lower().strip()
> if (lowered_statement.startswith('select') or
> lowered_statement.startswith('with') or
> (lowered_statement.startswith('set') and
> '=' not in lowered_statement)):
> {code}
> Queries which starts with show also returning results (like "show tables in
> default", or "show partitions in default.table_name").
> So I'm thinking it worth to add another statement which starts with "show":
> {code:java}
> lowered_statement.startswith('show')
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3054) Show statements don't work in Hive hook
[
https://issues.apache.org/jira/browse/AIRFLOW-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Anand resolved AIRFLOW-3054.
--
Resolution: Fixed
> Show statements don't work in Hive hook
>
>
> Key: AIRFLOW-3054
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3054
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks
>Reporter: Anton Protopopov
>Assignee: Anton Protopopov
>Priority: Minor
>
> Currently in airflow.hooks.hive_hooks.py in HiveServer2Hook class in
> _get_results method there are filter for incoming hql:
> source code:
> https://github.com/apache/incubator-airflow/blob/a79b92ae40c005d8239dfc7711ff44ae69b9c65b/airflow/hooks/hive_hooks.py#L809-L814
> {code:java}
> # we only get results of statements that returns
> lowered_statement = statement.lower().strip()
> if (lowered_statement.startswith('select') or
> lowered_statement.startswith('with') or
> (lowered_statement.startswith('set') and
> '=' not in lowered_statement)):
> {code}
> Queries which starts with show also returning results (like "show tables in
> default", or "show partitions in default.table_name").
> So I'm thinking it worth to add another statement which starts with "show":
> {code:java}
> lowered_statement.startswith('show')
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] r39132 closed pull request #3940: [AIRFLOW-3054] Add show statements to hql filtering
r39132 closed pull request #3940: [AIRFLOW-3054] Add show statements to hql
filtering
URL: https://github.com/apache/incubator-airflow/pull/3940
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/hooks/hive_hooks.py b/airflow/hooks/hive_hooks.py
index ffa1c863a2..178606aa4e 100644
--- a/airflow/hooks/hive_hooks.py
+++ b/airflow/hooks/hive_hooks.py
@@ -810,6 +810,7 @@ def _get_results(self, hql, schema='default',
fetch_size=None, hive_conf=None):
lowered_statement = statement.lower().strip()
if (lowered_statement.startswith('select') or
lowered_statement.startswith('with') or
+lowered_statement.startswith('show') or
(lowered_statement.startswith('set') and
'=' not in lowered_statement)):
description = [c for c in cur.description]
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (AIRFLOW-409) Add option to not configure logging (keep existing loggers)
[ https://issues.apache.org/jira/browse/AIRFLOW-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand resolved AIRFLOW-409. - Resolution: Won't Fix > Add option to not configure logging (keep existing loggers) > --- > > Key: AIRFLOW-409 > URL: https://issues.apache.org/jira/browse/AIRFLOW-409 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Reporter: Michael DeWulf >Assignee: Luke Maycock >Priority: Minor > > The airflow.settings module calls > [configure_logging|https://github.com/apache/incubator-airflow/blob/b18c9959142f3f1e2cb031c8709225af01192e32/airflow/settings.py#L127], > which removes all existing loggers and installs the basic logging config. > For projects with loggers already configured, it would be nice if this could > be disabled. > It seems like we could add a config option which is checked before logging is > configured by airflow. If the option is false, then logging would not be > configured. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io commented on issue #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing
codecov-io commented on issue #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing URL: https://github.com/apache/incubator-airflow/pull/3933#issuecomment-424202526 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3933?src=pr=h1) Report > Merging [#3933](https://codecov.io/gh/apache/incubator-airflow/pull/3933?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/dd85126f2667e46c78c5aaf2026462aa2787dd1a?src=pr=desc) will **decrease** coverage by `2.65%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3933?src=pr=tree) ```diff @@Coverage Diff@@ ## master #3933 +/- ## = - Coverage 77.55% 74.9% -2.66% = Files 200 200 Lines 15969 15968 -1 = - Hits12385 11961 -424 - Misses 35844007 +423 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3933?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZG9ja2VyX29wZXJhdG9yLnB5) | `0% <0%> (-97.73%)` | :arrow_down: | | [airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==) | `0% <0%> (-94.02%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `0% <0%> (-71.43%)` | :arrow_down: | | [airflow/operators/latest\_only\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbGF0ZXN0X29ubHlfb3BlcmF0b3IucHk=) | `25% <0%> (-65%)` | :arrow_down: | | [airflow/operators/subdag\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc3ViZGFnX29wZXJhdG9yLnB5) | `70.96% <0%> (-19.36%)` | :arrow_down: | | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `58% <0%> (-18%)` | :arrow_down: | | ... and [17 more](https://codecov.io/gh/apache/incubator-airflow/pull/3933/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3933?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3933?src=pr=footer). Last update [dd85126...db7279c](https://codecov.io/gh/apache/incubator-airflow/pull/3933?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sid88in commented on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow
sid88in commented on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow URL: https://github.com/apache/incubator-airflow/pull/3504#issuecomment-424198483 @OElesin AWS Glue Operator will be a great addition! Let's fix this PR and get this out? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3102) Processor exited with return code -9
[
https://issues.apache.org/jira/browse/AIRFLOW-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beau Barker updated AIRFLOW-3102:
-
Description:
I am finding tasks aren't starting for me.
The scheduler logs are showing "Processor exited with return code -9".
{code:java}
airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
(PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
airflow.jobs.SchedulerJob INFO:
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
--- - -
-- ---
/usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s 2018-09-22T01:27:53
airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
/usr/local/airflow/dags/definitions/my_dag.py finished
airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
/usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
{code}
After much searching I've narrowed it down to a line in logging_mixin.py:
{code:java}
self.logger.log(self.level, self._buffer)
{code}
Link on Github:
[https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95]
Commenting that line, the DagFileProcessorManager succeeds.
was:
I am finding tasks aren't starting for me.
The scheduler logs are showing "Processor exited with return code -9".
{code:java}
airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
(PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
airflow.jobs.SchedulerJob INFO:
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
--- - -
-- ---
/usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s 2018-09-22T01:27:53
airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
/usr/local/airflow/dags/definitions/my_dag.py finished
airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
/usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
{code}
After much searching I've narrowed it down to a line in logging_mixin.py:
{code:java}
self.logger.log(self.level, self._buffer)
{code}
Link on Github:
[https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95]
Commenting that line, the DagFileProcessorManager succeeds.
*Solution*
This ended up being due to the airflow.processor logger being set to
"propagate".
Closing!
> Processor exited with return code -9
>
>
> Key: AIRFLOW-3102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3102
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.0
> Environment: Docker
>Reporter: Beau Barker
>Assignee: Beau Barker
>Priority: Major
>
> I am finding tasks aren't starting for me.
> The scheduler logs are showing "Processor exited with return code -9".
> {code:java}
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
> (PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
> airflow.jobs.SchedulerJob INFO:
>
> DAG File Processing Stats
> File Path PID Runtime Last Runtime Last Run
> --- - -
> -- ---
> /usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s
> 2018-09-22T01:27:53
>
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py finished
> airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
> {code}
> After much searching I've narrowed it down to a line in logging_mixin.py:
> {code:java}
> self.logger.log(self.level, self._buffer)
> {code}
> Link on Github:
>
> [https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95]
> Commenting that line, the DagFileProcessorManager succeeds.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3102) Processor exited with return code -9
[
https://issues.apache.org/jira/browse/AIRFLOW-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beau Barker resolved AIRFLOW-3102.
--
Resolution: Not A Problem
This ended up being due to the airflow.processor logger being set to
"propagate".
> Processor exited with return code -9
>
>
> Key: AIRFLOW-3102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3102
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.0
> Environment: Docker
>Reporter: Beau Barker
>Assignee: Beau Barker
>Priority: Major
>
> I am finding tasks aren't starting for me.
> The scheduler logs are showing "Processor exited with return code -9".
> {code:java}
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
> (PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
> airflow.jobs.SchedulerJob INFO:
>
> DAG File Processing Stats
> File Path PID Runtime Last Runtime Last Run
> --- - -
> -- ---
> /usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s
> 2018-09-22T01:27:53
>
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py finished
> airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
> {code}
> After much searching I've narrowed it down to a line in logging_mixin.py:
> {code:java}
> self.logger.log(self.level, self._buffer)
> {code}
> Link on Github:
>
> [https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95]
> Commenting that line, the DagFileProcessorManager succeeds.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3102) Processor exited with return code -9
[
https://issues.apache.org/jira/browse/AIRFLOW-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beau Barker reassigned AIRFLOW-3102:
Assignee: Beau Barker
> Processor exited with return code -9
>
>
> Key: AIRFLOW-3102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3102
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.0
> Environment: Docker
>Reporter: Beau Barker
>Assignee: Beau Barker
>Priority: Major
>
> I am finding tasks aren't starting for me.
> The scheduler logs are showing "Processor exited with return code -9".
> {code:java}
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
> (PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
> airflow.jobs.SchedulerJob INFO:
>
> DAG File Processing Stats
> File Path PID Runtime Last Runtime Last Run
> --- - -
> -- ---
> /usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s
> 2018-09-22T01:27:53
>
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py finished
> airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
> {code}
> After much searching I've narrowed it down to a line in logging_mixin.py:
> {code:java}
> self.logger.log(self.level, self._buffer)
> {code}
> Link on Github:
>
> [https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95]
> Commenting that line, the DagFileProcessorManager succeeds.
>
> *Solution*
> This ended up being due to the airflow.processor logger being set to
> "propagate".
> Closing!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3102) Processor exited with return code -9
[
https://issues.apache.org/jira/browse/AIRFLOW-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beau Barker updated AIRFLOW-3102:
-
Description:
I am finding tasks aren't starting for me.
The scheduler logs are showing "Processor exited with return code -9".
{code:java}
airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
(PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
airflow.jobs.SchedulerJob INFO:
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
--- - -
-- ---
/usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s 2018-09-22T01:27:53
airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
/usr/local/airflow/dags/definitions/my_dag.py finished
airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
/usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
{code}
After much searching I've narrowed it down to a line in logging_mixin.py:
{code:java}
self.logger.log(self.level, self._buffer)
{code}
Link on Github:
[https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95]
Commenting that line, the DagFileProcessorManager succeeds.
*Solution*
This ended up being due to the airflow.processor logger being set to
"propagate".
Closing!
was:
I am finding tasks aren't starting for me.
The scheduler logs are showing "Processor exited with return code -9".
{code:java}
airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
(PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
airflow.jobs.SchedulerJob INFO:
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
--- - -
-- ---
/usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s 2018-09-22T01:27:53
airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
/usr/local/airflow/dags/definitions/my_dag.py finished
airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
/usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
{code}
After much searching I've narrowed it down to a line in logging_mixin.py:
{code:java}
self.logger.log(self.level, self._buffer)
{code}
Link on Github:
[https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95]
Commenting that line, the DagFileProcessorManager succeeds.
> Processor exited with return code -9
>
>
> Key: AIRFLOW-3102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3102
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.0
> Environment: Docker
>Reporter: Beau Barker
>Priority: Major
>
> I am finding tasks aren't starting for me.
> The scheduler logs are showing "Processor exited with return code -9".
> {code:java}
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
> (PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
> airflow.jobs.SchedulerJob INFO:
>
> DAG File Processing Stats
> File Path PID Runtime Last Runtime Last Run
> --- - -
> -- ---
> /usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s
> 2018-09-22T01:27:53
>
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py finished
> airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
> {code}
> After much searching I've narrowed it down to a line in logging_mixin.py:
> {code:java}
> self.logger.log(self.level, self._buffer)
> {code}
> Link on Github:
>
> [https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95]
> Commenting that line, the DagFileProcessorManager succeeds.
>
> *Solution*
> This ended up being due to the airflow.processor logger being set to
> "propagate".
> Closing!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3109) Default user permission should contain 'can_clear'
[
https://issues.apache.org/jira/browse/AIRFLOW-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626709#comment-16626709
]
ASF GitHub Bot commented on AIRFLOW-3109:
-
r39132 closed pull request #3943: [AIRFLOW-3109] Bugfix to allow user/op roles
to clear task intance vi…
URL: https://github.com/apache/incubator-airflow/pull/3943
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/www_rbac/security.py b/airflow/www_rbac/security.py
index 55570debf6..c0d8559275 100644
--- a/airflow/www_rbac/security.py
+++ b/airflow/www_rbac/security.py
@@ -113,6 +113,7 @@
'set_running',
'set_success',
'clear',
+'can_clear',
}
op_perms = {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Default user permission should contain 'can_clear'
> --
>
> Key: AIRFLOW-3109
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3109
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Joy Gao
>Assignee: Joy Gao
>Priority: Major
>
> The default user role is missing 'can_clear' permission which allows user to
> clear DAG runs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] r39132 closed pull request #3943: [AIRFLOW-3109] Bugfix to allow user/op roles to clear task intance vi…
r39132 closed pull request #3943: [AIRFLOW-3109] Bugfix to allow user/op roles
to clear task intance vi…
URL: https://github.com/apache/incubator-airflow/pull/3943
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/www_rbac/security.py b/airflow/www_rbac/security.py
index 55570debf6..c0d8559275 100644
--- a/airflow/www_rbac/security.py
+++ b/airflow/www_rbac/security.py
@@ -113,6 +113,7 @@
'set_running',
'set_success',
'clear',
+'can_clear',
}
op_perms = {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-2846) devel requirement is not sufficient to run tests
[ https://issues.apache.org/jira/browse/AIRFLOW-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626708#comment-16626708 ] ASF GitHub Bot commented on AIRFLOW-2846: - holdenk closed pull request #3691: [AIRFLOW-2846] Add missing python test dependency to setup.py URL: https://github.com/apache/incubator-airflow/pull/3691 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/setup.py b/setup.py index aecc218170..8d6f8686eb 100644 --- a/setup.py +++ b/setup.py @@ -256,7 +256,8 @@ def write_version(filename=os.path.join(*['airflow', 'qds-sdk>=1.9.6', 'rednose', 'requests_mock', -'flake8' +'flake8', +'tox' ] if not PY3: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > devel requirement is not sufficient to run tests > > > Key: AIRFLOW-2846 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2846 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Reporter: holdenk >Assignee: holdenk >Priority: Trivial > > The devel requirement doesn't list tox, but `python setup.py test` requires > it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] holdenk commented on issue #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing
holdenk commented on issue #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing URL: https://github.com/apache/incubator-airflow/pull/3933#issuecomment-424196317 The problem with installing it in the `run-ci.sh` command is if folks want to use docker compose on their own env they're missing some of the components. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] holdenk closed pull request #3691: [AIRFLOW-2846] Add missing python test dependency to setup.py
holdenk closed pull request #3691: [AIRFLOW-2846] Add missing python test dependency to setup.py URL: https://github.com/apache/incubator-airflow/pull/3691 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/setup.py b/setup.py index aecc218170..8d6f8686eb 100644 --- a/setup.py +++ b/setup.py @@ -256,7 +256,8 @@ def write_version(filename=os.path.join(*['airflow', 'qds-sdk>=1.9.6', 'rednose', 'requests_mock', -'flake8' +'flake8', +'tox' ] if not PY3: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #3855: [AIRFLOW-3022] Add volume mount to KubernetesExecutorConfig
codecov-io commented on issue #3855: [AIRFLOW-3022] Add volume mount to KubernetesExecutorConfig URL: https://github.com/apache/incubator-airflow/pull/3855#issuecomment-424196236 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3855?src=pr=h1) Report > Merging [#3855](https://codecov.io/gh/apache/incubator-airflow/pull/3855?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/19515e816785a6be828ec0191b9a7d83c9648110?src=pr=desc) will **increase** coverage by `2.62%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3855?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3855 +/- ## == + Coverage74.9% 77.53% +2.62% == Files 200 200 Lines 1596815815 -153 == + Hits1196112262 +301 + Misses 4007 3553 -454 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3855?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/helpers.py](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `71.16% <0%> (-13.21%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `84.07% <0%> (-5.19%)` | :arrow_down: | | [airflow/utils/state.py](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `93.33% <0%> (-3.34%)` | :arrow_down: | | [airflow/utils/email.py](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9lbWFpbC5weQ==) | `97.36% <0%> (-2.64%)` | :arrow_down: | | [airflow/ti\_deps/deps/base\_ti\_dep.py](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvYmFzZV90aV9kZXAucHk=) | `88.37% <0%> (-2.33%)` | :arrow_down: | | [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `96.66% <0%> (-1.21%)` | :arrow_down: | | [airflow/utils/db.py](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `33.33% <0%> (-0.27%)` | :arrow_down: | | [airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/www\_rbac/security.py](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9zZWN1cml0eS5weQ==) | `91.27% <0%> (ø)` | :arrow_up: | | [airflow/exceptions.py](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100% <0%> (ø)` | :arrow_up: | | ... and [37 more](https://codecov.io/gh/apache/incubator-airflow/pull/3855/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3855?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3855?src=pr=footer). Last update [19515e8...ce7b1eb](https://codecov.io/gh/apache/incubator-airflow/pull/3855?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3102) Processor exited with return code -9
[
https://issues.apache.org/jira/browse/AIRFLOW-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beau Barker updated AIRFLOW-3102:
-
Description:
I am finding tasks aren't starting for me.
The scheduler logs are showing "Processor exited with return code -9".
{code:java}
airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
(PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
airflow.jobs.SchedulerJob INFO:
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
--- - -
-- ---
/usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s 2018-09-22T01:27:53
airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
/usr/local/airflow/dags/definitions/my_dag.py finished
airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
/usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
{code}
After much searching I've narrowed it down to a line in logging_mixin.py:
{code:java}
self.logger.log(self.level, self._buffer)
{code}
Link on Github:
[https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95]
Commenting that line, the DagFileProcessorManager succeeds.
was:
I am finding tasks aren't starting for me.
The scheduler logs are showing "Processor exited with return code -9".
{code:java}
airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
(PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
airflow.jobs.SchedulerJob INFO:
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
--- - -
-- ---
/usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s 2018-09-22T01:27:53
airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
/usr/local/airflow/dags/definitions/my_dag.py finished
airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
/usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
{code}
After much searching I've narrowed it down to a line in logging_mixin.py:
{code:java}
self.logger.log(self.level, self._buffer)
{code}
Link on Github:
https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95
If I comment that line, the DagFileProcessorManager succeeds and tasks do start.
> Processor exited with return code -9
>
>
> Key: AIRFLOW-3102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3102
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.0
> Environment: Docker
>Reporter: Beau Barker
>Priority: Major
>
> I am finding tasks aren't starting for me.
> The scheduler logs are showing "Processor exited with return code -9".
> {code:java}
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
> (PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
> airflow.jobs.SchedulerJob INFO:
>
> DAG File Processing Stats
> File Path PID Runtime Last Runtime Last Run
> --- - -
> -- ---
> /usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s
> 2018-09-22T01:27:53
>
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py finished
> airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
> {code}
> After much searching I've narrowed it down to a line in logging_mixin.py:
> {code:java}
> self.logger.log(self.level, self._buffer)
> {code}
> Link on Github:
>
> [https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95]
> Commenting that line, the DagFileProcessorManager succeeds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] codecov-io commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file
codecov-io commented on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file URL: https://github.com/apache/incubator-airflow/pull/3893#issuecomment-424189836 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=h1) Report > Merging [#3893](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/19515e816785a6be828ec0191b9a7d83c9648110?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #3893 +/- ## == Coverage74.9% 74.9% == Files 200 200 Lines 15968 15968 == Hits11961 11961 Misses 40074007 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=footer). Last update [19515e8...4435893](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file
codecov-io edited a comment on issue #3893: [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file URL: https://github.com/apache/incubator-airflow/pull/3893#issuecomment-424189836 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=h1) Report > Merging [#3893](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/19515e816785a6be828ec0191b9a7d83c9648110?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #3893 +/- ## == Coverage74.9% 74.9% == Files 200 200 Lines 15968 15968 == Hits11961 11961 Misses 40074007 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=footer). Last update [19515e8...4435893](https://codecov.io/gh/apache/incubator-airflow/pull/3893?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] holdenk commented on issue #2: Install some Python requirements for supporting local testing.
holdenk commented on issue #2: Install some Python requirements for supporting local testing. URL: https://github.com/apache/incubator-airflow-ci/pull/2#issuecomment-424187184 cc @Fokko :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3102) Processor exited with return code -9
[
https://issues.apache.org/jira/browse/AIRFLOW-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beau Barker updated AIRFLOW-3102:
-
Description:
I am finding tasks aren't starting for me.
The scheduler logs are showing "Processor exited with return code -9".
{code:java}
airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
(PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
airflow.jobs.SchedulerJob INFO:
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
--- - -
-- ---
/usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s 2018-09-22T01:27:53
airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
/usr/local/airflow/dags/definitions/my_dag.py finished
airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
/usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
{code}
After much searching I've narrowed it down to a line in logging_mixin.py:
{code:java}
self.logger.log(self.level, self._buffer)
{code}
Link on Github:
https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95
If I comment that line, the DagFileProcessorManager succeeds and tasks do start.
was:
I am finding tasks aren't starting for me.
The scheduler logs are showing "Processor exited with return code -9".
{code:java}
airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
(PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
airflow.jobs.SchedulerJob INFO:
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
--- - -
-- ---
/usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s 2018-09-22T01:27:53
airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
/usr/local/airflow/dags/definitions/my_dag.py finished
airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
/usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
{code}
After much searching I've narrowed it down to [this line in
jobs.py|https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/jobs.py#L372]:
{code:java}
sys.stdout = stdout
{code}
If I comment that lines, the DagFileProcessorManager succeeds and tasks do
start.
(Still investigating)
> Processor exited with return code -9
>
>
> Key: AIRFLOW-3102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3102
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.0
> Environment: Docker
>Reporter: Beau Barker
>Priority: Major
>
> I am finding tasks aren't starting for me.
> The scheduler logs are showing "Processor exited with return code -9".
>
> {code:java}
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
> (PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
> airflow.jobs.SchedulerJob INFO:
>
> DAG File Processing Stats
> File Path PID Runtime Last Runtime Last Run
> --- - -
> -- ---
> /usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s
> 2018-09-22T01:27:53
>
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py finished
> airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
> {code}
> After much searching I've narrowed it down to a line in logging_mixin.py:
> {code:java}
> self.logger.log(self.level, self._buffer)
> {code}
> Link on Github:
> https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/logging_mixin.py#L95
> If I comment that line, the DagFileProcessorManager succeeds and tasks do
> start.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3102) Processor exited with return code -9
[
https://issues.apache.org/jira/browse/AIRFLOW-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beau Barker updated AIRFLOW-3102:
-
Summary: Processor exited with return code -9 (was: Tasks not starting -
Processor exited with return code -9)
> Processor exited with return code -9
>
>
> Key: AIRFLOW-3102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3102
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.0
> Environment: Docker
>Reporter: Beau Barker
>Priority: Major
>
> I am finding tasks aren't starting for me.
> The scheduler logs are showing "Processor exited with return code -9".
>
> {code:java}
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
> (PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
> airflow.jobs.SchedulerJob INFO:
>
> DAG File Processing Stats
> File Path PID Runtime Last Runtime Last Run
> --- - -
> -- ---
> /usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s
> 2018-09-22T01:27:53
>
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py finished
> airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
> {code}
> After much searching I've narrowed it down to [this line in
> jobs.py|https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/jobs.py#L372]:
> {code:java}
> sys.stdout = stdout
> {code}
> If I comment that lines, the DagFileProcessorManager succeeds and tasks do
> start.
> (Still investigating)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3102) Tasks not starting - Processor exited with return code -9
[
https://issues.apache.org/jira/browse/AIRFLOW-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beau Barker updated AIRFLOW-3102:
-
Description:
I am finding tasks aren't starting for me.
The scheduler logs are showing "Processor exited with return code -9".
{code:java}
airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
(PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
airflow.jobs.SchedulerJob INFO:
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
--- - -
-- ---
/usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s 2018-09-22T01:27:53
airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
/usr/local/airflow/dags/definitions/my_dag.py finished
airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
/usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
{code}
After much searching I've narrowed it down to [this line in
jobs.py|https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/jobs.py#L372]:
{code:java}
sys.stdout = stdout
{code}
If I comment that lines, the DagFileProcessorManager succeeds and tasks do
start.
(Still investigating)
was:
I am finding tasks aren't starting for me.
The scheduler logs are showing "Processor exited with return code -9".
{code:java}
airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
(PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
airflow.jobs.SchedulerJob INFO:
DAG File Processing Stats
File Path PID Runtime Last Runtime Last Run
--- - -
-- ---
/usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s 2018-09-22T01:27:53
airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
/usr/local/airflow/dags/definitions/my_dag.py finished
airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
/usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
{code}
After much searching I've narrowed it down to [these lines in
jobs.py|https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/jobs.py#L372]:
{code:java}
# redirect stdout/stderr to log
sys.stdout = stdoutt
sys.stderr = stderr
{code}
If I comment those lines, the error doesn't appear and tasks do start.
> Tasks not starting - Processor exited with return code -9
> -
>
> Key: AIRFLOW-3102
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3102
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.0
> Environment: Docker
>Reporter: Beau Barker
>Priority: Major
>
> I am finding tasks aren't starting for me.
> The scheduler logs are showing "Processor exited with return code -9".
>
> {code:java}
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Started a process
> (PID: 23) to generate tasks for /usr/local/airflow/dags/definitions/my_dag.py
> airflow.jobs.SchedulerJob INFO:
>
> DAG File Processing Stats
> File Path PID Runtime Last Runtime Last Run
> --- - -
> -- ---
> /usr/local/airflow/dags/definitions/my_dag.py 23 2.39s 7.66s
> 2018-09-22T01:27:53
>
> airflow.utils.dag_processing.DagFileProcessorManager INFO: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py finished
> airflow.utils.dag_processing.DagFileProcessorManager WARNING: Processor for
> /usr/local/airflow/dags/definitions/my_dag.py exited with return code -9.
> {code}
> After much searching I've narrowed it down to [this line in
> jobs.py|https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/jobs.py#L372]:
> {code:java}
> sys.stdout = stdout
> {code}
> If I comment that lines, the DagFileProcessorManager succeeds and tasks do
> start.
> (Still investigating)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-3104) .airflowignore feature is not mentioned at all in documentation
[ https://issues.apache.org/jira/browse/AIRFLOW-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG reopened AIRFLOW-3104: > .airflowignore feature is not mentioned at all in documentation > --- > > Key: AIRFLOW-3104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3104 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 2.0.0 > > > This is a nice feature, but not mentioned at all in anywhere of the > documentation. > Relevant information can only be found in source code comment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3067) (www_rbac) Flask flash messages are not displayed properly (no background color)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG resolved AIRFLOW-3067. Resolution: Fixed Fixed in commit https://github.com/apache/incubator-airflow/commit/19515e816785a6be828ec0191b9a7d83c9648110 > (www_rbac) Flask flash messages are not displayed properly (no background > color) > > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). > > *Latest update:* > we update bootstrap-theme.css, to add CSS classes 'alert-message' and > 'alert-error'. This may help minimize the changes we need to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3067) (www_rbac) Flask flash messages are not displayed properly (no background color)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG closed AIRFLOW-3067. -- Resolution: Fixed > (www_rbac) Flask flash messages are not displayed properly (no background > color) > > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). > > *Latest update:* > we update bootstrap-theme.css, to add CSS classes 'alert-message' and > 'alert-error'. This may help minimize the changes we need to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-3067) (www_rbac) Flask flash messages are not displayed properly (no background color)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG reopened AIRFLOW-3067: > (www_rbac) Flask flash messages are not displayed properly (no background > color) > > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). > > *Latest update:* > we update bootstrap-theme.css, to add CSS classes 'alert-message' and > 'alert-error'. This may help minimize the changes we need to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3067) (www_rbac) Flask flash messages are not displayed properly (no background color)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3067: --- Summary: (www_rbac) Flask flash messages are not displayed properly (no background color) (was: (www_rbac) Flask flash messages are not categoried properly (result in 'incomplete' UI display)) > (www_rbac) Flask flash messages are not displayed properly (no background > color) > > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3067) (www_rbac) Flask flash messages are not displayed properly (no background color)
[ https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3067: --- Description: The Flask flash messages are not categorized properly (ref: [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] Then the default category would be 'message'. Flask AppBuilder would assign CSS class `alert-[category]` to that message when displaying it (ref: [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes the the flash messages in the _*www_rbac*_ UI come with no background color (please check the screenshots attached. We don't see this issue in _*www*_ UI as all flash messages without categories are classed into 'info'). Solution: * Add proper category when we invoke *flash()*. * We can't use category 'error'. In bootstrap-theme.css, we don't have `alert-error` (we have `alert-danger` instead). *Latest update:* we update bootstrap-theme.css, to add CSS classes 'alert-message' and 'alert-error'. This may help minimize the changes we need to do. was: The Flask flash messages are not categorized properly (ref: [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] Then the default category would be 'message'. Flask AppBuilder would assign CSS class `alert-[category]` to that message when displaying it (ref: [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes the the flash messages in the _*www_rbac*_ UI come with no background color (please check the screenshots attached. We don't see this issue in _*www*_ UI as all flash messages without categories are classed into 'info'). Solution: * Add proper category when we invoke *flash()*. * We can't use category 'error'. In bootstrap-theme.css, we don't have `alert-error` (we have `alert-danger` instead). > (www_rbac) Flask flash messages are not displayed properly (no background > color) > > > Key: AIRFLOW-3067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3067 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Affects Versions: 1.10.0 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.1 > > Attachments: 1.png, 2.png > > > The Flask flash messages are not categorized properly (ref: > [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).] > Then the default category would be 'message'. Flask AppBuilder would assign > CSS class `alert-[category]` to that message when displaying it (ref: > [https://github.com/dpgaspar/Flask-AppBuilder/blob/master/flask_appbuilder/templates/appbuilder/flash.html] > ). But 'alert-message' is not specified in *bootstrap-theme.css*. This makes > the the flash messages in the _*www_rbac*_ UI come with no background color > (please check the screenshots attached. We don't see this issue in _*www*_ UI > as all flash messages without categories are classed into 'info'). > Solution: > * Add proper category when we invoke *flash()*. > * We can't use category 'error'. In bootstrap-theme.css, we don't have > `alert-error` (we have `alert-danger` instead). > > *Latest update:* > we update bootstrap-theme.css, to add CSS classes 'alert-message' and > 'alert-error'. This may help minimize the changes we need to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3111) Confusing comments and instructions for log templates in UPDATING.md and default_airflow.cfg
[
https://issues.apache.org/jira/browse/AIRFLOW-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626578#comment-16626578
]
ASF GitHub Bot commented on AIRFLOW-3111:
-
chang opened a new pull request #3944: [AIRFLOW-3111] Fix instructions in
UPDATING.md and remove comment artifacts
URL: https://github.com/apache/incubator-airflow/pull/3944
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- fixed incorrect instructions in `UPDATING.md` about adding the options
`core.log_filename_template` and `elasticsearch.elasticsearch_log_id_template`
- removed comments referencing "additional curly braces" from
`default_airflow.cfg` since they're irrelevant to the rendered `airflow.cfg`
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Just updating documentation and removing comments.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Confusing comments and instructions for log templates in UPDATING.md and
> default_airflow.cfg
>
>
> Key: AIRFLOW-3111
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3111
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Eric Chang
>Assignee: Eric Chang
>Priority: Minor
>
> The new 1.10 release includes instructions on updating the *airflow.cfg*
> options core.log_filename_template and
> elasticsearch.elasticsearch_log_id_template that are technically incorrect:
>
> {code:java}
> // UPDATING.md
> ...the following defaults need to be added.
> [core]
> log_filename_template = ti.dag_id / ti.task_id / ts
> / try_number .log
> [elasticsearch]
> elasticsearch_log_id_template =
> {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}{code}
>
> Inserting the above options into an existing *airflow.cfg* will result in a
> jinja parse failure because `` is not a valid template string. The reason
> the extra braces are necessary is because the default *airflow.cfg* copied to
> AIRFLOW_HOME is a rendered version of *default_airflow.cfg* (rendered by
> *airflow.configuration.parameterized_config*).
> The confusion is compounded by several comments copied over to *airflow.cfg*
> stating that
>
> {code:java}
> # we need to escape the curly braces by adding an additional curly brace
> {code}
> But that's is only true for *default_airflow.cfg* and doesn't apply to the
> actual *airflow.cfg*.
> I'll submit a PR that corrects this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] chang opened a new pull request #3944: [AIRFLOW-3111] Fix instructions in UPDATING.md and remove comment artifacts
chang opened a new pull request #3944: [AIRFLOW-3111] Fix instructions in
UPDATING.md and remove comment artifacts
URL: https://github.com/apache/incubator-airflow/pull/3944
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- fixed incorrect instructions in `UPDATING.md` about adding the options
`core.log_filename_template` and `elasticsearch.elasticsearch_log_id_template`
- removed comments referencing "additional curly braces" from
`default_airflow.cfg` since they're irrelevant to the rendered `airflow.cfg`
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Just updating documentation and removing comments.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3111) Confusing comments and instructions for log templates in UPDATING.md and default_airflow.cfg
Eric Chang created AIRFLOW-3111:
---
Summary: Confusing comments and instructions for log templates in
UPDATING.md and default_airflow.cfg
Key: AIRFLOW-3111
URL: https://issues.apache.org/jira/browse/AIRFLOW-3111
Project: Apache Airflow
Issue Type: Bug
Reporter: Eric Chang
Assignee: Eric Chang
The new 1.10 release includes instructions on updating the *airflow.cfg*
options core.log_filename_template and
elasticsearch.elasticsearch_log_id_template that are technically incorrect:
{code:java}
// UPDATING.md
...the following defaults need to be added.
[core]
log_filename_template = ti.dag_id / ti.task_id / ts
/ try_number .log
[elasticsearch]
elasticsearch_log_id_template =
{{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}{code}
Inserting the above options into an existing *airflow.cfg* will result in a
jinja parse failure because `` is not a valid template string. The reason
the extra braces are necessary is because the default *airflow.cfg* copied to
AIRFLOW_HOME is a rendered version of *default_airflow.cfg* (rendered by
*airflow.configuration.parameterized_config*).
The confusion is compounded by several comments copied over to *airflow.cfg*
stating that
{code:java}
# we need to escape the curly braces by adding an additional curly brace
{code}
But that's is only true for *default_airflow.cfg* and doesn't apply to the
actual *airflow.cfg*.
I'll submit a PR that corrects this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3110) Session not committed before failure callback, but is committed before success.
[ https://issues.apache.org/jira/browse/AIRFLOW-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subu updated AIRFLOW-3110: -- Description: Folks, The below line ideally should have a ``` if not test_mode: session.merge(self) session.commit() ``` [https://github.com/HanaDB/incubator-airflow/blob/master/airflow/models.py#L1815] Else as of today we are not able to access the latest session information using sqlalchemy from failure callbacks. I don't know if this change is safe because I lack context elsewhere -so cutting a JIRA. Thanks, Subu was: Folks, The below line ideally should have a session.commit() [https://github.com/HanaDB/incubator-airflow/blob/master/airflow/models.py#L1815] Else as of today we are not able to access the latest session information using sqlalchemy from failure callbacks. I don't know if this change is safe because I lack context elsewhere -so cutting a JIRA. Thanks, Subu > Session not committed before failure callback, but is committed before > success. > --- > > Key: AIRFLOW-3110 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3110 > Project: Apache Airflow > Issue Type: Bug >Reporter: Subu >Priority: Major > > Folks, > The below line ideally should have a > ``` > if not test_mode: > session.merge(self) > session.commit() > ``` > [https://github.com/HanaDB/incubator-airflow/blob/master/airflow/models.py#L1815] > > Else as of today we are not able to access the latest session information > using sqlalchemy from failure callbacks. > > I don't know if this change is safe because I lack context elsewhere -so > cutting a JIRA. > > Thanks, Subu -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3110) Session not committed before failure callback, but is committed before success.
Subu created AIRFLOW-3110: - Summary: Session not committed before failure callback, but is committed before success. Key: AIRFLOW-3110 URL: https://issues.apache.org/jira/browse/AIRFLOW-3110 Project: Apache Airflow Issue Type: Bug Reporter: Subu Folks, The below line ideally should have a session.commit() [https://github.com/HanaDB/incubator-airflow/blob/master/airflow/models.py#L1815] Else as of today we are not able to access the latest session information using sqlalchemy from failure callbacks. I don't know if this change is safe because I lack context elsewhere -so cutting a JIRA. Thanks, Subu -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3109) Default user permission should contain 'can_clear'
[
https://issues.apache.org/jira/browse/AIRFLOW-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626524#comment-16626524
]
ASF GitHub Bot commented on AIRFLOW-3109:
-
jgao54 opened a new pull request #3943: [AIRFLOW-3109] Bugfix to allow user/op
roles to clear task intance vi…
URL: https://github.com/apache/incubator-airflow/pull/3943
…a UI by default
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3109
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
The default user role is missing 'can_clear' permission which allows user to
clear DAG runs.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Verified via the RBAC UI.
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Default user permission should contain 'can_clear'
> --
>
> Key: AIRFLOW-3109
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3109
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Joy Gao
>Assignee: Joy Gao
>Priority: Major
>
> The default user role is missing 'can_clear' permission which allows user to
> clear DAG runs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] jgao54 opened a new pull request #3943: [AIRFLOW-3109] Bugfix to allow user/op roles to clear task intance vi…
jgao54 opened a new pull request #3943: [AIRFLOW-3109] Bugfix to allow user/op
roles to clear task intance vi…
URL: https://github.com/apache/incubator-airflow/pull/3943
…a UI by default
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3109
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
The default user role is missing 'can_clear' permission which allows user to
clear DAG runs.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Verified via the RBAC UI.
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3109) Default user permission should contain 'can_clear'
[ https://issues.apache.org/jira/browse/AIRFLOW-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joy Gao updated AIRFLOW-3109: - Description: The default user role is missing 'can_clear' permission which allows user to clear DAG runs. (was: There's a bug in the default user permission. 'clear' should have been 'can_clear' as FAB automatically prepend model permissions with 'can_' prefix.) > Default user permission should contain 'can_clear' > -- > > Key: AIRFLOW-3109 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3109 > Project: Apache Airflow > Issue Type: Bug >Reporter: Joy Gao >Assignee: Joy Gao >Priority: Major > > The default user role is missing 'can_clear' permission which allows user to > clear DAG runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3109) Default user permission should contain 'can_clear'
[ https://issues.apache.org/jira/browse/AIRFLOW-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joy Gao updated AIRFLOW-3109: - Summary: Default user permission should contain 'can_clear' (was: Default user permission should be 'can_clear' instead of 'clear') > Default user permission should contain 'can_clear' > -- > > Key: AIRFLOW-3109 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3109 > Project: Apache Airflow > Issue Type: Bug >Reporter: Joy Gao >Assignee: Joy Gao >Priority: Major > > There's a bug in the default user permission. 'clear' should have been > 'can_clear' as FAB automatically prepend model permissions with 'can_' prefix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3109) Default user permission should be 'can_clear' instead of 'clear'
Joy Gao created AIRFLOW-3109: Summary: Default user permission should be 'can_clear' instead of 'clear' Key: AIRFLOW-3109 URL: https://issues.apache.org/jira/browse/AIRFLOW-3109 Project: Apache Airflow Issue Type: Bug Reporter: Joy Gao Assignee: Joy Gao There's a bug in the default user permission. 'clear' should have been 'can_clear' as FAB automatically prepend model permissions with 'can_' prefix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2442) Airflow run command leaves database connections open
[
https://issues.apache.org/jira/browse/AIRFLOW-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626515#comment-16626515
]
Kevin Yang commented on AIRFLOW-2442:
-
I suppose this issue is resolved. let's resolve the ticket.
> Airflow run command leaves database connections open
>
>
> Key: AIRFLOW-2442
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2442
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: 1.8.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: connection_duration_1_hour.png, db_connections.png,
> fixed_before_and_after.jpg, monthly_db_connections.png, running_tasks.png
>
>
> *Summary*
> The "airflow run" command creates a connection to the database and leaves it
> open (until killed by SQLALchemy later). The number of these connections can
> skyrocket whenever hundreds/thousands of tasks are launched simultaneously,
> and potentially hit the database connection limit.
> The problem is that in cli.py, the run() method first calls
> {code:java}
> settings.configure_orm(disable_connection_pool=True){code}
> correctly
> to use a NullPool, but then parses any custom configs and again calls
> {code:java}
> settings.configure_orm(){code}
> , thereby overriding the desired behavior by instead using a QueuePool.
> The QueuePool uses the default configs for SQL_ALCHEMY_POOL_SIZE and
> SQL_ALCHEMY_POOL_RECYCLE. This means that while the task is running and the
> executor is sending heartbeats, the sleeping connection is idle until it is
> killed by SQLAlchemy.
> This fixes a bug introduced by
> [https://github.com/apache/incubator-airflow/pull/1934] in
> [https://github.com/apache/incubator-airflow/pull/1934/commits/b380013634b02bb4c1b9d1cc587ccd12383820b6#diff-1c2404a3a60f829127232842250ff406R344]
>
> which is present in branches 1-8-stable, 1-9-stable, and 1-10-test
> NOTE: Will create a PR once I've done more testing since I'm on an older
> branch. For now, attaching a patch file [^AIRFLOW-2442.patch]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync
codecov-io edited a comment on issue #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync URL: https://github.com/apache/incubator-airflow/pull/3770#issuecomment-423662259 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=h1) Report > Merging [#3770](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/cde74c4c52a30c0e363e87ce28f84c9beee22b52?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=tree) ```diff @@Coverage Diff@@ ## master #3770 +/- ## = - Coverage74.9% 74.9% -0.01% = Files 200 201 +1 Lines 15968 15969 +1 = Hits11961 11961 - Misses 40074008 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=tree) | Coverage Δ | | |---|---|---| | [...e\_dags\_kubernetes/example\_kubernetes\_annotation.py](https://codecov.io/gh/apache/incubator-airflow/pull/3770/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Nfa3ViZXJuZXRlcy9leGFtcGxlX2t1YmVybmV0ZXNfYW5ub3RhdGlvbi5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [airflow/example\_dags\_kubernetes/libs/helper.py](https://codecov.io/gh/apache/incubator-airflow/pull/3770/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Nfa3ViZXJuZXRlcy9saWJzL2hlbHBlci5weQ==) | `0% <0%> (ø)` | | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3770/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.04% <0%> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=footer). Last update [cde74c4...45fe19e](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync
codecov-io edited a comment on issue #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync URL: https://github.com/apache/incubator-airflow/pull/3770#issuecomment-423662259 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=h1) Report > Merging [#3770](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/cde74c4c52a30c0e363e87ce28f84c9beee22b52?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=tree) ```diff @@Coverage Diff@@ ## master #3770 +/- ## = - Coverage74.9% 74.9% -0.01% = Files 200 201 +1 Lines 15968 15969 +1 = Hits11961 11961 - Misses 40074008 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=tree) | Coverage Δ | | |---|---|---| | [...e\_dags\_kubernetes/example\_kubernetes\_annotation.py](https://codecov.io/gh/apache/incubator-airflow/pull/3770/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Nfa3ViZXJuZXRlcy9leGFtcGxlX2t1YmVybmV0ZXNfYW5ub3RhdGlvbi5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [airflow/example\_dags\_kubernetes/libs/helper.py](https://codecov.io/gh/apache/incubator-airflow/pull/3770/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Nfa3ViZXJuZXRlcy9saWJzL2hlbHBlci5weQ==) | `0% <0%> (ø)` | | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3770/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `72.04% <0%> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=footer). Last update [cde74c4...45fe19e](https://codecov.io/gh/apache/incubator-airflow/pull/3770?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] odracci commented on issue #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing
odracci commented on issue #3933: [AIRFLOW-3100][AIRFLOW-3101] Improve docker compose local testing URL: https://github.com/apache/incubator-airflow/pull/3933#issuecomment-424109192 @holdenk `scripts/ci/kubernetes/docker/Dockerfile` is the image used to run airflow on Kubernetes and it should not contain any dev / testing tool IMO. FYI `pip` is installed in `scripts/ci/run-ci.sh`: https://github.com/apache/incubator-airflow/blob/19515e816785a6be828ec0191b9a7d83c9648110/scripts/ci/run-ci.sh#L29-L36 One option would be to install `tox` in https://github.com/apache/incubator-airflow-ci This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3067) (www_rbac) Flask flash messages are not categoried properly (result in 'incomplete' UI display)
[
https://issues.apache.org/jira/browse/AIRFLOW-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626209#comment-16626209
]
ASF GitHub Bot commented on AIRFLOW-3067:
-
feng-tao closed pull request #3903: [AIRFLOW-3067] Display www_rbac Flask flash
msg properly
URL: https://github.com/apache/incubator-airflow/pull/3903
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/www_rbac/static/css/bootstrap-theme.css
b/airflow/www_rbac/static/css/bootstrap-theme.css
index 734f940fee..3adaec1d3b 100644
--- a/airflow/www_rbac/static/css/bootstrap-theme.css
+++ b/airflow/www_rbac/static/css/bootstrap-theme.css
@@ -4949,6 +4949,28 @@ a.thumbnail.active {
.alert-danger .alert-link {
color: #843534;
}
+.alert-message {
+ background-color: #d9edf7;
+ border-color: #bce8f1;
+ color: #31708f;
+}
+.alert-message hr {
+ border-top-color: #a6e1ec;
+}
+.alert-message .alert-link {
+ color: #245269;
+}
+.alert-error {
+ background-color: #f2dede;
+ border-color: #ebccd1;
+ color: #a94442;
+}
+.alert-error hr {
+ border-top-color: #e4b9c0;
+}
+.alert-error .alert-link {
+ color: #843534;
+}
@-webkit-keyframes progress-bar-stripes {
from {
background-position: 40px 0;
diff --git a/airflow/www_rbac/views.py b/airflow/www_rbac/views.py
index 3dc3400968..0fc3d67a80 100644
--- a/airflow/www_rbac/views.py
+++ b/airflow/www_rbac/views.py
@@ -782,11 +782,12 @@ def delete(self):
try:
delete_dag.delete_dag(dag_id)
except DagNotFound:
-flash("DAG with id {} not found. Cannot delete".format(dag_id))
+flash("DAG with id {} not found. Cannot delete".format(dag_id),
'error')
return redirect(request.referrer)
except DagFileExists:
flash("Dag id {} is still in DagBag. "
- "Remove the DAG file first.".format(dag_id))
+ "Remove the DAG file first.".format(dag_id),
+ 'error')
return redirect(request.referrer)
flash("Deleting DAG with id {}. May take a couple minutes to fully"
@@ -2065,7 +2066,7 @@ def varimport(self):
else:
d = json.loads(out)
except Exception:
-flash("Missing file or syntax error.")
+flash("Missing file or syntax error.", 'error')
else:
suc_count = fail_count = 0
for k, v in d.items():
@@ -2076,7 +2077,7 @@ def varimport(self):
fail_count += 1
else:
suc_count += 1
-flash("{} variable(s) successfully updated.".format(suc_count),
'info')
+flash("{} variable(s) successfully updated.".format(suc_count))
if fail_count:
flash("{} variables(s) failed to be
updated.".format(fail_count), 'error')
self.update_redirect()
@@ -2166,10 +2167,9 @@ def action_set_running(self, drs, session=None):
dr.state = State.RUNNING
models.DagStat.update(dirty_ids, session=session)
session.commit()
-flash(
-"{count} dag runs were set to running".format(**locals()))
+flash("{count} dag runs were set to running".format(**locals()))
except Exception as ex:
-flash(str(ex))
+flash(str(ex), 'error')
flash('Failed to set state', 'error')
return redirect(self.route_base + '/list')
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> (www_rbac) Flask flash messages are not categoried properly (result in
> 'incomplete' UI display)
> ---
>
> Key: AIRFLOW-3067
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3067
> Project: Apache Airflow
> Issue Type: Improvement
> Components: webapp
>Affects Versions: 1.10.0
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Fix For: 1.10.1
>
> Attachments: 1.png, 2.png
>
>
> The Flask flash messages are not categorized properly (ref:
> [http://flask.pocoo.org/docs/1.0/patterns/flashing/#flashing-with-categories).]
> Then the default category would be 'message'. Flask AppBuilder would assign
> CSS class `alert-[category]` to that message when displaying it (ref:
>
[GitHub] feng-tao commented on issue #3903: [AIRFLOW-3067] Display www_rbac Flask flash msg properly
feng-tao commented on issue #3903: [AIRFLOW-3067] Display www_rbac Flask flash msg properly URL: https://github.com/apache/incubator-airflow/pull/3903#issuecomment-424074486 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao closed pull request #3903: [AIRFLOW-3067] Display www_rbac Flask flash msg properly
feng-tao closed pull request #3903: [AIRFLOW-3067] Display www_rbac Flask flash
msg properly
URL: https://github.com/apache/incubator-airflow/pull/3903
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/www_rbac/static/css/bootstrap-theme.css
b/airflow/www_rbac/static/css/bootstrap-theme.css
index 734f940fee..3adaec1d3b 100644
--- a/airflow/www_rbac/static/css/bootstrap-theme.css
+++ b/airflow/www_rbac/static/css/bootstrap-theme.css
@@ -4949,6 +4949,28 @@ a.thumbnail.active {
.alert-danger .alert-link {
color: #843534;
}
+.alert-message {
+ background-color: #d9edf7;
+ border-color: #bce8f1;
+ color: #31708f;
+}
+.alert-message hr {
+ border-top-color: #a6e1ec;
+}
+.alert-message .alert-link {
+ color: #245269;
+}
+.alert-error {
+ background-color: #f2dede;
+ border-color: #ebccd1;
+ color: #a94442;
+}
+.alert-error hr {
+ border-top-color: #e4b9c0;
+}
+.alert-error .alert-link {
+ color: #843534;
+}
@-webkit-keyframes progress-bar-stripes {
from {
background-position: 40px 0;
diff --git a/airflow/www_rbac/views.py b/airflow/www_rbac/views.py
index 3dc3400968..0fc3d67a80 100644
--- a/airflow/www_rbac/views.py
+++ b/airflow/www_rbac/views.py
@@ -782,11 +782,12 @@ def delete(self):
try:
delete_dag.delete_dag(dag_id)
except DagNotFound:
-flash("DAG with id {} not found. Cannot delete".format(dag_id))
+flash("DAG with id {} not found. Cannot delete".format(dag_id),
'error')
return redirect(request.referrer)
except DagFileExists:
flash("Dag id {} is still in DagBag. "
- "Remove the DAG file first.".format(dag_id))
+ "Remove the DAG file first.".format(dag_id),
+ 'error')
return redirect(request.referrer)
flash("Deleting DAG with id {}. May take a couple minutes to fully"
@@ -2065,7 +2066,7 @@ def varimport(self):
else:
d = json.loads(out)
except Exception:
-flash("Missing file or syntax error.")
+flash("Missing file or syntax error.", 'error')
else:
suc_count = fail_count = 0
for k, v in d.items():
@@ -2076,7 +2077,7 @@ def varimport(self):
fail_count += 1
else:
suc_count += 1
-flash("{} variable(s) successfully updated.".format(suc_count),
'info')
+flash("{} variable(s) successfully updated.".format(suc_count))
if fail_count:
flash("{} variables(s) failed to be
updated.".format(fail_count), 'error')
self.update_redirect()
@@ -2166,10 +2167,9 @@ def action_set_running(self, drs, session=None):
dr.state = State.RUNNING
models.DagStat.update(dirty_ids, session=session)
session.commit()
-flash(
-"{count} dag runs were set to running".format(**locals()))
+flash("{count} dag runs were set to running".format(**locals()))
except Exception as ex:
-flash(str(ex))
+flash(str(ex), 'error')
flash('Failed to set state', 'error')
return redirect(self.route_base + '/list')
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on issue #3942: V1 10 stable
Fokko commented on issue #3942: V1 10 stable URL: https://github.com/apache/incubator-airflow/pull/3942#issuecomment-424034512 I think something went wrong @chenjialin. Closing this for now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko closed pull request #3942: V1 10 stable
Fokko closed pull request #3942: V1 10 stable URL: https://github.com/apache/incubator-airflow/pull/3942 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3941: [AIRFLOW-3106] Validate Postgres connection after saving it
Fokko commented on a change in pull request #3941: [AIRFLOW-3106] Validate Postgres connection after saving it URL: https://github.com/apache/incubator-airflow/pull/3941#discussion_r219897919 ## File path: airflow/hooks/postgres_hook.py ## @@ -60,6 +60,15 @@ def get_conn(self): self.conn = psycopg2.connect(**conn_args) return self.conn +def validate_conn(self): +try: +conn = self.get_conn() Review comment: Can you change this to `with self.get_conn()`? So it closes automagically. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3107) Create Dynamic External Task Sensor to handle non exact timeframes
[
https://issues.apache.org/jira/browse/AIRFLOW-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bert Desmet updated AIRFLOW-3107:
-
Labels: dependencies dynamic_dependencies sensors (was: )
> Create Dynamic External Task Sensor to handle non exact timeframes
> --
>
> Key: AIRFLOW-3107
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3107
> Project: Apache Airflow
> Issue Type: New Feature
> Components: core
>Affects Versions: 2.0.0
>Reporter: Bert Desmet
>Priority: Major
> Labels: dependencies, dynamic_dependencies, sensors
> Fix For: 2.0.0
>
>
> All,
> For a project I'm working on it is necessary to check if a specific task has
> run somewhere between 30 days ago and now.
> To facilitate this I have patched airflow to include the
> 'TimeRangeExternalTaskSensor' as provided by
> {color:#33}[omnilinguist|https://github.com/omnilinguist] in the
> following pull request:
> [https://github.com/apache/incubator-airflow/pull/1641] {color}
> I have updated his code so it aligns better with how the ExternalTaskSensor
> has been implemented.
> Currently we are running this patch in production on a 1.9 version of
> Airflow. I have now merged this with the master branch - but this code has
> not yet fully been tested.
>
> The code can be found here:
> [https://github.com/biertie/incubator-airflow/blob/master/airflow/sensors/time_range_external_task_sensor.py]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3107) Create Dynamic External Task Sensor to handle non exact timeframes
[
https://issues.apache.org/jira/browse/AIRFLOW-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bert Desmet updated AIRFLOW-3107:
-
Fix Version/s: (was: 1.10.0)
2.0.0
> Create Dynamic External Task Sensor to handle non exact timeframes
> --
>
> Key: AIRFLOW-3107
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3107
> Project: Apache Airflow
> Issue Type: New Feature
> Components: core
>Affects Versions: 2.0.0
>Reporter: Bert Desmet
>Priority: Major
> Labels: dependencies, dynamic_dependencies, sensors
> Fix For: 2.0.0
>
>
> All,
> For a project I'm working on it is necessary to check if a specific task has
> run somewhere between 30 days ago and now.
> To facilitate this I have patched airflow to include the
> 'TimeRangeExternalTaskSensor' as provided by
> {color:#33}[omnilinguist|https://github.com/omnilinguist] in the
> following pull request:
> [https://github.com/apache/incubator-airflow/pull/1641] {color}
> I have updated his code so it aligns better with how the ExternalTaskSensor
> has been implemented.
> Currently we are running this patch in production on a 1.9 version of
> Airflow. I have now merged this with the master branch - but this code has
> not yet fully been tested.
>
> The code can be found here:
> [https://github.com/biertie/incubator-airflow/blob/master/airflow/sensors/time_range_external_task_sensor.py]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3107) Create Dynamic External Task Sensor to handle non exact timeframes
[
https://issues.apache.org/jira/browse/AIRFLOW-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bert Desmet updated AIRFLOW-3107:
-
Affects Version/s: (was: 1.10.0)
2.0.0
> Create Dynamic External Task Sensor to handle non exact timeframes
> --
>
> Key: AIRFLOW-3107
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3107
> Project: Apache Airflow
> Issue Type: New Feature
> Components: core
>Affects Versions: 2.0.0
>Reporter: Bert Desmet
>Priority: Major
> Fix For: 1.10.0
>
>
> All,
> For a project I'm working on it is necessary to check if a specific task has
> run somewhere between 30 days ago and now.
> To facilitate this I have patched airflow to include the
> 'TimeRangeExternalTaskSensor' as provided by
> {color:#33}[omnilinguist|https://github.com/omnilinguist] in the
> following pull request:
> [https://github.com/apache/incubator-airflow/pull/1641] {color}
> I have updated his code so it aligns better with how the ExternalTaskSensor
> has been implemented.
> Currently we are running this patch in production on a 1.9 version of
> Airflow. I have now merged this with the master branch - but this code has
> not yet fully been tested.
>
> The code can be found here:
> [https://github.com/biertie/incubator-airflow/blob/master/airflow/sensors/time_range_external_task_sensor.py]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3108) MsSqlHook.run fails to commit if autocommit=False (Default config)
k created AIRFLOW-3108:
--
Summary: MsSqlHook.run fails to commit if autocommit=False
(Default config)
Key: AIRFLOW-3108
URL: https://issues.apache.org/jira/browse/AIRFLOW-3108
Project: Apache Airflow
Issue Type: Bug
Components: hooks
Affects Versions: 1.10.0
Reporter: k
The MsSqlHook.run method doesn't execute conn.commit() if autocommit is set to
False.
It looks like this bug has existed for a very long time, but wasn't apparent in
1.9, because the default value for autocommit was True. In 1.10 the default
value was changed to False and the MsSqlHook and Operator have started failing
silently.
The bug happens because the MssqlHook doesn't implement a custom
get_autocommit(self, conn) method.The superclass' DbiApiHook method always
returns True even if autocommit wasn't enabled in pymssql. Therefore the hook
doesn't call commit and pymsql doesn't autocommit.
The below patch fixes the issue. Please consider including this fix in airflow
1.10.1, because it is a very frustrating issue to debug.
{code:java}
--- mssql_hook.py
+++ mssql_hook.py
@@ -50,3 +50,13 @@ class MsSqlHook(DbApiHook):
def set_autocommit(self, conn, autocommit):
conn.autocommit(autocommit)
+
+ def get_autocommit(self, conn):
+ """
+ MS SQL connection gets autocommit in a different way.
+ :param conn: connection to get autocommit setting from.
+ :type conn: connection object.
+ :return: connection autocommit setting
+ :rtype bool
+ """
+ return conn.autocommit_state
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3107) Create Dynamic External Task Sensor to handle non exact timeframes
Bert Desmet created AIRFLOW-3107:
Summary: Create Dynamic External Task Sensor to handle non exact
timeframes
Key: AIRFLOW-3107
URL: https://issues.apache.org/jira/browse/AIRFLOW-3107
Project: Apache Airflow
Issue Type: New Feature
Components: core
Affects Versions: 1.10.0
Reporter: Bert Desmet
Fix For: 1.10.0
All,
For a project I'm working on it is necessary to check if a specific task has
run somewhere between 30 days ago and now.
To facilitate this I have patched airflow to include the
'TimeRangeExternalTaskSensor' as provided by
{color:#33}[omnilinguist|https://github.com/omnilinguist] in the following
pull request: [https://github.com/apache/incubator-airflow/pull/1641] {color}
I have updated his code so it aligns better with how the ExternalTaskSensor has
been implemented.
Currently we are running this patch in production on a 1.9 version of Airflow.
I have now merged this with the master branch - but this code has not yet fully
been tested.
The code can be found here:
[https://github.com/biertie/incubator-airflow/blob/master/airflow/sensors/time_range_external_task_sensor.py]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it
codecov-io edited a comment on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it URL: https://github.com/apache/incubator-airflow/pull/3941#issuecomment-423978333 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=h1) Report > Merging [#3941](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/cde74c4c52a30c0e363e87ce28f84c9beee22b52?src=pr=desc) will **decrease** coverage by `0.05%`. > The diff coverage is `14.28%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3941 +/- ## == - Coverage74.9% 74.85% -0.06% == Files 200 200 Lines 1596815982 +14 == + Hits1196111963 +2 - Misses 4007 4019 +12 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3941/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5) | `77.27% <12.5%> (-14.4%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3941/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.66% <16.66%> (-0.2%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=footer). Last update [cde74c4...ce9e0f5](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it
codecov-io commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it URL: https://github.com/apache/incubator-airflow/pull/3941#issuecomment-423978333 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=h1) Report > Merging [#3941](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/cde74c4c52a30c0e363e87ce28f84c9beee22b52?src=pr=desc) will **decrease** coverage by `0.05%`. > The diff coverage is `14.28%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3941 +/- ## == - Coverage74.9% 74.85% -0.06% == Files 200 200 Lines 1596815982 +14 == + Hits1196111963 +2 - Misses 4007 4019 +12 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3941/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5) | `77.27% <12.5%> (-14.4%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/3941/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.66% <16.66%> (-0.2%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=footer). Last update [cde74c4...ce9e0f5](https://codecov.io/gh/apache/incubator-airflow/pull/3941?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] chenjialin opened a new pull request #3942: V1 10 stable
chenjialin opened a new pull request #3942: V1 10 stable
URL: https://github.com/apache/incubator-airflow/pull/3942
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3106) Validate connection when saving connection
[
https://issues.apache.org/jira/browse/AIRFLOW-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625799#comment-16625799
]
ASF GitHub Bot commented on AIRFLOW-3106:
-
BasPH opened a new pull request #3941: [AIRFLOW-3106] Validate Postgres
connection after saving it
URL: https://github.com/apache/incubator-airflow/pull/3941
I want to create connection validation when saving a connection. I started
with Postgres. Would be nice to know if you agree with the PR, before I
implement validation for other connection types.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
-
[https://issues.apache.org/jira/browse/AIRFLOW-3106](https://issues.apache.org/jira/browse/AIRFLOW-3106)
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds validation for the Postgres connection type. After saving a
Connection, it validates by creating a new database connection and executing
`SELECT 1`. In case this fails, an error is displayed:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Validate connection when saving connection
> --
>
> Key: AIRFLOW-3106
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3106
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Bas Harenslak
>Priority: Major
>
> I've encountered failures in DAG runs at various occasions due to invalid
> connection credentials, or a domain was unreachable from the Airflow instance.
> It'd be nice to validate a connection when saving it, to directly know if a
> given connection can be made or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] BasPH opened a new pull request #3941: [AIRFLOW-3106] Validate Postgres connection after saving it
BasPH opened a new pull request #3941: [AIRFLOW-3106] Validate Postgres
connection after saving it
URL: https://github.com/apache/incubator-airflow/pull/3941
I want to create connection validation when saving a connection. I started
with Postgres. Would be nice to know if you agree with the PR, before I
implement validation for other connection types.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
-
[https://issues.apache.org/jira/browse/AIRFLOW-3106](https://issues.apache.org/jira/browse/AIRFLOW-3106)
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds validation for the Postgres connection type. After saving a
Connection, it validates by creating a new database connection and executing
`SELECT 1`. In case this fails, an error is displayed:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3106) Validate connection when saving connection
Bas Harenslak created AIRFLOW-3106: -- Summary: Validate connection when saving connection Key: AIRFLOW-3106 URL: https://issues.apache.org/jira/browse/AIRFLOW-3106 Project: Apache Airflow Issue Type: Improvement Reporter: Bas Harenslak I've encountered failures in DAG runs at various occasions due to invalid connection credentials, or a domain was unreachable from the Airflow instance. It'd be nice to validate a connection when saving it, to directly know if a given connection can be made or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #3940: [AIRFLOW-3054] Add show statements to hql filtering
codecov-io edited a comment on issue #3940: [AIRFLOW-3054] Add show statements to hql filtering URL: https://github.com/apache/incubator-airflow/pull/3940#issuecomment-423954321 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=h1) Report > Merging [#3940](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/cde74c4c52a30c0e363e87ce28f84c9beee22b52?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3940 +/- ## == - Coverage74.9% 74.89% -0.01% == Files 200 200 Lines 1596815968 == - Hits1196111960 -1 - Misses 4007 4008 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/incubator-airflow/pull/3940/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `72.89% <ø> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3940/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.6% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=footer). Last update [cde74c4...4866f21](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3940: [AIRFLOW-3054] Add show statements to hql filtering
codecov-io edited a comment on issue #3940: [AIRFLOW-3054] Add show statements to hql filtering URL: https://github.com/apache/incubator-airflow/pull/3940#issuecomment-423954321 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=h1) Report > Merging [#3940](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/cde74c4c52a30c0e363e87ce28f84c9beee22b52?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3940 +/- ## == - Coverage74.9% 74.89% -0.01% == Files 200 200 Lines 1596815968 == - Hits1196111960 -1 - Misses 4007 4008 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/incubator-airflow/pull/3940/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `72.89% <ø> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3940/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.6% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=footer). Last update [cde74c4...4866f21](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #3940: [AIRFLOW-3054] Add show statements to hql filtering
codecov-io commented on issue #3940: [AIRFLOW-3054] Add show statements to hql filtering URL: https://github.com/apache/incubator-airflow/pull/3940#issuecomment-423954321 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=h1) Report > Merging [#3940](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/cde74c4c52a30c0e363e87ce28f84c9beee22b52?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3940 +/- ## == - Coverage74.9% 74.89% -0.01% == Files 200 200 Lines 1596815968 == - Hits1196111960 -1 - Misses 4007 4008 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/incubator-airflow/pull/3940/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `72.89% <ø> (ø)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3940/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.6% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=footer). Last update [cde74c4...4866f21](https://codecov.io/gh/apache/incubator-airflow/pull/3940?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3054) Show statements don't work in Hive hook
[
https://issues.apache.org/jira/browse/AIRFLOW-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625700#comment-16625700
]
ASF GitHub Bot commented on AIRFLOW-3054:
-
aprotopopov opened a new pull request #3940: [AIRFLOW-3054] Add show statements
to hql filtering
URL: https://github.com/apache/incubator-airflow/pull/3940
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3054
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Queries which starts with show returning results (like "show tables in
default", or "show partitions in default.table_name" and now ingnored with `if`
statement. This minor change `if` statements.
### Tests
- [x] My PR does not need testing for this extremely good reason: minor
change
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Show statements don't work in Hive hook
>
>
> Key: AIRFLOW-3054
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3054
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks
>Reporter: Anton Protopopov
>Assignee: Anton Protopopov
>Priority: Minor
>
> Currently in airflow.hooks.hive_hooks.py in HiveServer2Hook class in
> _get_results method there are filter for incoming hql:
> source code:
> https://github.com/apache/incubator-airflow/blob/a79b92ae40c005d8239dfc7711ff44ae69b9c65b/airflow/hooks/hive_hooks.py#L809-L814
> {code:java}
> # we only get results of statements that returns
> lowered_statement = statement.lower().strip()
> if (lowered_statement.startswith('select') or
> lowered_statement.startswith('with') or
> (lowered_statement.startswith('set') and
> '=' not in lowered_statement)):
> {code}
> Queries which starts with show also returning results (like "show tables in
> default", or "show partitions in default.table_name").
> So I'm thinking it worth to add another statement which starts with "show":
> {code:java}
> lowered_statement.startswith('show')
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] aprotopopov opened a new pull request #3940: [AIRFLOW-3054] Add show statements to hql filtering
aprotopopov opened a new pull request #3940: [AIRFLOW-3054] Add show statements
to hql filtering
URL: https://github.com/apache/incubator-airflow/pull/3940
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3054
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Queries which starts with show returning results (like "show tables in
default", or "show partitions in default.table_name" and now ingnored with `if`
statement. This minor change `if` statements.
### Tests
- [x] My PR does not need testing for this extremely good reason: minor
change
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] XD-DENG commented on issue #3903: [AIRFLOW-3067] Display www_rbac Flask flash msg properly
XD-DENG commented on issue #3903: [AIRFLOW-3067] Display www_rbac Flask flash msg properly URL: https://github.com/apache/incubator-airflow/pull/3903#issuecomment-423927409 Hi @feng-tao , a gentle ping. Please check the UI screenshots attached earlier. Cheers! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] odracci commented on issue #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync
odracci commented on issue #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync URL: https://github.com/apache/incubator-airflow/pull/3770#issuecomment-423896978 @dimberman squas done, PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] odracci edited a comment on issue #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync
odracci edited a comment on issue #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync URL: https://github.com/apache/incubator-airflow/pull/3770#issuecomment-423896978 @dimberman squash done, PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] odracci commented on issue #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync
odracci commented on issue #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync URL: https://github.com/apache/incubator-airflow/pull/3770#issuecomment-423896308 - I refactored `worker_configuration.py` so that now `volumes` and `volume_mount` is a dictionary and is easier to remove the unneeded volume - Now the docker image contains the web assets used in `www_rbac` - Fixed Codecov build path This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] odracci commented on a change in pull request #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync
odracci commented on a change in pull request #3770: [AIRFLOW-XXX] Fix
Kubernetes operator with git-sync
URL: https://github.com/apache/incubator-airflow/pull/3770#discussion_r219747236
##
File path: scripts/ci/kubernetes/docker/build.sh
##
@@ -34,8 +34,18 @@ fi
echo "Airflow directory $AIRFLOW_ROOT"
echo "Airflow Docker directory $DIRNAME"
+if [[ ${PYTHON_VERSION} == '3' ]]; then
+ PYTHON_DOCKER_IMAGE=python:3.6-slim
+else
+ PYTHON_DOCKER_IMAGE=python:2.7-slim
+fi
+
cd $AIRFLOW_ROOT
-python setup.py sdist -q
+docker run -ti --rm -e SLUGIFY_USES_TEXT_UNIDECODE -v ${AIRFLOW_ROOT}:/airflow
\
+-w /airflow ${PYTHON_DOCKER_IMAGE}
./scripts/ci/kubernetes/docker/compile.sh
Review comment:
compile airflow in a container, in addition it will compile the web assets
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] odracci commented on a change in pull request #3770: [AIRFLOW-XXX] Fix Kubernetes operator with git-sync
odracci commented on a change in pull request #3770: [AIRFLOW-XXX] Fix
Kubernetes operator with git-sync
URL: https://github.com/apache/incubator-airflow/pull/3770#discussion_r219746951
##
File path: scripts/ci/5-run-tests.sh
##
@@ -94,5 +94,11 @@ fi
echo "Starting the unit tests with the following nose arguments: "$nose_args
nosetests $nose_args
+# Fix codecov build path
+if [ ! -h /home/travis/build/apache/incubator-airflow ]; then
+ sudo mkdir -p /home/travis/build/apache
+ sudo ln -s ${ROOTDIR} /home/travis/build/apache/incubator-airflow
Review comment:
Codecov uses Travis envs to get the build path but because the tests are
running in the container the path is `/app` instead of
`/home/travis/build/apache/incubator-airflow`
Codecov logs:
```
==> Preparing upload
/bin/sh: 1: cd: can't cd to /home/travis/build/apache/incubator-airflow
Error running `cd /home/travis/build/apache/incubator-airflow && git
ls-files`: None
==> Processing gcov (disable by -X gcov)
Executing gcov (find /home/travis/build/apache/incubator-airflow -not
-path './bower_components/**' -not -path './node_modules/**' -not -path
'./vendor/**' -type f -name '*.gcno' -exec gcov -pb {} +)
find: ‘/home/travis/build/apache/incubator-airflow’: No such file or
directory
Error running `find /home/travis/build/apache/incubator-airflow -not
-path './bower_components/**' -not -path './node_modules/**' -not -path
'./vendor/**' -type f -name '*.gcno' -exec gcov -pb {} +`: None
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] codecov-io edited a comment on issue #2334: [AIRFLOW-1252] API - Fix when conf is in JSON body
codecov-io edited a comment on issue #2334: [AIRFLOW-1252] API - Fix when conf is in JSON body URL: https://github.com/apache/incubator-airflow/pull/2334#issuecomment-337199496 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=h1) Report > Merging [#2334](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/cde74c4c52a30c0e363e87ce28f84c9beee22b52?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=tree) ```diff @@Coverage Diff@@ ## master #2334 +/- ## = + Coverage74.9% 74.9% +<.01% = Files 200 200 Lines 15968 15969 +1 = + Hits11961 11962 +1 Misses 40074007 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/incubator-airflow/pull/2334/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `90.9% <100%> (+0.76%)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/2334/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.6% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=footer). Last update [cde74c4...5c09991](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #2334: [AIRFLOW-1252] API - Fix when conf is in JSON body
codecov-io edited a comment on issue #2334: [AIRFLOW-1252] API - Fix when conf is in JSON body URL: https://github.com/apache/incubator-airflow/pull/2334#issuecomment-337199496 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=h1) Report > Merging [#2334](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/cde74c4c52a30c0e363e87ce28f84c9beee22b52?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=tree) ```diff @@Coverage Diff@@ ## master #2334 +/- ## = + Coverage74.9% 74.9% +<.01% = Files 200 200 Lines 15968 15969 +1 = + Hits11961 11962 +1 Misses 40074007 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/incubator-airflow/pull/2334/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `90.9% <100%> (+0.76%)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/2334/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.6% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=footer). Last update [cde74c4...5c09991](https://codecov.io/gh/apache/incubator-airflow/pull/2334?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
