[jira] [Commented] (AIRFLOW-3638) Add tests for PrestoToMySqlTransfer
[ https://issues.apache.org/jira/browse/AIRFLOW-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740137#comment-16740137 ] ASF GitHub Bot commented on AIRFLOW-3638: - feng-tao commented on pull request #4449: [AIRFLOW-3638] Add tests for PrestoToMySqlTransfer URL: https://github.com/apache/airflow/pull/4449 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add tests for PrestoToMySqlTransfer > --- > > Key: AIRFLOW-3638 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3638 > Project: Apache Airflow > Issue Type: Test >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Fokko commented on issue #4477: [AIRFLOW-3670] Add stages to Travis build
Fokko commented on issue #4477: [AIRFLOW-3670] Add stages to Travis build URL: https://github.com/apache/airflow/pull/4477#issuecomment-453408581 Awesome @drewsonne Thanks for picking this up! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-3670) Make Travis Fail quickly on Flake
[ https://issues.apache.org/jira/browse/AIRFLOW-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-3670: - Assignee: Drew Sonne > Make Travis Fail quickly on Flake > - > > Key: AIRFLOW-3670 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3670 > Project: Apache Airflow > Issue Type: Improvement > Components: travis >Affects Versions: 1.10.1 >Reporter: Drew Sonne >Assignee: Drew Sonne >Priority: Minor > Fix For: 1.10.2 > > > Add stages to the travis builds to split flake8 and functional/unit tests, so > that if flake8 tests fail, the rest of the build will fail quickly. > This will reduce turnaround time for bad linting notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3670) Make Travis Fail quickly on Flake
[ https://issues.apache.org/jira/browse/AIRFLOW-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-3670. - Resolution: Fixed > Make Travis Fail quickly on Flake > - > > Key: AIRFLOW-3670 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3670 > Project: Apache Airflow > Issue Type: Improvement > Components: travis >Affects Versions: 1.10.1 >Reporter: Drew Sonne >Assignee: Drew Sonne >Priority: Minor > Fix For: 1.10.2 > > > Add stages to the travis builds to split flake8 and functional/unit tests, so > that if flake8 tests fail, the rest of the build will fail quickly. > This will reduce turnaround time for bad linting notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #4477: [AIRFLOW-3670] Add stages to Travis build
feng-tao commented on issue #4477: [AIRFLOW-3670] Add stages to Travis build URL: https://github.com/apache/airflow/pull/4477#issuecomment-453393928 cc @kaxil, @ashb, @Fokko FYI, this helps to avoid long running CI if we have flake8 errors. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] singhania edited a comment on issue #4268: [AIRFLOW-3412] Fix kubernetes executor to delete pods after termination
singhania edited a comment on issue #4268: [AIRFLOW-3412] Fix kubernetes executor to delete pods after termination URL: https://github.com/apache/airflow/pull/4268#issuecomment-453374547 I have created a PR #4471 which along with this PR will solve the issue of pods not being created (https://issues.apache.org/jira/browse/AIRFLOW-3652). Would really like to get it reviewed as we are blocked by this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TheZepto opened a new pull request #4482: [AIRFLOW-XXX] Nine are using Airflow.
TheZepto opened a new pull request #4482: [AIRFLOW-XXX] Nine are using Airflow.
URL: https://github.com/apache/airflow/pull/4482
We are using Airflow at Nine.
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ckljohn opened a new pull request #4481: [AIRFLOW-3672] Add support for Mongo DB DNS Seedlist Connection Format
ckljohn opened a new pull request #4481: [AIRFLOW-3672] Add support for Mongo
DB DNS Seedlist Connection Format
URL: https://github.com/apache/airflow/pull/4481
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3672
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Add support for Mongo DB DNS Seedlist Connection Format. Starting with
version 3.6, PyMongo supports mongodb+srv:// URIs. Set `srv` to `true` in the
extra field of the connection will do.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] singhania commented on issue #4268: [AIRFLOW-3412] Fix kubernetes executor to delete pods after termination
singhania commented on issue #4268: [AIRFLOW-3412] Fix kubernetes executor to delete pods after termination URL: https://github.com/apache/airflow/pull/4268#issuecomment-453374547 I have created a PR which along with this PR will solve the issue of pods not being created (https://issues.apache.org/jira/browse/AIRFLOW-3652). Would really like to get it reviewed as we are blocked by this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ckljohn opened a new pull request #4480: [AIRFLOW-3671] Remove arg `replace` of MongoToS3Operator from `kwargs`
ckljohn opened a new pull request #4480: [AIRFLOW-3671] Remove arg `replace` of
MongoToS3Operator from `kwargs`
URL: https://github.com/apache/airflow/pull/4480
If the operator get arguments from `kwargs`, it will fire DeprecationWarning
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3671
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Remove arg `replace` of MongoToS3Operator from `kwargs`
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Trivial change.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] reubenvanammers commented on issue #4156: [AIRFLOW-3314] Changed auto inlets feature to work as described
reubenvanammers commented on issue #4156: [AIRFLOW-3314] Changed auto inlets feature to work as described URL: https://github.com/apache/airflow/pull/4156#issuecomment-453369593 > @reubenvanammers can you rebase? Done! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mik-laj commented on a change in pull request #4474: [AIRFLOW-3439] Decode logs with 'utf-8'
mik-laj commented on a change in pull request #4474: [AIRFLOW-3439] Decode logs
with 'utf-8'
URL: https://github.com/apache/airflow/pull/4474#discussion_r246997088
##
File path: airflow/utils/log/gcs_task_handler.py
##
@@ -129,7 +129,7 @@ def gcs_read(self, remote_log_location):
:type remote_log_location: str (path)
"""
bkt, blob = self.parse_gcs_url(remote_log_location)
-return self.hook.download(bkt, blob).decode()
+return self.hook.download(bkt, blob).decode('utf-8')
Review comment:
> So .decode('utf-8') is no difference from .decode().
I have a different opinion. Airflow support Python 2.7, 3.6=>
([Source](https://github.com/apache/airflow/blob/master/setup.py#L424))
In documentation for Python 2.7, you can read a fragment:
> Python’s default encoding is the ‘ascii’ encoding.
([Source](https://docs.python.org/2/howto/unicode.html))
It is also worth quoting another fragment
> str.decode([encoding[, errors]])
> Decodes the string using the codec registered for encoding. encoding
defaults to the default string encoding.
([Source]( https://docs.python.org/2.7/library/stdtypes.html#str.decode))
Taking into account the quotations above, the change proposed here changes
the behavior of the program.
I hope that the explanations are sufficient and clear.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] mik-laj commented on a change in pull request #4474: [AIRFLOW-3439] Decode logs with 'utf-8'
mik-laj commented on a change in pull request #4474: [AIRFLOW-3439] Decode logs
with 'utf-8'
URL: https://github.com/apache/airflow/pull/4474#discussion_r246997088
##
File path: airflow/utils/log/gcs_task_handler.py
##
@@ -129,7 +129,7 @@ def gcs_read(self, remote_log_location):
:type remote_log_location: str (path)
"""
bkt, blob = self.parse_gcs_url(remote_log_location)
-return self.hook.download(bkt, blob).decode()
+return self.hook.download(bkt, blob).decode('utf-8')
Review comment:
> So .decode('utf-8') is no difference from .decode().
I have a different opinion. Airflow support Python 2.7, 3.6=>
([Source](https://github.com/apache/airflow/blob/master/setup.py#L424))
In documentation for Python 2.7, you can read a fragment:
> Python’s default encoding is the ‘ascii’ encoding.
Source: https://docs.python.org/2/howto/unicode.html
It is also worth quoting another fragment
> str.decode([encoding[, errors]])
> Decodes the string using the codec registered for encoding. encoding
defaults to the default string encoding.
Source: https://docs.python.org/2.7/library/stdtypes.html#str.decode
Taking into account the quotations above, the change proposed here changes
the behavior of the program.I hope that the explanations are sufficient and
clear.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] mik-laj commented on a change in pull request #4474: [AIRFLOW-3439] Decode logs with 'utf-8'
mik-laj commented on a change in pull request #4474: [AIRFLOW-3439] Decode logs
with 'utf-8'
URL: https://github.com/apache/airflow/pull/4474#discussion_r246997088
##
File path: airflow/utils/log/gcs_task_handler.py
##
@@ -129,7 +129,7 @@ def gcs_read(self, remote_log_location):
:type remote_log_location: str (path)
"""
bkt, blob = self.parse_gcs_url(remote_log_location)
-return self.hook.download(bkt, blob).decode()
+return self.hook.download(bkt, blob).decode('utf-8')
Review comment:
> So .decode('utf-8') is no difference from .decode().
I have a different opinion. Airflow support Python 2.7, 3.6=>
([Source](https://github.com/apache/airflow/blob/master/setup.py#L424))
In documentation for Python 2.7, you can read a fragment:
> Python’s default encoding is the ‘ascii’ encoding.
Source: https://docs.python.org/2/howto/unicode.html
It is also worth quoting another fragment
> str.decode([encoding[, errors]])
> Decodes the string using the codec registered for encoding. encoding
defaults to the default string encoding.
Source: https://docs.python.org/2.7/library/stdtypes.html#str.decode
Taking into account the quotations above, the change proposed here changes
the behavior of the program.I hope that the explanations are sufficient and
clear.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3671) Wrong deprecation warning for MongoToS3Operator
John Cheng created AIRFLOW-3671:
---
Summary: Wrong deprecation warning for MongoToS3Operator
Key: AIRFLOW-3671
URL: https://issues.apache.org/jira/browse/AIRFLOW-3671
Project: Apache Airflow
Issue Type: New Feature
Components: operators
Affects Versions: 1.10.0
Reporter: John Cheng
Assignee: John Cheng
The operator should not get {{replace}} from {{kwargs}}.
{{/usr/local/lib/python3.6/site-packages/airflow/models.py:2418:
PendingDeprecationWarning: Invalid arguments were passed to MongoToS3Operator.
Support for passing such arguments will be dropped in Airflow 2.0. Invalid
arguments were:}}
{{*args: ()}}
{{**kwargs: \{'replace': True}}}
{{ category=PendingDeprecationWarning}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] mik-laj commented on a change in pull request #4474: [AIRFLOW-3439] Decode logs with 'utf-8'
mik-laj commented on a change in pull request #4474: [AIRFLOW-3439] Decode logs
with 'utf-8'
URL: https://github.com/apache/airflow/pull/4474#discussion_r246997088
##
File path: airflow/utils/log/gcs_task_handler.py
##
@@ -129,7 +129,7 @@ def gcs_read(self, remote_log_location):
:type remote_log_location: str (path)
"""
bkt, blob = self.parse_gcs_url(remote_log_location)
-return self.hook.download(bkt, blob).decode()
+return self.hook.download(bkt, blob).decode('utf-8')
Review comment:
> So .decode('utf-8') is no difference from .decode().
I have a different opinion. Airflow support Python 2.7, 3.6=>
([Source](https://github.com/apache/airflow/blob/master/setup.py#L424))
In documentation for Python 2.7, you can read a fragment:
> Python’s default encoding is the ‘ascii’ encoding.
Source: https://docs.python.org/2/howto/unicode.html
It is also worth quoting another fragment
> str.decode([encoding[, errors]])
> Decodes the string using the codec registered for encoding. encoding
defaults to the default string encoding.
Taking into account the quotations above, the change proposed here changes
the behavior of the program.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] codecov-io edited a comment on issue #4156: [AIRFLOW-3314] Changed auto inlets feature to work as described
codecov-io edited a comment on issue #4156: [AIRFLOW-3314] Changed auto inlets feature to work as described URL: https://github.com/apache/airflow/pull/4156#issuecomment-436867007 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=h1) Report > Merging [#4156](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d108d8d2aa0726574f985b44588d157579b8b97e?src=pr=desc) will **increase** coverage by `0.03%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4156 +/- ## == + Coverage 79.51% 79.54% +0.03% == Files 204 204 Lines 1655016561 +11 == + Hits1315913174 +15 + Misses 3391 3387 -4 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/lineage/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4156/diff?src=pr=tree#diff-YWlyZmxvdy9saW5lYWdlL19faW5pdF9fLnB5) | `97.14% <100%> (+0.53%)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4156/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.68% <100%> (+0.08%)` | :arrow_up: | | [airflow/lineage/datasets.py](https://codecov.io/gh/apache/airflow/pull/4156/diff?src=pr=tree#diff-YWlyZmxvdy9saW5lYWdlL2RhdGFzZXRzLnB5) | `87.32% <0%> (+2.81%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=footer). Last update [d108d8d...ff7e248](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4156: [AIRFLOW-3314] Changed auto inlets feature to work as described
codecov-io edited a comment on issue #4156: [AIRFLOW-3314] Changed auto inlets feature to work as described URL: https://github.com/apache/airflow/pull/4156#issuecomment-436867007 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=h1) Report > Merging [#4156](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d108d8d2aa0726574f985b44588d157579b8b97e?src=pr=desc) will **increase** coverage by `0.03%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4156 +/- ## == + Coverage 79.51% 79.54% +0.03% == Files 204 204 Lines 1655016561 +11 == + Hits1315913174 +15 + Misses 3391 3387 -4 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/lineage/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4156/diff?src=pr=tree#diff-YWlyZmxvdy9saW5lYWdlL19faW5pdF9fLnB5) | `97.14% <100%> (+0.53%)` | :arrow_up: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4156/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.68% <100%> (+0.08%)` | :arrow_up: | | [airflow/lineage/datasets.py](https://codecov.io/gh/apache/airflow/pull/4156/diff?src=pr=tree#diff-YWlyZmxvdy9saW5lYWdlL2RhdGFzZXRzLnB5) | `87.32% <0%> (+2.81%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=footer). Last update [d108d8d...ff7e248](https://codecov.io/gh/apache/airflow/pull/4156?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4479: [AIRFLOW-XXX] Docs: Fix paths to GCS transfer operator
codecov-io commented on issue #4479: [AIRFLOW-XXX] Docs: Fix paths to GCS transfer operator URL: https://github.com/apache/airflow/pull/4479#issuecomment-453364375 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4479?src=pr=h1) Report > Merging [#4479](https://codecov.io/gh/apache/airflow/pull/4479?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d108d8d2aa0726574f985b44588d157579b8b97e?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4479?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4479 +/- ## === Coverage 79.51% 79.51% === Files 204 204 Lines 1655016550 === Hits1315913159 Misses 3391 3391 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4479?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4479?src=pr=footer). Last update [d108d8d...702cfdf](https://codecov.io/gh/apache/airflow/pull/4479?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4478: [AIRFLOW-3616] Add aliases for schema with underscore
codecov-io edited a comment on issue #4478: [AIRFLOW-3616] Add aliases for schema with underscore URL: https://github.com/apache/airflow/pull/4478#issuecomment-453356007 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=h1) Report > Merging [#4478](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d108d8d2aa0726574f985b44588d157579b8b97e?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4478 +/- ## == + Coverage 79.51% 79.51% +<.01% == Files 204 204 Lines 1655016552 +2 == + Hits1315913161 +2 Misses 3391 3391 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/4478/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `88.75% <100%> (+0.14%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=footer). Last update [d108d8d...bbaa80a](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4478: [AIRFLOW-3616] Add aliases for schema with underscore
codecov-io commented on issue #4478: [AIRFLOW-3616] Add aliases for schema with underscore URL: https://github.com/apache/airflow/pull/4478#issuecomment-453356007 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=h1) Report > Merging [#4478](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d108d8d2aa0726574f985b44588d157579b8b97e?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4478 +/- ## == + Coverage 79.51% 79.51% +<.01% == Files 204 204 Lines 1655016552 +2 == + Hits1315913161 +2 Misses 3391 3391 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/4478/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `88.75% <100%> (+0.14%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=footer). Last update [d108d8d...bbaa80a](https://codecov.io/gh/apache/airflow/pull/4478?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mik-laj commented on issue #4479: [AIRFLOW-XXX] Docs: Fix paths to GCS transfer operator
mik-laj commented on issue #4479: [AIRFLOW-XXX] Docs: Fix paths to GCS transfer operator URL: https://github.com/apache/airflow/pull/4479#issuecomment-453353175 @antonimaciej, Can you look at this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3616) Connection parsed from URI - unacceptable underscore in schema part
[
https://issues.apache.org/jira/browse/AIRFLOW-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739945#comment-16739945
]
ASF GitHub Bot commented on AIRFLOW-3616:
-
mik-laj commented on pull request #4478: [AIRFLOW-3616] Add aliases for schema
with underscore
URL: https://github.com/apache/airflow/pull/4478
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-3616\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3616
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] A comprehensive description is included in JIRA Ticket, include
analysis of other possible solutions.
### Tests
- [ ] Add test_connection_from_uri_with_underscore in
tests.models:ConnectionTest
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Connection parsed from URI - unacceptable underscore in schema part
>
>
> Key: AIRFLOW-3616
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3616
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Major
>
> There is a problem with creating a new connection based on the state of the
> environment variable if the connection type contains the underscore character.
> If we want to configure a connection based on an environment variable, we
> must create an environment variable. The name of the environment must be
> given according to the scheme:
> {{AIRFLOW_CONN_[CONN_ID]}}
> where {{[CONN_ID]}} is the connection identifier to be used, written in
> upper case.
> The content of the variable defines the connection and is saved in the form
> of URI.
> Defining a URI is complex, but the key is that the connection type is given
> as the schema. There are many possible values to give. but the sample values
> are {{mysql}}, {{postgresql}} or {{google_cloud_platform}}. Unfortunately,
> the last case is not properly handled.
> This is caused by using {{urllib.parse}} to process the value. Unfortunately,
> this module does not support the uppercase character in schema port of URI -
> see below snippets showing the behaviour.
> Since urllib.parse is really there to parse URLs and it is not good for
> parsing non-URL URIs - we should likely use different parser which handles
> more generic URIs.
> Especially that it also creates other problems:
> https://issues.apache.org/jira/browse/AIRFLOW-3615
> Another solution is to create aliases for each connection type with a variant
> that does not contain an unacceptable character. For example
> {{google_cloud_platform => gcp}}. It is worth noting that one alias is
> currently defined - {{postgresql => postgres}}.
> Snippet showing urrlib.parse behaviour:
> Python 3.6.5 (default, Oct 3 2018, 10:03:09)
> Type 'copyright', 'credits' or 'license' for more information
> IPython 7.0.1 – An enhanced Interactive Python. Type '?' for help
> In [1]: from urllib.parse import urlparse
> In [2]:
> urlparse("google_cloud_platform://user:pass@hostname/path?extra_param=extra_param_value")
> Out[2]: ParseResult(scheme='', netloc='',
> path='google_cloud_platform://user:pass@hostname/path', params='',
> query='extra_param=extra_param_value', fragment='')
[jira] [Commented] (AIRFLOW-3616) Connection parsed from URI - unacceptable underscore in schema part
[
https://issues.apache.org/jira/browse/AIRFLOW-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739948#comment-16739948
]
ASF GitHub Bot commented on AIRFLOW-3616:
-
mik-laj commented on pull request #4478: [AIRFLOW-3616] Add aliases for schema
with underscore
URL: https://github.com/apache/airflow/pull/4478
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Connection parsed from URI - unacceptable underscore in schema part
>
>
> Key: AIRFLOW-3616
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3616
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Major
>
> There is a problem with creating a new connection based on the state of the
> environment variable if the connection type contains the underscore character.
> If we want to configure a connection based on an environment variable, we
> must create an environment variable. The name of the environment must be
> given according to the scheme:
> {{AIRFLOW_CONN_[CONN_ID]}}
> where {{[CONN_ID]}} is the connection identifier to be used, written in
> upper case.
> The content of the variable defines the connection and is saved in the form
> of URI.
> Defining a URI is complex, but the key is that the connection type is given
> as the schema. There are many possible values to give. but the sample values
> are {{mysql}}, {{postgresql}} or {{google_cloud_platform}}. Unfortunately,
> the last case is not properly handled.
> This is caused by using {{urllib.parse}} to process the value. Unfortunately,
> this module does not support the uppercase character in schema port of URI -
> see below snippets showing the behaviour.
> Since urllib.parse is really there to parse URLs and it is not good for
> parsing non-URL URIs - we should likely use different parser which handles
> more generic URIs.
> Especially that it also creates other problems:
> https://issues.apache.org/jira/browse/AIRFLOW-3615
> Another solution is to create aliases for each connection type with a variant
> that does not contain an unacceptable character. For example
> {{google_cloud_platform => gcp}}. It is worth noting that one alias is
> currently defined - {{postgresql => postgres}}.
> Snippet showing urrlib.parse behaviour:
> Python 3.6.5 (default, Oct 3 2018, 10:03:09)
> Type 'copyright', 'credits' or 'license' for more information
> IPython 7.0.1 – An enhanced Interactive Python. Type '?' for help
> In [1]: from urllib.parse import urlparse
> In [2]:
> urlparse("google_cloud_platform://user:pass@hostname/path?extra_param=extra_param_value")
> Out[2]: ParseResult(scheme='', netloc='',
> path='google_cloud_platform://user:pass@hostname/path', params='',
> query='extra_param=extra_param_value', fragment='')
> In [3]:
> urlparse("gcp://user:pass@hostname/path?extra_param=extra_param_value")
> Out[3]: ParseResult(scheme='gcp', netloc='user:pass@hostname', path='/path',
> params='', query='extra_param=extra_param_value', fragment='')
> Connection parsed from URI - unacceptable underscore in schema part
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] mik-laj opened a new pull request #4478: [AIRFLOW-3616] Add aliases for schema with underscore
mik-laj opened a new pull request #4478: [AIRFLOW-3616] Add aliases for schema
with underscore
URL: https://github.com/apache/airflow/pull/4478
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-3616\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3616
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] A comprehensive description is included in JIRA Ticket, include
analysis of other possible solutions.
### Tests
- [ ] Add test_connection_from_uri_with_underscore in
tests.models:ConnectionTest
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (AIRFLOW-3616) Connection parsed from URI - unacceptable underscore in schema part
[
https://issues.apache.org/jira/browse/AIRFLOW-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamil Bregula reassigned AIRFLOW-3616:
--
Assignee: Kamil Bregula
> Connection parsed from URI - unacceptable underscore in schema part
>
>
> Key: AIRFLOW-3616
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3616
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Major
>
> There is a problem with creating a new connection based on the state of the
> environment variable if the connection type contains the underscore character.
> If we want to configure a connection based on an environment variable, we
> must create an environment variable. The name of the environment must be
> given according to the scheme:
> {{AIRFLOW_CONN_[CONN_ID]}}
> where {{[CONN_ID]}} is the connection identifier to be used, written in
> upper case.
> The content of the variable defines the connection and is saved in the form
> of URI.
> Defining a URI is complex, but the key is that the connection type is given
> as the schema. There are many possible values to give. but the sample values
> are {{mysql}}, {{postgresql}} or {{google_cloud_platform}}. Unfortunately,
> the last case is not properly handled.
> This is caused by using {{urllib.parse}} to process the value. Unfortunately,
> this module does not support the uppercase character in schema port of URI -
> see below snippets showing the behaviour.
> Since urllib.parse is really there to parse URLs and it is not good for
> parsing non-URL URIs - we should likely use different parser which handles
> more generic URIs.
> Especially that it also creates other problems:
> https://issues.apache.org/jira/browse/AIRFLOW-3615
> Another solution is to create aliases for each connection type with a variant
> that does not contain an unacceptable character. For example
> {{google_cloud_platform => gcp}}. It is worth noting that one alias is
> currently defined - {{postgresql => postgres}}.
> Snippet showing urrlib.parse behaviour:
> Python 3.6.5 (default, Oct 3 2018, 10:03:09)
> Type 'copyright', 'credits' or 'license' for more information
> IPython 7.0.1 – An enhanced Interactive Python. Type '?' for help
> In [1]: from urllib.parse import urlparse
> In [2]:
> urlparse("google_cloud_platform://user:pass@hostname/path?extra_param=extra_param_value")
> Out[2]: ParseResult(scheme='', netloc='',
> path='google_cloud_platform://user:pass@hostname/path', params='',
> query='extra_param=extra_param_value', fragment='')
> In [3]:
> urlparse("gcp://user:pass@hostname/path?extra_param=extra_param_value")
> Out[3]: ParseResult(scheme='gcp', netloc='user:pass@hostname', path='/path',
> params='', query='extra_param=extra_param_value', fragment='')
> Connection parsed from URI - unacceptable underscore in schema part
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #4456: [AIRFLOW-3645] Add base_executor_config
codecov-io edited a comment on issue #4456: [AIRFLOW-3645] Add base_executor_config URL: https://github.com/apache/airflow/pull/4456#issuecomment-453308561 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=h1) Report > Merging [#4456](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b450065dc2422408a6cb828ade69804956174ca4?src=pr=desc) will **decrease** coverage by `62.13%`. > The diff coverage is `80%`. [](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4456 +/- ## === - Coverage 78.85% 16.72% -62.14% === Files 204 204 Lines 1649816569 +71 === - Hits13010 2771-10239 - Misses 348813798+10310 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `28.21% <75%> (-64.39%)` | :arrow_down: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `29.41% <87.5%> (-53.35%)` | :arrow_down: | | [airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/validators.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/api/common/experimental/get\_dag\_runs.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFnX3J1bnMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/forms.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | ... and [169 more](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=footer). Last update [b450065...3f14ff2](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4456: [AIRFLOW-3645] Add base_executor_config
codecov-io commented on issue #4456: [AIRFLOW-3645] Add base_executor_config URL: https://github.com/apache/airflow/pull/4456#issuecomment-453308561 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=h1) Report > Merging [#4456](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b450065dc2422408a6cb828ade69804956174ca4?src=pr=desc) will **decrease** coverage by `62.13%`. > The diff coverage is `80%`. [](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4456 +/- ## === - Coverage 78.85% 16.72% -62.14% === Files 204 204 Lines 1649816569 +71 === - Hits13010 2771-10239 - Misses 348813798+10310 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `28.21% <75%> (-64.39%)` | :arrow_down: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `29.41% <87.5%> (-53.35%)` | :arrow_down: | | [airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/validators.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/api/common/experimental/get\_dag\_runs.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFnX3J1bnMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/forms.py](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | ... and [169 more](https://codecov.io/gh/apache/airflow/pull/4456/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=footer). Last update [b450065...3f14ff2](https://codecov.io/gh/apache/airflow/pull/4456?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
feng-tao edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-453302551 cc @kaxil as he is driving the 10.2 release. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
feng-tao commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-453302551 cc @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] drewsonne commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
drewsonne commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-453300966 @feng-tao amazing, thank you! looking forward to the release for this This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3605) Load Plugins via entry_points
[
https://issues.apache.org/jira/browse/AIRFLOW-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739850#comment-16739850
]
ASF GitHub Bot commented on AIRFLOW-3605:
-
feng-tao commented on pull request #4412: [AIRFLOW-3605] Load plugins from
entry_points
URL: https://github.com/apache/airflow/pull/4412
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Load Plugins via entry_points
> -
>
> Key: AIRFLOW-3605
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3605
> Project: Apache Airflow
> Issue Type: Bug
> Components: plugins
>Affects Versions: 1.10.1
>Reporter: Drew Sonne
>Assignee: Drew Sonne
>Priority: Minor
> Labels: features, newbie, pull-request-available
> Fix For: 1.10.2
>
>
> Added logic to load AirflowPlugins from the
> [entry_points|https://setuptools.readthedocs.io/en/latest/pkg_resources.html#id16].
> Rather than moving files on the OS, this allows plugins to be installed and
> distributed via {{pip}}.
> Also added a callback to execute business logic from the plugin when the
> plugin is loaded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] feng-tao closed pull request #4412: [AIRFLOW-3605] Load plugins from entry_points
feng-tao closed pull request #4412: [AIRFLOW-3605] Load plugins from
entry_points
URL: https://github.com/apache/airflow/pull/4412
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/plugins_manager.py b/airflow/plugins_manager.py
index 0eee5f8f93..d0efe62da9 100644
--- a/airflow/plugins_manager.py
+++ b/airflow/plugins_manager.py
@@ -28,6 +28,7 @@
import os
import re
import sys
+import pkg_resources
from airflow import configuration
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -59,6 +60,59 @@ def validate(cls):
if not cls.name:
raise AirflowPluginException("Your plugin needs a name.")
+@classmethod
+def on_load(cls, *args, **kwargs):
+"""
+Executed when the plugin is loaded.
+This method is only called once during runtime.
+
+:param args: If future arguments are passed in on call.
+:param kwargs: If future arguments are passed in on call.
+"""
+pass
+
+
+def load_entrypoint_plugins(entry_points, airflow_plugins):
+"""
+Load AirflowPlugin subclasses from the entrypoints
+provided. The entry_point group should be 'airflow.plugins'.
+
+:param entry_points: A collection of entrypoints to search for plugins
+:type entry_points: Generator[setuptools.EntryPoint, None, None]
+:param airflow_plugins: A collection of existing airflow plugins to
+ensure we don't load duplicates
+:type airflow_plugins: List[AirflowPlugin]
+:return: List[Type[AirflowPlugin]]
+"""
+for entry_point in entry_points:
+log.debug('Importing entry_point plugin %s', entry_point.name)
+plugin_obj = entry_point.load()
+if is_valid_plugin(plugin_obj, airflow_plugins):
+if callable(getattr(plugin_obj, 'on_load', None)):
+plugin_obj.on_load()
+airflow_plugins.append(plugin_obj)
+return airflow_plugins
+
+
+def is_valid_plugin(plugin_obj, existing_plugins):
+"""
+Check whether a potential object is a subclass of
+the AirflowPlugin class.
+
+:param plugin_obj: potential subclass of AirflowPlugin
+:param existing_plugins: Existing list of AirflowPlugin subclasses
+:return: Whether or not the obj is a valid subclass of
+AirflowPlugin
+"""
+if (
+inspect.isclass(plugin_obj) and
+issubclass(plugin_obj, AirflowPlugin) and
+(plugin_obj is not AirflowPlugin)
+):
+plugin_obj.validate()
+return plugin_obj not in existing_plugins
+return False
+
plugins_folder = configuration.conf.get('core', 'plugins_folder')
if not plugins_folder:
@@ -90,19 +144,19 @@ def validate(cls):
m = imp.load_source(namespace, filepath)
for obj in list(m.__dict__.values()):
-if (
-inspect.isclass(obj) and
-issubclass(obj, AirflowPlugin) and
-obj is not AirflowPlugin):
-obj.validate()
-if obj not in plugins:
-plugins.append(obj)
+if is_valid_plugin(obj, plugins):
+plugins.append(obj)
except Exception as e:
log.exception(e)
log.error('Failed to import plugin %s', filepath)
import_errors[filepath] = str(e)
+plugins = load_entrypoint_plugins(
+pkg_resources.iter_entry_points('airflow.plugins'),
+plugins
+)
+
def make_module(name, objects):
log.debug('Creating module %s', name)
diff --git a/docs/plugins.rst b/docs/plugins.rst
index e2f37774de..3e72903a55 100644
--- a/docs/plugins.rst
+++ b/docs/plugins.rst
@@ -220,3 +220,56 @@ Note on role based views
Airflow 1.10 introduced role based views using FlaskAppBuilder. You can
configure which UI is used by setting
rbac = True. To support plugin views and links for both versions of the UI and
maintain backwards compatibility,
the fields appbuilder_views and appbuilder_menu_items were added to the
AirflowTestPlugin class.
+
+
+Plugins as Python packages
+--
+
+It is possible to load plugins via `setuptools'
entrypoint`_
mechanism. To do this link
+your plugin using an entrypoint in your package. If the package is installed,
airflow
+will automatically load the registered plugins from the entrypoint list.
+
+_Note_: Neither the entrypoint name (eg, `my_plugin`) nor the name of the
+plugin class will contribute towards the module and class name of the plugin
+itself. The structure is determined by
+`airflow.plugins_manager.AirflowPlugin.name` and the class name of the plugin
+component
[GitHub] feng-tao commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
feng-tao commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-453291298 thanks @drewsonne , merging now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-1491) Celery executor restarts result in duplicate tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739839#comment-16739839 ] ASF GitHub Bot commented on AIRFLOW-1491: - stale[bot] commented on pull request #2538: [AIRFLOW-1491] Recover celery queue on restart URL: https://github.com/apache/airflow/pull/2538 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Celery executor restarts result in duplicate tasks > -- > > Key: AIRFLOW-1491 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1491 > Project: Apache Airflow > Issue Type: Bug >Reporter: George Leslie-Waksman >Assignee: George Leslie-Waksman >Priority: Major > > Since the Celery executor uses internal state to determine what it has queued > already, anything that wipes the executor state (such as a scheduler restart) > will result in rescheduling of tasks. > This bug exacerbates a number of task scheduling race conditions and can > result in duplicate task execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-865) Configure FTP connection mode
[ https://issues.apache.org/jira/browse/AIRFLOW-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739840#comment-16739840 ] ASF GitHub Bot commented on AIRFLOW-865: stale[bot] commented on pull request #2069: [AIRFLOW-865] - Configure FTP connection mode URL: https://github.com/apache/airflow/pull/2069 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Configure FTP connection mode > - > > Key: AIRFLOW-865 > URL: https://issues.apache.org/jira/browse/AIRFLOW-865 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Ján Koščo >Assignee: Ján Koščo >Priority: Trivial > > As user I want to configure mode for my ftp connections to support also FTP > serves with passive mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #4083: [AIRFLOW-3211] Reattach to GCP Dataproc jobs upon Airflow restart
feng-tao commented on issue #4083: [AIRFLOW-3211] Reattach to GCP Dataproc jobs upon Airflow restart URL: https://github.com/apache/airflow/pull/4083#issuecomment-453288152 PTAL @fenglu-g @bolkedebruin This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-1836) Airflow select keycloak as OAuth Provider
[
https://issues.apache.org/jira/browse/AIRFLOW-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739837#comment-16739837
]
ASF GitHub Bot commented on AIRFLOW-1836:
-
stale[bot] commented on pull request #2799: [AIRFLOW-1836] airflow uses OAuth
Provider keycloak
URL: https://github.com/apache/airflow/pull/2799
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Airflow select keycloak as OAuth Provider
> -
>
> Key: AIRFLOW-1836
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1836
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib
>Affects Versions: 1.8.2
> Environment: Linux CentOS 7.3
> Python 2.7.13
>Reporter: Kevin Zhang
>Priority: Major
> Fix For: 1.8.2
>
>
> As the need of my project in hand, I had to integrate keycloak with
> airflow.During implemetation，I learnt from the github_enterprise_auth
> provided by airflow.When I found it used python module flask_oauthlib，I
> thought the fastest way for me was to modify the parameters about connecting
> to keycloak.So I tried the idea and found it's true.The next question is to
> resolve the token from keycloak. I referred to keycloak and decided two key
> -preferred_username and email.
> In my environment, the keycloak is accessed by http.So I had to add the code
> {code:java}
> import os
> os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1772) Google Updated Sensor doesnt work with CRON expressions
[ https://issues.apache.org/jira/browse/AIRFLOW-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739838#comment-16739838 ] ASF GitHub Bot commented on AIRFLOW-1772: - stale[bot] commented on pull request #2747: AIRFLOW-1772: Fix bug with handling cron expressions as an schedule i… URL: https://github.com/apache/airflow/pull/2747 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Google Updated Sensor doesnt work with CRON expressions > > > Key: AIRFLOW-1772 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1772 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Reporter: sam elamin >Priority: Major > > Hi > I noticed that the [Update > Sensor|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/sensors/gcs_sensor.py#L73] > does not work with a cron expression set as the schedule interval > According to the airflow docs a scheduler interval can be a timedelta or a > string > When its a cron expression the schedule is a string and hence this [line | > https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/sensors/gcs_sensor.py#L70] > breaks > the error is below > ''str' object is not callable' > Ideally that line should check if the interval is a cron expression and if it > is convert it to a time delta > Also as a side note i noticed that the update sensor doesnt match regex and > has to be exact, surly adding regex support here would make sense since the > gsutil cli does this as well -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1238) Connections CLI doesn't decode percent encodings
[
https://issues.apache.org/jira/browse/AIRFLOW-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739836#comment-16739836
]
ASF GitHub Bot commented on AIRFLOW-1238:
-
stale[bot] commented on pull request #3605: [AIRFLOW-1238] Decode URL-encoded
characters.
URL: https://github.com/apache/airflow/pull/3605
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Connections CLI doesn't decode percent encodings
>
>
> Key: AIRFLOW-1238
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1238
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: John Zeringue
>Assignee: Matt Chapman
>Priority: Minor
>
> For example, {{airflow connections -a --conn_id my_connection --conn_uri
> https://my%20username:my%20passw...@mydomain.com}} should create a connection
> with username {{my username}} and password {{my password}}. Instead, it
> creates a connection with username {{my%20username}} and password
> {{my%20password}}.
> As it stands now, email usernames (which have an {{@}}) or passwords with
> special characters like {{p@$$word}} can't be specified from the CLI either.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-722) Add Celery queue sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739841#comment-16739841 ] ASF GitHub Bot commented on AIRFLOW-722: stale[bot] commented on pull request #1964: [AIRFLOW-722] Add Celery queue sensor URL: https://github.com/apache/airflow/pull/1964 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Celery queue sensor > --- > > Key: AIRFLOW-722 > URL: https://issues.apache.org/jira/browse/AIRFLOW-722 > Project: Apache Airflow > Issue Type: New Feature > Components: celery >Reporter: Nicholas Duffy >Assignee: Nicholas Duffy >Priority: Major > > As an Airflow user, I would like a Celery queue sensor, in order to check > that a particular Celery queue is empty before continuing to the next task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] stale[bot] closed pull request #3643: [AIRFLOW-XXX] Reduce logging for used pool settings
stale[bot] closed pull request #3643: [AIRFLOW-XXX] Reduce logging for used
pool settings
URL: https://github.com/apache/airflow/pull/3643
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/settings.py b/airflow/settings.py
index 788ecc4a2a..31969efb52 100644
--- a/airflow/settings.py
+++ b/airflow/settings.py
@@ -172,8 +172,8 @@ def configure_orm(disable_connection_pool=False):
except conf.AirflowConfigException:
pool_recycle = 1800
-log.info("setting.configure_orm(): Using pool settings. pool_size={}, "
- "pool_recycle={}".format(pool_size, pool_recycle))
+log.debug("setting.configure_orm(): Using pool settings. pool_size={},
"
+ "pool_recycle={}".format(pool_size, pool_recycle))
engine_args['pool_size'] = pool_size
engine_args['pool_recycle'] = pool_recycle
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] stale[bot] closed pull request #2538: [AIRFLOW-1491] Recover celery queue on restart
stale[bot] closed pull request #2538: [AIRFLOW-1491] Recover celery queue on
restart
URL: https://github.com/apache/airflow/pull/2538
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/executors/base_executor.py
b/airflow/executors/base_executor.py
index 7a4065eb07..eb8bfc8205 100644
--- a/airflow/executors/base_executor.py
+++ b/airflow/executors/base_executor.py
@@ -120,7 +120,7 @@ def heartbeat(self):
self.queued_tasks.pop(key)
ti.refresh_from_db()
if ti.state != State.RUNNING:
-self.running[key] = command
+self.running[key] = True
self.execute_async(key, command=command, queue=queue)
else:
self.logger.debug(
diff --git a/airflow/executors/celery_executor.py
b/airflow/executors/celery_executor.py
index 17c343bd4a..ef8d82c13c 100644
--- a/airflow/executors/celery_executor.py
+++ b/airflow/executors/celery_executor.py
@@ -13,6 +13,7 @@
# limitations under the License.
from builtins import object
+
import logging
import subprocess
import ssl
@@ -20,10 +21,13 @@
import traceback
from celery import Celery
+from celery.result import AsyncResult
from celery import states as celery_states
from airflow.exceptions import AirflowConfigException, AirflowException
from airflow.executors.base_executor import BaseExecutor
+from airflow.models import ExecutorQueueManager
+
from airflow import configuration
PARALLELISM = configuration.get('core', 'PARALLELISM')
@@ -49,10 +53,13 @@ class CeleryConfig(object):
CELERY_DEFAULT_QUEUE = DEFAULT_QUEUE
CELERY_DEFAULT_EXCHANGE = DEFAULT_QUEUE
+#CELERY_SEND_EVENTS = True
+#CELERY_EVENT_QUEUE_EXPIRES = 120
+
celery_ssl_active = False
try:
celery_ssl_active = configuration.getboolean('celery',
'CELERY_SSL_ACTIVE')
-except AirflowConfigException as e:
+except AirflowConfigException:
logging.warning("Celery Executor will run without SSL")
try:
@@ -61,10 +68,10 @@ class CeleryConfig(object):
'certfile': configuration.get('celery',
'CELERY_SSL_CERT'),
'ca_certs': configuration.get('celery',
'CELERY_SSL_CACERT'),
'cert_reqs': ssl.CERT_REQUIRED}
-except AirflowConfigException as e:
+except AirflowConfigException:
raise AirflowException('AirflowConfigException: CELERY_SSL_ACTIVE is
True, please ensure CELERY_SSL_KEY, '
'CELERY_SSL_CERT and CELERY_SSL_CACERT are set')
-except Exception as e:
+except Exception:
raise AirflowException('Exception: There was an unknown Celery SSL
Error. Please ensure you want to use '
'SSL and/or have all necessary certs and key.')
@@ -74,13 +81,13 @@ class CeleryConfig(object):
@app.task
-def execute_command(command):
-logging.info("Executing command in Celery " + command)
+def execute_command(command, key):
+logging.info("[celery] executing command {} for {} ".format(command, key))
try:
subprocess.check_call(command, shell=True)
except subprocess.CalledProcessError as e:
logging.error(e)
-raise AirflowException('Celery command failed')
+raise AirflowException('Celery command failed for {}'.format(key))
class CeleryExecutor(BaseExecutor):
@@ -92,49 +99,60 @@ class CeleryExecutor(BaseExecutor):
vast amounts of messages, while providing operations with the tools
required to maintain such a system.
"""
+def __init__(self, parallelism=PARALLELISM):
+super(CeleryExecutor, self).__init__(parallelism=parallelism)
+self.tasks = ExecutorQueueManager()
def start(self):
-self.tasks = {}
-self.last_state = {}
+self._recover_queue()
def execute_async(self, key, command, queue=DEFAULT_QUEUE):
-self.logger.info( "[celery] queuing {key} through celery, "
- "queue={queue}".format(**locals()))
+self.logger.info("[celery] queuing {key} through celery, "
+ "queue={queue}".format(**locals()))
self.tasks[key] = execute_command.apply_async(
-args=[command], queue=queue)
-self.last_state[key] = celery_states.PENDING
+args=[command, key], queue=queue)
def sync(self):
-
self.logger.debug(
"Inquiring about {} celery task(s)".format(len(self.tasks)))
-for key, async in list(self.tasks.items()):
+for key, uuid in list(self.tasks.items()):
+async = AsyncResult(id=uuid, app=app)
try:
state = async.state
-if
[GitHub] stale[bot] closed pull request #1964: [AIRFLOW-722] Add Celery queue sensor
stale[bot] closed pull request #1964: [AIRFLOW-722] Add Celery queue sensor
URL: https://github.com/apache/airflow/pull/1964
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/sensors/celery_queue_sensor.py
b/airflow/contrib/sensors/celery_queue_sensor.py
new file mode 100644
index 00..3cb7d5dc5a
--- /dev/null
+++ b/airflow/contrib/sensors/celery_queue_sensor.py
@@ -0,0 +1,87 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from airflow.operators.sensors import BaseSensorOperator
+from airflow.utils import apply_defaults
+from celery.task.control import inspect
+
+
+class CeleryQueueSensor(BaseSensorOperator):
+"""
+Waits for a Celery queue to be empty. By default, in order to be
considered
+empty, the queue must not have any tasks in the ``reserved``,
``scheduled``
+or ``active`` states.
+
+:param celery_queue: The name of the Celery queue to wait for.
+:type celery_queue: str
+"""
+@apply_defaults
+def __init__(
+self,
+celery_queue,
+af_task_id=None,
+*args,
+**kwargs):
+"""The constructor."""
+super(CeleryQueueSensor, self).__init__(*args, **kwargs)
+self.celery_queue = celery_queue
+self.af_task_id = af_task_id
+
+def _check_task_id(self, context):
+"""
+Gets the returned Celery result from the Airflow task
+ID provided to the sensor, and returns True if the
+celery result has been finished execution.
+
+:param context: Airflow's execution context
+:type context: dict
+:return: True if task has been executed, otherwise False
+:rtype: bool
+"""
+ti = context['ti']
+celery_result = ti.xcom_pull(task_ids=self.af_task_id)
+return celery_result.ready()
+
+def poke(self, context):
+"""Poke."""
+from celery.task.control import inspect
+
+if self.af_task_id:
+return self._check_task_id(context)
+
+i = inspect()
+reserved = i.reserved()
+scheduled = i.scheduled()
+active = i.active()
+
+try:
+reserved = len(reserved[self.celery_queue])
+scheduled = len(scheduled[self.celery_queue])
+active = len(active[self.celery_queue])
+
+logging.info(
+'Checking if celery queue {0} is empty.'.format(
+self.celery_queue
+)
+)
+
+return reserved == 0 and scheduled == 0 and active == 0
+except KeyError:
+raise KeyError(
+'Could not locate Celery queue {0}'.format(
+self.celery_queue
+)
+)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] stale[bot] closed pull request #2069: [AIRFLOW-865] - Configure FTP connection mode
stale[bot] closed pull request #2069: [AIRFLOW-865] - Configure FTP connection
mode
URL: https://github.com/apache/airflow/pull/2069
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/hooks/ftp_hook.py
b/airflow/contrib/hooks/ftp_hook.py
index 2f2e4c287f..ac10bc9ebb 100644
--- a/airflow/contrib/hooks/ftp_hook.py
+++ b/airflow/contrib/hooks/ftp_hook.py
@@ -58,8 +58,9 @@ class FTPHook(BaseHook):
"""
Interact with FTP.
-Errors that may occur throughout but should be handled
-downstream.
+Errors that may occur throughout but should be handled downstream.
+You can specify mode for data transfers in the extra field of your
+connection as ``{"passive": "true"}``.
"""
def __init__(self, ftp_conn_id='ftp_default'):
@@ -79,7 +80,10 @@ def get_conn(self):
"""
if self.conn is None:
params = self.get_connection(self.ftp_conn_id)
+pasv = params.extra_dejson.get("passive", True)
+
self.conn = ftplib.FTP(params.host, params.login, params.password)
+self.conn.set_pasv(pasv)
return self.conn
@@ -237,7 +241,11 @@ def get_conn(self):
"""
if self.conn is None:
params = self.get_connection(self.ftp_conn_id)
+pasv = params.extra_dejson.get("passive", True)
+
self.conn = ftplib.FTP_TLS(
params.host, params.login, params.password
)
+self.conn.set_pasv(pasv)
+
return self.conn
diff --git a/tests/contrib/hooks/test_ftp_hook.py
b/tests/contrib/hooks/test_ftp_hook.py
index ab6f459aa5..f053406af6 100644
--- a/tests/contrib/hooks/test_ftp_hook.py
+++ b/tests/contrib/hooks/test_ftp_hook.py
@@ -79,5 +79,52 @@ def test_rename(self):
self.conn_mock.quit.assert_called_once_with()
+class TestIntegrationFTPHook(unittest.TestCase):
+
+def setUp(self):
+super(TestIntegrationFTPHook, self).setUp()
+
+from airflow import configuration
+from airflow.utils import db
+from airflow import models
+
+configuration.load_test_config()
+db.merge_conn(
+models.Connection(
+conn_id='ftp_passive', conn_type='ftp',
+host='localhost',
+extra='{"passive": true}'))
+
+db.merge_conn(
+models.Connection(
+conn_id='ftp_active', conn_type='ftp',
+host='localhost',
+extra='{"passive": false}'))
+
+def _test_mode(self, hook_type, connection_id, expected_mode):
+hook = hook_type(connection_id)
+conn = hook.get_conn()
+conn.set_pasv.assert_called_with(expected_mode)
+
+@mock.patch("ftplib.FTP")
+def test_ftp_passive_mode(self, ftp_mock):
+from airflow.contrib.hooks.ftp_hook import FTPHook
+self._test_mode(FTPHook, "ftp_passive", True)
+
+@mock.patch("ftplib.FTP")
+def test_ftp_active_mode(self, ftp_mock):
+from airflow.contrib.hooks.ftp_hook import FTPHook
+self._test_mode(FTPHook, "ftp_active", False)
+
+@mock.patch("ftplib.FTP_TLS")
+def test_ftps_passive_mode(self, ftps_mock):
+from airflow.contrib.hooks.ftp_hook import FTPSHook
+self._test_mode(FTPSHook, "ftp_passive", True)
+
+@mock.patch("ftplib.FTP_TLS")
+def test_ftps_active_mode(self, ftps_mock):
+from airflow.contrib.hooks.ftp_hook import FTPSHook
+self._test_mode(FTPSHook, "ftp_active", False)
+
if __name__ == '__main__':
unittest.main()
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] stale[bot] closed pull request #2799: [AIRFLOW-1836] airflow uses OAuth Provider keycloak
stale[bot] closed pull request #2799: [AIRFLOW-1836] airflow uses OAuth
Provider keycloak
URL: https://github.com/apache/airflow/pull/2799
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/auth/backends/keycloak_auth.py
b/airflow/contrib/auth/backends/keycloak_auth.py
new file mode 100644
index 00..3252021c8a
--- /dev/null
+++ b/airflow/contrib/auth/backends/keycloak_auth.py
@@ -0,0 +1,189 @@
+# Copyright 2015 Matthew Pelland (m...@pelland.io)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import logging
+
+import flask_login
+
+# Need to expose these downstream
+# pylint: disable=unused-import
+from flask_login import (current_user,
+ logout_user,
+ login_required,
+ login_user)
+# pylint: enable=unused-import
+
+from flask import url_for, redirect, request
+
+from flask_oauthlib.client import OAuth
+
+from airflow import models, configuration, settings
+from airflow.configuration import AirflowConfigException
+
+import os
+os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'
+
+_log = logging.getLogger(__name__)
+
+
+def get_config_param(param):
+return str(configuration.get('keycloak', param))
+
+
+class KeyCloakUser(models.User):
+
+def __init__(self, user):
+self.user = user
+
+def is_active(self):
+'''Required by flask_login'''
+return True
+
+def is_authenticated(self):
+'''Required by flask_login'''
+return True
+
+def is_anonymous(self):
+'''Required by flask_login'''
+return False
+
+def get_id(self):
+'''Returns the current user id as required by flask_login'''
+return self.user.get_id()
+
+def data_profiling(self):
+'''Provides access to data profiling tools'''
+return True
+
+def is_superuser(self):
+'''Access all the things'''
+return True
+
+
+class AuthenticationError(Exception):
+pass
+
+
+class GHEAuthBackend(object):
+
+def __init__(self):
+self.ghe_host = get_config_param('host')
+self.login_manager = flask_login.LoginManager()
+self.login_manager.login_view = 'airflow.login'
+self.flask_app = None
+self.ghe_oauth = None
+
+def ghe_api_route(self, leaf):
+
+return '/'.join(['http:/',
+ self.ghe_host,
+ 'auth/realms/brent/protocol/openid-connect'])
+
+def init_app(self, flask_app):
+self.flask_app = flask_app
+
+self.login_manager.init_app(self.flask_app)
+
+self.ghe_oauth = OAuth(self.flask_app).remote_app(
+'ghe',
+consumer_key=get_config_param('client_id'),
+consumer_secret=get_config_param('client_secret'),
+# need read:org to get team member list
+request_token_params={'scope': 'openid'},
+base_url=self.ghe_host,
+request_token_url=None,
+access_token_method='POST',
+access_token_url=''.join(['http://',
+ self.ghe_host,
+
'/auth/realms/brent/protocol/openid-connect/token']),
+authorize_url=''.join(['http://',
+ self.ghe_host,
+
'/auth/realms/brent/protocol/openid-connect/auth']))
+
+self.login_manager.user_loader(self.load_user)
+
+self.flask_app.add_url_rule(get_config_param('oauth_callback_route'),
+'ghe_oauth_callback',
+self.oauth_callback)
+
+def login(self, request):
+_log.debug('Redirecting user to GHE login')
+return self.ghe_oauth.authorize(callback=url_for(
+'ghe_oauth_callback',
+_external=True,
+next=request.args.get('next') or request.referrer or None))
+
+def get_ghe_user_profile_info(self, ghe_token):
+userinfo_url=''.join(['http://',
self.ghe_host,'/auth/realms/brent/protocol/openid-connect/userinfo'])
+resp=self.ghe_oauth.get(userinfo_url,token=(ghe_token,''))
+
+
+if not resp or resp.status != 200:
+raise
[GitHub] stale[bot] closed pull request #3850: [AIRFLOW-3010] added support for date format to postgres_to_gcs_operator
stale[bot] closed pull request #3850: [AIRFLOW-3010] added support for date
format to postgres_to_gcs_operator
URL: https://github.com/apache/airflow/pull/3850
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/operators/postgres_to_gcs_operator.py
b/airflow/contrib/operators/postgres_to_gcs_operator.py
index 88b4d00e39..505ad7e815 100644
--- a/airflow/contrib/operators/postgres_to_gcs_operator.py
+++ b/airflow/contrib/operators/postgres_to_gcs_operator.py
@@ -207,8 +207,10 @@ def convert_types(cls, value):
JSON/Google Cloud Storage/BigQuery. Dates are converted to UTC seconds.
Decimals are converted to floats. Times are converted to seconds.
"""
-if type(value) in (datetime.datetime, datetime.date):
+if type(value) == datetime.datetime:
return time.mktime(value.timetuple())
+elif type(value) == datetime.date:
+return value.strftime('%Y-%m-%d')
elif type(value) == datetime.time:
formated_time = time.strptime(str(value), "%H:%M:%S")
return datetime.timedelta(
@@ -229,8 +231,8 @@ def type_map(cls, postgres_type):
d = {
1114: 'TIMESTAMP',
1184: 'TIMESTAMP',
-1082: 'TIMESTAMP',
1083: 'TIMESTAMP',
+1082: 'DATE',
1005: 'INTEGER',
1007: 'INTEGER',
1016: 'INTEGER',
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] stale[bot] closed pull request #2747: AIRFLOW-1772: Fix bug with handling cron expressions as an schedule i…
stale[bot] closed pull request #2747: AIRFLOW-1772: Fix bug with handling cron expressions as an schedule i… URL: https://github.com/apache/airflow/pull/2747 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/contrib/sensors/gcs_sensor.py b/airflow/contrib/sensors/gcs_sensor.py index 384e26fe04..1106cfdee0 100644 --- a/airflow/contrib/sensors/gcs_sensor.py +++ b/airflow/contrib/sensors/gcs_sensor.py @@ -15,7 +15,6 @@ from airflow.operators.sensors import BaseSensorOperator from airflow.utils.decorators import apply_defaults - class GoogleCloudStorageObjectSensor(BaseSensorOperator): """ Checks for the existence of a file in Google Cloud Storage. @@ -67,8 +66,7 @@ def ts_function(context): behaviour is check for the object being updated after execution_date + schedule_interval. """ -return context['execution_date'] + context['dag'].schedule_interval - +return context['dag'].following_schedule(context['execution_date']) class GoogleCloudStorageObjectUpdatedSensor(BaseSensorOperator): """ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] stale[bot] closed pull request #3605: [AIRFLOW-1238] Decode URL-encoded characters.
stale[bot] closed pull request #3605: [AIRFLOW-1238] Decode URL-encoded characters. URL: https://github.com/apache/airflow/pull/3605 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/models.py b/airflow/models.py index 57ef9d3d33..ea8eeca13f 100755 --- a/airflow/models.py +++ b/airflow/models.py @@ -52,7 +52,7 @@ import uuid from datetime import datetime -from urllib.parse import urlparse, quote, parse_qsl +from urllib.parse import urlparse, quote, parse_qsl, unquote from sqlalchemy import ( Column, Integer, String, DateTime, Text, Boolean, ForeignKey, PickleType, @@ -686,8 +686,8 @@ def parse_from_uri(self, uri): self.conn_type = conn_type self.host = hostname self.schema = temp_uri.path[1:] -self.login = temp_uri.username -self.password = temp_uri.password +self.login = unquote(temp_uri.username or '') +self.password = unquote(temp_uri.password or '') self.port = temp_uri.port if temp_uri.query: self.extra = json.dumps(dict(parse_qsl(temp_uri.query))) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-3663) add requirements.txt file
[ https://issues.apache.org/jira/browse/AIRFLOW-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-3663. - Resolution: Won't Fix > add requirements.txt file > - > > Key: AIRFLOW-3663 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3663 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > We specify a range for setup.py. We should pin deps with requirements.txt > file as a recommendation for development. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #4470: [AIRFLOW-3663] Add requirements.txt file
feng-tao commented on issue #4470: [AIRFLOW-3663] Add requirements.txt file URL: https://github.com/apache/airflow/pull/4470#issuecomment-453275359 thanks @r39132 for the comment. Close the pr for now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao closed pull request #4470: [AIRFLOW-3663] Add requirements.txt file
feng-tao closed pull request #4470: [AIRFLOW-3663] Add requirements.txt file URL: https://github.com/apache/airflow/pull/4470 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/start.rst b/docs/start.rst index e3b16b28f7..465dbabcba 100644 --- a/docs/start.rst +++ b/docs/start.rst @@ -30,6 +30,11 @@ The installation is quick and straightforward. # install from pypi using pip pip install apache-airflow +# (optional) install locally with the master branch +pip install -r requirements.txt # for python2.7 +pip install -r requirements3.txt # for python3.x + + # initialize the database airflow initdb diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 00..066041219f --- /dev/null +++ b/requirements.txt @@ -0,0 +1,95 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# +# This file is autogenerated by pip-compile +# To update, run: +# +#pip-compile --output-file requirements.txt setup.py +# +alembic==0.9.10 +babel==2.6.0 # via flask-babel +bleach==2.1.4 +certifi==2018.11.29 # via requests +chardet==3.0.4# via requests +click==7.0# via flask, flask-appbuilder +colorama==0.4.1 # via flask-appbuilder +configparser==3.5.0 +croniter==0.3.26 +defusedxml==0.5.0 # via python3-openid +dill==0.2.8.2 +docutils==0.14# via python-daemon +enum34==1.1.6 +flask-admin==1.5.2 +flask-appbuilder==1.12.1 +flask-babel==0.12.2 # via flask-appbuilder +flask-caching==1.3.3 +flask-login==0.4.1# via flask-appbuilder +flask-openid==1.2.5 # via flask-appbuilder +flask-sqlalchemy==2.3.2 # via flask-appbuilder +flask-swagger==0.2.13 +flask-wtf==0.14.2 # via flask-appbuilder +flask==0.12.4 # via flask-admin, flask-appbuilder, flask-babel, flask-caching, flask-login, flask-openid, flask-sqlalchemy, flask-swagger, flask-wtf +funcsigs==1.0.0 +future==0.16.0 +gitdb2==2.0.5 # via gitpython +gitpython==2.1.11 +gunicorn==19.9.0 +html5lib==1.0.1 # via bleach +idna==2.8 # via requests +iso8601==0.1.12 +itsdangerous==1.1.0 # via flask +jinja2==2.10 # via flask, flask-babel, python-nvd3 +json-merge-patch==0.2 +lockfile==0.12.2 # via python-daemon +mako==1.0.7 # via alembic +markdown==2.6.11 +markupsafe==1.1.0 # via jinja2, mako +monotonic==1.5# via tenacity +numpy==1.15.4 # via pandas +ordereddict==1.1 # via funcsigs +pandas==0.23.4 +pendulum==1.4.4 +psutil==5.4.8 +pygments==2.3.1 +python-daemon==2.1.2 +python-dateutil==2.7.5# via alembic, croniter, flask-appbuilder, pandas, pendulum +python-editor==1.0.3 # via alembic +python-nvd3==0.15.0 +python-slugify==2.0.1 # via python-nvd3 +python3-openid==3.1.0 # via flask-openid +pytz==2018.9 # via babel, pandas, tzlocal +pytzdata==2018.7 # via pendulum +pyyaml==3.13 # via flask-swagger +requests==2.21.0 +setproctitle==1.1.10 +six==1.12.0 # via bleach, html5lib, python-dateutil, tenacity, thrift +smmap2==2.0.5 # via gitdb2 +sqlalchemy==1.2.15# via alembic, flask-sqlalchemy +tabulate==0.8.2 +tenacity==4.8.0 +thrift==0.11.0 +tzlocal==1.5.1# via pendulum +unicodecsv==0.14.1 +unidecode==1.0.23 # via python-slugify +urllib3==1.24.1 # via requests +webencodings==0.5.1 # via html5lib +werkzeug==0.14.1 # via flask, flask-caching +wtforms==2.2.1# via flask-admin, flask-wtf +zope.deprecation==4.4.0 diff --git a/requirements3.txt b/requirements3.txt new file mode 100644 index 00..066041219f --- /dev/null +++ b/requirements3.txt @@ -0,0 +1,95 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +#
[jira] [Commented] (AIRFLOW-3663) add requirements.txt file
[ https://issues.apache.org/jira/browse/AIRFLOW-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739825#comment-16739825 ] ASF GitHub Bot commented on AIRFLOW-3663: - feng-tao commented on pull request #4470: [AIRFLOW-3663] Add requirements.txt file URL: https://github.com/apache/airflow/pull/4470 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > add requirements.txt file > - > > Key: AIRFLOW-3663 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3663 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > We specify a range for setup.py. We should pin deps with requirements.txt > file as a recommendation for development. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] r39132 commented on issue #4470: [AIRFLOW-3663] Add requirements.txt file
r39132 commented on issue #4470: [AIRFLOW-3663] Add requirements.txt file URL: https://github.com/apache/airflow/pull/4470#issuecomment-453273177 For historical reference, we used to have a requirements.txt file and it was frequently out of sync with the setup.py file, so got rid of it. I'd recommend sticking with one source of truth. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3670) Make Travis Fail quickly on Flake
[
https://issues.apache.org/jira/browse/AIRFLOW-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739814#comment-16739814
]
ASF GitHub Bot commented on AIRFLOW-3670:
-
drewsonne commented on pull request #4477: [AIRFLOW-3670] Add stages to Travis
build
URL: https://github.com/apache/airflow/pull/4477
Allow travis to fail the entire build quickly, if the flake8 build fails
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-3670
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This makes no changes to the application code, and is designed to improve
the development ccle by reducing the cycle time from test start to notification
that test has failed
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
No tests as this is a config for the test running. An example of the fast
fail can be seen here https://travis-ci.org/drewsonne/airflow/builds/478054124
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Make Travis Fail quickly on Flake
> -
>
> Key: AIRFLOW-3670
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3670
> Project: Apache Airflow
> Issue Type: Improvement
> Components: travis
>Affects Versions: 1.10.1
>Reporter: Drew Sonne
>Priority: Minor
> Fix For: 1.10.2
>
>
> Add stages to the travis builds to split flake8 and functional/unit tests, so
> that if flake8 tests fail, the rest of the build will fail quickly.
> This will reduce turnaround time for bad linting notifications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] drewsonne edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
drewsonne edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-453268397 Argh, stupid mistake. Here's a PR to try to shorten the turnaround time for travis alerting on lint failures https://github.com/apache/airflow/pull/4477/files This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] drewsonne commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
drewsonne commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-453268397 Argh, stupid mistake and here's a PR to try to shorten the turnaround time for travis alerting on lint failures https://github.com/apache/airflow/pull/4477/files This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] drewsonne opened a new pull request #4477: [AIRFLOW-3670] Add stages to Travis build
drewsonne opened a new pull request #4477: [AIRFLOW-3670] Add stages to Travis
build
URL: https://github.com/apache/airflow/pull/4477
Allow travis to fail the entire build quickly, if the flake8 build fails
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-3670
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This makes no changes to the application code, and is designed to improve
the development ccle by reducing the cycle time from test start to notification
that test has failed
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
No tests as this is a config for the test running. An example of the fast
fail can be seen here https://travis-ci.org/drewsonne/airflow/builds/478054124
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3670) Make Travis Fail quickly on Flake
Drew Sonne created AIRFLOW-3670: --- Summary: Make Travis Fail quickly on Flake Key: AIRFLOW-3670 URL: https://issues.apache.org/jira/browse/AIRFLOW-3670 Project: Apache Airflow Issue Type: Improvement Components: travis Affects Versions: 1.10.1 Reporter: Drew Sonne Fix For: 1.10.2 Add stages to the travis builds to split flake8 and functional/unit tests, so that if flake8 tests fail, the rest of the build will fail quickly. This will reduce turnaround time for bad linting notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
feng-tao commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-453262065 @drewsonne , FYI, two flake8 errors: ```flake8 runtests: commands[0] | flake8 ./tests/plugins/test_plugins_manager_www.py:33:1: F401 'airflow.plugins_manager.AirflowPlugin' imported but unused ./tests/plugins/test_plugins_manager_www.py:90:1: E302 expected 2 blank lines, found 1``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
feng-tao edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-453262065 @drewsonne , FYI, two flake8 errors: ```flake8 runtests: commands[0] | flake8 ./tests/plugins/test_plugins_manager_www.py:33:1: F401 'airflow.plugins_manager.AirflowPlugin' imported but unused ./tests/plugins/test_plugins_manager_www.py:90:1: E302 expected 2 blank lines, found 1 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] drewsonne commented on a change in pull request #4412: [AIRFLOW-3605] Load plugins from entry_points
drewsonne commented on a change in pull request #4412: [AIRFLOW-3605] Load
plugins from entry_points
URL: https://github.com/apache/airflow/pull/4412#discussion_r246912963
##
File path: tests/plugins/test_plugins_manager_www.py
##
@@ -83,3 +85,66 @@ def test_menu_links(self):
[menu_link] = [ml for ml in category.get_children()
if isinstance(ml, MenuLink)]
self.assertEqual('Test Menu Link', menu_link.name)
+
+
+class MockPluginA(AirflowPlugin):
Review comment:
Done, I just have to wait for the tests to run again
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] drewsonne commented on a change in pull request #4412: [AIRFLOW-3605] Load plugins from entry_points
drewsonne commented on a change in pull request #4412: [AIRFLOW-3605] Load
plugins from entry_points
URL: https://github.com/apache/airflow/pull/4412#discussion_r246912582
##
File path: tests/plugins/test_plugins_manager_www.py
##
@@ -83,3 +85,66 @@ def test_menu_links(self):
[menu_link] = [ml for ml in category.get_children()
if isinstance(ml, MenuLink)]
self.assertEqual('Test Menu Link', menu_link.name)
+
+
+class MockPluginA(AirflowPlugin):
+name = 'plugin-a'
+
+
+class MockPluginB(AirflowPlugin):
+name = 'plugin-b'
+
+
+class MockPluginC(AirflowPlugin):
+name = 'plugin-c'
+
+
+class PluginsTestEntrypointLoad(unittest.TestCase):
Review comment:
I'm not sure, as I haven't touched on the UI parts of the code base yet.
This code is not really anything to do with the UI as far as I'm aware. Or
at least, it's not specific to one UI or the other. When I pulled the above PR
changes in, this is where my code was move to, so the old
test/plugin_manager.py file was renamed to this.
tl;dr it's not UI specific(?), so I'm not sure where it goes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] drewsonne commented on a change in pull request #4412: [AIRFLOW-3605] Load plugins from entry_points
drewsonne commented on a change in pull request #4412: [AIRFLOW-3605] Load
plugins from entry_points
URL: https://github.com/apache/airflow/pull/4412#discussion_r246912582
##
File path: tests/plugins/test_plugins_manager_www.py
##
@@ -83,3 +85,66 @@ def test_menu_links(self):
[menu_link] = [ml for ml in category.get_children()
if isinstance(ml, MenuLink)]
self.assertEqual('Test Menu Link', menu_link.name)
+
+
+class MockPluginA(AirflowPlugin):
+name = 'plugin-a'
+
+
+class MockPluginB(AirflowPlugin):
+name = 'plugin-b'
+
+
+class MockPluginC(AirflowPlugin):
+name = 'plugin-c'
+
+
+class PluginsTestEntrypointLoad(unittest.TestCase):
Review comment:
I'm not sure, as I haven't touched on the UI parts of the code base yet.
This code is not really anything to do with the UI as far as I'm aware. Or
at least, it's not specific to one UI or the other. When I pulled the above PR
changes in, this is where my code was move to, so the old
test/plugin_manager.py file was renamed to this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] feng-tao commented on a change in pull request #4412: [AIRFLOW-3605] Load plugins from entry_points
feng-tao commented on a change in pull request #4412: [AIRFLOW-3605] Load
plugins from entry_points
URL: https://github.com/apache/airflow/pull/4412#discussion_r246911641
##
File path: tests/plugins/test_plugins_manager_www.py
##
@@ -83,3 +85,66 @@ def test_menu_links(self):
[menu_link] = [ml for ml in category.get_children()
if isinstance(ml, MenuLink)]
self.assertEqual('Test Menu Link', menu_link.name)
+
+
+class MockPluginA(AirflowPlugin):
+name = 'plugin-a'
+
+
+class MockPluginB(AirflowPlugin):
+name = 'plugin-b'
+
+
+class MockPluginC(AirflowPlugin):
+name = 'plugin-c'
+
+
+class PluginsTestEntrypointLoad(unittest.TestCase):
Review comment:
are these two unit test only worked for old UI? If not I would suggest to
move to test_plugins_manager_rbac.py once we delete the old UI code, we will
remove this unit test file as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] feng-tao commented on a change in pull request #4412: [AIRFLOW-3605] Load plugins from entry_points
feng-tao commented on a change in pull request #4412: [AIRFLOW-3605] Load
plugins from entry_points
URL: https://github.com/apache/airflow/pull/4412#discussion_r246911327
##
File path: tests/plugins/test_plugins_manager_www.py
##
@@ -83,3 +85,66 @@ def test_menu_links(self):
[menu_link] = [ml for ml in category.get_children()
if isinstance(ml, MenuLink)]
self.assertEqual('Test Menu Link', menu_link.name)
+
+
+class MockPluginA(AirflowPlugin):
Review comment:
sorry, but could you move the MockPlugin to test_plugin file which is a
centralized place for all the test plugins?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3249) unify do_xcom_push for all operators
[
https://issues.apache.org/jira/browse/AIRFLOW-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739724#comment-16739724
]
ASF GitHub Bot commented on AIRFLOW-3249:
-
marengaz commented on pull request #4345: [AIRFLOW-3249] unify (and fix)
pushing result to xcom
URL: https://github.com/apache/airflow/pull/4345
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> unify do_xcom_push for all operators
>
>
> Key: AIRFLOW-3249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3249
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Ben Marengo
>Assignee: Ben Marengo
>Priority: Major
>
> following the implementation of AIRFLOW-3207 (global option to stop task
> pushing result to xcom), i did a quick search around to find out which
> operators have a custom implementation of this {{do_xcom_push}} flag:
> ||operator||instance var||__init__ arg||will change be backward comp?||
> |DatabricksRunNowOperator|do_xcom_push | do_xcom_push|(/)|
> |DatabricksSubmitRunOperator|do_xcom_push| do_xcom_push|(/)|
> |DatastoreExportOperator|xcom_push| xcom_push|(x)|
> |DatastoreImportOperator|xcom_push| xcom_push|(x)|
> |KubernetesPodOperator|xcom_push|xcom_push |(x)|
> |SSHOperator|xcom_push|xcom_push |(x)|
> |WinRMOperator|xcom_push| xcom_push|(x)|
> |BashOperator|xcom_push_flag|xcom_push|(x)|
> |DockerOperator|xcom_push_flag|xcom_push|(x)|
> |SimpleHttpOperator|xcom_push_flag|xcom_push|(x)|
> this custom implementation should be removed.
> i presume also that the operators with instance var = xcom_push conflict with
> method BaseOperator.xcom_push() and probably aren't working properly anyway!?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3249) unify do_xcom_push for all operators
[
https://issues.apache.org/jira/browse/AIRFLOW-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739725#comment-16739725
]
ASF GitHub Bot commented on AIRFLOW-3249:
-
marengaz commented on pull request #4345: [AIRFLOW-3249] unify (and fix)
pushing result to xcom
URL: https://github.com/apache/airflow/pull/4345
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://jira.apache.org/jira/browse/AIRFLOW-3249) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://jira.apache.org/jira/browse/AIRFLOW-3249
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
see https://jira.apache.org/jira/browse/AIRFLOW-3249
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> unify do_xcom_push for all operators
>
>
> Key: AIRFLOW-3249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3249
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Ben Marengo
>Assignee: Ben Marengo
>Priority: Major
>
> following the implementation of AIRFLOW-3207 (global option to stop task
> pushing result to xcom), i did a quick search around to find out which
> operators have a custom implementation of this {{do_xcom_push}} flag:
> ||operator||instance var||__init__ arg||will change be backward comp?||
> |DatabricksRunNowOperator|do_xcom_push | do_xcom_push|(/)|
> |DatabricksSubmitRunOperator|do_xcom_push| do_xcom_push|(/)|
> |DatastoreExportOperator|xcom_push| xcom_push|(x)|
> |DatastoreImportOperator|xcom_push| xcom_push|(x)|
> |KubernetesPodOperator|xcom_push|xcom_push |(x)|
> |SSHOperator|xcom_push|xcom_push |(x)|
> |WinRMOperator|xcom_push| xcom_push|(x)|
> |BashOperator|xcom_push_flag|xcom_push|(x)|
> |DockerOperator|xcom_push_flag|xcom_push|(x)|
> |SimpleHttpOperator|xcom_push_flag|xcom_push|(x)|
> this custom implementation should be removed.
> i presume also that the operators with instance var = xcom_push conflict with
> method BaseOperator.xcom_push() and probably aren't working properly anyway!?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] marengaz closed pull request #4345: [AIRFLOW-3249] unify (and fix) pushing result to xcom
marengaz closed pull request #4345: [AIRFLOW-3249] unify (and fix) pushing
result to xcom
URL: https://github.com/apache/airflow/pull/4345
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/UPDATING.md b/UPDATING.md
index 7114dc99c7..9a1ff532b5 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -117,6 +117,12 @@ session.add(admin)
session.commit()
```
+### Unification of `do_xcom_push` flag
+The `do_xcom_push` flag (a switch to push the result of an operator to xcom or
not) was appearing in different incarnations in different operators. It's
function has been unified under a common name (`do_xcom_push`) on
`BaseOperator`. This way it is also easy to globally disable pushing results to
xcom.
+
+See [AIRFLOW-3249](https://jira.apache.org/jira/browse/AIRFLOW-3249) to check
if your operator was affected.
+
+
## Airflow 1.10.1
### StatsD Metrics
diff --git a/airflow/contrib/operators/datastore_export_operator.py
b/airflow/contrib/operators/datastore_export_operator.py
index 9d95eadc74..2e6da4ada6 100644
--- a/airflow/contrib/operators/datastore_export_operator.py
+++ b/airflow/contrib/operators/datastore_export_operator.py
@@ -54,8 +54,6 @@ class DatastoreExportOperator(BaseOperator):
:param overwrite_existing: if the storage bucket + namespace is not empty,
it will be
emptied prior to exports. This enables overwriting existing backups.
:type overwrite_existing: bool
-:param xcom_push: push operation name to xcom for reference
-:type xcom_push: bool
"""
@apply_defaults
@@ -69,7 +67,6 @@ def __init__(self,
labels=None,
polling_interval_in_seconds=10,
overwrite_existing=False,
- xcom_push=False,
*args,
**kwargs):
super(DatastoreExportOperator, self).__init__(*args, **kwargs)
@@ -82,7 +79,8 @@ def __init__(self,
self.labels = labels
self.polling_interval_in_seconds = polling_interval_in_seconds
self.overwrite_existing = overwrite_existing
-self.xcom_push = xcom_push
+if kwargs.get('xcom_push') is not None:
+raise AirflowException("'xcom_push' was deprecated, use
'BaseOperator.do_xcom_push' instead")
def execute(self, context):
self.log.info('Exporting data to Cloud Storage bucket ' + self.bucket)
@@ -106,5 +104,4 @@ def execute(self, context):
if state != 'SUCCESSFUL':
raise AirflowException('Operation failed:
result={}'.format(result))
-if self.xcom_push:
-return result
+return result
diff --git a/airflow/contrib/operators/datastore_import_operator.py
b/airflow/contrib/operators/datastore_import_operator.py
index c79767f35e..b7897cf051 100644
--- a/airflow/contrib/operators/datastore_import_operator.py
+++ b/airflow/contrib/operators/datastore_import_operator.py
@@ -50,8 +50,6 @@ class DatastoreImportOperator(BaseOperator):
:param polling_interval_in_seconds: number of seconds to wait before
polling for
execution status again
:type polling_interval_in_seconds: int
-:param xcom_push: push operation name to xcom for reference
-:type xcom_push: bool
"""
@apply_defaults
@@ -64,7 +62,6 @@ def __init__(self,
datastore_conn_id='google_cloud_default',
delegate_to=None,
polling_interval_in_seconds=10,
- xcom_push=False,
*args,
**kwargs):
super(DatastoreImportOperator, self).__init__(*args, **kwargs)
@@ -76,7 +73,8 @@ def __init__(self,
self.entity_filter = entity_filter
self.labels = labels
self.polling_interval_in_seconds = polling_interval_in_seconds
-self.xcom_push = xcom_push
+if kwargs.get('xcom_push') is not None:
+raise AirflowException("'xcom_push' was deprecated, use
'BaseOperator.do_xcom_push' instead")
def execute(self, context):
self.log.info('Importing data from Cloud Storage bucket %s',
self.bucket)
@@ -94,5 +92,4 @@ def execute(self, context):
if state != 'SUCCESSFUL':
raise AirflowException('Operation failed:
result={}'.format(result))
-if self.xcom_push:
-return result
+return result
diff --git a/airflow/contrib/operators/kubernetes_pod_operator.py
b/airflow/contrib/operators/kubernetes_pod_operator.py
index 3a5bef5381..1635ae4d07 100644
--- a/airflow/contrib/operators/kubernetes_pod_operator.py
+++ b/airflow/contrib/operators/kubernetes_pod_operator.py
@@ -80,14 +80,14 @@ class KubernetesPodOperator(BaseOperator):
:type node_selectors: dict
:param config_file: The path to the Kubernetes config
[GitHub] marengaz opened a new pull request #4345: [AIRFLOW-3249] unify (and fix) pushing result to xcom
marengaz opened a new pull request #4345: [AIRFLOW-3249] unify (and fix)
pushing result to xcom
URL: https://github.com/apache/airflow/pull/4345
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://jira.apache.org/jira/browse/AIRFLOW-3249) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://jira.apache.org/jira/browse/AIRFLOW-3249
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
see https://jira.apache.org/jira/browse/AIRFLOW-3249
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] drewsonne commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
drewsonne commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-453224881 Ok, this looks good from my side now. The codecov says it's reducing coverage, but in files this PR doesn't touch, so I'm not sure why :-/ Anything else or is this good to go? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jj-ian commented on a change in pull request #4083: [AIRFLOW-3211] Reattach to GCP Dataproc jobs upon Airflow restart
jj-ian commented on a change in pull request #4083: [AIRFLOW-3211] Reattach to
GCP Dataproc jobs upon Airflow restart
URL: https://github.com/apache/airflow/pull/4083#discussion_r246891824
##
File path: airflow/contrib/hooks/gcp_dataproc_hook.py
##
@@ -33,12 +33,82 @@ def __init__(self, dataproc_api, project_id, job,
region='global',
self.dataproc_api = dataproc_api
self.project_id = project_id
self.region = region
+
+# Check if the job to submit is already running on the cluster.
+# If so, don't resubmit the job.
+try:
+cluster_name = job['job']['placement']['clusterName']
+except KeyError:
+self.log.error('Job to submit is incorrectly configured.')
+raise
+
+jobs_on_cluster_response =
dataproc_api.projects().regions().jobs().list(
+projectId=self.project_id,
+region=self.region,
+clusterName=cluster_name).execute()
+
+UUID_LENGTH = 9
Review comment:
I'm going to go ahead with implementing the solution I proposed above
@fenglu-g; if you have any reservations please let me know.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3669) Add HashiCorp Vault hook
Will Eaton created AIRFLOW-3669:
---
Summary: Add HashiCorp Vault hook
Key: AIRFLOW-3669
URL: https://issues.apache.org/jira/browse/AIRFLOW-3669
Project: Apache Airflow
Issue Type: Improvement
Components: hooks
Reporter: Will Eaton
Assignee: Will Eaton
I'd like to propose a new {{contrib}} hook for interacting with HashiCorp's
Vault.
*Motivation:* Useful for teams who need access to secrets during the run-time
of an operator that can't easily be stored as a default connection in the UI,
eg. a Python operator getting a temporary token to access a web service.
Airflow jobs can also be used to automate credential management, such as using
database triggers to revoke a particular token.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] feluelle edited a comment on issue #4476: [AIRFLOW-3552] Add ImapToS3TransferOperator
feluelle edited a comment on issue #4476: [AIRFLOW-3552] Add ImapToS3TransferOperator URL: https://github.com/apache/airflow/pull/4476#issuecomment-453213757 PTAL: @feng-tao @kaxil @ashb Especially because of this: > NOTE: This operator only transfers the latest attachment by name. I am thinking of changing this so that you can transfer multiple attachments that for example match the regular expression you pass in but I don't know if this is really a good idea and if so how would be the best way to implement this. Something like adding `imap_latest_only` (set to False) + passing `s3_bucket` so that the attachments just keep its names looks very strange to me. :/ ..but I don't know any other way of doing this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feluelle commented on issue #4476: [AIRFLOW-3552] Add ImapToS3TransferOperator
feluelle commented on issue #4476: [AIRFLOW-3552] Add ImapToS3TransferOperator URL: https://github.com/apache/airflow/pull/4476#issuecomment-453213757 PTAL: @feng-tao @kaxil @ashb Especially because of this: > NOTE: This operator only transfers the latest attachment by name. I am thinking of changing this so that you can transfer multiple attachments that for example match the regular expression you pass in but I don't know if this is really a good idea and if so how would be the best way to implement this. Something like adding `imap_latest_only` (set to False) + passing `s3_bucket` and then just naming the attachments like the list you could pass in to `s3_key` looks very strange to me. :/ ..but I don't know any other way of doing this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3552) Add ImapToS3TransferOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739676#comment-16739676
]
ASF GitHub Bot commented on AIRFLOW-3552:
-
feluelle commented on pull request #4476: [AIRFLOW-3552] Add
ImapToS3TransferOperator
URL: https://github.com/apache/airflow/pull/4476
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3552
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds an operator that transfers a mail attachment into an s3 bucket.
It also:
- adds tests
**NOTE:** This operator only transfers the latest attachment by name.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add ImapToS3TransferOperator
>
>
> Key: AIRFLOW-3552
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3552
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
> This operator transfers mail attachments from a mail server to an amazon s3
> bucket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] feluelle opened a new pull request #4476: [AIRFLOW-3552] Add ImapToS3TransferOperator
feluelle opened a new pull request #4476: [AIRFLOW-3552] Add
ImapToS3TransferOperator
URL: https://github.com/apache/airflow/pull/4476
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3552
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds an operator that transfers a mail attachment into an s3 bucket.
It also:
- adds tests
**NOTE:** This operator only transfers the latest attachment by name.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3602) Change ImapHook to raise an Exception when no attachments have been found
[
https://issues.apache.org/jira/browse/AIRFLOW-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739647#comment-16739647
]
ASF GitHub Bot commented on AIRFLOW-3602:
-
feluelle commented on pull request #4475: [AIRFLOW-3602] Changes ImapHook to
raise an exception on no attachments
URL: https://github.com/apache/airflow/pull/4475
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR changes the ImapHook so it will fail when no attachments have been
found.
It also:
- fixes escaping chars and symlink tests not testing correctly
- adds note for handling symlinks
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Change ImapHook to raise an Exception when no attachments have been found
> -
>
> Key: AIRFLOW-3602
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3602
> Project: Apache Airflow
> Issue Type: Improvement
>Affects Versions: 1.10.1
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
> At the moment the
> {code:python}
> def retrieve_mail_attachments(self, name, mail_folder='INBOX',
> check_regex=False,
> latest_only=False):
> {code}
> returns an empty list if no attachments have been found.
> And the
> {code:python}
> def download_mail_attachments(self, name, local_output_directory,
> mail_folder='INBOX',
> check_regex=False, latest_only=False):
> {code}
> also does not log anything if no attachment has been found.
> Is this useful? Or would it be better to raise an exception instead?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
codecov-io edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-450662547 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4412?src=pr=h1) Report > Merging [#4412](https://codecov.io/gh/apache/airflow/pull/4412?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/24bc74bc5f45455aee21f6b506d282ee052efa19?src=pr=desc) will **decrease** coverage by `0.19%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4412?src=pr=tree) ```diff @@Coverage Diff@@ ## master#4412 +/- ## = - Coverage 79.47% 79.28% -0.2% = Files 204 204 Lines 1653416550 +16 = - Hits1314013121 -19 - Misses 3394 3429 +35 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4412?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `97.14% <100%> (+5%)` | :arrow_up: | | [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `32.5% <0%> (-60%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `79.83% <0%> (-3.23%)` | :arrow_down: | | [airflow/task/task\_runner/base\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL2Jhc2VfdGFza19ydW5uZXIucHk=) | `77.58% <0%> (-1.73%)` | :arrow_down: | | [airflow/operators/docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZG9ja2VyX29wZXJhdG9yLnB5) | `96.51% <0%> (-1.17%)` | :arrow_down: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `96.07% <0%> (-0.99%)` | :arrow_down: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.52% <0%> (-0.25%)` | :arrow_down: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `75% <0%> (-0.22%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `70.29% <0%> (-0.13%)` | :arrow_down: | | [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `92.55% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4412?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4412?src=pr=footer). Last update [24bc74b...2ca1c60](https://codecov.io/gh/apache/airflow/pull/4412?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-3657) version 2.7.0 and above of zdesk breaks airflow - setup.py should specify zdesk pypy version
[ https://issues.apache.org/jira/browse/AIRFLOW-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-3657. - Resolution: Fixed > version 2.7.0 and above of zdesk breaks airflow - setup.py should specify > zdesk pypy version > > > Key: AIRFLOW-3657 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3657 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.1 >Reporter: Tobias Kaymak >Assignee: Tobias Kaymak >Priority: Major > > Currently, the zdesk package version is undefined, resulting in the newest > available being installed when selecting zendesk during setup. > The init method of the package has changed between version 2.6.0 and 2.7.0, > therefore airflow will not work with the newer version, as it does use > unnamed variable arguments. > Quickfix is to bind the zdesk package version to 2.6.0 in setup.py - better > is to use named arguments in the get_conn() method in the zendesk_hook. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3657) version 2.7.0 and above of zdesk breaks airflow - setup.py should specify zdesk pypy version
[ https://issues.apache.org/jira/browse/AIRFLOW-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739648#comment-16739648 ] ASF GitHub Bot commented on AIRFLOW-3657: - feng-tao commented on pull request #4466: [AIRFLOW-3657] Fix zendesk integration URL: https://github.com/apache/airflow/pull/4466 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > version 2.7.0 and above of zdesk breaks airflow - setup.py should specify > zdesk pypy version > > > Key: AIRFLOW-3657 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3657 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.1 >Reporter: Tobias Kaymak >Assignee: Tobias Kaymak >Priority: Major > > Currently, the zdesk package version is undefined, resulting in the newest > available being installed when selecting zendesk during setup. > The init method of the package has changed between version 2.6.0 and 2.7.0, > therefore airflow will not work with the newer version, as it does use > unnamed variable arguments. > Quickfix is to bind the zdesk package version to 2.6.0 in setup.py - better > is to use named arguments in the get_conn() method in the zendesk_hook. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao closed pull request #4466: [AIRFLOW-3657] Fix zendesk integration
feng-tao closed pull request #4466: [AIRFLOW-3657] Fix zendesk integration
URL: https://github.com/apache/airflow/pull/4466
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/hooks/zendesk_hook.py b/airflow/hooks/zendesk_hook.py
index 3cf8353344..a6f7583450 100644
--- a/airflow/hooks/zendesk_hook.py
+++ b/airflow/hooks/zendesk_hook.py
@@ -33,7 +33,8 @@ def __init__(self, zendesk_conn_id):
def get_conn(self):
conn = self.get_connection(self.__zendesk_conn_id)
self.__url = "https://; + conn.host
-return Zendesk(self.__url, conn.login, conn.password, True)
+return Zendesk(zdesk_url=self.__url, zdesk_email=conn.login,
zdesk_password=conn.password,
+ zdesk_token=True)
def __handle_rate_limit_exception(self, rate_limit_exception):
"""
diff --git a/tests/contrib/hooks/test_zendesk_hook.py
b/tests/contrib/hooks/test_zendesk_hook.py
index 99dd8926dc..51be042086 100644
--- a/tests/contrib/hooks/test_zendesk_hook.py
+++ b/tests/contrib/hooks/test_zendesk_hook.py
@@ -92,8 +92,8 @@ def test_zdesk_is_inited_correctly(self, mock_zendesk):
zendesk_hook = ZendeskHook("conn_id")
zendesk_hook.get_connection = mock.Mock(return_value=conn_mock)
zendesk_hook.get_conn()
-mock_zendesk.assert_called_with('https://conn_host', 'conn_login',
-'conn_pass', True)
+mock_zendesk.assert_called_with(zdesk_url='https://conn_host',
zdesk_email='conn_login',
+zdesk_password='conn_pass',
zdesk_token=True)
@mock.patch("airflow.hooks.zendesk_hook.Zendesk")
def test_zdesk_sideloading_works_correctly(self, mock_zendesk):
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] feng-tao commented on issue #4466: [AIRFLOW-3657] Fix zendesk integration
feng-tao commented on issue #4466: [AIRFLOW-3657] Fix zendesk integration URL: https://github.com/apache/airflow/pull/4466#issuecomment-453192079 , thanks @tkaymak This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feluelle opened a new pull request #4475: [AIRFLOW-3602] Changes ImapHook to raise an exception on no attachments
feluelle opened a new pull request #4475: [AIRFLOW-3602] Changes ImapHook to
raise an exception on no attachments
URL: https://github.com/apache/airflow/pull/4475
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR changes the ImapHook so it will fail when no attachments have been
found.
It also:
- fixes escaping chars and symlink tests not testing correctly
- adds note for handling symlinks
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] feng-tao commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
feng-tao commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points URL: https://github.com/apache/airflow/pull/4412#issuecomment-453181676 I restart the fail jenkins CI. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-3602) Change ImapHook to raise an Exception when no attachments have been found
[
https://issues.apache.org/jira/browse/AIRFLOW-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall reassigned AIRFLOW-3602:
-
Assignee: Felix Uellendall
> Change ImapHook to raise an Exception when no attachments have been found
> -
>
> Key: AIRFLOW-3602
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3602
> Project: Apache Airflow
> Issue Type: Improvement
>Affects Versions: 1.10.1
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
> At the moment the
> {code:python}
> def retrieve_mail_attachments(self, name, mail_folder='INBOX',
> check_regex=False,
> latest_only=False):
> {code}
> returns an empty list if no attachments have been found.
> And the
> {code:python}
> def download_mail_attachments(self, name, local_output_directory,
> mail_folder='INBOX',
> check_regex=False, latest_only=False):
> {code}
> also does not log anything if no attachment has been found.
> Is this useful? Or would it be better to raise an exception instead?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] feng-tao commented on issue #4163: [AIRFLOW-3319] - KubernetsExecutor: Need in try_number in labels if getting them later
feng-tao commented on issue #4163: [AIRFLOW-3319] - KubernetsExecutor: Need in try_number in labels if getting them later URL: https://github.com/apache/airflow/pull/4163#issuecomment-453169768 PTAL @dimberman This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #4268: [AIRFLOW-3412] Fix kubernetes executor to delete pods after termination
feng-tao commented on issue #4268: [AIRFLOW-3412] Fix kubernetes executor to delete pods after termination URL: https://github.com/apache/airflow/pull/4268#issuecomment-453169731 PTAL @dimberman This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #4470: [AIRFLOW-3663] Add requirements.txt file
feng-tao commented on issue #4470: [AIRFLOW-3663] Add requirements.txt file URL: https://github.com/apache/airflow/pull/4470#issuecomment-453168748 @mistercrunch Max, would like to get your take as well. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ckljohn commented on issue #4456: [AIRFLOW-3645] Add base_executor_config
ckljohn commented on issue #4456: [AIRFLOW-3645] Add base_executor_config URL: https://github.com/apache/airflow/pull/4456#issuecomment-453145610 @Mokubyow I have given you a PR https://github.com/Mokubyow/airflow/pull/1/files This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on a change in pull request #4474: [AIRFLOW-3439] Decode logs with 'utf-8'
XD-DENG commented on a change in pull request #4474: [AIRFLOW-3439] Decode logs
with 'utf-8'
URL: https://github.com/apache/airflow/pull/4474#discussion_r246807120
##
File path: airflow/utils/log/gcs_task_handler.py
##
@@ -129,7 +129,7 @@ def gcs_read(self, remote_log_location):
:type remote_log_location: str (path)
"""
bkt, blob = self.parse_gcs_url(remote_log_location)
-return self.hook.download(bkt, blob).decode()
+return self.hook.download(bkt, blob).decode('utf-8')
Review comment:
The default encoding of `.decode()` is already "utf-8". Please refer to
https://docs.python.org/3/library/stdtypes.html#bytes.decode
So `.decode('utf-8')` is no difference from `.decode()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] XD-DENG removed a comment on issue #4474: [AIRFLOW-3439] Decode logs with 'utf-8'
XD-DENG removed a comment on issue #4474: [AIRFLOW-3439] Decode logs with
'utf-8'
URL: https://github.com/apache/airflow/pull/4474#issuecomment-453138962
The default encoding of `.decode()` is already "utf-8". Please refer to
https://docs.python.org/3/library/stdtypes.html#bytes.decode
So `.decode('utf-8')` is no difference from `.decode()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Mokubyow commented on issue #4456: [AIRFLOW-3645] Add base_executor_config
Mokubyow commented on issue #4456: [AIRFLOW-3645] Add base_executor_config URL: https://github.com/apache/airflow/pull/4456#issuecomment-453137477 @ckljohn I don't know that it matters either way. I chose a dict because the operator level executor_config is already a dict so merging is easy that way. What are your thoughts? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mik-laj commented on issue #4472: [AIRFLOW-3664] Fix interpreter errors in test_python_operator.py
mik-laj commented on issue #4472: [AIRFLOW-3664] Fix interpreter errors in test_python_operator.py URL: https://github.com/apache/airflow/pull/4472#issuecomment-453127892 Documenation says: > If no expressions are present, raise re-raises the last exception that was active in the current scope. If no exception is active in the current scope, a RuntimeError exception is raised indicating that this is an error. https://docs.python.org/3.7/reference/simple_stmts.html#the-raise-statement This should not cause problems. Can you provide more information that allows me to reproduce the error? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mik-laj edited a comment on issue #4472: [AIRFLOW-3664] Fix interpreter errors in test_python_operator.py
mik-laj edited a comment on issue #4472: [AIRFLOW-3664] Fix interpreter errors in test_python_operator.py URL: https://github.com/apache/airflow/pull/4472#issuecomment-453127892 Documenation says: > If no expressions are present, raise re-raises the last exception that was active in the current scope. If no exception is active in the current scope, a RuntimeError exception is raised indicating that this is an error. https://docs.python.org/3.7/reference/simple_stmts.html#the-raise-statement This should not cause problems. Can you provide more information that allows me to reproduce the error? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3439) Logs with non-ascii characters can't be read from GCS
[
https://issues.apache.org/jira/browse/AIRFLOW-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739452#comment-16739452
]
ASF GitHub Bot commented on AIRFLOW-3439:
-
RasPavel commented on pull request #4474: [AIRFLOW-3439] Decode logs with
'utf-8'
URL: https://github.com/apache/airflow/pull/4474
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Logs with non-ascii characters can't be read from GCS
> -
>
> Key: AIRFLOW-3439
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3439
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging, webserver
>Affects Versions: 1.10.1
>Reporter: Pavel Raschetnov
>Priority: Major
> Labels: UTF-8, encoding, gcs_task_handler
>
> Can't see task logs in web interface due to
> {{*** Unable to read remote log from
> gs://bucket/dag/abcd61826/2018-11-23T00:00:00+00:00/1.log *** 'ascii' codec
> can't decode byte 0xc3 in position 4421445: ordinal not in range(128)}}
> GCSTaskHandler should use `.decode('utf-8') instead of `.decode` in
> gcs_read() method
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] RasPavel opened a new pull request #4474: [AIRFLOW-3439] Decode logs with 'utf-8'
RasPavel opened a new pull request #4474: [AIRFLOW-3439] Decode logs with
'utf-8'
URL: https://github.com/apache/airflow/pull/4474
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] drewsonne commented on issue #4412: [AIRFLOW-3605] Load plugins from entry_points
drewsonne commented on issue #4412: [AIRFLOW-3605] Load plugins from
entry_points
URL: https://github.com/apache/airflow/pull/4412#issuecomment-453110777
Uh... it's failing on writing a log
https://travis-ci.org/apache/airflow/jobs/477803046#L4897
```==
37) ERROR: test_failed_load_entrpoint_plugins
(tests.plugins.test_plugins_manager_www.PluginsTestEntrypointLoad)
--
Traceback (most recent call last):
tests/plugins/test_plugins_manager_www.py line 128 in
test_failed_load_entrpoint_plugins
)], []),
airflow/plugins_manager.py line 88 in load_entrypoint_plugins
log.debug('Importing entry_point plugin %s', entry_point.name)
/usr/lib/python2.7/logging/__init__.py line 1155 in debug
self._log(DEBUG, msg, args, **kwargs)
/usr/lib/python2.7/logging/__init__.py line 1286 in _log
self.handle(record)
/usr/lib/python2.7/logging/__init__.py line 1296 in handle
self.callHandlers(record)
/usr/lib/python2.7/logging/__init__.py line 1336 in callHandlers
hdlr.handle(record)
/usr/lib/python2.7/logging/__init__.py line 759 in handle
self.emit(record)
.tox/py27-backend_mysql-env_docker/lib/python2.7/site-packages/nose/plugins/logcapture.py
line 82 in emit
self.buffer.append(self.format(record))
/usr/lib/python2.7/logging/__init__.py line 734 in format
return fmt.format(record)
/usr/lib/python2.7/logging/__init__.py line 465 in format
record.message = record.getMessage()
/usr/lib/python2.7/logging/__init__.py line 329 in getMessage
msg = msg % self.args
TypeError: coercing to Unicode: need string or buffer, MagicMock found```
Initial inspection looks like I'm using logging the same as everywhere else,
but I will have to have a closer look later. It looks like a problem with the
method of test implementation rather than the code itself... I think.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] codecov-io edited a comment on issue #4473: [AIRFLOW-3668] 'Zoom into sub DAG' button fix
codecov-io edited a comment on issue #4473: [AIRFLOW-3668] 'Zoom into sub DAG' button fix URL: https://github.com/apache/airflow/pull/4473#issuecomment-453103432 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=h1) Report > Merging [#4473](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/24bc74bc5f45455aee21f6b506d282ee052efa19?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4473 +/- ## === Coverage 79.47% 79.47% === Files 204 204 Lines 1653416534 === Hits1314013140 Misses 3394 3394 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=footer). Last update [24bc74b...f9f0ba3](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4473: [AIRFLOW-3668] 'Zoom into sub DAG' button fix
codecov-io edited a comment on issue #4473: [AIRFLOW-3668] 'Zoom into sub DAG' button fix URL: https://github.com/apache/airflow/pull/4473#issuecomment-453103432 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=h1) Report > Merging [#4473](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/24bc74bc5f45455aee21f6b506d282ee052efa19?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4473 +/- ## === Coverage 79.47% 79.47% === Files 204 204 Lines 1653416534 === Hits1314013140 Misses 3394 3394 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=footer). Last update [24bc74b...f9f0ba3](https://codecov.io/gh/apache/airflow/pull/4473?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
