[GitHub] [airflow] clacke commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
clacke commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-538615069 We have solved the problem, and uncovered another problem. Working on navigating my company's open source contribution process. Thank you for your patience. :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #5077: [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator
mik-laj commented on issue #5077: [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator URL: https://github.com/apache/airflow/pull/5077#issuecomment-538610165 Can you create a bug report in Jira? Please add the "gcp" component. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work started] (AIRFLOW-5578) The root url "/" should be configurable, not hardcoded
[ https://issues.apache.org/jira/browse/AIRFLOW-5578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5578 started by David Dossett. -- > The root url "/" should be configurable, not hardcoded > -- > > Key: AIRFLOW-5578 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5578 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.5 >Reporter: David Dossett >Assignee: David Dossett >Priority: Minor > > The webserver root url "/" is hardcoded in airflow/www/blueprints.py to > redirect to the Airflow.index view function. If you want to build a front-end > website to airflow, with the normal /home as an admin area, then you will > always have the problem that this will redirect to the admin area first and > this is not configurable. > Flask Appbuilder has the IndexView class which can be set dynamically when > creating the app. This should allow for some configuration of custom > IndexView classes using the default_webserver_config.py > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log
codecov-io edited a comment on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log URL: https://github.com/apache/airflow/pull/6265#issuecomment-538605802 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=h1) Report > Merging [#6265](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6265 +/- ## == - Coverage 80.05% 80.05% -0.01% == Files 610 610 Lines 3526435264 == - Hits2823228229 -3 - Misses 7032 7035 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6265/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.24% <0%> (-0.51%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=footer). Last update [32fd2af...6c52c19](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log
codecov-io commented on issue #6265: [AIRFLOW-5597] Linkify urls in task instance log URL: https://github.com/apache/airflow/pull/6265#issuecomment-538605802 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=h1) Report > Merging [#6265](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6265 +/- ## == - Coverage 80.05% 80.05% -0.01% == Files 610 610 Lines 3526435264 == - Hits2823228229 -3 - Misses 7032 7035 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6265/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.24% <0%> (-0.51%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=footer). Last update [32fd2af...6c52c19](https://codecov.io/gh/apache/airflow/pull/6265?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6263: [AIRFLOW-5598] Improve MLEngine typehints
codecov-io commented on issue #6263: [AIRFLOW-5598] Improve MLEngine typehints URL: https://github.com/apache/airflow/pull/6263#issuecomment-538600271 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6263?src=pr=h1) Report > Merging [#6263](https://codecov.io/gh/apache/airflow/pull/6263?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6263?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#6263 +/- ## === Coverage 80.05% 80.05% === Files 610 610 Lines 3526435264 === Hits2823228232 Misses 7032 7032 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6263?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/operators/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6263/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL21sZW5naW5lLnB5) | `75.72% <ø> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6263?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6263?src=pr=footer). Last update [32fd2af...e61dbd8](https://codecov.io/gh/apache/airflow/pull/6263?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5597) Linkify urls in task instance log
[
https://issues.apache.org/jira/browse/AIRFLOW-5597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944934#comment-16944934
]
ASF GitHub Bot commented on AIRFLOW-5597:
-
ZxMYS commented on pull request #6265: [AIRFLOW-5597] Linkify urls in task
instance log
URL: https://github.com/apache/airflow/pull/6265
#6070 # Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5597
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
Very often we have urls in our task instances' logs. This change allows
these urls to be rendered as clickable links.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Just a simple JS change
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Linkify urls in task instance log
> -
>
> Key: AIRFLOW-5597
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5597
> Project: Apache Airflow
> Issue Type: Wish
> Components: webserver
>Affects Versions: 1.10.5
>Reporter: Xiao Zhu
>Priority: Minor
>
> It would be great if urls in task instance log can be linkified thus user can
> click on the links and open them instead of having to copy and paste.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] ZxMYS opened a new pull request #6265: [AIRFLOW-5597] Linkify urls in task instance log
ZxMYS opened a new pull request #6265: [AIRFLOW-5597] Linkify urls in task
instance log
URL: https://github.com/apache/airflow/pull/6265
#6070 # Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5597
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
Very often we have urls in our task instances' logs. This change allows
these urls to be rendered as clickable links.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Just a simple JS change
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (AIRFLOW-4660) Make airflow/bin Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arkadiusz Bach reassigned AIRFLOW-4660: --- Assignee: Arkadiusz Bach > Make airflow/bin Pylint compatible > -- > > Key: AIRFLOW-4660 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4660 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Arkadiusz Bach >Priority: Major > > Fix all Pylint messages in airflow/bin. To start; running > scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the > files mentioned in your issue from the blacklist. (2) Run > scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted > files. (3) Fix all messages and create PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stpavinash commented on issue #5077: [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator
stpavinash commented on issue #5077: [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator URL: https://github.com/apache/airflow/pull/5077#issuecomment-538594334 I'm trying to connect to sql server instance using this operator and trying to a query a table which has got 20M records and the airflow task is in running state for atleast 8 hours and it never generated a json file in GCS. However, tables that have like 200k records were queried in like 1 min and json was created in GCS. Could you please help? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5598) Improve MLEngine typehint
[
https://issues.apache.org/jira/browse/AIRFLOW-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944901#comment-16944901
]
ASF GitHub Bot commented on AIRFLOW-5598:
-
mik-laj commented on pull request #6263: [AIRFLOW-5598] Improve MLEngine
typehints
URL: https://github.com/apache/airflow/pull/6263
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve MLEngine typehint
> -
>
> Key: AIRFLOW-5598
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5598
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
>Affects Versions: 1.10.5
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5600) Add MLEngine system tests
[
https://issues.apache.org/jira/browse/AIRFLOW-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944908#comment-16944908
]
ASF GitHub Bot commented on AIRFLOW-5600:
-
mik-laj commented on pull request #6264: [AIRFLOW-5600][depends on
AIRFLOW-5598,AIRFLOW-5599] Add MLEngine system tests
URL: https://github.com/apache/airflow/pull/6264
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add MLEngine system tests
> -
>
> Key: AIRFLOW-5600
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5600
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
>Affects Versions: 1.10.5
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6264: [AIRFLOW-5600][depends on AIRFLOW-5598, AIRFLOW-5599] Add MLEngine system tests
mik-laj opened a new pull request #6264: [AIRFLOW-5600][depends on
AIRFLOW-5598,AIRFLOW-5599] Add MLEngine system tests
URL: https://github.com/apache/airflow/pull/6264
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5599) Imporve Python 3 support in MLEngine integration
[
https://issues.apache.org/jira/browse/AIRFLOW-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944900#comment-16944900
]
ASF GitHub Bot commented on AIRFLOW-5599:
-
mik-laj commented on pull request #6262: [AIRFLOW-5599] Improve Python 3
support in MLEngine integration
URL: https://github.com/apache/airflow/pull/6262
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Imporve Python 3 support in MLEngine integration
>
>
> Key: AIRFLOW-5599
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5599
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
>Affects Versions: 1.10.5
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6263: [AIRFLOW-5598] Improve MLEngine typehints
mik-laj opened a new pull request #6263: [AIRFLOW-5598] Improve MLEngine
typehints
URL: https://github.com/apache/airflow/pull/6263
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj opened a new pull request #6262: [AIRFLOW-5599] Improve Python 3 support in MLEngine integration
mik-laj opened a new pull request #6262: [AIRFLOW-5599] Improve Python 3
support in MLEngine integration
URL: https://github.com/apache/airflow/pull/6262
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod…
codecov-io commented on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod… URL: https://github.com/apache/airflow/pull/6261#issuecomment-538590785 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=h1) Report > Merging [#6261](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `84.29%`. [](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6261 +/- ## == + Coverage 80.05% 80.07% +0.01% == Files 610 611 +1 Lines 3526435382 +118 == + Hits2823228331 +99 - Misses 7032 7051 +19 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=tree) | Coverage Δ | | |---|---|---| | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6261/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `98.57% <100%> (+0.06%)` | :arrow_up: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6261/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `90.47% <76.92%> (-1.5%)` | :arrow_down: | | [airflow/kubernetes/pod\_usage\_metrics\_logger.py](https://codecov.io/gh/apache/airflow/pull/6261/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF91c2FnZV9tZXRyaWNzX2xvZ2dlci5weQ==) | `84.76% <84.76%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=footer). Last update [32fd2af...75c33ab](https://codecov.io/gh/apache/airflow/pull/6261?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5600) Add MLEngine system tests
Kamil Bregula created AIRFLOW-5600: -- Summary: Add MLEngine system tests Key: AIRFLOW-5600 URL: https://issues.apache.org/jira/browse/AIRFLOW-5600 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.5 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5599) Imporve Python 3 support in MLEngine integration
Kamil Bregula created AIRFLOW-5599: -- Summary: Imporve Python 3 support in MLEngine integration Key: AIRFLOW-5599 URL: https://issues.apache.org/jira/browse/AIRFLOW-5599 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.5 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5598) Improve MLEngine typehint
Kamil Bregula created AIRFLOW-5598: -- Summary: Improve MLEngine typehint Key: AIRFLOW-5598 URL: https://issues.apache.org/jira/browse/AIRFLOW-5598 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.5 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5597) Linkify urls in task instance log
Xiao Zhu created AIRFLOW-5597: - Summary: Linkify urls in task instance log Key: AIRFLOW-5597 URL: https://issues.apache.org/jira/browse/AIRFLOW-5597 Project: Apache Airflow Issue Type: Wish Components: webserver Affects Versions: 1.10.5 Reporter: Xiao Zhu It would be great if urls in task instance log can be linkified thus user can click on the links and open them instead of having to copy and paste. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5141) KubernetesPodOperator to print resource utilization at end of job
[
https://issues.apache.org/jira/browse/AIRFLOW-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944867#comment-16944867
]
ASF GitHub Bot commented on AIRFLOW-5141:
-
Sinsin1367 commented on pull request #6261: [AIRFLOW-5141] Added optional
resource usage logging to KubernetesPod…
URL: https://github.com/apache/airflow/pull/6261
…Operator.
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5141
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
This PR implement periodic resource fetching and keeping a max resource
usage (CPU and memory) per container for pod in KubernetesPodOperator. The
fetch frequency and log frequency are configurable to avoid excessive logging.
There will be reports on maximum CPU usage and memory usage per container at
the end of the job.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good
The PodUsageMetricsLogger class is tested in all sensible ways I could think
of.
reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> KubernetesPodOperator to print resource utilization at end of job
> -
>
> Key: AIRFLOW-5141
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5141
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Affects Versions: 1.10.4
>Reporter: Chris McLennon
>Assignee: Sina Asadallahi
>Priority: Minor
>
> It would be fantastic if the KubernetesPodOperator printed out a pod's max
> CPU/mem after it completes. This would give users a better idea of how many
> resources they should be requesting for their jobs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4493) emr_add_step_operator needs region_name as parameter
[ https://issues.apache.org/jira/browse/AIRFLOW-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944869#comment-16944869 ] Francesco Macagno commented on AIRFLOW-4493: I believe this is still a problem. an aws connection can work for multiple regions, and locking it to a specific one requires creating redundant connections, ruining their purpose. > emr_add_step_operator needs region_name as parameter > > > Key: AIRFLOW-4493 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4493 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Reporter: Guang Hu >Priority: Minor > > when adding EmrAddStepsOperator to dag, the boto3 lib cannot find the > corresponding job flow id if we do not specify the region_name if our emr > cluster located in no default az. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5596) EmrJobFlowSensor does not allow configuring region
Francesco Macagno created AIRFLOW-5596: -- Summary: EmrJobFlowSensor does not allow configuring region Key: AIRFLOW-5596 URL: https://issues.apache.org/jira/browse/AIRFLOW-5596 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 1.10.3 Reporter: Francesco Macagno While it is possible to set the region from the aws_connection, it makes it more difficult to reuse the connection, since i might want to use it for multiple regions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] Sinsin1367 commented on a change in pull request #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod…
Sinsin1367 commented on a change in pull request #6261: [AIRFLOW-5141] Added
optional resource usage logging to KubernetesPod…
URL: https://github.com/apache/airflow/pull/6261#discussion_r331702649
##
File path: airflow/kubernetes/pod_launcher.py
##
@@ -116,9 +124,27 @@ def run_pod(
time.sleep(1)
self.log.debug('Pod not yet started')
-return self._monitor_pod(pod, get_logs)
+return self._monitor_pod(pod,
+ get_logs,
+ get_resource_usage_logs,
+ resource_usage_fetch_interval,
+ resource_usage_log_interval)
-def _monitor_pod(self, pod: V1Pod, get_logs: bool) -> Tuple[State,
Optional[str]]:
+def _monitor_pod(self,
+ pod: V1Pod,
+ get_logs: bool,
+ get_resource_usage_logs: bool,
+ resource_usage_fetch_interval: int,
+ resource_usage_log_interval: int) -> Tuple[State,
Optional[str]]:
+# Initialize the resource monitoring thread to do nothing.
+resource_monitoring_thread = threading.Thread(target=lambda: None)
+if get_resource_usage_logs:
+metric_server_logger = PodUsageMetricsLogger(self,
Review comment:
Right now the logger object keeps a reference of the pod_launcher object to
be able to call pod_is_running method to know when to finish making api calls.
A cleaner way would be separating the pod status monitoring (like
pod_is_running method) into a separate pod status util class so that all others
can use without keeping pod_launcher. I can implement this util class and move
the status methods out of pod_launcher if committers agree it is a cleaner
implementation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] Sinsin1367 opened a new pull request #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod…
Sinsin1367 opened a new pull request #6261: [AIRFLOW-5141] Added optional
resource usage logging to KubernetesPod…
URL: https://github.com/apache/airflow/pull/6261
…Operator.
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5141
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
This PR implement periodic resource fetching and keeping a max resource
usage (CPU and memory) per container for pod in KubernetesPodOperator. The
fetch frequency and log frequency are configurable to avoid excessive logging.
There will be reports on maximum CPU usage and memory usage per container at
the end of the job.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good
The PodUsageMetricsLogger class is tested in all sensible ways I could think
of.
reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil merged pull request #6066: [AIRFLOW-XXX] Update to new logo
kaxil merged pull request #6066: [AIRFLOW-XXX] Update to new logo URL: https://github.com/apache/airflow/pull/6066 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil merged pull request #6250: [AIRFLOW-XXX] Move examples note
kaxil merged pull request #6250: [AIRFLOW-XXX] Move examples note URL: https://github.com/apache/airflow/pull/6250 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Tagar commented on a change in pull request #6090: [AIRFLOW-5470] Add Apache Livy REST operator
Tagar commented on a change in pull request #6090: [AIRFLOW-5470] Add Apache
Livy REST operator
URL: https://github.com/apache/airflow/pull/6090#discussion_r331673490
##
File path: airflow/contrib/hooks/livy_hook.py
##
@@ -0,0 +1,297 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains the Apache Livy hook.
+"""
+
+import re
+from enum import Enum
+import json
+import requests
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class BatchState(Enum):
+"""
+Batch session states
+"""
+NOT_STARTED = 'not_started'
+STARTING = 'starting'
+RUNNING = 'running'
+IDLE = 'idle'
+BUSY = 'busy'
+SHUTTING_DOWN = 'shutting_down'
+ERROR = 'error'
+DEAD = 'dead'
+KILLED = 'killed'
+SUCCESS = 'success'
+
+
+TERMINAL_STATES = {
+BatchState.SUCCESS,
+BatchState.DEAD,
+BatchState.KILLED,
+BatchState.ERROR,
+}
+
+
+class LivyHook(BaseHook, LoggingMixin):
+"""
+Hook for Apache Livy through the REST API.
+
+For more information about the API refer to
+https://livy.apache.org/docs/latest/rest-api.html
+
+:param livy_conn_id: reference to a pre-defined Livy Connection.
+:type livy_conn_id: str
+"""
+def __init__(self, livy_conn_id='livy_default'):
+super(LivyHook, self).__init__(livy_conn_id)
+self._livy_conn_id = livy_conn_id
+self._build_base_url()
+
+def _build_base_url(self):
+"""
+Build connection URL
+"""
+params = self.get_connection(self._livy_conn_id)
+
+base_url = params.host
+
+if not base_url:
+raise AirflowException("Missing Livy endpoint hostname")
+
+if '://' not in base_url:
+base_url = '{}://{}'.format('http', base_url)
+if not re.search(r':\d+$', base_url):
+base_url = '{}:{}'.format(base_url, str(params.port or 8998))
+
+self._base_url = base_url
+
+def get_conn(self):
+pass
+
+def post_batch(self, *args, **kwargs):
+"""
+Perform request to submit batch
+"""
+
+batch_submit_body = json.dumps(LivyHook.build_post_batch_body(*args,
**kwargs))
+headers = {'Content-Type': 'application/json'}
+
+self.log.info("Submitting job {} to {}".format(batch_submit_body,
self._base_url))
+response = requests.post(self._base_url + '/batches',
data=batch_submit_body, headers=headers)
Review comment:
@ashb `HttpHook` doesn't support currently `auth` if I understand that right
https://github.com/apache/airflow/blob/master/airflow/hooks/http_hook.py#L165
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5586) Improve error messaging for role creation and updates
[ https://issues.apache.org/jira/browse/AIRFLOW-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944791#comment-16944791 ] ASF subversion and git services commented on AIRFLOW-5586: -- Commit 178f6be215c256ea8558c39b3c79a4f7c33f1a7c in airflow's branch refs/heads/master from Leah E. Cole [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=178f6be ] [AIRFLOW-5586] Improve CLI error messaging (#6246) > Improve error messaging for role creation and updates > - > > Key: AIRFLOW-5586 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5586 > Project: Apache Airflow > Issue Type: Improvement > Components: cli >Affects Versions: 1.10.5 >Reporter: Leah >Assignee: Leah >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kpathak13 commented on a change in pull request #6237: AIRFLOW-5581: Join KubernetesJobWatcher in terminate call and unblock queues from blocking forever
kpathak13 commented on a change in pull request #6237: AIRFLOW-5581: Join
KubernetesJobWatcher in terminate call and unblock queues from blocking forever
URL: https://github.com/apache/airflow/pull/6237#discussion_r331660938
##
File path: airflow/executors/kubernetes_executor.py
##
@@ -768,9 +787,45 @@ def _change_state(self, key, state, pod_id: str) -> None:
self.log.debug('Could not find key: %s', str(key))
self.event_buffer[key] = state
+def _flush_task_queue(self):
+self.log.debug('Executor shutting down, task_queue approximate
size=%d', self.task_queue.qsize())
+while True:
+try:
+task = self.task_queue.get_nowait()
+# This is a new task to run thus ok to ignore.
+self.log.warning('Executor shutting down, will NOT run
task=%s', task)
+self.task_queue.task_done()
+except Empty:
+break
+
+def _flush_result_queue(self):
+self.log.debug('Executor shutting down, result_queue approximate
size=%d', self.result_queue.qsize())
+while True: # pylint: disable=too-many-nested-blocks
+try:
+results = self.result_queue.get_nowait()
+self.log.warning('Executor shutting down, flushing
results=%s', results)
+try:
+key, state, pod_id, resource_version = results
+self.log.info('Changing state of %s to %s :
resource_version=%d', results, state,
+ resource_version)
+try:
+self._change_state(key, state, pod_id)
+except Exception as e: # pylint: disable=broad-except
+self.log.exception('Ignoring exception: %s when
attempting to change state of %s '
+ 'to %s.', e, results, state)
+finally:
+self.result_queue.task_done()
+except Empty:
+break
+
def end(self):
"""Called when the executor shuts down"""
self.log.info('Shutting down Kubernetes executor')
+self.log.debug('Flushing task_queue...')
+self._flush_task_queue()
Review comment:
That should not be the case since `executor.end()` is called after exiting
the executor loop.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (AIRFLOW-5586) Improve error messaging for role creation and updates
[ https://issues.apache.org/jira/browse/AIRFLOW-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-5586. --- Fix Version/s: 2.0.0 Resolution: Fixed > Improve error messaging for role creation and updates > - > > Key: AIRFLOW-5586 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5586 > Project: Apache Airflow > Issue Type: Improvement > Components: cli >Affects Versions: 1.10.5 >Reporter: Leah >Assignee: Leah >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5586) Improve error messaging for role creation and updates
[ https://issues.apache.org/jira/browse/AIRFLOW-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944790#comment-16944790 ] ASF GitHub Bot commented on AIRFLOW-5586: - Fokko commented on pull request #6246: [AIRFLOW-5586] Improve CLI error messaging URL: https://github.com/apache/airflow/pull/6246 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve error messaging for role creation and updates > - > > Key: AIRFLOW-5586 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5586 > Project: Apache Airflow > Issue Type: Improvement > Components: cli >Affects Versions: 1.10.5 >Reporter: Leah >Assignee: Leah >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] Fokko merged pull request #6246: [AIRFLOW-5586] Improve CLI error messaging
Fokko merged pull request #6246: [AIRFLOW-5586] Improve CLI error messaging URL: https://github.com/apache/airflow/pull/6246 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #6246: [AIRFLOW-5586] Improve CLI error messaging
Fokko commented on issue #6246: [AIRFLOW-5586] Improve CLI error messaging URL: https://github.com/apache/airflow/pull/6246#issuecomment-538536813 My pleasure @leahecole, thank you for the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle merged pull request #6244: [AIRFLOW-XXX] Improve linking
feluelle merged pull request #6244: [AIRFLOW-XXX] Improve linking URL: https://github.com/apache/airflow/pull/6244 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Tagar commented on a change in pull request #6090: [AIRFLOW-5470] Add Apache Livy REST operator
Tagar commented on a change in pull request #6090: [AIRFLOW-5470] Add Apache
Livy REST operator
URL: https://github.com/apache/airflow/pull/6090#discussion_r331621383
##
File path: airflow/contrib/hooks/livy_hook.py
##
@@ -0,0 +1,297 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains the Apache Livy hook.
+"""
+
+import re
+from enum import Enum
+import json
+import requests
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class BatchState(Enum):
+"""
+Batch session states
+"""
+NOT_STARTED = 'not_started'
+STARTING = 'starting'
+RUNNING = 'running'
+IDLE = 'idle'
+BUSY = 'busy'
+SHUTTING_DOWN = 'shutting_down'
+ERROR = 'error'
+DEAD = 'dead'
+KILLED = 'killed'
+SUCCESS = 'success'
+
+
+TERMINAL_STATES = {
+BatchState.SUCCESS,
+BatchState.DEAD,
+BatchState.KILLED,
+BatchState.ERROR,
+}
+
+
+class LivyHook(BaseHook, LoggingMixin):
+"""
+Hook for Apache Livy through the REST API.
+
+For more information about the API refer to
+https://livy.apache.org/docs/latest/rest-api.html
+
+:param livy_conn_id: reference to a pre-defined Livy Connection.
+:type livy_conn_id: str
+"""
+def __init__(self, livy_conn_id='livy_default'):
+super(LivyHook, self).__init__(livy_conn_id)
+self._livy_conn_id = livy_conn_id
+self._build_base_url()
+
+def _build_base_url(self):
+"""
+Build connection URL
+"""
+params = self.get_connection(self._livy_conn_id)
+
+base_url = params.host
+
+if not base_url:
+raise AirflowException("Missing Livy endpoint hostname")
+
+if '://' not in base_url:
+base_url = '{}://{}'.format('http', base_url)
+if not re.search(r':\d+$', base_url):
+base_url = '{}:{}'.format(base_url, str(params.port or 8998))
+
+self._base_url = base_url
+
+def get_conn(self):
+pass
+
+def post_batch(self, *args, **kwargs):
+"""
+Perform request to submit batch
+"""
+
+batch_submit_body = json.dumps(LivyHook.build_post_batch_body(*args,
**kwargs))
+headers = {'Content-Type': 'application/json'}
+
+self.log.info("Submitting job {} to {}".format(batch_submit_body,
self._base_url))
+response = requests.post(self._base_url + '/batches',
data=batch_submit_body, headers=headers)
Review comment:
Thanks @ashb
Would we be able to add
```
'X-Requested-By': 'livy-client'
```
too that I referenced above?
Through `extra` argument when LivyHook will be based off of HttpHook?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] Tagar commented on a change in pull request #6090: [AIRFLOW-5470] Add Apache Livy REST operator
Tagar commented on a change in pull request #6090: [AIRFLOW-5470] Add Apache
Livy REST operator
URL: https://github.com/apache/airflow/pull/6090#discussion_r331620655
##
File path: airflow/contrib/hooks/livy_hook.py
##
@@ -0,0 +1,297 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This module contains the Apache Livy hook.
+"""
+
+import re
+from enum import Enum
+import json
+import requests
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class BatchState(Enum):
+"""
+Batch session states
+"""
+NOT_STARTED = 'not_started'
+STARTING = 'starting'
+RUNNING = 'running'
+IDLE = 'idle'
+BUSY = 'busy'
+SHUTTING_DOWN = 'shutting_down'
+ERROR = 'error'
+DEAD = 'dead'
+KILLED = 'killed'
+SUCCESS = 'success'
+
+
+TERMINAL_STATES = {
+BatchState.SUCCESS,
+BatchState.DEAD,
+BatchState.KILLED,
+BatchState.ERROR,
+}
+
+
+class LivyHook(BaseHook, LoggingMixin):
Review comment:
+1
Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook
mik-laj merged pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook URL: https://github.com/apache/airflow/pull/6260 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6260: [AIRFLOW-XXX] Fix minor typo in slack hook
mik-laj commented on issue #6260: [AIRFLOW-XXX] Fix minor typo in slack hook URL: https://github.com/apache/airflow/pull/6260#issuecomment-538476155 Thanks. Awesome I merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook
mik-laj commented on a change in pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook URL: https://github.com/apache/airflow/pull/6260#discussion_r331592427 ## File path: airflow/hooks/slack_hook.py ## @@ -53,7 +53,7 @@ def __get_token(self, token, slack_conn_id): def call(self, method: str, api_params: dict) -> None: """ -Calls the clack client. +Calls the slack client. Review comment: This you can commit using the Github UI. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook
mik-laj commented on a change in pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook URL: https://github.com/apache/airflow/pull/6260#discussion_r331592427 ## File path: airflow/hooks/slack_hook.py ## @@ -53,7 +53,7 @@ def __get_token(self, token, slack_conn_id): def call(self, method: str, api_params: dict) -> None: """ -Calls the clack client. +Calls the slack client. Review comment: This you can commit from the Github UI level. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook
mik-laj commented on a change in pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook URL: https://github.com/apache/airflow/pull/6260#discussion_r331592020 ## File path: airflow/hooks/slack_hook.py ## @@ -53,7 +53,7 @@ def __get_token(self, token, slack_conn_id): def call(self, method: str, api_params: dict) -> None: """ -Calls the clack client. +Calls the slack client. Review comment: ```suggestion Calls the Slack client. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on a change in pull request #6237: AIRFLOW-5581: Join KubernetesJobWatcher in terminate call and unblock queues from blocking forever
dimberman commented on a change in pull request #6237: AIRFLOW-5581: Join
KubernetesJobWatcher in terminate call and unblock queues from blocking forever
URL: https://github.com/apache/airflow/pull/6237#discussion_r331591457
##
File path: airflow/executors/kubernetes_executor.py
##
@@ -768,9 +787,45 @@ def _change_state(self, key, state, pod_id: str) -> None:
self.log.debug('Could not find key: %s', str(key))
self.event_buffer[key] = state
+def _flush_task_queue(self):
+self.log.debug('Executor shutting down, task_queue approximate
size=%d', self.task_queue.qsize())
+while True:
+try:
+task = self.task_queue.get_nowait()
+# This is a new task to run thus ok to ignore.
+self.log.warning('Executor shutting down, will NOT run
task=%s', task)
+self.task_queue.task_done()
+except Empty:
+break
+
+def _flush_result_queue(self):
+self.log.debug('Executor shutting down, result_queue approximate
size=%d', self.result_queue.qsize())
+while True: # pylint: disable=too-many-nested-blocks
+try:
+results = self.result_queue.get_nowait()
+self.log.warning('Executor shutting down, flushing
results=%s', results)
+try:
+key, state, pod_id, resource_version = results
+self.log.info('Changing state of %s to %s :
resource_version=%d', results, state,
+ resource_version)
+try:
+self._change_state(key, state, pod_id)
+except Exception as e: # pylint: disable=broad-except
+self.log.exception('Ignoring exception: %s when
attempting to change state of %s '
+ 'to %s.', e, results, state)
+finally:
+self.result_queue.task_done()
+except Empty:
+break
+
def end(self):
"""Called when the executor shuts down"""
self.log.info('Shutting down Kubernetes executor')
+self.log.debug('Flushing task_queue...')
+self._flush_task_queue()
Review comment:
Is there any chance of a race condition where it continues putting in new
tasks after the queue has been flushed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6260: [AIRFLOW-XXX] Fix minor typo in slack hook
codecov-io commented on issue #6260: [AIRFLOW-XXX] Fix minor typo in slack hook URL: https://github.com/apache/airflow/pull/6260#issuecomment-538470638 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6260?src=pr=h1) Report > Merging [#6260](https://codecov.io/gh/apache/airflow/pull/6260?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2bea3d74952d0d68d90e8bbc307ac3dfe8fcf2ff?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6260?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#6260 +/- ## === Coverage 80.05% 80.05% === Files 610 610 Lines 3526135261 === Hits2822928229 Misses 7032 7032 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6260?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/slack\_hook.py](https://codecov.io/gh/apache/airflow/pull/6260/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9zbGFja19ob29rLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6260/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.37% <0%> (-0.52%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6260/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.75% <0%> (+0.5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6260?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6260?src=pr=footer). Last update [2bea3d7...f89f14e](https://codecov.io/gh/apache/airflow/pull/6260?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5585) Docker-context file check gives only marginal improvements for build
[ https://issues.apache.org/jira/browse/AIRFLOW-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944639#comment-16944639 ] ASF subversion and git services commented on AIRFLOW-5585: -- Commit f8fa984b8e15d6de649ac4cd042e610ae22923fb in airflow's branch refs/heads/v1-10-test from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f8fa984 ] [AIRFLOW-5585] Remove docker context from build (#6239) (cherry picked from commit bf4cea0b2f82d709893203b396ae6e05683349a6) > Docker-context file check gives only marginal improvements for build > > > Key: AIRFLOW-5585 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5585 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > After recent merges of breeze where we check whether Docker needs rebuild we > do not need to fix all files in the context. > Most of the times the build will not be triggered (and setup.py, Dockerfile > package.json etc. will not change). So it is enough just to fix permissions > for those sensitive files and do not loose time for listing and fixing all > the files in the context. > This will save 5 or so seconds for every build and decrease overall build > script complexity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] l0n3r4n83r commented on a change in pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook
l0n3r4n83r commented on a change in pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook URL: https://github.com/apache/airflow/pull/6260#discussion_r331581263 ## File path: README.md ## @@ -124,6 +124,7 @@ Currently **officially** using Airflow: 1. [8fit](https://8fit.com/) [[@nicor88](https://github.com/nicor88), [@frnzska](https://github.com/frnzska)] 1. [90 Seconds](https://90seconds.tv/) [[@aaronmak](https://github.com/aaronmak)] 1. [99](https://99taxis.com) [[@fbenevides](https://github.com/fbenevides), [@gustavoamigo](https://github.com/gustavoamigo) & [@mmmaia](https://github.com/mmmaia)] +1. [Acorns Grow](https://www.acorns.com/) [[Acorns](https://github.com/Acornsgrow)] Review comment: Removed from this PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook
mik-laj commented on a change in pull request #6260: [AIRFLOW-XXX] Fix minor typo in slack hook URL: https://github.com/apache/airflow/pull/6260#discussion_r331580279 ## File path: README.md ## @@ -124,6 +124,7 @@ Currently **officially** using Airflow: 1. [8fit](https://8fit.com/) [[@nicor88](https://github.com/nicor88), [@frnzska](https://github.com/frnzska)] 1. [90 Seconds](https://90seconds.tv/) [[@aaronmak](https://github.com/aaronmak)] 1. [99](https://99taxis.com) [[@fbenevides](https://github.com/fbenevides), [@gustavoamigo](https://github.com/gustavoamigo) & [@mmmaia](https://github.com/mmmaia)] +1. [Acorns Grow](https://www.acorns.com/) [[Acorns](https://github.com/Acornsgrow)] Review comment: Can you do it in a separate PR? Otherwise, this change will be difficult to cherry-pick to 1.10.X. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5585) Docker-context file check gives only marginal improvements for build
[ https://issues.apache.org/jira/browse/AIRFLOW-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944633#comment-16944633 ] ASF GitHub Bot commented on AIRFLOW-5585: - potiuk commented on pull request #6239: [AIRFLOW-5585] Remove docker context from build URL: https://github.com/apache/airflow/pull/6239 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Docker-context file check gives only marginal improvements for build > > > Key: AIRFLOW-5585 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5585 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > After recent merges of breeze where we check whether Docker needs rebuild we > do not need to fix all files in the context. > Most of the times the build will not be triggered (and setup.py, Dockerfile > package.json etc. will not change). So it is enough just to fix permissions > for those sensitive files and do not loose time for listing and fixing all > the files in the context. > This will save 5 or so seconds for every build and decrease overall build > script complexity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5585) Docker-context file check gives only marginal improvements for build
[ https://issues.apache.org/jira/browse/AIRFLOW-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944634#comment-16944634 ] ASF subversion and git services commented on AIRFLOW-5585: -- Commit bf4cea0b2f82d709893203b396ae6e05683349a6 in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=bf4cea0 ] [AIRFLOW-5585] Remove docker context from build > Docker-context file check gives only marginal improvements for build > > > Key: AIRFLOW-5585 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5585 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > After recent merges of breeze where we check whether Docker needs rebuild we > do not need to fix all files in the context. > Most of the times the build will not be triggered (and setup.py, Dockerfile > package.json etc. will not change). So it is enough just to fix permissions > for those sensitive files and do not loose time for listing and fixing all > the files in the context. > This will save 5 or so seconds for every build and decrease overall build > script complexity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6239: [AIRFLOW-5585] Remove docker context from build
potiuk merged pull request #6239: [AIRFLOW-5585] Remove docker context from build URL: https://github.com/apache/airflow/pull/6239 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6259: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator
codecov-io commented on issue #6259: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator URL: https://github.com/apache/airflow/pull/6259#issuecomment-538457664 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=h1) Report > Merging [#6259](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/104a151d6a19b1ba1281cb00c66a2c3409e1bb13?src=pr=desc) will **increase** coverage by `0.02%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6259 +/- ## == + Coverage 80.04% 80.07% +0.02% == Files 610 611 +1 Lines 3526135270 +9 == + Hits2822628241 +15 + Misses 7035 7029 -6 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=tree) | Coverage Δ | | |---|---|---| | [...rflow/example\_dags/example\_bigquery\_to\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/6259/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9iaWdxdWVyeV90b19iaWdxdWVyeS5weQ==) | `100% <100%> (ø)` | | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6259/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+5%)` | :arrow_up: | | [airflow/operators/bigquery\_to\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/6259/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYmlncXVlcnlfdG9fYmlncXVlcnkucHk=) | `100% <0%> (+6.45%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=footer). Last update [104a151...f23de31](https://codecov.io/gh/apache/airflow/pull/6259?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] l0n3r4n83r opened a new pull request #6260: Fix minor typo in slack hook
l0n3r4n83r opened a new pull request #6260: Fix minor typo in slack hook URL: https://github.com/apache/airflow/pull/6260 Fixes a minor typo in the slack hook This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] davlum commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S pod to be configured from JSON/YAML file
davlum commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S pod to be configured from JSON/YAML file URL: https://github.com/apache/airflow/pull/6230#discussion_r331551757 ## File path: airflow/kubernetes/pod_generator.py ## @@ -62,75 +68,53 @@ class PodGenerator: Contains Kubernetes Airflow Worker configuration logic Represents a kubernetes pod and manages execution of a single pod. -:param image: The docker image Review comment: I found the docstring on the parameter level to be frequently incomplete and/or inaccurate. The `:type` aspect in particular often conflicts with the types that are guaranteed to be correct in the type annotations. It creates two places where code needs to be updated, but with no static check to actually guarantee that it has been done. I realize that the standard is having the docstring for every parameter so I will add it back, but I do feel that in some cases this is creating more maintenance cost and bloat than value being added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-5575) Cannot use the mark future success/failure feature in the UI
[ https://issues.apache.org/jira/browse/AIRFLOW-5575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Hyatt updated AIRFLOW-5575: --- Attachment: pip_freeze.txt > Cannot use the mark future success/failure feature in the UI > > > Key: AIRFLOW-5575 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5575 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, DagRun, webserver >Affects Versions: 1.10.4, 1.10.5 > Environment: airflow@dfe51753a035:~$ pip list > PackageVersion > -- - > alembic1.2.0 > amqp 2.5.1 > apache-airflow 1.10.4 > apispec3.0.0 > arrow 0.13.0 > asn1crypto 0.24.0 > atomicwrites 1.3.0 > attrs 19.1.0 > aws-xray-sdk 0.95 > Babel 2.7.0 > backcall 0.1.0 > bcrypt 3.1.7 > billiard 3.6.1.0 > boto 2.49.0 > boto3 1.7.76 > botocore 1.10.84 > cached-property1.5.1 > celery 4.3.0 > certifi2019.9.11 > cffi 1.12.3 > chardet3.0.4 > Click 7.0 > click-datetime 0.2 > colorama 0.4.1 > colorlog 4.0.2 > configparser 3.5.3 > cookies2.2.1 > croniter 0.3.30 > cryptography 2.7 > cycler 0.10.0 > Cython 0.29.13 > decorator 4.4.0 > defusedxml 0.6.0 > dill 0.2.9 > docker 4.0.2 > docutils 0.15.2 > dumb-init 1.2.2 > ecdsa 0.13.2 > Flask 1.1.1 > Flask-Admin1.5.3 > Flask-AppBuilder 1.13.1 > Flask-Babel0.12.2 > Flask-Caching 1.3.3 > Flask-JWT-Extended 3.23.0 > Flask-Login0.4.1 > Flask-OpenID 1.2.5 > Flask-SQLAlchemy 2.4.0 > flask-swagger 0.2.13 > Flask-WTF 0.14.2 > flower 0.9.3 > funcsigs 1.0.0 > future 0.16.0 > gunicorn 19.9.0 > hmsclient 0.1.1 > idna 2.8 > importlib-metadata 0.23 > ipython7.8.0 > ipython-genutils 0.2.0 > iso86010.1.12 > itsdangerous 1.1.0 > JayDeBeApi 1.1.1 > jedi 0.15.1 > Jinja2 2.10.1 > jmespath 0.9.4 > joblib 0.13.2 > JPype1 0.7.0 > json-merge-patch 0.2 > jsondiff 1.1.1 > jsonpickle 1.2 > jsonschema 3.0.2 > kiwisolver 1.1.0 > kombu 4.6.4 > lazy-object-proxy 1.4.2 > lockfile 0.12.2 > Mako 1.1.0 > Markdown 2.6.11 > MarkupSafe 1.1.1 > marshmallow2.19.5 > marshmallow-enum 1.5.1 > marshmallow-sqlalchemy 0.19.0 > matplotlib 3.1.1 > mock 3.0.5 > more-itertools 7.2.0 > moto 1.3.6 > ndg-httpsclient0.5.1 > nose 1.3.7 > numpy 1.16.2 > ordereddict1.1 > packaging 19.2 > pandas 0.25.1 > paramiko 2.6.0 > parso 0.5.1 > patsy 0.5.1 > pendulum 1.4.4 > pexpect4.7.0 > pickleshare0.7.5 > pip19.2.3 > pluggy 0.13.0 > pmdarima 1.3.0 > prison 0.1.0 > prompt-toolkit 2.0.9 > psutil 5.6.3 > psycopg2 2.7.7 > psycopg2-binary2.8.3 > ptyprocess 0.6.0 > py 1.8.0 > pyaml 19.4.1 > pyasn1 0.4.7 > PyAstronomy0.13.0 > pycparser 2.19 > pycryptodome 3.9.0 > pydlm 0.1.1.11 > Pygments 2.4.2 > PyHive 0.6.1 > PyJWT 1.7.1 > PyNaCl 1.3.0 > pyOpenSSL 19.0.0 > pyparsing 2.4.2 > pyrsistent 0.15.4 > pysftp 0.2.9 > pytest 5.1.3 > python-daemon 2.1.2 > python-dateutil2.8.0 > python-editor 1.0.4 > python-jose2.0.2 > python3-openid 3.1.0 > pytz 2019.2 > pytzdata 2019.3 > PyYAML 5.1.2 > redis 3.3.8 > requests 2.22.0 > responses 0.10.6 > s3transfer 0.1.13 > scikit-learn 0.21.3 > scipy 1.3.1 > setproctitle
[GitHub] [airflow] codecov-io edited a comment on issue #6239: [AIRFLOW-5585] Remove docker context from build
codecov-io edited a comment on issue #6239: [AIRFLOW-5585] Remove docker context from build URL: https://github.com/apache/airflow/pull/6239#issuecomment-537570182 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6239?src=pr=h1) Report > Merging [#6239](https://codecov.io/gh/apache/airflow/pull/6239?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d719e1fd6705a93a0dfefef4b46478ade5e006ea?src=pr=desc) will **decrease** coverage by `0.35%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6239?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6239 +/- ## == - Coverage 80.06% 79.71% -0.36% == Files 610 610 Lines 3526135441 +180 == + Hits2823128251 +20 - Misses 7030 7190 +160 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6239?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6239/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `47.61% <0%> (-52.39%)` | :arrow_down: | | [airflow/contrib/utils/sendgrid.py](https://codecov.io/gh/apache/airflow/pull/6239/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3V0aWxzL3NlbmRncmlkLnB5) | `68.91% <0%> (-19.02%)` | :arrow_down: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6239/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `79.08% <0%> (-16.2%)` | :arrow_down: | | [airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6239/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=) | `81.81% <0%> (-11.37%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6239/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | [airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6239/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=) | `63.26% <0%> (-4.09%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6239/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `54.13% <0%> (-2.59%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6239/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `73.19% <0%> (-1.21%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6239/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.24% <0%> (-0.51%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6239?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6239?src=pr=footer). Last update [d719e1f...8d1bd7c](https://codecov.io/gh/apache/airflow/pull/6239?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] davlum commented on a change in pull request #6230: [AIRFLOW-5413] enable K8S pod config from file
davlum commented on a change in pull request #6230: [AIRFLOW-5413] enable K8S pod config from file URL: https://github.com/apache/airflow/pull/6230#discussion_r331533436 ## File path: airflow/executors/kubernetes_executor.py ## @@ -45,199 +47,6 @@ MAX_LABEL_LEN = 63 -class KubeConfig: # pylint: disable=too-many-instance-attributes Review comment: The issue with `KubeConfig` existing in the `kubernetes_executor` module is that `WorkerConfiguration` takes `KubeConfig` as an argument. In order to get better linting, [I annotated that argument](https://github.com/apache/airflow/blob/9d0d3e3039aa37766003b639b11605a5675e4746/airflow/kubernetes/worker_configuration.py#L45), however this creates a circular dependency. `kubernetes_executor` module requires `WorkerConfiguration` and `WorkerConfiguration` requires `kubernetes_executor`. This refactor is also being done in #5256, but I feel as thought that PR had been a bit forgotten, and the conflicts might be a bit tedious at this point. I agree this change is unrelated and creates a lot of noise. I could quickly move this into another PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6257: [AIRFLOW-XXX] Split extra packages table in multiple
codecov-io commented on issue #6257: [AIRFLOW-XXX] Split extra packages table in multiple URL: https://github.com/apache/airflow/pull/6257#issuecomment-538420620 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6257?src=pr=h1) Report > Merging [#6257](https://codecov.io/gh/apache/airflow/pull/6257?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e3731b1e33059b7e01813ae54e74f8dbef9add73?src=pr=desc) will **increase** coverage by `0.08%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6257?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6257 +/- ## == + Coverage 80.06% 80.14% +0.08% == Files 610 610 Lines 3526135425 +164 == + Hits2823228393 +161 - Misses 7029 7032 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6257?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6257/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.55% <0%> (-0.35%)` | :arrow_down: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6257/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.51% <0%> (+1.23%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6257?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6257?src=pr=footer). Last update [e3731b1...a92937f](https://codecov.io/gh/apache/airflow/pull/6257?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-5575) Cannot use the mark future success/failure feature in the UI
[ https://issues.apache.org/jira/browse/AIRFLOW-5575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Hyatt updated AIRFLOW-5575: --- Attachment: backfill_test.py > Cannot use the mark future success/failure feature in the UI > > > Key: AIRFLOW-5575 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5575 > Project: Apache Airflow > Issue Type: Bug > Components: DAG, DagRun, webserver >Affects Versions: 1.10.4, 1.10.5 > Environment: airflow@dfe51753a035:~$ pip list > PackageVersion > -- - > alembic1.2.0 > amqp 2.5.1 > apache-airflow 1.10.4 > apispec3.0.0 > arrow 0.13.0 > asn1crypto 0.24.0 > atomicwrites 1.3.0 > attrs 19.1.0 > aws-xray-sdk 0.95 > Babel 2.7.0 > backcall 0.1.0 > bcrypt 3.1.7 > billiard 3.6.1.0 > boto 2.49.0 > boto3 1.7.76 > botocore 1.10.84 > cached-property1.5.1 > celery 4.3.0 > certifi2019.9.11 > cffi 1.12.3 > chardet3.0.4 > Click 7.0 > click-datetime 0.2 > colorama 0.4.1 > colorlog 4.0.2 > configparser 3.5.3 > cookies2.2.1 > croniter 0.3.30 > cryptography 2.7 > cycler 0.10.0 > Cython 0.29.13 > decorator 4.4.0 > defusedxml 0.6.0 > dill 0.2.9 > docker 4.0.2 > docutils 0.15.2 > dumb-init 1.2.2 > ecdsa 0.13.2 > Flask 1.1.1 > Flask-Admin1.5.3 > Flask-AppBuilder 1.13.1 > Flask-Babel0.12.2 > Flask-Caching 1.3.3 > Flask-JWT-Extended 3.23.0 > Flask-Login0.4.1 > Flask-OpenID 1.2.5 > Flask-SQLAlchemy 2.4.0 > flask-swagger 0.2.13 > Flask-WTF 0.14.2 > flower 0.9.3 > funcsigs 1.0.0 > future 0.16.0 > gunicorn 19.9.0 > hmsclient 0.1.1 > idna 2.8 > importlib-metadata 0.23 > ipython7.8.0 > ipython-genutils 0.2.0 > iso86010.1.12 > itsdangerous 1.1.0 > JayDeBeApi 1.1.1 > jedi 0.15.1 > Jinja2 2.10.1 > jmespath 0.9.4 > joblib 0.13.2 > JPype1 0.7.0 > json-merge-patch 0.2 > jsondiff 1.1.1 > jsonpickle 1.2 > jsonschema 3.0.2 > kiwisolver 1.1.0 > kombu 4.6.4 > lazy-object-proxy 1.4.2 > lockfile 0.12.2 > Mako 1.1.0 > Markdown 2.6.11 > MarkupSafe 1.1.1 > marshmallow2.19.5 > marshmallow-enum 1.5.1 > marshmallow-sqlalchemy 0.19.0 > matplotlib 3.1.1 > mock 3.0.5 > more-itertools 7.2.0 > moto 1.3.6 > ndg-httpsclient0.5.1 > nose 1.3.7 > numpy 1.16.2 > ordereddict1.1 > packaging 19.2 > pandas 0.25.1 > paramiko 2.6.0 > parso 0.5.1 > patsy 0.5.1 > pendulum 1.4.4 > pexpect4.7.0 > pickleshare0.7.5 > pip19.2.3 > pluggy 0.13.0 > pmdarima 1.3.0 > prison 0.1.0 > prompt-toolkit 2.0.9 > psutil 5.6.3 > psycopg2 2.7.7 > psycopg2-binary2.8.3 > ptyprocess 0.6.0 > py 1.8.0 > pyaml 19.4.1 > pyasn1 0.4.7 > PyAstronomy0.13.0 > pycparser 2.19 > pycryptodome 3.9.0 > pydlm 0.1.1.11 > Pygments 2.4.2 > PyHive 0.6.1 > PyJWT 1.7.1 > PyNaCl 1.3.0 > pyOpenSSL 19.0.0 > pyparsing 2.4.2 > pyrsistent 0.15.4 > pysftp 0.2.9 > pytest 5.1.3 > python-daemon 2.1.2 > python-dateutil2.8.0 > python-editor 1.0.4 > python-jose2.0.2 > python3-openid 3.1.0 > pytz 2019.2 > pytzdata 2019.3 > PyYAML 5.1.2 > redis 3.3.8 > requests 2.22.0 > responses 0.10.6 > s3transfer 0.1.13 > scikit-learn 0.21.3 > scipy 1.3.1 > setproctitle
[GitHub] [airflow] davlum commented on issue #6196: [AIRFLOW-3152] Kubernetes Pod Operator should support init containers.
davlum commented on issue #6196: [AIRFLOW-3152] Kubernetes Pod Operator should support init containers. URL: https://github.com/apache/airflow/pull/6196#issuecomment-538418468 1. Yeah, I'm referring to those custom objects. Each of them duplicates code that is available in `kubernetes.client.models` 2. Exactly. Users pass `List[k8s.V1Contiainer]`. No need to then convert the `InitContainer` object to the `k8s.V1Container` object. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-5575) Cannot use the mark future success/failure feature in the UI
[ https://issues.apache.org/jira/browse/AIRFLOW-5575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Hyatt updated AIRFLOW-5575: --- Description: Every time we try to mark tasks to in the future success or failure we get an oops message: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69, in inner return self._run_view(f, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 368, in _run_view return fn(self, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/flask_login/utils.py", line 258, in decorated_view return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/www/utils.py", line 281, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/www/utils.py", line 328, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line 1471, in success future, past, State.SUCCESS) File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line 1422, in _mark_task_instance_state commit=False) File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/api/common/experimental/mark_tasks.py", line 103, in set_state dates = get_execution_dates(dag, execution_date, future, past) File "/usr/local/lib/python3.6/site-packages/airflow/api/common/experimental/mark_tasks.py", line 256, in get_execution_dates dates = dag.date_range(start_date=start_date, end_date=end_date) File "/usr/local/lib/python3.6/site-packages/airflow/models/dag.py", line 357, in date_range num=num, delta=self._schedule_interval) File "/usr/local/lib/python3.6/site-packages/airflow/utils/dates.py", line 88, in date_range end_date = timezone.make_naive(end_date, tz) File "/usr/local/lib/python3.6/site-packages/airflow/utils/timezone.py", line 146, in make_naive o = value.astimezone(timezone) File "/usr/local/lib/python3.6/site-packages/pendulum/tz/timezone_info.py", line 99, in fromutc tzinfo = self._tz._tzinfos[self._tz._transitions[idx]._tzinfo_index] IndexError: list index out of range was: Every time we try to mark tasks to in the future success or failure we get an oops message: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69, in inner return self._run_view(f, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 368, in _run_view return fn(self, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/flask_login/utils.py", line 258, in decorated_view return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/www/utils.py", line 281, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/www/utils.py", line 328, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line 1471, in success future, past, State.SUCCESS) File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line 1422, in _mark_task_instance_state commit=False) File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File
[GitHub] [airflow] feluelle merged pull request #6243: [AIRFLOW-XXX] Highlight code blocks
feluelle merged pull request #6243: [AIRFLOW-XXX] Highlight code blocks URL: https://github.com/apache/airflow/pull/6243 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] seelmann commented on issue #5908: Revert "[AIRFLOW-4797] Improve performance and behaviour of zombie de…
seelmann commented on issue #5908: Revert "[AIRFLOW-4797] Improve performance and behaviour of zombie de… URL: https://github.com/apache/airflow/pull/5908#issuecomment-538412584 I finally was able to test, zombie detection works great and deterministic with the 2nd commit. Thanks @KevinYang21 for the change, can you please solve the conflicts? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] aKumpan edited a comment on issue #6256: [AIRFLOW-5590] Add run_id to trigger DAG run API response
aKumpan edited a comment on issue #6256: [AIRFLOW-5590] Add run_id to trigger DAG run API response URL: https://github.com/apache/airflow/pull/6256#issuecomment-538399007 Hi @XD-DENG, could you please provide more details why you do not like the [AIRFLOW-5593 proposal](https://issues.apache.org/jira/browse/AIRFLOW-5593)? So I will have better understanding if it is worth implementing. The only concern I have - API breaking changes in the implementation. But since API is experimental - better to do it now, then never. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6237: AIRFLOW-5581: Join KubernetesJobWatcher in terminate call and unblock queues from blocking forever
codecov-io commented on issue #6237: AIRFLOW-5581: Join KubernetesJobWatcher in terminate call and unblock queues from blocking forever URL: https://github.com/apache/airflow/pull/6237#issuecomment-538409875 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=h1) Report > Merging [#6237](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e9ab9d6a92256a42607c12630fe89368f3316b1c?src=pr=desc) will **increase** coverage by `0.08%`. > The diff coverage is `6.66%`. [](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6237 +/- ## == + Coverage79.8% 79.89% +0.08% == Files 610 610 Lines 3526135306 +45 == + Hits2814028207 +67 + Misses 7121 7099 -22 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `58.89% <6.66%> (-6.34%)` | :arrow_down: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5) | `94.73% <0%> (-1.76%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `86.44% <0%> (-0.85%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.55% <0%> (+0.17%)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `65% <0%> (+1.11%)` | :arrow_up: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `77.6% <0%> (+1.78%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+5%)` | :arrow_up: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `91.52% <0%> (+5.08%)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (+100%)` | :arrow_up: | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/6237/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=footer). Last update [e9ab9d6...1ab4c59](https://codecov.io/gh/apache/airflow/pull/6237?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] sann3 opened a new pull request #6259: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator
sann3 opened a new pull request #6259: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator URL: https://github.com/apache/airflow/pull/6259 Example for BigQuery to BigQuery operator. Accommodated all the changes mentioned [here](https://github.com/apache/airflow/pull/6227) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-5592) Add lint for operators-and-hooks-ref.rst
[ https://issues.apache.org/jira/browse/AIRFLOW-5592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-5592. -- Assignee: Kamil Bregula Resolution: Fixed > Add lint for operators-and-hooks-ref.rst > > > Key: AIRFLOW-5592 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5592 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj edited a comment on issue #6243: [AIRFLOW-XXX] Highlight code blocks
mik-laj edited a comment on issue #6243: [AIRFLOW-XXX] Highlight code blocks URL: https://github.com/apache/airflow/pull/6243#issuecomment-538405316 @feluelle Can you look at this agian? It gently blocks my other activities, because it affects many files. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6243: [AIRFLOW-XXX] Highlight code blocks
mik-laj commented on issue #6243: [AIRFLOW-XXX] Highlight code blocks URL: https://github.com/apache/airflow/pull/6243#issuecomment-538405316 @feluelle Can you look at this agian? It gently blocks my other activities. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5592) Add lint for operators-and-hooks-ref.rst
[ https://issues.apache.org/jira/browse/AIRFLOW-5592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944511#comment-16944511 ] ASF GitHub Bot commented on AIRFLOW-5592: - mik-laj commented on pull request #6255: [AIRFLOW-5592] Add lint for operators-and-hooks-ref.rst URL: https://github.com/apache/airflow/pull/6255 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add lint for operators-and-hooks-ref.rst > > > Key: AIRFLOW-5592 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5592 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5592) Add lint for operators-and-hooks-ref.rst
[ https://issues.apache.org/jira/browse/AIRFLOW-5592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944512#comment-16944512 ] ASF subversion and git services commented on AIRFLOW-5592: -- Commit 104a151d6a19b1ba1281cb00c66a2c3409e1bb13 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=104a151 ] [AIRFLOW-5592] Add lint for operators-and-hooks-ref.rst (#6255) > Add lint for operators-and-hooks-ref.rst > > > Key: AIRFLOW-5592 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5592 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #6255: [AIRFLOW-5592] Add lint for operators-and-hooks-ref.rst
mik-laj merged pull request #6255: [AIRFLOW-5592] Add lint for operators-and-hooks-ref.rst URL: https://github.com/apache/airflow/pull/6255 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] sann3 commented on a change in pull request #6227: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator
sann3 commented on a change in pull request #6227: [AIRFLOW-XXX] Example for
BigQuery to BigQuery operator
URL: https://github.com/apache/airflow/pull/6227#discussion_r331504672
##
File path: airflow/example_dags/example_bigquery_to_bigquery.py
##
@@ -0,0 +1,34 @@
+import airflow
+from airflow.models import DAG
+from airflow.operators import bash_operator
+from airflow.contrib.operators.bigquery_to_bigquery import
BigQueryToBigQueryOperator
+
+args = {
+'owner': 'Airflow',
+'start_date': airflow.utils.dates.days_ago(2),
+}
+
+dag = DAG(
+dag_id='example_bigquery_to_bigquery',
+default_args=args,
+schedule_interval="@daily",
+)
+
+# [START howto_operator_bq_to_bq]
+copy_table = BigQueryToBigQueryOperator(
+task_id='bq_to_bq_example',
+source_project_dataset_tables='bigquery-public-data.baseball.schedules',
+destination_project_dataset_table='bigquery-public-data.test.schedules',
+write_disposition='WRITE_EMPTY',
+create_disposition='CREATE_IF_NEEDED',
+bigquery_conn_id='bigquery_default',
+encryption_configuration = {
+"kmsKeyName":
"projects/testp/locations/us/keyRings/test-kr/cryptoKeys/test-key"
+},
Review comment:
It need extra setup, I will remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] sann3 commented on a change in pull request #6227: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator
sann3 commented on a change in pull request #6227: [AIRFLOW-XXX] Example for
BigQuery to BigQuery operator
URL: https://github.com/apache/airflow/pull/6227#discussion_r331504767
##
File path: airflow/example_dags/example_bigquery_to_bigquery.py
##
@@ -0,0 +1,34 @@
+import airflow
+from airflow.models import DAG
+from airflow.operators import bash_operator
+from airflow.contrib.operators.bigquery_to_bigquery import
BigQueryToBigQueryOperator
+
+args = {
+'owner': 'Airflow',
+'start_date': airflow.utils.dates.days_ago(2),
+}
+
+dag = DAG(
Review comment:
Sure
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] aKumpan commented on issue #6256: [AIRFLOW-5590] Add run_id to trigger DAG run API response
aKumpan commented on issue #6256: [AIRFLOW-5590] Add run_id to trigger DAG run API response URL: https://github.com/apache/airflow/pull/6256#issuecomment-538399007 Hi @XD-DENG, could you please provide more details why you do not like the proposal? So I will have better understanding if it is worth implementing. The only concern I have - API breaking changes in the implementation. But since API is experimental - better to do it now, then never. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #6249: [AIRFLOW-XXX] Add fundaments operators and hooks
mik-laj merged pull request #6249: [AIRFLOW-XXX] Add fundaments operators and hooks URL: https://github.com/apache/airflow/pull/6249 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] sann3 commented on issue #6227: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator
sann3 commented on issue #6227: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator URL: https://github.com/apache/airflow/pull/6227#issuecomment-538385090 Thank you for your suggestions, I will accommodate all the changes and raise a new pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] sann3 closed pull request #6227: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator
sann3 closed pull request #6227: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator URL: https://github.com/apache/airflow/pull/6227 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] sann3 commented on a change in pull request #6227: [AIRFLOW-XXX] Example for BigQuery to BigQuery operator
sann3 commented on a change in pull request #6227: [AIRFLOW-XXX] Example for
BigQuery to BigQuery operator
URL: https://github.com/apache/airflow/pull/6227#discussion_r331485310
##
File path: airflow/example_dags/example_bigquery_to_bigquery.py
##
@@ -0,0 +1,34 @@
+import airflow
+from airflow.models import DAG
+from airflow.operators import bash_operator
+from airflow.contrib.operators.bigquery_to_bigquery import
BigQueryToBigQueryOperator
+
+args = {
+'owner': 'Airflow',
+'start_date': airflow.utils.dates.days_ago(2),
+}
+
+dag = DAG(
+dag_id='example_bigquery_to_bigquery',
+default_args=args,
+schedule_interval="@daily",
+)
+
+# [START howto_operator_bq_to_bq]
+copy_table = BigQueryToBigQueryOperator(
+task_id='bq_to_bq_example',
Review comment:
Thank you for your suggestions, I will accommodate all the changes and raise
a new pull request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-5594) Remove excessive logging for connections
[
https://issues.apache.org/jira/browse/AIRFLOW-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johannes Wienke updated AIRFLOW-5594:
-
Priority: Minor (was: Major)
> Remove excessive logging for connections
>
>
> Key: AIRFLOW-5594
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5594
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.10.5
>Reporter: Johannes Wienke
>Priority: Minor
>
> Currently, acquiring a connection with the {{get_connection}} method in
> {{BaseHook}} always logs a message with info level of the format:
> {noformat}
> [2019-10-04 10:18:35,845] {{base_hook.py:84}} INFO - Using connection to: id:
> xxx. Host: xxx, Port: xxx, Schema: xxx, Login: xxx, Password: ,
> extra: {}
> {noformat}
> I am using stored connections to construct the environment variables for
> docker tasks that are being run. That means, connections have to be accessed
> each time a DAG is being refreshed / parsed. Due to this message being logged
> with level INFO, this results in a huge amount of log volume on our airflow
> installation.
> To my mind, a debug message would be sufficient for this one single message
> that is printed each iteration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5595) gcs_to_bq operator - Throws error when BQ dataset isn't located in US or EU region.
[
https://issues.apache.org/jira/browse/AIRFLOW-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944451#comment-16944451
]
ASF GitHub Bot commented on AIRFLOW-5595:
-
Nagalenoj commented on pull request #6258: [AIRFLOW-5595] Pass job location
when initializing the operator
URL: https://github.com/apache/airflow/pull/6258
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
* As per Google docs, passing location is required to get job status except
for US and EU region. Added the optional parameter for gcs_to_bq operator and
handled it.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
* Only an optional location is passed to the underlying hook from the
operator.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> gcs_to_bq operator - Throws error when BQ dataset isn't located in US or EU
> region.
> ---
>
> Key: AIRFLOW-5595
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5595
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp, operators
>Affects Versions: 1.10.5
>Reporter: Lenoj
>Assignee: Lenoj
>Priority: Major
>
> I have my BQ dataset in "Asia-South1" region and tried using gcs_to_bq
> operator. I started seeing errors after calling the operator, when the hook
> tries to check the status of the job. I realized there is no way to pass
> "location" parameter to the underlying hook from the gcs_to_bq operator,
> which results in no job found error when the hook tries to get the status of
> the job.
> As per the
> doc([https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get),] the
> geographic location of the job is required except for US and EU.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5595) gcs_to_bq operator - Throws error when BQ dataset isn't located in US or EU region.
[ https://issues.apache.org/jira/browse/AIRFLOW-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lenoj updated AIRFLOW-5595: --- Labels: easyfix pull-request-available (was: ) > gcs_to_bq operator - Throws error when BQ dataset isn't located in US or EU > region. > --- > > Key: AIRFLOW-5595 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5595 > Project: Apache Airflow > Issue Type: Bug > Components: gcp, operators >Affects Versions: 1.10.5 >Reporter: Lenoj >Assignee: Lenoj >Priority: Major > Labels: easyfix, pull-request-available > > I have my BQ dataset in "Asia-South1" region and tried using gcs_to_bq > operator. I started seeing errors after calling the operator, when the hook > tries to check the status of the job. I realized there is no way to pass > "location" parameter to the underlying hook from the gcs_to_bq operator, > which results in no job found error when the hook tries to get the status of > the job. > As per the > doc([https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get),] the > geographic location of the job is required except for US and EU. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on issue #6258: [AIRFLOW-5595] Pass job location when initializing the operator
mik-laj commented on issue #6258: [AIRFLOW-5595] Pass job location when initializing the operator URL: https://github.com/apache/airflow/pull/6258#issuecomment-538374534 @nuclearpinguin Can you look at it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Nagalenoj opened a new pull request #6258: [AIRFLOW-5595] Pass job location when initializing the operator
Nagalenoj opened a new pull request #6258: [AIRFLOW-5595] Pass job location
when initializing the operator
URL: https://github.com/apache/airflow/pull/6258
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
* As per Google docs, passing location is required to get job status except
for US and EU region. Added the optional parameter for gcs_to_bq operator and
handled it.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
* Only an optional location is passed to the underlying hook from the
operator.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6249: [AIRFLOW-XXX] Add fundaments operators and hooks
codecov-io edited a comment on issue #6249: [AIRFLOW-XXX] Add fundaments operators and hooks URL: https://github.com/apache/airflow/pull/6249#issuecomment-538161078 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6249?src=pr=h1) Report > Merging [#6249](https://codecov.io/gh/apache/airflow/pull/6249?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e3731b1e33059b7e01813ae54e74f8dbef9add73?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6249?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#6249 +/- ## === Coverage 80.06% 80.06% === Files 610 610 Lines 3526135261 === Hits2823228232 Misses 7029 7029 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6249?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6249?src=pr=footer). Last update [e3731b1...f36](https://codecov.io/gh/apache/airflow/pull/6249?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4415) skip status stops propagation randomly.
[ https://issues.apache.org/jira/browse/AIRFLOW-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944411#comment-16944411 ] ASF GitHub Bot commented on AIRFLOW-4415: - ashb commented on pull request #5189: [AIRFLOW-4415] skip status propagation URL: https://github.com/apache/airflow/pull/5189 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > skip status stops propagation randomly. > --- > > Key: AIRFLOW-4415 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4415 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.8.1 >Reporter: Feng Mao >Priority: Major > Fix For: 2.0.0 > > > Issue: skip status stop propogation to down streams and get randomly stopped > with the dag status marked as failed. > The issue is located in the version 1.8.1. > In version 1.8.0 there is a temp fix but removed after this version. > https://github.com/apache/airflow/commit/4077c6de297566a4c598065867a9a27324ae6eb1 > https://github.com/apache/airflow/commit/92965e8275c6f2ec2282ad46c09950bab10c1cb2 > > root casue: > In a loop, the scheduler evaluate each dag and all its task dependcies > around by around. > Each round evaluation happens twice in the context of flag_upstream_failed > = false and true. > > The dag run update method mark the dag run deadlocked which stops the dag > and all its tasks from be processed furture. > https://github.com/apache/airflow/blob/1.8.1/airflow/models.py#L4184 > It is due to in no_dependencies_met. All_sccucess trigger rule misses > skipped status check and marks the task as failed when upstream only has > skipped tasks. > https://github.com/apache/airflow/blob/1.8.1/airflow/models.py#L4152 > > https://github.com/apache/airflow/blob/1.8.1/airflow/ti_deps/deps/trigger_rule_dep.py#L165 > > Each dag update will checks all its task deps and sent ready tasks to run > in the context of flag_upstream_failed=false (defalt) > https://github.com/apache/airflow/blob/1.8.1/airflow/models.py#L4156 > which wont handle skip status propogation. > > After dag update, dag will checks all its task deps and sent ready tasks to > run in the context of flag_upstream_failed=true > https://github.com/apache/airflow/blob/1.8.1/airflow/jobs.py#L904 > which handles skip status propogration. > > https://github.com/apache/airflow/blob/1.8.1/airflow/ti_deps/deps/trigger_rule_dep.py#L138 > > Two potential causes that will trigger dag update detect a deadlock. > The skip status proprogatation rely on detected skipped upstreams (which > happens asyncly by other nodes writing to db). > If the tasks been evaluated are not following topoloy order(random order) > by priority_weigth. It requried multipe loop rounds to propogate skip statue > to all downsteam tasks. > Depending on how close the topoloy order the tasks fetched, the > proprogation may go further or shorter. > > The deadlock detetion can be avoid only the following conditions happen at > the same time: > 1. the skip task (shortcurit operation async process) update db with > skipped task status, right after dag update (flag_upstream_failed=false > )before dag task checks(flag_upstream_failed=true) in scheduler process. > 2. dag checks(flag_upstream_failed=true) have all tasks fectch/evaluated in > the topology order that skip status can propogate in one evaluations round. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on issue #5189: [AIRFLOW-4415] skip status propagation
ashb commented on issue #5189: [AIRFLOW-4415] skip status propagation URL: https://github.com/apache/airflow/pull/5189#issuecomment-538354175 Closing this PR as it stands as: - It has no tests. Given this is a bug fix we must have unit tests that cover it, otherwise the feature might get broken in the future - It massively changes the default behaviour as yugian90 points out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb closed pull request #5189: [AIRFLOW-4415] skip status propagation
ashb closed pull request #5189: [AIRFLOW-4415] skip status propagation URL: https://github.com/apache/airflow/pull/5189 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4939) Ability to specify default retries for all tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944406#comment-16944406 ] ASF GitHub Bot commented on AIRFLOW-4939: - kaxil commented on pull request #6233: [AIRFLOW-4939] Simplify Code for Default Task Retries URL: https://github.com/apache/airflow/pull/6233 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Ability to specify default retries for all tasks > > > Key: AIRFLOW-4939 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4939 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration >Affects Versions: 1.10.3 >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari >Priority: Major > Fix For: 1.10.6 > > > There should be a config which sets the default retry count for all the > tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4939) Ability to specify default retries for all tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944407#comment-16944407 ] ASF subversion and git services commented on AIRFLOW-4939: -- Commit 29caf1f50c9fcb27565152dd4a04e9a9824b6121 in airflow's branch refs/heads/master from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=29caf1f ] [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) > Ability to specify default retries for all tasks > > > Key: AIRFLOW-4939 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4939 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration >Affects Versions: 1.10.3 >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari >Priority: Major > Fix For: 1.10.6 > > > There should be a config which sets the default retry count for all the > tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #6233: [AIRFLOW-4939] Simplify Code for Default Task Retries
kaxil merged pull request #6233: [AIRFLOW-4939] Simplify Code for Default Task Retries URL: https://github.com/apache/airflow/pull/6233 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work started] (AIRFLOW-5595) gcs_to_bq operator - Throws error when BQ dataset isn't located in US or EU region.
[ https://issues.apache.org/jira/browse/AIRFLOW-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5595 started by Lenoj. -- > gcs_to_bq operator - Throws error when BQ dataset isn't located in US or EU > region. > --- > > Key: AIRFLOW-5595 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5595 > Project: Apache Airflow > Issue Type: Bug > Components: gcp, operators >Affects Versions: 1.10.5 >Reporter: Lenoj >Assignee: Lenoj >Priority: Major > > I have my BQ dataset in "Asia-South1" region and tried using gcs_to_bq > operator. I started seeing errors after calling the operator, when the hook > tries to check the status of the job. I realized there is no way to pass > "location" parameter to the underlying hook from the gcs_to_bq operator, > which results in no job found error when the hook tries to get the status of > the job. > As per the > doc([https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get),] the > geographic location of the job is required except for US and EU. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5595) gcs_to_bq operator - Throws error when BQ dataset isn't located in US or EU region.
Lenoj created AIRFLOW-5595: -- Summary: gcs_to_bq operator - Throws error when BQ dataset isn't located in US or EU region. Key: AIRFLOW-5595 URL: https://issues.apache.org/jira/browse/AIRFLOW-5595 Project: Apache Airflow Issue Type: Bug Components: gcp, operators Affects Versions: 1.10.5 Reporter: Lenoj Assignee: Lenoj I have my BQ dataset in "Asia-South1" region and tried using gcs_to_bq operator. I started seeing errors after calling the operator, when the hook tries to check the status of the job. I realized there is no way to pass "location" parameter to the underlying hook from the gcs_to_bq operator, which results in no job found error when the hook tries to get the status of the job. As per the doc([https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get),] the geographic location of the job is required except for US and EU. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6255: [AIRFLOW-5592][depends on #6249] Add lint for operators-and-hooks-ref.rst
codecov-io edited a comment on issue #6255: [AIRFLOW-5592][depends on #6249] Add lint for operators-and-hooks-ref.rst URL: https://github.com/apache/airflow/pull/6255#issuecomment-538187681 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6255?src=pr=h1) Report > Merging [#6255](https://codecov.io/gh/apache/airflow/pull/6255?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e3731b1e33059b7e01813ae54e74f8dbef9add73?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6255?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6255 +/- ## == - Coverage 80.06% 80.06% -0.01% == Files 610 610 Lines 3526135261 == - Hits2823228231 -1 - Misses 7029 7030 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6255?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6255/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.72% <0%> (-0.18%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6255?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6255?src=pr=footer). Last update [e3731b1...afce85a](https://codecov.io/gh/apache/airflow/pull/6255?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] arkadiuszbach commented on a change in pull request #6252: [AIRFLOW-4676] Make airflow/macros Pylint compatible
arkadiuszbach commented on a change in pull request #6252: [AIRFLOW-4676] Make
airflow/macros Pylint compatible
URL: https://github.com/apache/airflow/pull/6252#discussion_r331438987
##
File path: airflow/macros/__init__.py
##
@@ -91,4 +92,4 @@ def _integrate_plugins():
from airflow.plugins_manager import macros_modules
for macros_module in macros_modules:
sys.modules[macros_module.__name__] = macros_module
-globals()[macros_module._name] = macros_module
+globals()[macros_module.__name__.split('.')[-1]] = macros_module
Review comment:
It is used only in the following five
[inits](https://github.com/apache/airflow/search?q=_integrate_plugins_q=_integrate_plugins)
in order to integrate plugins:
-
[macros](https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/macros/__init__.py)
-
[operators](https://github.com/apache/airflow/blob/5c7e4e4e9c2014a78830a81162532c0f2eecd6e4/airflow/operators/__init__.py)
-
[sensors](https://github.com/apache/airflow/blob/5c7e4e4e9c2014a78830a81162532c0f2eecd6e4/airflow/sensors/__init__.py)
-
[hooks](https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/hooks/__init__.py)
-
[executors](https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/executors/__init__.py)
We could change ```_name``` to ```name```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5594) Remove excessive logging for connections
Johannes Wienke created AIRFLOW-5594:
Summary: Remove excessive logging for connections
Key: AIRFLOW-5594
URL: https://issues.apache.org/jira/browse/AIRFLOW-5594
Project: Apache Airflow
Issue Type: Improvement
Components: core
Affects Versions: 1.10.5
Reporter: Johannes Wienke
Currently, acquiring a connection with the {{get_connection}} method in
{{BaseHook}} always logs a message with info level of the format:
{noformat}
[2019-10-04 10:18:35,845] {{base_hook.py:84}} INFO - Using connection to: id:
xxx. Host: xxx, Port: xxx, Schema: xxx, Login: xxx, Password: , extra:
{}
{noformat}
I am using stored connections to construct the environment variables for docker
tasks that are being run. That means, connections have to be accessed each time
a DAG is being refreshed / parsed. Due to this message being logged with level
INFO, this results in a huge amount of log volume on our airflow installation.
To my mind, a debug message would be sufficient for this one single message
that is printed each iteration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle commented on issue #5994: [AIRFLOW-5395] Fix unloading with column names containing a dot in RedshiftToS3Operator
feluelle commented on issue #5994: [AIRFLOW-5395] Fix unloading with column names containing a dot in RedshiftToS3Operator URL: https://github.com/apache/airflow/pull/5994#issuecomment-538339594 > @feluelle Thought if you want to "take-over" the other PR that might be better - open a new PR, fix the issues and annotate the commit with `Co-authored-by:`? https://github.blog/2018-01-29-commit-together-with-co-authors/ Will do that - on monday when I am back at a "good" computer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb merged pull request #6226: [AIRFLOW-XXX] Add message about breaking change in DAG#get_task_instances
ashb merged pull request #6226: [AIRFLOW-XXX] Add message about breaking change in DAG#get_task_instances URL: https://github.com/apache/airflow/pull/6226 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6226: [AIRFLOW-XXX] Add message about breaking change in DAG#get_task_instances
codecov-io commented on issue #6226: [AIRFLOW-XXX] Add message about breaking change in DAG#get_task_instances URL: https://github.com/apache/airflow/pull/6226#issuecomment-538336789 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6226?src=pr=h1) Report > Merging [#6226](https://codecov.io/gh/apache/airflow/pull/6226?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e3731b1e33059b7e01813ae54e74f8dbef9add73?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6226?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#6226 +/- ## === Coverage 80.06% 80.06% === Files 610 610 Lines 3526135261 === Hits2823228232 Misses 7029 7029 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6226?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6226?src=pr=footer). Last update [e3731b1...d35b047](https://codecov.io/gh/apache/airflow/pull/6226?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6252: [AIRFLOW-4676] Make airflow/macros Pylint compatible
feluelle commented on a change in pull request #6252: [AIRFLOW-4676] Make
airflow/macros Pylint compatible
URL: https://github.com/apache/airflow/pull/6252#discussion_r331423282
##
File path: airflow/macros/__init__.py
##
@@ -91,4 +92,4 @@ def _integrate_plugins():
from airflow.plugins_manager import macros_modules
for macros_module in macros_modules:
sys.modules[macros_module.__name__] = macros_module
-globals()[macros_module._name] = macros_module
+globals()[macros_module.__name__.split('.')[-1]] = macros_module
Review comment:
The `_name` attribute will be set during the
[make_module](https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/plugins_manager.py#L159).
So related to that it needs to be:
```suggestion
globals()[macros_module.__name__.lower().split('.')[-1]] =
macros_module
```
But I am wondering why it is "private" when it gets only accessed from
[inits](https://github.com/apache/airflow/search?utf8=%E2%9C%93=._name=)
outside of its class.
I suggest to ignore it via `pylint: disable=message`, because if you want to
rename the `_name` attribute it will affect all other inits as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
