[GitHub] [airflow] potiuk opened a new pull request #6266: Production image support including refactoring of build scripts
potiuk opened a new pull request #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. - In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)). - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Description - [ ] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain docstrings that explain what it does - If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] seelmann commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor
seelmann commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor URL: https://github.com/apache/airflow/pull/5499#issuecomment-538650847 @YingboWang may I ask for the status of the smart sensor change? We are also running into issues with sensors being too heavy weight. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor
feng-tao commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor URL: https://github.com/apache/airflow/pull/5499#issuecomment-538661311 +1 on @BasPH , it will be good to change the default sensor behavior. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on issue #6237: AIRFLOW-5581: Join KubernetesJobWatcher in terminate call and unblock queues from blocking forever
dimberman commented on issue #6237: AIRFLOW-5581: Join KubernetesJobWatcher in terminate call and unblock queues from blocking forever URL: https://github.com/apache/airflow/pull/6237#issuecomment-538670434 @kpathak13 LGTM. Can you rebase? I'll merge when ready :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S pod to be configured from JSON/YAML file
dimberman commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S pod to be configured from JSON/YAML file URL: https://github.com/apache/airflow/pull/6230#discussion_r331754671 ## File path: airflow/executors/kubernetes_executor.py ## @@ -45,199 +47,6 @@ MAX_LABEL_LEN = 63 -class KubeConfig: # pylint: disable=too-many-instance-attributes Review comment: @davlum Could you move the Kubeconfig back and just make that a seperate PR? I'll get that merged quickly once this one goes through. I think with front-facing changes it better to put refactors in separately. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-5601) Use built-in pagination mechanism in MLEngine hook
Kamil Bregula created AIRFLOW-5601: -- Summary: Use built-in pagination mechanism in MLEngine hook Key: AIRFLOW-5601 URL: https://issues.apache.org/jira/browse/AIRFLOW-5601 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.5 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5602) Use unittest.mock in MLEngine hook tests
Kamil Bregula created AIRFLOW-5602: -- Summary: Use unittest.mock in MLEngine hook tests Key: AIRFLOW-5602 URL: https://issues.apache.org/jira/browse/AIRFLOW-5602 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.5 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6267: [AIRFLOW-5601] Use built-in pagination mechanism in MLEngine hook
mik-laj opened a new pull request #6267: [AIRFLOW-5601] Use built-in pagination mechanism in MLEngine hook URL: https://github.com/apache/airflow/pull/6267 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-5601 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. - In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)). - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Description - [ ] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain docstrings that explain what it does - If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5601) Use built-in pagination mechanism in MLEngine hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16945124#comment-16945124 ] ASF GitHub Bot commented on AIRFLOW-5601: - mik-laj commented on pull request #6267: [AIRFLOW-5601] Use built-in pagination mechanism in MLEngine hook URL: https://github.com/apache/airflow/pull/6267 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-5601 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. - In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)). - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Description - [ ] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain docstrings that explain what it does - If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use built-in pagination mechanism in MLEngine hook > -- > > Key: AIRFLOW-5601 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5601 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests
mik-laj opened a new pull request #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests URL: https://github.com/apache/airflow/pull/6268 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. - In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)). - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Description - [ ] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain docstrings that explain what it does - If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5602) Use unittest.mock in MLEngine hook tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16945125#comment-16945125 ] ASF GitHub Bot commented on AIRFLOW-5602: - mik-laj commented on pull request #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests URL: https://github.com/apache/airflow/pull/6268 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. - In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)). - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Description - [ ] Here are some details about my PR, including screenshots of any UI changes: ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: ### Commits - [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain docstrings that explain what it does - If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use unittest.mock in MLEngine hook tests > > > Key: AIRFLOW-5602 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5602 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on issue #6266: Production image support including refactoring of build scripts
mik-laj commented on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538674172 Can you add documentation? I would be happy with a short article similar to [quickstart(https://airflow.readthedocs.io/en/latest/start.html)]. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #6266: Production image support including refactoring of build scripts
mik-laj edited a comment on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538674172 Can you add documentation? I would be happy with a short article similar to [Quick start](https://airflow.readthedocs.io/en/latest/start.html). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #6266: Production image support including refactoring of build scripts
mik-laj edited a comment on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538674172 Can you add documentation? I would be happy with a short article similar to [Quick start](https://airflow.readthedocs.io/en/latest/start.html). It's ok for me to copy this article and replace the commands in this article. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] YingboWang commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor
YingboWang commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor URL: https://github.com/apache/airflow/pull/5499#issuecomment-538674356 @seelmann We are deploying the smart sensor inside our company right now. Testing and monitoring it for further refinement. I will update the code for this PR and AIP document with the overall structure and some statistics based on our testing for the past several weeks. @BasPH @feng-tao, Thanks for your feedbacks. It's my bad not update this PR on time. The smart sensor operator itself is only used to create a control dag for taking over all sensors in the airflow cluster when the cluster is enable to "use_smart_sensor" by configuration. The smart sensor task is running as a service keep refreshing all active sensor tasks from DB and poking them. Users doesn't need to change their original sensor tasks when writing DAGs. When use smart sensor, the sensor tasks state changing is different with regular sensor. The scheduler will persist sensor's poke context to DB and set the state to "Smart_pending" so that the task will not be queued and taking a separate process in any worker. Smart sensor task instance which is a long running process on the work will refresh DB to get all sensors that are in "Smart" state (smart_running or smart_pending) and poke them. If the poke return True it will set the corresponding task_instance state to "success" so that the scheduler will be able to kick off the downstream task_instance in the next parsing loop. There is only a cluster configuration change for using smart sensor. No need to write DAGs in a different way. I will update the PR and the description with latest change and also update the AIP to include all details for smart sensor design. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6267: [AIRFLOW-5601] Use built-in pagination mechanism in MLEngine hook
codecov-io commented on issue #6267: [AIRFLOW-5601] Use built-in pagination mechanism in MLEngine hook URL: https://github.com/apache/airflow/pull/6267#issuecomment-538677593 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6267?src=pr&el=h1) Report > Merging [#6267](https://codecov.io/gh/apache/airflow/pull/6267?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr&el=desc) will **increase** coverage by `<.01%`. > The diff coverage is `0%`. [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/6267/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6267?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6267 +/- ## == + Coverage 80.05% 80.06% +<.01% == Files 610 610 Lines 3526435260 -4 == - Hits2823228231 -1 + Misses 7032 7029 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6267?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6267/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=) | `21.36% <0%> (+0.7%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6267/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.72% <0%> (-0.18%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6267?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6267?src=pr&el=footer). Last update [32fd2af...2b24c6f](https://codecov.io/gh/apache/airflow/pull/6267?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16945135#comment-16945135 ] jack commented on AIRFLOW-3978: --- The question is what happens if your MySQL table has BINARY field and you use MySqlToGoogleCloudStorageOperator and then load it to BigQuery Will it be recognized as {{BYTES}} in BigQuery with auto detect? > Add missing types in MySqlToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-3978 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3978 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.2 >Reporter: Roster >Assignee: Roster >Priority: Minor > Labels: gcs > > There fields are missing and can not be mapped: > TIME, BINARY , VARBINARY -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests
codecov-io commented on issue #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests URL: https://github.com/apache/airflow/pull/6268#issuecomment-538680046 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=h1) Report > Merging [#6268](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr&el=desc) will **increase** coverage by `0.16%`. > The diff coverage is `100%`. [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/6268/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6268 +/- ## == + Coverage 80.05% 80.22% +0.16% == Files 610 610 Lines 3526435260 -4 == + Hits2823228288 +56 + Misses 7032 6972 -60 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6268/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=) | `78.63% <100%> (+57.97%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6268/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6268/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `89.9% <0%> (-1.53%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6268/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.55% <0%> (-0.35%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=footer). Last update [32fd2af...36eb8d0](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests
codecov-io edited a comment on issue #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests URL: https://github.com/apache/airflow/pull/6268#issuecomment-538680046 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=h1) Report > Merging [#6268](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr&el=desc) will **increase** coverage by `0.19%`. > The diff coverage is `100%`. [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/6268/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6268 +/- ## == + Coverage 80.05% 80.25% +0.19% == Files 610 610 Lines 3526435260 -4 == + Hits2823228299 +67 + Misses 7032 6961 -71 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6268/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=) | `78.63% <100%> (+57.97%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=footer). Last update [32fd2af...36eb8d0](https://codecov.io/gh/apache/airflow/pull/6268?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6266: Production image support including refactoring of build scripts
potiuk commented on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538683562 Sure. This is still WIP/Draft. I will add documentation when it is ready. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6266: Production image support including refactoring of build scripts
potiuk commented on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538684102 Also - we will be asking community for help with documenting the official/prod image when the PR is ready. I hope we can get community to contribute on how they are using current prod images. I hope @mik-laj we can get some feedback and help from you/the people involved in improving the documentation. Looking forward to it. Any input and fixups with doc improvements are welcome. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #6266: Production image support including refactoring of build scripts
potiuk edited a comment on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538684102 Also - we will be asking community for help with documenting the official/prod image usage when the PR is ready. I hope we can get community to contribute on how they are using current prod images. I hope @mik-laj we can get some feedback and help from you/the people involved in improving the documentation. Looking forward to it. Any input and fixups with doc improvements are welcome. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6262: [AIRFLOW-5599] Improve Python 3 support in MLEngine integration
codecov-io commented on issue #6262: [AIRFLOW-5599] Improve Python 3 support in MLEngine integration URL: https://github.com/apache/airflow/pull/6262#issuecomment-538684323 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6262?src=pr&el=h1) Report > Merging [#6262](https://codecov.io/gh/apache/airflow/pull/6262?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr&el=desc) will **not change** coverage. > The diff coverage is `50%`. [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/6262/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6262?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#6262 +/- ## === Coverage 80.05% 80.05% === Files 610 610 Lines 3526435264 === Hits2823228232 Misses 7032 7032 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6262?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/utils/mlengine\_prediction\_summary.py](https://codecov.io/gh/apache/airflow/pull/6262/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfcHJlZGljdGlvbl9zdW1tYXJ5LnB5) | `0% <0%> (ø)` | :arrow_up: | | [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6262/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <100%> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6262?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6262?src=pr&el=footer). Last update [32fd2af...9134958](https://codecov.io/gh/apache/airflow/pull/6262?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator
jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#discussion_r33176 ## File path: airflow/models/base_async_operator.py ## @@ -0,0 +1,133 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +Base Asynchronous Operator for kicking off a long running +operations and polling for completion with reschedule mode. +""" +from abc import abstractmethod +from typing import Dict, List, Union, Optional + +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.exceptions import AirflowException +from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY +from airflow.models import SkipMixin, TaskReschedule +from airflow.utils.decorators import apply_defaults + +PLACEHOLDER_RESOURCE_ID = 'RESOURCE_ID_NOT_APPLICABLE' + +class BaseAsyncOperator(BaseSensorOperator, SkipMixin): +""" +AsyncOperators are derived from this class and inherit these attributes. + +AsyncOperators must override the following methods: +:meth:`submit_request`: fire a request for a long running operation +:meth:`poke`: a method to check if the long running operation is +complete it should return True when a success criteria is met. +Optionally, AsyncOperators can override: +:meth: `process_result` to perform any operations after the success +criteria is met in :meth: `poke` +:meth: `poke` is executed at a time interval and succeed when a +criteria is met and fail if and when they time out. They are effctively +an opinionated way use combine an Operator and a Sensor in order to kick +off a long running process and poll for completion without tying up a +worker slot while waiting between pokes leveraging reschedule mode. + +:param soft_fail: Set to true to mark the task as SKIPPED on failure +:type soft_fail: bool +:param poke_interval: Time in seconds that the job should wait in +between each tries +:type poke_interval: int +:param timeout: Time, in seconds before the task times out and fails. +:type timeout: int +""" +ui_color = '#9933ff' # type: str + +@apply_defaults +def __init__(self, + *args, + **kwargs) -> None: +super().__init__(mode='reschedule', *args, **kwargs) + +@abstractmethod +def submit_request(self, context) -> Optional[Union[str, List, Dict]]: +""" +This method should kick off a long running operation. +This method should return the ID for the long running operation if +applicable. +Context is the same dictionary used as when rendering jinja templates. + +Refer to get_template_context for more context. + +:returns: a resource_id for the long running operation. +:rtype: Optional[Union[String, List, Dict]] +""" +raise AirflowException('Async Operators must override the `submit_request` method.') + +def process_result(self, context): +""" +This method can optionally be overriden to process the result of a long running operation. +Context is the same dictionary used as when rendering jinja templates. + +Refer to get_template_context for more context. +""" +self.log.info('Using default process_result. Got result of %s. Done.', + self.get_external_resource_id(context)) + +def execute(self, context): +# On the first execute call submit_request and set the +# external resource id. +task_reschedules = TaskReschedule.find_for_task_instance(context['ti']) +if not task_reschedules: +resource_id = self.submit_request(context) +if not resource_id: +resource_id = PLACEHOLDER_RESOURCE_ID +self.set_external_resource_id(context, resource_id) + +super().execute(context) + +# TODO: find out why code below here never called. +resource_id = self.get_external_resource_id(context) +if resource_id == PLACEHOLDER_RESOURCE_ID: +self.log.info("Calling process_result.
[GitHub] [airflow] jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator
jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#discussion_r331763207 ## File path: airflow/models/base_async_operator.py ## @@ -0,0 +1,133 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +Base Asynchronous Operator for kicking off a long running +operations and polling for completion with reschedule mode. +""" +from abc import abstractmethod +from typing import Dict, List, Union, Optional + +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.exceptions import AirflowException +from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY +from airflow.models import SkipMixin, TaskReschedule +from airflow.utils.decorators import apply_defaults + +PLACEHOLDER_RESOURCE_ID = 'RESOURCE_ID_NOT_APPLICABLE' + +class BaseAsyncOperator(BaseSensorOperator, SkipMixin): +""" +AsyncOperators are derived from this class and inherit these attributes. + +AsyncOperators must override the following methods: +:meth:`submit_request`: fire a request for a long running operation +:meth:`poke`: a method to check if the long running operation is +complete it should return True when a success criteria is met. +Optionally, AsyncOperators can override: +:meth: `process_result` to perform any operations after the success +criteria is met in :meth: `poke` +:meth: `poke` is executed at a time interval and succeed when a +criteria is met and fail if and when they time out. They are effctively +an opinionated way use combine an Operator and a Sensor in order to kick +off a long running process and poll for completion without tying up a +worker slot while waiting between pokes leveraging reschedule mode. + +:param soft_fail: Set to true to mark the task as SKIPPED on failure +:type soft_fail: bool +:param poke_interval: Time in seconds that the job should wait in +between each tries +:type poke_interval: int +:param timeout: Time, in seconds before the task times out and fails. +:type timeout: int +""" +ui_color = '#9933ff' # type: str + +@apply_defaults +def __init__(self, + *args, + **kwargs) -> None: +super().__init__(mode='reschedule', *args, **kwargs) + +@abstractmethod +def submit_request(self, context) -> Optional[Union[str, List, Dict]]: +""" +This method should kick off a long running operation. +This method should return the ID for the long running operation if +applicable. +Context is the same dictionary used as when rendering jinja templates. + +Refer to get_template_context for more context. + +:returns: a resource_id for the long running operation. +:rtype: Optional[Union[String, List, Dict]] +""" +raise AirflowException('Async Operators must override the `submit_request` method.') + +def process_result(self, context): +""" +This method can optionally be overriden to process the result of a long running operation. +Context is the same dictionary used as when rendering jinja templates. + +Refer to get_template_context for more context. +""" +self.log.info('Using default process_result. Got result of %s. Done.', + self.get_external_resource_id(context)) + +def execute(self, context): +# On the first execute call submit_request and set the +# external resource id. +task_reschedules = TaskReschedule.find_for_task_instance(context['ti']) +if not task_reschedules: +resource_id = self.submit_request(context) +if not resource_id: +resource_id = PLACEHOLDER_RESOURCE_ID +self.set_external_resource_id(context, resource_id) + +super().execute(context) + +# TODO: find out why code below here never called. +resource_id = self.get_external_resource_id(context) +if resource_id == PLACEHOLDER_RESOURCE_ID: +self.log.info("Calling process_result.
[GitHub] [airflow] yegeniy opened a new pull request #6269: Use poke_interval of 0 to make EMR tests run green [AIRFLOW-XXX]
yegeniy opened a new pull request #6269: Use poke_interval of 0 to make EMR tests run green [AIRFLOW-XXX] URL: https://github.com/apache/airflow/pull/6269 ``` As part of introducing EmrRunJobFlows in a separate PR, I noticed that the EMR tests were so slow they were yellow and that a poke_interval of 0 is a valid poke_interval for the tests to run at. Practically speaking, though, this is only saving a dozen seconds in the test suite at most. ``` Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. - In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)). - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: See above ### Tests - [x] My PR ~adds the following unit tests __OR__~ does not need testing for this extremely good reason: I'm modifying existing tests only. ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain docstrings that explain what it does - If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6269: Use poke_interval of 0 to make EMR tests run green [AIRFLOW-XXX]
codecov-io commented on issue #6269: Use poke_interval of 0 to make EMR tests run green [AIRFLOW-XXX] URL: https://github.com/apache/airflow/pull/6269#issuecomment-538703206 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6269?src=pr&el=h1) Report > Merging [#6269](https://codecov.io/gh/apache/airflow/pull/6269?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr&el=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/6269/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6269?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6269 +/- ## == - Coverage 80.05% 80.03% -0.03% == Files 610 610 Lines 3526435264 == - Hits2823228223 -9 - Misses 7032 7041 +9 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6269?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6269/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6269/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.24% <0%> (-0.51%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6269/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.55% <0%> (-0.35%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6269?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6269?src=pr&el=footer). Last update [32fd2af...1a45456](https://codecov.io/gh/apache/airflow/pull/6269?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] friedmanCadre opened a new pull request #6270: [AIRFLOW-5566] Fix false positive deprecation warning
friedmanCadre opened a new pull request #6270: [AIRFLOW-5566] Fix false positive deprecation warning URL: https://github.com/apache/airflow/pull/6270 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. - In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)). - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: This PR fixes the following issue: [This deprecation warning](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L551-L568) is run regardless of whether an 'airflow_home' entry exists in the config file when the 'AIRFLOW_HOME' environment variables are set. Traceback: [config entry check](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L551) -> [config.has_option(...)](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L296-L304) -> [get(...)]( https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L214-L267) in get: ``` option = self._get_env_var_option(section, key) if option is not None: return option ``` 'get' returns True, then 'has_option' returns True, which triggers the warning This PR adds an additional `file_has_option` method to the `AirflowConfigParser` that does not check the environment variables for the specified option. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: The additional method in this PR simply provides a way to access Python's `ConfigParser` `has_option` method, which was overwritten in the `AirflowConfigParser` class. ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain docstrings that explain what it does - If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5566) Invalid deprecation warning for airflow home specification within config file
[ https://issues.apache.org/jira/browse/AIRFLOW-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16945240#comment-16945240 ] ASF GitHub Bot commented on AIRFLOW-5566: - friedmanCadre commented on pull request #6270: [AIRFLOW-5566] Fix false positive deprecation warning URL: https://github.com/apache/airflow/pull/6270 Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-XXX - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. - In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)). - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Description - [x] Here are some details about my PR, including screenshots of any UI changes: This PR fixes the following issue: [This deprecation warning](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L551-L568) is run regardless of whether an 'airflow_home' entry exists in the config file when the 'AIRFLOW_HOME' environment variables are set. Traceback: [config entry check](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L551) -> [config.has_option(...)](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L296-L304) -> [get(...)]( https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L214-L267) in get: ``` option = self._get_env_var_option(section, key) if option is not None: return option ``` 'get' returns True, then 'has_option' returns True, which triggers the warning This PR adds an additional `file_has_option` method to the `AirflowConfigParser` that does not check the environment variables for the specified option. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: The additional method in this PR simply provides a way to access Python's `ConfigParser` `has_option` method, which was overwritten in the `AirflowConfigParser` class. ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [x] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain docstrings that explain what it does - If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Invalid deprecation warning for airflow home specification within config file > - > > Key: AIRFLOW-5566 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5566 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: 1.10.5 >Reporter: Ian Friedman >Assignee: Ian Friedman >Priority: Minor > > [This deprecation > warning|https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L551-L568] > is run regardless of whether an 'airflow_home' entry exists in the config > file when the 'AIRFLOW_HOME' environment va