[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9503: add date-time format validation for API spec

[GitBox]

ephraimbuddy commented on a change in pull request #9503:
URL: https://github.com/apache/airflow/pull/9503#discussion_r445270916



##
File path: tests/api_connexion/test_parameters.py
##
@@ -1,69 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.

Review comment:
   This file will be added back in #9431





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

mik-laj commented on pull request #9497:
URL: https://github.com/apache/airflow/pull/9497#issuecomment-649264988


   > @I think we need to give up one response format.
   > zalando/connexion#860
   > 
   This is weird because Dag Source and Log uses different types of responses 
and it probably works there.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

mik-laj commented on pull request #9497:
URL: https://github.com/apache/airflow/pull/9497#issuecomment-649263837


   @zikun Here is an example of testing using mock 
   
https://github.com/mik-laj/airflow/commit/48e412761feb80cadb016be06a4cc965831f7932



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

mik-laj commented on pull request #9497:
URL: https://github.com/apache/airflow/pull/9497#issuecomment-649254084


   I think we need to give up one response format.
   https://github.com/zalando/connexion/issues/860



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

mik-laj commented on pull request #9497:
URL: https://github.com/apache/airflow/pull/9497#issuecomment-649230245


   @zikun I'm starting to look at it



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] zikun commented on pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

zikun commented on pull request #9497:
URL: https://github.com/apache/airflow/pull/9497#issuecomment-649229712


   Hi @mik-laj I need some help for the unit test. Because the original 
`airflow.configuration.conf` variable contains many sections and options, the 
expected API response is too long to put in the test. It is also not 
maintainable as the default config can change. So I want to use a small `conf` 
to mock it. I tried both pytest monkeypatch and mock.patch, but the API still 
returns the original config. Any idea?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #9363: Use 'airflow.task' Logger for LocalTaskJob

[GitBox]

aneesh-joseph commented on a change in pull request #9363:
URL: https://github.com/apache/airflow/pull/9363#discussion_r445298308



##
File path: airflow/task/task_runner/standard_task_runner.py
##
@@ -73,11 +74,24 @@ def _start_by_fork(self):  # pylint: 
disable=inconsistent-return-statements
 # [1:] - remove "airflow" from the start of the command
 args = parser.parse_args(self._command[1:])
 
+self.log.info('Running: %s', self._command)
+self.log.info('Job %s: Subtask %s', self._task_instance.job_id, 
self._task_instance.task_id)
+
 proc_title = "airflow task runner: {0.dag_id} {0.task_id} 
{0.execution_date}"
 if hasattr(args, "job_id"):
 proc_title += " {0.job_id}"
 setproctitle(proc_title.format(args))
 
+# Get all the Handlers from 'airflow.task' logger
+# Add these handlers to the root logger so that we can get logs 
from
+# any custom loggers defined in the DAG
+airflow_logger_handlers = 
logging.getLogger('airflow.task').handlers
+root_logger = logging.getLogger()
+for handler in airflow_logger_handlers:
+if isinstance(handler, FileTaskHandler):

Review comment:
   not very sure on this, but can this be 
   
   `if handler.name == 'task' or isinstance(handler, FileTaskHandler):`
   
   instead? This will help if we are using a custom task log Handler which 
doesn't extend from `FileTaskHandler`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] aneesh-joseph commented on a change in pull request #9363: Use 'airflow.task' Logger for LocalTaskJob

[GitBox]

aneesh-joseph commented on a change in pull request #9363:
URL: https://github.com/apache/airflow/pull/9363#discussion_r445298308



##
File path: airflow/task/task_runner/standard_task_runner.py
##
@@ -73,11 +74,24 @@ def _start_by_fork(self):  # pylint: 
disable=inconsistent-return-statements
 # [1:] - remove "airflow" from the start of the command
 args = parser.parse_args(self._command[1:])
 
+self.log.info('Running: %s', self._command)
+self.log.info('Job %s: Subtask %s', self._task_instance.job_id, 
self._task_instance.task_id)
+
 proc_title = "airflow task runner: {0.dag_id} {0.task_id} 
{0.execution_date}"
 if hasattr(args, "job_id"):
 proc_title += " {0.job_id}"
 setproctitle(proc_title.format(args))
 
+# Get all the Handlers from 'airflow.task' logger
+# Add these handlers to the root logger so that we can get logs 
from
+# any custom loggers defined in the DAG
+airflow_logger_handlers = 
logging.getLogger('airflow.task').handlers
+root_logger = logging.getLogger()
+for handler in airflow_logger_handlers:
+if isinstance(handler, FileTaskHandler):

Review comment:
   not sure on this, but can this be 
   
   `if handler.name == 'task':`
   
   instead? This will help if we are using a custom task log Handler which 
doesn't extend from `FileTaskHandler`

##
File path: airflow/task/task_runner/standard_task_runner.py
##
@@ -73,11 +74,24 @@ def _start_by_fork(self):  # pylint: 
disable=inconsistent-return-statements
 # [1:] - remove "airflow" from the start of the command
 args = parser.parse_args(self._command[1:])
 
+self.log.info('Running: %s', self._command)
+self.log.info('Job %s: Subtask %s', self._task_instance.job_id, 
self._task_instance.task_id)
+
 proc_title = "airflow task runner: {0.dag_id} {0.task_id} 
{0.execution_date}"
 if hasattr(args, "job_id"):
 proc_title += " {0.job_id}"
 setproctitle(proc_title.format(args))
 
+# Get all the Handlers from 'airflow.task' logger
+# Add these handlers to the root logger so that we can get logs 
from
+# any custom loggers defined in the DAG
+airflow_logger_handlers = 
logging.getLogger('airflow.task').handlers
+root_logger = logging.getLogger()
+for handler in airflow_logger_handlers:
+if isinstance(handler, FileTaskHandler):

Review comment:
   not very sure on this, but can this be 
   
   `if handler.name == 'task':`
   
   instead? This will help if we are using a custom task log Handler which 
doesn't extend from `FileTaskHandler`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] aneesh-joseph commented on pull request #9363: Use 'airflow.task' Logger for LocalTaskJob

[GitBox]

aneesh-joseph commented on pull request #9363:
URL: https://github.com/apache/airflow/pull/9363#issuecomment-649206229


   thank you @kaxil , I tried out this change on 1.10.10 and it fixed my 
logging issues, hope this makes into 1.10.11 :)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] tag nightly-master updated (23faab5 -> 2b61912)

[github-bot]

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


*** WARNING: tag nightly-master was modified! ***

from 23faab5  (commit)
  to 2b61912  (commit)
from 23faab5  [AIRFLOW-8057] [AIP-31]  Add @task decorator (#8962)
 add 3190db5  [AIRFLOW-9347] Fix QuboleHook unable to add list to tags 
(#9349)
 add c703ce2  Move python import path from operationId into 
x-openapi-router-controller (#9495)
 add 2b61912  Add extra links endpoint (#9475)

No new revisions were added by this update.

Summary of changes:
 .../api_connexion/endpoints/extra_link_endpoint.py |  37 +++-
 airflow/api_connexion/openapi/v1.yaml  | 138 ++-
 airflow/providers/qubole/hooks/qubole.py   |   2 +-
 .../endpoints/test_extra_link_endpoint.py  | 190 -
 tests/cli/commands/test_plugins_command.py |  56 +-
 .../test_sqs.py => qubole/hooks/test_qubole.py}|  28 +--
 tests/test_project_structure.py|   1 -
 tests/test_utils/db.py |   7 +-
 .../mock_plugins.py}   |  61 ++-
 9 files changed, 347 insertions(+), 173 deletions(-)
 copy tests/providers/{amazon/aws/hooks/test_sqs.py => 
qubole/hooks/test_qubole.py} (53%)
 copy tests/{cli/commands/test_plugins_command.py => 
test_utils/mock_plugins.py} (56%)

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9503: add date-time format validation for API spec

[GitBox]

ephraimbuddy commented on a change in pull request #9503:
URL: https://github.com/apache/airflow/pull/9503#discussion_r445270916



##
File path: tests/api_connexion/test_parameters.py
##
@@ -1,69 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.

Review comment:
   This file will be added back in #9431





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj opened a new pull request #9504: Add Redoc Open API preview

[GitBox]

mik-laj opened a new pull request #9504:
URL: https://github.com/apache/airflow/pull/9504


   Some users prefer Redox documentation because it has a more modern look.
   
   https://user-images.githubusercontent.com/12058428/85645099-0e09db80-b699-11ea-996c-62b06a00dcd1.png";>
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Target Github ISSUE in description if exists
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] zikun commented on a change in pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

zikun commented on a change in pull request #9497:
URL: https://github.com/apache/airflow/pull/9497#discussion_r445267538



##
File path: airflow/api_connexion/openapi/v1.yaml
##
@@ -1760,6 +1757,9 @@ components:
 value:
   type: string
   readOnly: true
+source:

Review comment:
   This is inspired by the table in the web configuration page, which has 
four columns - section, key, value and source. Isn't source information useful 
for admin users to change and debug the configuration? Especially when it comes 
from multiple sources like airflow.cfg, env var, cmd.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments

[GitBox]

ephraimbuddy commented on pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#issuecomment-649169418


   Depends on #9503 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy opened a new pull request #9503: add date-time format validation for API spec

[GitBox]

ephraimbuddy opened a new pull request #9503:
URL: https://github.com/apache/airflow/pull/9503


   ---
   Currently, connexion does not validate date-time because `strict_rfc3339` 
which jsonschema uses, is not compatible with Apache License.
   Now jsonschema have started using `rfc3339-validator` to validate date-time 
as can be seen here: 
https://github.com/Julian/jsonschema/blob/9d5edb4749ab1f6194aa5c7c099c6e6fd402c4cf/jsonschema/_format.py#L305-L324
   
   This PR adds `rfc3339-validator` as a dependency so connexion can validate 
date-time
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Target Github ISSUE in description if exists
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj edited a comment on issue #9080: Build and publish API clients automatically

[GitBox]

mik-laj edited a comment on issue #9080:
URL: https://github.com/apache/airflow/issues/9080#issuecomment-649155378


   I have prepared a spreadsheet in which we can collect requirements about 
what the client must have in order to be released.
   
https://docs.google.com/spreadsheets/d/11VaVxdZyYcVHtPwL0JL_qKJ4YzIyvefGcoUleBofn28/edit#gid=0
   For now, this is a very short list, but I hope that we will develop it in 
time.
   If someone has other requirements, please submit a comment / suggestion, I 
will add them.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #9080: Build and publish API clients automatically

[GitBox]

mik-laj commented on issue #9080:
URL: https://github.com/apache/airflow/issues/9080#issuecomment-649155378


   I have prepared a spreadsheet in which we can collect requirements about 
what the client must have in order to be released.
   
https://docs.google.com/spreadsheets/d/11VaVxdZyYcVHtPwL0JL_qKJ4YzIyvefGcoUleBofn28/edit#gid=0



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] KIRY4 closed issue #9374: AIRFLOW__KUBERNETES__AFFINITY AIRFLOW__KUBERNETES__TOLERATIONS how to use?

[GitBox]

KIRY4 closed issue #9374:
URL: https://github.com/apache/airflow/issues/9374


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] KIRY4 edited a comment on issue #9374: AIRFLOW__KUBERNETES__AFFINITY AIRFLOW__KUBERNETES__TOLERATIONS how to use?

[GitBox]

KIRY4 edited a comment on issue #9374:
URL: https://github.com/apache/airflow/issues/9374#issuecomment-649149025


   Example of usage:
   ```
   apiVersion: v1
   kind: ConfigMap
   metadata:
 name: airflow-cm
 namespace: airflow
   data:
 k8spodstolerations: '[{ "key": "type", "operator": "Equal", "value": 
"tool", "effect": "NoSchedule" }]'
 k8spodsaffinity: 
'{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"type","operator":"In","values":["tool"]}]}]}}}'
   ```
   
   In values.yaml:
   ```
 extraEnv:
   - name: AIRFLOW__KUBERNETES__TOLERATIONS
 valueFrom:
   configMapKeyRef:
 name: airflow-cm
 key: k8spodstolerations
   - name: AIRFLOW__KUBERNETES__AFFINITY
 valueFrom:
   configMapKeyRef:
 name: airflow-cm
 key: k8spodsaffinity
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] KIRY4 commented on issue #9374: AIRFLOW__KUBERNETES__AFFINITY AIRFLOW__KUBERNETES__TOLERATIONS how to use?

[GitBox]

KIRY4 commented on issue #9374:
URL: https://github.com/apache/airflow/issues/9374#issuecomment-649149025


   Example of usage:
   ```
   apiVersion: v1
   kind: ConfigMap
   metadata:
 name: airflow-cm
 namespace: airflow
   data:
 k8spodstolerations: '[{ "key": "type", "operator": "Equal", "value": 
"tool", "effect": "NoSchedule" }]'
 k8spodsaffinity: 
'{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"type","operator":"In","values":["tool"]}]}]}}}'
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on pull request #9482: Add CRUD endpoint for XCom

[GitBox]

ephraimbuddy commented on pull request #9482:
URL: https://github.com/apache/airflow/pull/9482#issuecomment-649149124


   Depends on : #9170 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] houqp edited a comment on pull request #9502: [WIP] generate go client from openapi spec

[GitBox]

houqp edited a comment on pull request #9502:
URL: https://github.com/apache/airflow/pull/9502#issuecomment-649108009


   Here is what it looks like to use the go client in a real world project: 
https://github.com/houqp/terraform-provider-airflow/blob/openapi/resource_variable.go,
 https://github.com/houqp/terraform-provider-airflow/blob/openapi/provider.go



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9431: Move API page limit and offset parameters to views as kwargs Arguments

[GitBox]

mik-laj commented on pull request #9431:
URL: https://github.com/apache/airflow/pull/9431#issuecomment-649109375


   Can you move datetime format related stuff to new PR?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] houqp commented on pull request #9502: [WIP] generate go client from openapi spec

[GitBox]

houqp commented on pull request #9502:
URL: https://github.com/apache/airflow/pull/9502#issuecomment-649108009


   Here is what it looks like to use the go client in a real world project: 
https://github.com/houqp/terraform-provider-airflow/blob/openapi/resource_variable.go



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] houqp opened a new pull request #9502: [WIP] generate go client from openapi spec

[GitBox]

houqp opened a new pull request #9502:
URL: https://github.com/apache/airflow/pull/9502


   relates to #9080.
   
   TODO:
   - [ ] Create custom README template
   - [ ] Create documentation for go client release process
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #9401: All "private" images and binaries used by Airflow should be manage by community

[GitBox]

mik-laj commented on issue #9401:
URL: https://github.com/apache/airflow/issues/9401#issuecomment-649059524


   I looked at these dependencies and I am not worried about most because they 
are based on Open-Source licenses or are used only during development. However, 
these 3 of them are runtime dependencies is based on the proprietary/unknown(?) 
license and have no specific alternative.
   
   > astronomerinc/ap-statsd-exporter:0.11.0
   > astronomerinc/ap-pgbouncer:1.8.1
   > astronomerinc/ap-pgbouncer-exporter:0.5.0-1
   
   Thank Jarek for paying attention to this issue. I know I couldn't use the 
Helm Chart in its current form in any of my work. I am afraid that many members 
of our community would face similar problems if they tried to use it in a 
production environment.
   
   I described my fears on the mailing list in more detail:
   
https://lists.apache.org/thread.html/r9d13709f5a968c849fb57733436d330561e5278a595ddad7c9b96a2b%40%3Cdev.airflow.apache.org%3E
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] houqp commented on pull request #9493: [WIP] generate python, go, javascript, java clients from openapi spec

[GitBox]

houqp commented on pull request #9493:
URL: https://github.com/apache/airflow/pull/9493#issuecomment-649051296


   will replace this PR with change for one language at a time starting with Go.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] houqp closed pull request #9493: [WIP] generate python, go, javascript, java clients from openapi spec

[GitBox]

houqp closed pull request #9493:
URL: https://github.com/apache/airflow/pull/9493


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #9501: Add stats to backport packages

[GitBox]

potiuk commented on pull request #9501:
URL: https://github.com/apache/airflow/pull/9501#issuecomment-649050540


   I refactored it a bit as we had many repeating parts for different types of 
classes. Now a lot of it is common and we also have stats for the classes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request #9501: Add stats to backport packages

[GitBox]

potiuk opened a new pull request #9501:
URL: https://github.com/apache/airflow/pull/9501


   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] houqp commented on issue #9080: Build and publish API clients automatically

[GitBox]

houqp commented on issue #9080:
URL: https://github.com/apache/airflow/issues/9080#issuecomment-649011544


   @mik-laj sure, i will take this ticket.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch v1-10-test updated (238d841 -> 2f17299)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.


omit 238d841  Monitor pods by labels instead of names (#6377)
 new 2f17299  Monitor pods by labels instead of names (#6377)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (238d841)
\
 N -- N -- N   refs/heads/v1-10-test (2f17299)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .github/workflows/ci.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[airflow] 01/01: Monitor pods by labels instead of names (#6377)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 2f1729944688a40f3cc7caa622216bbab05bf258
Author: Daniel Imberman 
AuthorDate: Tue Jun 23 11:49:51 2020 -0700

Monitor pods by labels instead of names (#6377)

* Monitor k8sPodOperator pods by labels

To prevent situations where the scheduler starts a
second k8sPodOperator pod after a restart, we now check
for existing pods using kubernetes labels

* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

Co-authored-by: Kaxil Naik 

* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

Co-authored-by: Kaxil Naik 

* add docs

* Update airflow/kubernetes/pod_launcher.py

Co-authored-by: Kaxil Naik 

Co-authored-by: Daniel Imberman 
Co-authored-by: Kaxil Naik 
(cherry picked from commit 8985df0bfcb5f2b2cd69a21b9814021f9f8ce953)
---
 .github/workflows/ci.yml   |   2 +-
 .../contrib/operators/kubernetes_pod_operator.py   | 300 ++---
 airflow/executors/kubernetes_executor.py   |  58 ++--
 airflow/kubernetes/k8s_model.py|   3 +-
 airflow/kubernetes/pod_generator.py|  50 +++-
 airflow/kubernetes/pod_launcher.py |  40 ++-
 chart/charts/postgresql-6.3.12.tgz | Bin 0 -> 22754 bytes
 kubernetes_tests/test_kubernetes_pod_operator.py   | 185 ++---
 tests/executors/test_kubernetes_executor.py|  20 +-
 tests/kubernetes/models/test_pod.py|   6 +-
 tests/kubernetes/models/test_secret.py |  18 +-
 tests/kubernetes/test_pod_generator.py |  18 +-
 .../kubernetes/operators/test_kubernetes_pod.py|  45 ++--
 13 files changed, 527 insertions(+), 218 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index ee799dd..fb16aaf 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -132,7 +132,7 @@ jobs:
   - uses: actions/checkout@master
   - uses: actions/setup-python@v1
 with:
-  python-version: '3.x'
+  python-version: '3.6'
   - name: "Free space"
 run: ./scripts/ci/ci_free_space_on_ci.sh
   - name: "Build PROD image ${{ matrix.python-version }}"
diff --git a/airflow/contrib/operators/kubernetes_pod_operator.py 
b/airflow/contrib/operators/kubernetes_pod_operator.py
index b89a37f..8adb131 100644
--- a/airflow/contrib/operators/kubernetes_pod_operator.py
+++ b/airflow/contrib/operators/kubernetes_pod_operator.py
@@ -15,7 +15,6 @@
 # specific language governing permissions and limitations
 # under the License.
 """Executes task in a Kubernetes POD"""
-import warnings
 
 import re
 
@@ -80,6 +79,12 @@ class KubernetesPodOperator(BaseOperator):  # pylint: 
disable=too-many-instance-
 :param cluster_context: context that points to kubernetes cluster.
 Ignored when in_cluster is True. If None, current-context is used.
 :type cluster_context: str
+:param reattach_on_restart: if the scheduler dies while the pod is 
running, reattach and monitor
+:type reattach_on_restart: bool
+:param labels: labels to apply to the Pod.
+:type labels: dict
+:param startup_timeout_seconds: timeout in seconds to startup the pod.
+:type startup_timeout_seconds: int
 :param get_logs: get the stdout of the container as logs of the tasks.
 :type get_logs: bool
 :param annotations: non-identifying metadata you can attach to the Pod.
@@ -126,90 +131,11 @@ class KubernetesPodOperator(BaseOperator):  # pylint: 
disable=too-many-instance-
 """
 template_fields = ('cmds', 'arguments', 'env_vars', 'config_file')
 
-def execute(self, context):
-try:
-client = kube_client.get_kube_client(in_cluster=self.in_cluster,
- 
cluster_context=self.cluster_context,
- config_file=self.config_file)
-# Add Airflow Version to the label
-# And a label to identify that pod is launched by 
KubernetesPodOperator
-self.labels.update(
-{
-'airflow_version': airflow_version.replace('+', '-'),
-'kubernetes_pod_operator': 'True',
-}
-)
-
-pod = pod_generator.PodGenerator(
-image=self.image,
-namespace=self.namespace,
-cmds=self.cmds,
-args=self.arguments,
-labels=self.labels,
-name=self.name,
-envs=self.env_vars,
-extract_xcom=self.do_xcom_push,
-image_pull_policy=self.image_pull_policy,
-node_selectors=self.node_selectors,
-priority_clas

[GitHub] [airflow] mik-laj commented on issue #9080: Build and publish API clients automatically

[GitBox]

mik-laj commented on issue #9080:
URL: https://github.com/apache/airflow/issues/9080#issuecomment-649001242


   @houqp  Can you reply here? I can't assign you to a ticket without it.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] albertocalderari commented on issue #8903: BigQueryHook refactor + deterministic BQ Job ID

[GitBox]

albertocalderari commented on issue #8903:
URL: https://github.com/apache/airflow/issues/8903#issuecomment-649000670


   @turbaszek yeah sort of, it’s not as simple, I really rather have a quick 
call, are you on airflow’s slack?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

mik-laj commented on a change in pull request #9497:
URL: https://github.com/apache/airflow/pull/9497#discussion_r445095375



##
File path: airflow/api_connexion/openapi/v1.yaml
##
@@ -1760,6 +1757,9 @@ components:
 value:
   type: string
   readOnly: true
+source:

Review comment:
   I think they can't do anything about this information. They will not 
change their behavior because the information comes from environment variable.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pogson commented on issue #8061: Support DAGS folder being in different location on scheduler and worker.

[GitBox]

pogson commented on issue #8061:
URL: https://github.com/apache/airflow/issues/8061#issuecomment-648958459


   I just encountered this. Strange thing is that a regular scheduled dag is 
sent to the celery worker with the full path... but if you click run and ignore 
all the celery worker get `DAGS_FOLDER/dagname.py`. I'm on 1.10.5 right now.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on issue #9498: Remove adding Operators via plugins

[GitBox]

ashb commented on issue #9498:
URL: https://github.com/apache/airflow/issues/9498#issuecomment-648951691


   Good call. We can adopt something from 
https://www.astronomer.io/guides/airflow-importing-custom-hooks-operators/



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch v1-10-test updated (da9c63e -> 238d841)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.


omit da9c63e  Monitor pods by labels instead of names (#6377)
 new 238d841  Monitor pods by labels instead of names (#6377)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (da9c63e)
\
 N -- N -- N   refs/heads/v1-10-test (238d841)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 tests/kubernetes/models/test_pod.py|  6 --
 tests/kubernetes/models/test_secret.py | 18 +++---
 2 files changed, 15 insertions(+), 9 deletions(-)

[airflow] 01/01: Monitor pods by labels instead of names (#6377)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 238d841a0865205ba9a18eaf2d1202cde0be0477
Author: Daniel Imberman 
AuthorDate: Tue Jun 23 11:49:51 2020 -0700

Monitor pods by labels instead of names (#6377)

* Monitor k8sPodOperator pods by labels

To prevent situations where the scheduler starts a
second k8sPodOperator pod after a restart, we now check
for existing pods using kubernetes labels

* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

Co-authored-by: Kaxil Naik 

* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

Co-authored-by: Kaxil Naik 

* add docs

* Update airflow/kubernetes/pod_launcher.py

Co-authored-by: Kaxil Naik 

Co-authored-by: Daniel Imberman 
Co-authored-by: Kaxil Naik 
(cherry picked from commit 8985df0bfcb5f2b2cd69a21b9814021f9f8ce953)
---
 .../contrib/operators/kubernetes_pod_operator.py   | 300 ++---
 airflow/executors/kubernetes_executor.py   |  58 ++--
 airflow/kubernetes/k8s_model.py|   3 +-
 airflow/kubernetes/pod_generator.py|  50 +++-
 airflow/kubernetes/pod_launcher.py |  40 ++-
 chart/charts/postgresql-6.3.12.tgz | Bin 0 -> 22754 bytes
 kubernetes_tests/test_kubernetes_pod_operator.py   | 185 ++---
 tests/executors/test_kubernetes_executor.py|  20 +-
 tests/kubernetes/models/test_pod.py|   6 +-
 tests/kubernetes/models/test_secret.py |  18 +-
 tests/kubernetes/test_pod_generator.py |  18 +-
 .../kubernetes/operators/test_kubernetes_pod.py|  45 ++--
 12 files changed, 526 insertions(+), 217 deletions(-)

diff --git a/airflow/contrib/operators/kubernetes_pod_operator.py 
b/airflow/contrib/operators/kubernetes_pod_operator.py
index b89a37f..8adb131 100644
--- a/airflow/contrib/operators/kubernetes_pod_operator.py
+++ b/airflow/contrib/operators/kubernetes_pod_operator.py
@@ -15,7 +15,6 @@
 # specific language governing permissions and limitations
 # under the License.
 """Executes task in a Kubernetes POD"""
-import warnings
 
 import re
 
@@ -80,6 +79,12 @@ class KubernetesPodOperator(BaseOperator):  # pylint: 
disable=too-many-instance-
 :param cluster_context: context that points to kubernetes cluster.
 Ignored when in_cluster is True. If None, current-context is used.
 :type cluster_context: str
+:param reattach_on_restart: if the scheduler dies while the pod is 
running, reattach and monitor
+:type reattach_on_restart: bool
+:param labels: labels to apply to the Pod.
+:type labels: dict
+:param startup_timeout_seconds: timeout in seconds to startup the pod.
+:type startup_timeout_seconds: int
 :param get_logs: get the stdout of the container as logs of the tasks.
 :type get_logs: bool
 :param annotations: non-identifying metadata you can attach to the Pod.
@@ -126,90 +131,11 @@ class KubernetesPodOperator(BaseOperator):  # pylint: 
disable=too-many-instance-
 """
 template_fields = ('cmds', 'arguments', 'env_vars', 'config_file')
 
-def execute(self, context):
-try:
-client = kube_client.get_kube_client(in_cluster=self.in_cluster,
- 
cluster_context=self.cluster_context,
- config_file=self.config_file)
-# Add Airflow Version to the label
-# And a label to identify that pod is launched by 
KubernetesPodOperator
-self.labels.update(
-{
-'airflow_version': airflow_version.replace('+', '-'),
-'kubernetes_pod_operator': 'True',
-}
-)
-
-pod = pod_generator.PodGenerator(
-image=self.image,
-namespace=self.namespace,
-cmds=self.cmds,
-args=self.arguments,
-labels=self.labels,
-name=self.name,
-envs=self.env_vars,
-extract_xcom=self.do_xcom_push,
-image_pull_policy=self.image_pull_policy,
-node_selectors=self.node_selectors,
-priority_class_name=self.priority_class_name,
-annotations=self.annotations,
-affinity=self.affinity,
-init_containers=self.init_containers,
-image_pull_secrets=self.image_pull_secrets,
-service_account_name=self.service_account_name,
-hostnetwork=self.hostnetwork,
-tolerations=self.tolerations,
-configmaps=self.configmaps,
-security_context=self.security_context,
-dnspolicy=self.dnspolicy,
-  

[GitHub] [airflow] mik-laj commented on issue #9498: Remove adding Operators via plugins

[GitBox]

mik-laj commented on issue #9498:
URL: https://github.com/apache/airflow/issues/9498#issuecomment-648927474


   First of all, we should prepare documentation that describes how loading 
operators works to increase awareness of the problem. Many people use plugins 
because they know no other alternative.
   https://github.com/apache/airflow/issues/8715



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb opened a new issue #9500: Deprecate adding Operators via plugins

[GitBox]

ashb opened a new issue #9500:
URL: https://github.com/apache/airflow/issues/9500


   This is the companion issue to 
https://github.com/apache/airflow/issues/9498, but for adding a deprecation 
warning to 1.10.12 so we can remove it in Airflow 2.0
   
   
   ```
   from airflow.operators.my_plugin import MyOperator
   ```
   
   can become
   
   ```
   from my_plugin import MyOperator
   ```
   
   In 1.10.12 we should issue a FutureDeprecationWarning in the plugin if it 
includes anything in `operators = []`, and additionally when anything imports 
from `airflow.operators.my_plugin` (this last part may already by happening. We 
should double check and confirm.)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] 01/01: Monitor pods by labels instead of names (#6377)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit da9c63e7f3c03db31140b1df0aea3e2750a5e53d
Author: Daniel Imberman 
AuthorDate: Tue Jun 23 11:49:51 2020 -0700

Monitor pods by labels instead of names (#6377)

* Monitor k8sPodOperator pods by labels

To prevent situations where the scheduler starts a
second k8sPodOperator pod after a restart, we now check
for existing pods using kubernetes labels

* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

Co-authored-by: Kaxil Naik 

* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

Co-authored-by: Kaxil Naik 

* add docs

* Update airflow/kubernetes/pod_launcher.py

Co-authored-by: Kaxil Naik 

Co-authored-by: Daniel Imberman 
Co-authored-by: Kaxil Naik 
(cherry picked from commit 8985df0bfcb5f2b2cd69a21b9814021f9f8ce953)
---
 .../contrib/operators/kubernetes_pod_operator.py   | 300 ++---
 airflow/executors/kubernetes_executor.py   |  58 ++--
 airflow/kubernetes/k8s_model.py|   3 +-
 airflow/kubernetes/pod_generator.py|  50 +++-
 airflow/kubernetes/pod_launcher.py |  40 ++-
 chart/charts/postgresql-6.3.12.tgz | Bin 0 -> 22754 bytes
 kubernetes_tests/test_kubernetes_pod_operator.py   | 185 ++---
 tests/executors/test_kubernetes_executor.py|  20 +-
 tests/kubernetes/test_pod_generator.py |  18 +-
 .../kubernetes/operators/test_kubernetes_pod.py|  45 ++--
 10 files changed, 511 insertions(+), 208 deletions(-)

diff --git a/airflow/contrib/operators/kubernetes_pod_operator.py 
b/airflow/contrib/operators/kubernetes_pod_operator.py
index b89a37f..8adb131 100644
--- a/airflow/contrib/operators/kubernetes_pod_operator.py
+++ b/airflow/contrib/operators/kubernetes_pod_operator.py
@@ -15,7 +15,6 @@
 # specific language governing permissions and limitations
 # under the License.
 """Executes task in a Kubernetes POD"""
-import warnings
 
 import re
 
@@ -80,6 +79,12 @@ class KubernetesPodOperator(BaseOperator):  # pylint: 
disable=too-many-instance-
 :param cluster_context: context that points to kubernetes cluster.
 Ignored when in_cluster is True. If None, current-context is used.
 :type cluster_context: str
+:param reattach_on_restart: if the scheduler dies while the pod is 
running, reattach and monitor
+:type reattach_on_restart: bool
+:param labels: labels to apply to the Pod.
+:type labels: dict
+:param startup_timeout_seconds: timeout in seconds to startup the pod.
+:type startup_timeout_seconds: int
 :param get_logs: get the stdout of the container as logs of the tasks.
 :type get_logs: bool
 :param annotations: non-identifying metadata you can attach to the Pod.
@@ -126,90 +131,11 @@ class KubernetesPodOperator(BaseOperator):  # pylint: 
disable=too-many-instance-
 """
 template_fields = ('cmds', 'arguments', 'env_vars', 'config_file')
 
-def execute(self, context):
-try:
-client = kube_client.get_kube_client(in_cluster=self.in_cluster,
- 
cluster_context=self.cluster_context,
- config_file=self.config_file)
-# Add Airflow Version to the label
-# And a label to identify that pod is launched by 
KubernetesPodOperator
-self.labels.update(
-{
-'airflow_version': airflow_version.replace('+', '-'),
-'kubernetes_pod_operator': 'True',
-}
-)
-
-pod = pod_generator.PodGenerator(
-image=self.image,
-namespace=self.namespace,
-cmds=self.cmds,
-args=self.arguments,
-labels=self.labels,
-name=self.name,
-envs=self.env_vars,
-extract_xcom=self.do_xcom_push,
-image_pull_policy=self.image_pull_policy,
-node_selectors=self.node_selectors,
-priority_class_name=self.priority_class_name,
-annotations=self.annotations,
-affinity=self.affinity,
-init_containers=self.init_containers,
-image_pull_secrets=self.image_pull_secrets,
-service_account_name=self.service_account_name,
-hostnetwork=self.hostnetwork,
-tolerations=self.tolerations,
-configmaps=self.configmaps,
-security_context=self.security_context,
-dnspolicy=self.dnspolicy,
-pod=self.full_pod_spec,
-).gen_pod()
-
-pod = append_to_pod(
-pod,
-

[airflow] branch v1-10-test updated (1c507c8 -> da9c63e)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.


omit 1c507c8  Monitor pods by labels instead of names (#6377)
 new da9c63e  Monitor pods by labels instead of names (#6377)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (1c507c8)
\
 N -- N -- N   refs/heads/v1-10-test (da9c63e)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 tests/executors/test_kubernetes_executor.py | 1 -
 1 file changed, 1 deletion(-)

[GitHub] [airflow] blimmer opened a new pull request #9499: Replace "bail." with "Cancel"

[GitBox]

blimmer opened a new pull request #9499:
URL: https://github.com/apache/airflow/pull/9499


   "Bail" is a word that's unlikely to be widely understood by non-native 
English speakers. "Cancel" is used much more frequently in software interfaces 
and is likely to be more understood.
   
   This PR updates the two user-interface usages of "bail." and replaces them 
with "Cancel"
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9499: Replace "bail." with "Cancel"

[GitBox]

boring-cyborg[bot] commented on pull request #9499:
URL: https://github.com/apache/airflow/pull/9499#issuecomment-648886957


   Congratulations on your first Pull Request and welcome to the Apache Airflow 
community! If you have any issues or are unsure about any anything please check 
our Contribution Guide 
(https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (flake8, pylint and type 
annotations). Our [pre-commits]( 
https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
 will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in 
`docs/` directory). Adding a new operator? Check this short 
[guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst)
 Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze 
environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for 
testing locally, it’s a heavy docker but it ships with a working Airflow and a 
lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get 
the final approval from Committers.
   - Please follow [ASF Code of 
Conduct](https://www.apache.org/foundation/policies/conduct) for all 
communication including (but not limited to) comments on Pull Requests, Mailing 
list and Slack.
   - Be sure to read the [Airflow Coding style]( 
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
   Apache Airflow is a community-driven project and together we are making it 
better 🚀.
   In case of doubts contact the developers at:
   Mailing List: d...@airflow.apache.org
   Slack: https://apache-airflow-slack.herokuapp.com/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch v1-10-test updated (5304352 -> 1c507c8)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.


omit 5304352  Monitor pods by labels instead of names (#6377)
 new 1c507c8  Monitor pods by labels instead of names (#6377)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (5304352)
\
 N -- N -- N   refs/heads/v1-10-test (1c507c8)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 tests/executors/test_kubernetes_executor.py | 3 +++
 tests/kubernetes/test_pod_generator.py  | 4 ++--
 2 files changed, 5 insertions(+), 2 deletions(-)

[airflow] 01/01: Monitor pods by labels instead of names (#6377)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 1c507c8d2ce52b6d23735d06c1a05b8babb79139
Author: Daniel Imberman 
AuthorDate: Tue Jun 23 11:49:51 2020 -0700

Monitor pods by labels instead of names (#6377)

* Monitor k8sPodOperator pods by labels

To prevent situations where the scheduler starts a
second k8sPodOperator pod after a restart, we now check
for existing pods using kubernetes labels

* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

Co-authored-by: Kaxil Naik 

* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

Co-authored-by: Kaxil Naik 

* add docs

* Update airflow/kubernetes/pod_launcher.py

Co-authored-by: Kaxil Naik 

Co-authored-by: Daniel Imberman 
Co-authored-by: Kaxil Naik 
(cherry picked from commit 8985df0bfcb5f2b2cd69a21b9814021f9f8ce953)
---
 .../contrib/operators/kubernetes_pod_operator.py   | 300 ++---
 airflow/executors/kubernetes_executor.py   |  58 ++--
 airflow/kubernetes/k8s_model.py|   3 +-
 airflow/kubernetes/pod_generator.py|  50 +++-
 airflow/kubernetes/pod_launcher.py |  40 ++-
 chart/charts/postgresql-6.3.12.tgz | Bin 0 -> 22754 bytes
 kubernetes_tests/test_kubernetes_pod_operator.py   | 185 ++---
 tests/executors/test_kubernetes_executor.py|  21 +-
 tests/kubernetes/test_pod_generator.py |  18 +-
 .../kubernetes/operators/test_kubernetes_pod.py|  45 ++--
 10 files changed, 512 insertions(+), 208 deletions(-)

diff --git a/airflow/contrib/operators/kubernetes_pod_operator.py 
b/airflow/contrib/operators/kubernetes_pod_operator.py
index b89a37f..8adb131 100644
--- a/airflow/contrib/operators/kubernetes_pod_operator.py
+++ b/airflow/contrib/operators/kubernetes_pod_operator.py
@@ -15,7 +15,6 @@
 # specific language governing permissions and limitations
 # under the License.
 """Executes task in a Kubernetes POD"""
-import warnings
 
 import re
 
@@ -80,6 +79,12 @@ class KubernetesPodOperator(BaseOperator):  # pylint: 
disable=too-many-instance-
 :param cluster_context: context that points to kubernetes cluster.
 Ignored when in_cluster is True. If None, current-context is used.
 :type cluster_context: str
+:param reattach_on_restart: if the scheduler dies while the pod is 
running, reattach and monitor
+:type reattach_on_restart: bool
+:param labels: labels to apply to the Pod.
+:type labels: dict
+:param startup_timeout_seconds: timeout in seconds to startup the pod.
+:type startup_timeout_seconds: int
 :param get_logs: get the stdout of the container as logs of the tasks.
 :type get_logs: bool
 :param annotations: non-identifying metadata you can attach to the Pod.
@@ -126,90 +131,11 @@ class KubernetesPodOperator(BaseOperator):  # pylint: 
disable=too-many-instance-
 """
 template_fields = ('cmds', 'arguments', 'env_vars', 'config_file')
 
-def execute(self, context):
-try:
-client = kube_client.get_kube_client(in_cluster=self.in_cluster,
- 
cluster_context=self.cluster_context,
- config_file=self.config_file)
-# Add Airflow Version to the label
-# And a label to identify that pod is launched by 
KubernetesPodOperator
-self.labels.update(
-{
-'airflow_version': airflow_version.replace('+', '-'),
-'kubernetes_pod_operator': 'True',
-}
-)
-
-pod = pod_generator.PodGenerator(
-image=self.image,
-namespace=self.namespace,
-cmds=self.cmds,
-args=self.arguments,
-labels=self.labels,
-name=self.name,
-envs=self.env_vars,
-extract_xcom=self.do_xcom_push,
-image_pull_policy=self.image_pull_policy,
-node_selectors=self.node_selectors,
-priority_class_name=self.priority_class_name,
-annotations=self.annotations,
-affinity=self.affinity,
-init_containers=self.init_containers,
-image_pull_secrets=self.image_pull_secrets,
-service_account_name=self.service_account_name,
-hostnetwork=self.hostnetwork,
-tolerations=self.tolerations,
-configmaps=self.configmaps,
-security_context=self.security_context,
-dnspolicy=self.dnspolicy,
-pod=self.full_pod_spec,
-).gen_pod()
-
-pod = append_to_pod(
-pod,
-

[GitHub] [airflow] pulsar314 commented on issue #9316: Tasks get failed if capacity of a pool is exceeded

[GitBox]

pulsar314 commented on issue #9316:
URL: https://github.com/apache/airflow/issues/9316#issuecomment-648880188


   There's a patch with fix of scheduler. It is based on 1.10.10 tag
   
   
[0001-Fixes-treatment-of-open-slots-in-scheduler-job.patch.txt](https://github.com/apache/airflow/files/4826240/0001-Fixes-treatment-of-open-slots-in-scheduler-job.patch.txt)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb opened a new issue #9498: Remove adding Operators via plugins

[GitBox]

ashb opened a new issue #9498:
URL: https://github.com/apache/airflow/issues/9498


   In Airflow 2.0 we should remove the ability to add operators and hooks via 
plugins. 
   
   I think we should deprecate adding Operators and Hooks via the Airflow 
plugin mechanism.
   
   I think plugins should be reserved for any mechanism that a plain-ol python 
module import won't work for (which is basically anything that needs to tie 
deeply in to the Webserver or Scheduler process).
   
   To that end I think we should deprecate adding operators via plugins:
   
   ```python
   from airflow.operators.my_plugin import MyOperator
   ```
   
   can become
   
   ```python
   from my_plugin import MyOperator
   ```
   
   with no impact on functionality.
   
   For this not to be a hard/sudden breaking change we should issue a 
deprecation warning for this in 1.10.12 Issue goes here)
   
   
   Discussed here:
   
   
https://lists.apache.org/thread.html/a1453d6a6f113709386b61c68c3f5cd61b258fe78f07811169500fe3%40%3Cdev.airflow.apache.org%3E
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] aneesh-joseph edited a comment on pull request #8777: Add Production Helm chart support

[GitBox]

aneesh-joseph edited a comment on pull request #8777:
URL: https://github.com/apache/airflow/pull/8777#issuecomment-648874370


   > Another issue I am facing is mounting secrets to task pods. Basically I am 
trying to use ssh_key file for ssh operator. I added additional volumes and 
volume mounts for the scheduler. Will this reflect on the task pods as well?
   
   No, I  don't think it will mount them automatically to the worker pods 
created by the KubernetesExecutor. If this is for a specific task, you  could 
probably specify the `volumes` and `volumeMounts` via the `executor_config` 
param of that task. If you want it to be mounted on all worker pods, you may 
have to use a pod mutation hook via 
[airflowLocalSettings](https://github.com/apache/airflow/blob/66e738296a81a80e56457981c3ac93f835200c30/chart/values.yaml#L199)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] aneesh-joseph commented on pull request #8777: Add Production Helm chart support

[GitBox]

aneesh-joseph commented on pull request #8777:
URL: https://github.com/apache/airflow/pull/8777#issuecomment-648874370


   > Another issue I am facing is mounting secrets to task pods. Basically I am 
trying to use ssh_key file for ssh operator. I added additional volumes and 
volume mounts for the scheduler. Will this reflect on the task pods as well?
   
   No, I  don't think it will mount them automatically to the worker pods 
created by the KubernetesExecutor. If this is for a specific task, you  could 
probably specify the `volumes` and `volumeMounts` via the `executor_config` 
param of that task. If you want it to be mounted on all worker pods, you may 
have to use a pod mutation hook via 
[airflowLocalSettings](https://github.com/apache/airflow/blob/master/chart/values.yaml#L199)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb closed pull request #9226: Adding functionality to shuffle HMS connections to facilitate load balancing

[GitBox]

ashb closed pull request #9226:
URL: https://github.com/apache/airflow/pull/9226


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on pull request #9226: Adding functionality to shuffle HMS connections to facilitate load balancing

[GitBox]

ashb commented on pull request #9226:
URL: https://github.com/apache/airflow/pull/9226#issuecomment-648868596


   Closing in favour of #9280



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a change in pull request #9280: Functionality to shuffle HMS connections to be used by HiveMetastoreHook facilitating load balancing

[GitBox]

ashb commented on a change in pull request #9280:
URL: https://github.com/apache/airflow/pull/9280#discussion_r444951860



##
File path: tests/hooks/test_hive_hook.py
##
@@ -390,6 +390,25 @@ def test_table_exists(self):
 self.hook.table_exists(str(random.randint(1, 1)))
 )
 
+def test_check_hms_clients_load_balance(self):
+#   checks if every time HMS Hook is instantiated, it gets to
+#   different HMS server most of the time and not to the same HMS 
server.
+connection_count = {}
+hms_hook = HiveMetastoreHook()
+hms_server_count = len(hms_hook.get_connections('metastore_default'))
+
+if hms_server_count > 2:
+for index in range(2 * hms_server_count):
+conn = HiveMetastoreHook()._find_valid_server().host
+if conn in connection_count:
+if connection_count[conn] >= (2 * hms_server_count) - 1:
+self.assertTrue(1 == 2)
+else:
+connection_count[conn] = connection_count[conn] + 1
+else:
+connection_count[conn] = 1
+self.assertTrue(1 == 1)

Review comment:
   Yeah, that sounds more stable to me.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] VinayGb665 commented on a change in pull request #9477: [WIP] YAML file format in LocalFilesystemBackend

[GitBox]

VinayGb665 commented on a change in pull request #9477:
URL: https://github.com/apache/airflow/pull/9477#discussion_r444938432



##
File path: tests/secrets/test_local_filesystem.py
##
@@ -104,6 +105,19 @@ def test_missing_file(self, mock_exists):
 ):
 local_filesystem.load_variables("a.json")
 
+@parameterized.expand(
+(
+({}, {}),
+({"KEY": "AAA"}, {"KEY": "AAA"}),
+({"KEY_A": "AAA", "KEY_B": "BBB"}, {"KEY_A": "AAA", "KEY_B": 
"BBB"}),

Review comment:
   Added  YAML files for both variables and connection tests





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9331: add log endpoint

[GitBox]

ephraimbuddy commented on a change in pull request #9331:
URL: https://github.com/apache/airflow/pull/9331#discussion_r444904773



##
File path: tests/api_connexion/endpoints/test_log_endpoint.py
##
@@ -14,25 +14,254 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+import copy
+import logging.config
+import os
+import shutil
+import sys
+import tempfile
 import unittest
+from unittest import mock
+from unittest.mock import PropertyMock
 
-import pytest
+from itsdangerous.url_safe import URLSafeSerializer
 
+from airflow import DAG, settings
+from airflow.config_templates.airflow_local_settings import 
DEFAULT_LOGGING_CONFIG
+from airflow.models import DagRun, TaskInstance
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.utils import timezone
+from airflow.utils.session import create_session, provide_session
+from airflow.utils.types import DagRunType
 from airflow.www import app
+from tests.test_utils.config import conf_vars
+from tests.test_utils.db import clear_db_runs
 
 
 class TestGetLog(unittest.TestCase):
+DAG_ID = 'dag_for_testing_log_endpoint'
+DAG_ID_REMOVED = 'removed_dag_for_testing_log_endpoint'
+TASK_ID = 'task_for_testing_log_endpoint'
+TRY_NUMBER = 1
+
 @classmethod
-def setUpClass(cls) -> None:
-super().setUpClass()
-cls.app = app.create_app(testing=True)  # type:ignore
+def setUpClass(cls):
+settings.configure_orm()
+cls.session = settings.Session
+cls.app = app.create_app(testing=True)
 
 def setUp(self) -> None:
-self.client = self.app.test_client()  # type:ignore
+self.default_time = "2020-06-10T20:00:00+00:00"
+self.client = self.app.test_client()
+self.log_dir = tempfile.mkdtemp()
+# Make sure that the configure_logging is not cached
+self.old_modules = dict(sys.modules)
+self._prepare_log_files()
+self._configure_loggers()
+self._prepare_db()
+
+def _create_dagrun(self, session):
+dagrun_model = DagRun(
+dag_id=self.DAG_ID,
+run_id='TEST_DAG_RUN_ID',
+run_type=DagRunType.MANUAL.value,
+execution_date=timezone.parse(self.default_time),
+start_date=timezone.parse(self.default_time),
+external_trigger=True,
+)
+session.add(dagrun_model)
+session.commit()
+
+def _configure_loggers(self):
+# Create a custom logging configuration
+logging_config = copy.deepcopy(DEFAULT_LOGGING_CONFIG)
+logging_config['handlers']['task']['base_log_folder'] = self.log_dir
+
+logging_config['handlers']['task']['filename_template'] = \
+'{{ ti.dag_id }}/{{ ti.task_id }}/' \
+'{{ ts | replace(":", ".") }}/{{ try_number }}.log'
+
+# Write the custom logging configuration to a file
+self.settings_folder = tempfile.mkdtemp()
+settings_file = os.path.join(self.settings_folder, 
"airflow_local_settings.py")
+new_logging_file = "LOGGING_CONFIG = {}".format(logging_config)
+with open(settings_file, 'w') as handle:
+handle.writelines(new_logging_file)
+sys.path.append(self.settings_folder)
+
+with conf_vars({('logging', 'logging_config_class'): 
'airflow_local_settings.LOGGING_CONFIG'}):
+self.app = app.create_app(testing=True)
+self.client = self.app.test_client()
+settings.configure_logging()
+
+def _prepare_db(self):
+dagbag = self.app.dag_bag  # pylint: disable=no-member
+dag = DAG(self.DAG_ID, start_date=timezone.parse(self.default_time))
+dag.sync_to_db()
+dag_removed = DAG(self.DAG_ID_REMOVED, 
start_date=timezone.parse(self.default_time))
+dag_removed.sync_to_db()
+dagbag.bag_dag(dag, parent_dag=dag, root_dag=dag)
+with create_session() as session:
+self.ti = TaskInstance(
+task=DummyOperator(task_id=self.TASK_ID, dag=dag),
+execution_date=timezone.parse(self.default_time)
+)
+self.ti.try_number = 1
+self.ti_removed_dag = TaskInstance(
+task=DummyOperator(task_id=self.TASK_ID, dag=dag_removed),
+execution_date=timezone.parse(self.default_time)
+)
+self.ti_removed_dag.try_number = 1
+
+session.merge(self.ti)
+session.merge(self.ti_removed_dag)
+
+def _prepare_log_files(self):
+dir_path = f"{self.log_dir}/{self.DAG_ID}/{self.TASK_ID}/" \
+   f"{self.default_time.replace(':', '.')}/"
+os.makedirs(dir_path)
+with open(f"{dir_path}/1.log", "w+") as file:
+file.write("Log for testing.")
+file.flush()
 
-@pytest.mark.skip(reason="Not implemented yet")
-def test_should_response_200

[airflow] branch v1-10-test updated (08b8769 -> 5304352)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.


 discard 08b8769  Monitor pods by labels instead of names (#6377)
omit 2324175  Move KubernetesPodOperator into providers package
 add 5304352  Monitor pods by labels instead of names (#6377)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (08b8769)
\
 N -- N -- N   refs/heads/v1-10-test (5304352)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:

[airflow] branch v1-10-test updated (08b8769 -> 5304352)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.


 discard 08b8769  Monitor pods by labels instead of names (#6377)
omit 2324175  Move KubernetesPodOperator into providers package
 add 5304352  Monitor pods by labels instead of names (#6377)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (08b8769)
\
 N -- N -- N   refs/heads/v1-10-test (5304352)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:

[airflow] branch v1-10-test updated (a4cb652 -> 08b8769)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.


omit a4cb652  Monitor pods by labels instead of names (#6377)
 new 08b8769  Monitor pods by labels instead of names (#6377)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (a4cb652)
\
 N -- N -- N   refs/heads/v1-10-test (08b8769)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../contrib/operators/gcp_container_operator.py|   2 +-
 .../contrib/operators/kubernetes_pod_operator.py   | 388 +++-
 airflow/providers/__init__.py  |  18 -
 airflow/providers/cncf/__init__.py |  18 -
 airflow/providers/cncf/kubernetes/__init__.py  |  18 -
 .../cncf/kubernetes/operators/__init__.py  |  18 -
 .../cncf/kubernetes/operators/kubernetes_pod.py| 400 -
 .../operators/test_gcp_container_operator.py   |   6 +-
 8 files changed, 380 insertions(+), 488 deletions(-)
 delete mode 100644 airflow/providers/__init__.py
 delete mode 100644 airflow/providers/cncf/__init__.py
 delete mode 100644 airflow/providers/cncf/kubernetes/__init__.py
 delete mode 100644 airflow/providers/cncf/kubernetes/operators/__init__.py
 delete mode 100644 
airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

[airflow] 01/01: Monitor pods by labels instead of names (#6377)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 08b87690fc2cfc775394e5d036a8bf3b653e9bd9
Author: Daniel Imberman 
AuthorDate: Sat May 16 14:13:58 2020 -0700

Monitor pods by labels instead of names (#6377)

* Monitor k8sPodOperator pods by labels

To prevent situations where the scheduler starts a
second k8sPodOperator pod after a restart, we now check
for existing pods using kubernetes labels

* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

Co-authored-by: Kaxil Naik 

* Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py

Co-authored-by: Kaxil Naik 

* add docs

* Update airflow/kubernetes/pod_launcher.py

Co-authored-by: Kaxil Naik 

Co-authored-by: Daniel Imberman 
Co-authored-by: Kaxil Naik 
(cherry picked from commit 8985df0bfcb5f2b2cd69a21b9814021f9f8ce953)
---
 .../contrib/operators/kubernetes_pod_operator.py   | 388 -
 airflow/executors/kubernetes_executor.py   |  58 +--
 airflow/kubernetes/k8s_model.py|   3 +-
 airflow/kubernetes/pod_generator.py|  50 ++-
 airflow/kubernetes/pod_launcher.py |  40 ++-
 .../cncf/kubernetes/operators/kubernetes_pod.py| 288 ---
 kubernetes_tests/test_kubernetes_pod_operator.py   | 185 --
 tests/executors/test_kubernetes_executor.py|  18 +-
 tests/kubernetes/test_pod_generator.py |  14 +-
 .../kubernetes/operators/test_kubernetes_pod.py|  45 ++-
 10 files changed, 677 insertions(+), 412 deletions(-)

diff --git a/airflow/contrib/operators/kubernetes_pod_operator.py 
b/airflow/contrib/operators/kubernetes_pod_operator.py
index 382f965..8adb131 100644
--- a/airflow/contrib/operators/kubernetes_pod_operator.py
+++ b/airflow/contrib/operators/kubernetes_pod_operator.py
@@ -15,22 +15,386 @@
 # specific language governing permissions and limitations
 # under the License.
 """Executes task in a Kubernetes POD"""
-import warnings
 
-from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import 
KubernetesPodOperator as K8sPodOp
+import re
 
+from airflow.exceptions import AirflowException
+from airflow.kubernetes import kube_client, pod_generator, pod_launcher
+from airflow.kubernetes.k8s_model import append_to_pod
+from airflow.kubernetes.pod import Resources
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.utils.helpers import validate_key
+from airflow.utils.state import State
+from airflow.version import version as airflow_version
 
-class KubernetesPodOperator(K8sPodOp):
+
+class KubernetesPodOperator(BaseOperator):  # pylint: 
disable=too-many-instance-attributes
 """
-This class is deprecated.
-Please use 
`airflow.providers.cncd.kubernetes.operators.kubernetes_pod.KubernetesPodOperator
+Execute a task in a Kubernetes Pod
+
+.. note::
+If you use `Google Kubernetes Engine 
`__, use
+:class:`~airflow.gcp.operators.kubernetes_engine.GKEPodOperator`, which
+simplifies the authorization process.
+
+:param image: Docker image you wish to launch. Defaults to hub.docker.com,
+but fully qualified URLS will point to custom repositories.
+:type image: str
+:param name: name of the pod in which the task will run, will be used 
(plus a random
+suffix) to generate a pod id (DNS-1123 subdomain, containing only 
[a-z0-9.-]).
+:type name: str
+:param cmds: entrypoint of the container. (templated)
+The docker images's entrypoint is used if this is not provided.
+:type cmds: list[str]
+:param arguments: arguments of the entrypoint. (templated)
+The docker image's CMD is used if this is not provided.
+:type arguments: list[str]
+:param image_pull_policy: Specify a policy to cache or always pull an 
image.
+:type image_pull_policy: str
+:param image_pull_secrets: Any image pull secrets to be given to the pod.
+   If more than one secret is required, provide a
+   comma separated list: secret_a,secret_b
+:type image_pull_secrets: str
+:param ports: ports for launched pod.
+:type ports: list[airflow.kubernetes.pod.Port]
+:param volume_mounts: volumeMounts for launched pod.
+:type volume_mounts: list[airflow.kubernetes.volume_mount.VolumeMount]
+:param volumes: volumes for launched pod. Includes ConfigMaps and 
PersistentVolumes.
+:type volumes: list[airflow.kubernetes.volume.Volume]
+:param labels: labels to apply to the Pod.
+:type labels: dict
+:param startup_timeout_seconds: timeout in seconds to startup the pod.
+:type startup_timeout_seconds: int
+:param n

[GitHub] [airflow] turbaszek commented on pull request #9488: Improve queries number SchedulerJob._process_executor_events

[GitBox]

turbaszek commented on pull request #9488:
URL: https://github.com/apache/airflow/pull/9488#issuecomment-648819429


   @mik-laj would you mind taking a look at the CI? Does 
`TestServeLogs.test_should_serve_file` seem flaky?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Siddharthk edited a comment on pull request #8777: Add Production Helm chart support

[GitBox]

Siddharthk edited a comment on pull request #8777:
URL: https://github.com/apache/airflow/pull/8777#issuecomment-648819272


   > Well I did that too actually but it was with the unofficial helm chart. 
Are you using the official one here?
   
   Yes, using the official one.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Siddharthk commented on pull request #8777: Add Production Helm chart support

[GitBox]

Siddharthk commented on pull request #8777:
URL: https://github.com/apache/airflow/pull/8777#issuecomment-648819272


   > Well I did that too actually but it was with the unofficial helm chart. 
Are you using the official one here?
   Yes, using the official one.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] feluelle commented on a change in pull request #9394: Fix PythonVirtualenvOperator not working with Airflow context

[GitBox]

feluelle commented on a change in pull request #9394:
URL: https://github.com/apache/airflow/pull/9394#discussion_r444889415



##
File path: airflow/operators/python.py
##
@@ -412,140 +427,113 @@ def __init__(  # pylint: disable=too-many-arguments
 templates_exts=templates_exts,
 *args,
 **kwargs)
-self.requirements = requirements or []
+self.requirements = list(requirements or [])
 self.string_args = string_args or []
 self.python_version = python_version
 self.use_dill = use_dill
 self.system_site_packages = system_site_packages
-# check that dill is present if needed
-dill_in_requirements = map(lambda x: x.lower().startswith('dill'),
-   self.requirements)
-if (not system_site_packages) and use_dill and not 
any(dill_in_requirements):
-raise AirflowException('If using dill, dill must be in the 
environment ' +
-   'either via system_site_packages or 
requirements')
-# check that a function is passed, and that it is not a lambda
-if (not isinstance(self.python_callable,
-   types.FunctionType) or 
(self.python_callable.__name__ ==
-   (lambda x: 0).__name__)):
-raise AirflowException('{} only supports functions for 
python_callable arg'.format(
-self.__class__.__name__))
-# check that args are passed iff python major version matches
-if (python_version is not None and
-   str(python_version)[0] != str(sys.version_info[0]) and
-   self._pass_op_args()):
-raise AirflowException("Passing op_args or op_kwargs is not 
supported across "
-   "different Python major versions "
-   "for PythonVirtualenvOperator. "
-   "Please use string_args.")
+if not self.system_site_packages and self.use_dill and 'dill' not in 
self.requirements:
+self.requirements.append('dill')
+self.pickling_library = dill if self.use_dill else pickle
 
 def execute_callable(self):
 with TemporaryDirectory(prefix='venv') as tmp_dir:
 if self.templates_dict:
 self.op_kwargs['templates_dict'] = self.templates_dict
-# generate filenames
+
 input_filename = os.path.join(tmp_dir, 'script.in')
 output_filename = os.path.join(tmp_dir, 'script.out')
 string_args_filename = os.path.join(tmp_dir, 'string_args.txt')
 script_filename = os.path.join(tmp_dir, 'script.py')
 
-# set up virtualenv
-python_bin = 'python' + str(self.python_version) if 
self.python_version else None
 prepare_virtualenv(
 venv_directory=tmp_dir,
-python_bin=python_bin,
+python_bin=f'python{self.python_version}' if 
self.python_version else None,
 system_site_packages=self.system_site_packages,
-requirements=self.requirements,
+requirements=self.requirements
 )
 
 self._write_args(input_filename)
-self._write_script(script_filename)
 self._write_string_args(string_args_filename)
+self._write_script(script_filename)
+
+execute_in_subprocess(cmd=[
+f'{tmp_dir}/bin/python',
+script_filename,
+input_filename,
+output_filename,
+string_args_filename
+])
 
-# execute command in virtualenv
-execute_in_subprocess(
-self._generate_python_cmd(tmp_dir,
-  script_filename,
-  input_filename,
-  output_filename,
-  string_args_filename))
 return self._read_result(output_filename)
 
-def _pass_op_args(self):
-# we should only pass op_args if any are given to us
-return len(self.op_args) + len(self.op_kwargs) > 0
+def _write_args(self, filename):
+if self.op_args or self.op_kwargs:
+if self.op_kwargs:
+# some items from context can't be loaded in virtual env
+self._keep_serializable_op_kwargs()
+with open(filename, 'wb') as file:
+self.pickling_library.dump({'args': self.op_args, 'kwargs': 
self.op_kwargs}, file)
+
+def _keep_serializable_op_kwargs(self):
+# Remove unserializable objects
+# otherwise "KeyError: 'Variable __getstate__ does not exist'" would 
be raised.
+self.op_kwargs.pop('var', None)
+# otherwise "TypeError: cannot serialize '_io.FileIO' object" would be 
raised.
+

[GitHub] [airflow] marclamberti commented on pull request #8777: Add Production Helm chart support

[GitBox]

marclamberti commented on pull request #8777:
URL: https://github.com/apache/airflow/pull/8777#issuecomment-648815552


   Well I did that too actually but it was with the unofficial helm chart. Are 
you using the official one here?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Siddharthk commented on pull request #8777: Add Production Helm chart support

[GitBox]

Siddharthk commented on pull request #8777:
URL: https://github.com/apache/airflow/pull/8777#issuecomment-648814376


   Thanks @aneesh-joseph @marclamberti 
   
   I have made webserver as HA for now and configured S3 for logs. For python 
module error, I have created below env for now which worked:
   ```
- name: "AIRFLOW__KUBERNETES__RUN_AS_USER"
  value: "5"
   ```
   Another issue I am facing is mounting secrets to task pods. Basically I am 
trying to use ssh_key file for ssh operator. I added additional volumes and 
volume mounts for the scheduler. Will this reflect on the task pods as well? 
   
   ```
   Task pod:
   airflow@tutorial1sleep300-2f920b4c375b4d54bd15a4535b49d368:/opt/airflow$ ls
   airflow.cfg  config  dags  logs  requirements.txt  unittests.cfg  
webserver_config.py
   
   Scheduler pod:
   airflow@airflow-scheduler-5474b69b67-j5c2c:/opt/airflow$ ls
   airflow.cfg  dags  **id_rsa.pub**  logs  requirements.txt  unittests.cfg  
webserver_config.py
   ```




This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on issue #8903: BigQueryHook refactor + deterministic BQ Job ID

[GitBox]

turbaszek commented on issue #8903:
URL: https://github.com/apache/airflow/issues/8903#issuecomment-648784763


   > @turbaszek In case of downscale or pod dying you'd want to check if the 
job is still running, hence the need of having a job id derived from the task 
name and execution date.
   
   Now I got it. So if we use execution date in job_id like this 
`airflow_task_id_20200623T00+` then in case of failure re-running the 
DAg will make the operator reattach to the existing job. I like it. @edejong 
WDYT?
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

mik-laj commented on a change in pull request #9497:
URL: https://github.com/apache/airflow/pull/9497#discussion_r444846952



##
File path: airflow/api_connexion/endpoints/config_endpoint.py
##
@@ -15,12 +15,41 @@
 # specific language governing permissions and limitations
 # under the License.
 
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at: 
https://github.com/apache/airflow/issues/8136
+from flask import Response, request
 
+from airflow.api_connexion.schemas.config_schema import config_schema
+from airflow.configuration import conf
 
-def get_config():
+
+def get_config() -> Response:
 """
 Get current configuration.
 """
-raise NotImplementedError("Not implemented yet.")
+response_types = ['text/plain', 'application/json']
+content_type = request.accept_mimetypes.best_match(response_types)
+conf_dict = conf.as_dict(display_source=True, display_sensitive=True)
+if content_type == 'text/plain':
+config = ''
+for section, parameters in conf_dict.items():
+config += f'[{section}]\n'
+for key, (value, source) in parameters.items():
+config += f'{key} = {value}  # source: {source}\n'
+else:
+config = {
+'sections': [

Review comment:
   Can you use NamedTuple here?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

mik-laj commented on pull request #9497:
URL: https://github.com/apache/airflow/pull/9497#issuecomment-648780496


   @zikun I agree. That's weird. Let's delete these parameters.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] zikun commented on pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

zikun commented on pull request #9497:
URL: https://github.com/apache/airflow/pull/9497#issuecomment-648779230


   I see there are pagenization parameters:
   
https://github.com/apache/airflow/blob/d531cd6334c0edf24d2d023f09f2a2104bd112bb/airflow/api_connexion/openapi/v1.yaml#L1136-L1138
   But to me it seems werid to have pagenization for config. I prefer to remove 
it. WDYT? @mik-laj 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] zikun opened a new pull request #9497: [WIP] Add read-only Config endpoint

[GitBox]

zikun opened a new pull request #9497:
URL: https://github.com/apache/airflow/pull/9497


   Closes #8136 
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Target Github ISSUE in description if exists
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a change in pull request #9363: Use 'airflow.task' Logger for LocalTaskJob

[GitBox]

ashb commented on a change in pull request #9363:
URL: https://github.com/apache/airflow/pull/9363#discussion_r444833072



##
File path: airflow/task/task_runner/standard_task_runner.py
##
@@ -16,16 +16,17 @@
 # specific language governing permissions and limitations
 # under the License.
 """Standard task runner"""
-
+import logging
 import os
 
 import psutil
 from setproctitle import setproctitle  # pylint: disable=no-name-in-module
 
 from airflow.task.task_runner.base_task_runner import BaseTaskRunner
+from airflow.utils.log.file_task_handler import FileTaskHandler
 from airflow.utils.process_utils import reap_process_group
 
-CAN_FORK = hasattr(os, 'fork')
+CAN_FORK = True

Review comment:
   Don't think you want this change, do you?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj edited a comment on pull request #9493: [WIP] generate python, go, javascript, java clients from openapi spec

[GitBox]

mik-laj edited a comment on pull request #9493:
URL: https://github.com/apache/airflow/pull/9493#issuecomment-648767743


   Can you split these changes into several PRs? Each language has different 
requirements that we will have to consider to generate the perfect client 
library. 
   Here is an example of operations that are done by the Kubernetes team.
   
https://github.com/kubernetes-client/gen/blob/master/openapi/preprocess_spec.py
   I will also have to involve other reviewers who have better language. It 
will be easier if each language will be in a separate PR.
   
   > Do we want to also implement artifact publish for python, javascript and 
java?
   
   We can use Github Action to store development artifacts.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9493: [WIP] generate python, go, javascript, java clients from openapi spec

[GitBox]

mik-laj commented on pull request #9493:
URL: https://github.com/apache/airflow/pull/9493#issuecomment-648767743


   Can you split these changes into several PRs? Each language has different 
requirements that we will have to consider to generate the perfect client 
library. 
   Here is an example of operations that are done by the Kubernetes team.
   
https://github.com/kubernetes-client/gen/blob/master/openapi/preprocess_spec.py
   I will also have to involve other reviewers who have better language. It 
will be easier if each language will be in a separate PR.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] auvipy commented on pull request #9496: update celery and kombu

[GitBox]

auvipy commented on pull request #9496:
URL: https://github.com/apache/airflow/pull/9496#issuecomment-648766414


   this release fixes some regression.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] nullhack commented on issue #9492: DockerOperator support for Nvidia-Docker

[GitBox]

nullhack commented on issue #9492:
URL: https://github.com/apache/airflow/issues/9492#issuecomment-648765141


   DockerOperator uses docker-py internally, so It'll not be possible to think 
about including It until that 
[PR](https://github.com/docker/docker-py/pull/2471) is merged.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] albertocalderari commented on issue #8903: BigQueryHook refactor + deterministic BQ Job ID

[GitBox]

albertocalderari commented on issue #8903:
URL: https://github.com/apache/airflow/issues/8903#issuecomment-648763105


   @turbaszek In case of downscale or pod dying you'd want to check if the job 
is still running, hence the need of having a job id derived from the task name 
and execution date.
   We had several instances where the job is still running and a new one is 
started generating extra costs for no reason.
   
   If you want we can have a call and I can explain better.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #9330: Add read-only Task endpoint

[GitBox]

turbaszek commented on a change in pull request #9330:
URL: https://github.com/apache/airflow/pull/9330#discussion_r444818466



##
File path: tests/cli/commands/test_dag_command.py
##
@@ -59,8 +60,14 @@ class TestCliDags(unittest.TestCase):
 @classmethod
 def setUpClass(cls):
 cls.dagbag = DagBag(include_examples=True)
+DAG.bulk_sync_to_db([d[1] for d in cls.dagbag.dags.items()])

Review comment:
   Done, thanks!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #9330: Add read-only Task endpoint

[GitBox]

turbaszek commented on a change in pull request #9330:
URL: https://github.com/apache/airflow/pull/9330#discussion_r444818356



##
File path: airflow/api_connexion/openapi/v1.yaml
##
@@ -1964,13 +1972,14 @@ components:
 __type: {type: string}
 days: {type: integer}
 seconds: {type: integer}
-microsecond: {type: integer}
+microseconds: {type: integer}
 
 RelativeDelta:
   # TODO: Why we need these fields?
   type: object
   required:
 - __type
+- years

Review comment:
   Done

##
File path: airflow/api_connexion/openapi/v1.yaml
##
@@ -1964,13 +1972,14 @@ components:
 __type: {type: string}

Review comment:
   Done





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] feluelle commented on pull request #9246: Few new functionalities and changes in S3ToRedshiftOperator

[GitBox]

feluelle commented on pull request #9246:
URL: https://github.com/apache/airflow/pull/9246#issuecomment-648755344


   I think also the "truncate" functionality can be done by a different 
PostgresOperator task. WDYT?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #9488: Improve queries number SchedulerJob._process_executor_events

[GitBox]

turbaszek commented on a change in pull request #9488:
URL: https://github.com/apache/airflow/pull/9488#discussion_r444818015



##
File path: airflow/jobs/scheduler_job.py
##
@@ -1476,28 +1476,32 @@ def _process_executor_events(self, simple_dag_bag, 
session=None):
 """
 Respond to executor events.
 """
-# TODO: this shares quite a lot of code with _manage_executor_state
-for key, value in 
self.executor.get_event_buffer(simple_dag_bag.dag_ids).items():
+event_buffer = self.executor.get_event_buffer(simple_dag_bag.dag_ids)
+tis_with_right_state: List[TaskInstanceKeyType] = []
+
+# Report execution
+for key, value in event_buffer.items():
 state, info = value
 dag_id, task_id, execution_date, try_number = key
 self.log.info(
 "Executor reports execution of %s.%s execution_date=%s "
 "exited with status %s for try_number %s",
 dag_id, task_id, execution_date, state, try_number
 )
-if state not in (State.FAILED, State.SUCCESS):
-continue
+if state in (State.FAILED, State.SUCCESS):
+tis_with_right_state.append(key)
 
-# Process finished tasks
-qry = session.query(TI).filter(
-TI.dag_id == dag_id,
-TI.task_id == task_id,
-TI.execution_date == execution_date
-)
-ti = qry.first()
-if not ti:
-self.log.warning("TaskInstance %s went missing from the 
database", ti)
-continue

Review comment:
   This log message is useless as the ti is `None` so I'm not sure if we 
have to keep it





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #9394: Fix PythonVirtualenvOperator not working with Airflow context

[GitBox]

turbaszek commented on a change in pull request #9394:
URL: https://github.com/apache/airflow/pull/9394#discussion_r444816940



##
File path: airflow/operators/python.py
##
@@ -412,140 +427,113 @@ def __init__(  # pylint: disable=too-many-arguments
 templates_exts=templates_exts,
 *args,
 **kwargs)
-self.requirements = requirements or []
+self.requirements = list(requirements or [])
 self.string_args = string_args or []
 self.python_version = python_version
 self.use_dill = use_dill
 self.system_site_packages = system_site_packages
-# check that dill is present if needed
-dill_in_requirements = map(lambda x: x.lower().startswith('dill'),
-   self.requirements)
-if (not system_site_packages) and use_dill and not 
any(dill_in_requirements):
-raise AirflowException('If using dill, dill must be in the 
environment ' +
-   'either via system_site_packages or 
requirements')
-# check that a function is passed, and that it is not a lambda
-if (not isinstance(self.python_callable,
-   types.FunctionType) or 
(self.python_callable.__name__ ==
-   (lambda x: 0).__name__)):
-raise AirflowException('{} only supports functions for 
python_callable arg'.format(
-self.__class__.__name__))
-# check that args are passed iff python major version matches
-if (python_version is not None and
-   str(python_version)[0] != str(sys.version_info[0]) and
-   self._pass_op_args()):
-raise AirflowException("Passing op_args or op_kwargs is not 
supported across "
-   "different Python major versions "
-   "for PythonVirtualenvOperator. "
-   "Please use string_args.")
+if not self.system_site_packages and self.use_dill and 'dill' not in 
self.requirements:
+self.requirements.append('dill')
+self.pickling_library = dill if self.use_dill else pickle
 
 def execute_callable(self):
 with TemporaryDirectory(prefix='venv') as tmp_dir:
 if self.templates_dict:
 self.op_kwargs['templates_dict'] = self.templates_dict
-# generate filenames
+
 input_filename = os.path.join(tmp_dir, 'script.in')
 output_filename = os.path.join(tmp_dir, 'script.out')
 string_args_filename = os.path.join(tmp_dir, 'string_args.txt')
 script_filename = os.path.join(tmp_dir, 'script.py')
 
-# set up virtualenv
-python_bin = 'python' + str(self.python_version) if 
self.python_version else None
 prepare_virtualenv(
 venv_directory=tmp_dir,
-python_bin=python_bin,
+python_bin=f'python{self.python_version}' if 
self.python_version else None,
 system_site_packages=self.system_site_packages,
-requirements=self.requirements,
+requirements=self.requirements
 )
 
 self._write_args(input_filename)
-self._write_script(script_filename)
 self._write_string_args(string_args_filename)
+self._write_script(script_filename)
+
+execute_in_subprocess(cmd=[
+f'{tmp_dir}/bin/python',
+script_filename,
+input_filename,
+output_filename,
+string_args_filename
+])
 
-# execute command in virtualenv
-execute_in_subprocess(
-self._generate_python_cmd(tmp_dir,
-  script_filename,
-  input_filename,
-  output_filename,
-  string_args_filename))
 return self._read_result(output_filename)
 
-def _pass_op_args(self):
-# we should only pass op_args if any are given to us
-return len(self.op_args) + len(self.op_kwargs) > 0
+def _write_args(self, filename):
+if self.op_args or self.op_kwargs:
+if self.op_kwargs:
+# some items from context can't be loaded in virtual env
+self._keep_serializable_op_kwargs()
+with open(filename, 'wb') as file:
+self.pickling_library.dump({'args': self.op_args, 'kwargs': 
self.op_kwargs}, file)
+
+def _keep_serializable_op_kwargs(self):
+# Remove unserializable objects
+# otherwise "KeyError: 'Variable __getstate__ does not exist'" would 
be raised.
+self.op_kwargs.pop('var', None)
+# otherwise "TypeError: cannot serialize '_io.FileIO' object" would be 
raised.

[GitHub] [airflow] turbaszek commented on a change in pull request #9394: Fix PythonVirtualenvOperator not working with Airflow context

[GitBox]

turbaszek commented on a change in pull request #9394:
URL: https://github.com/apache/airflow/pull/9394#discussion_r444816320



##
File path: airflow/utils/python_virtualenv_script.tpl
##
@@ -0,0 +1,23 @@
+import {{ pickling_library }}
+import sys
+
+# Read args
+{% if op_args or op_kwargs %}
+with open(sys.argv[1], "rb") as file:
+arg_dict = {{ pickling_library }}.load(file)
+{% else %}
+arg_dict = {"args": [], "kwargs": {}}
+{% endif %}
+
+# Read string args
+with open(sys.argv[3], "r") as file:
+virtualenv_string_args = list(map(lambda x: x.strip(), list(file)))
+
+# Script
+{{ python_callable_source }}
+res = {{ python_callable }}(*arg_dict["args"], **arg_dict["kwargs"])
+
+# Write output
+with open(sys.argv[2], "wb") as file:
+if res:
+{{ pickling_library }}.dump(res, file)

Review comment:
   Nice!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] feluelle commented on pull request #9246: Few new functionalities and changes in S3ToRedshiftOperator

[GitBox]

feluelle commented on pull request #9246:
URL: https://github.com/apache/airflow/pull/9246#issuecomment-648752515


   > Got it. So I will remove from the PR the functionalities of deleting s3 
file, duplicates checking and executing previous queries, and leave the other 
things. Is that okay?
   
   By different tasks I mean different Airflow tasks - not different PR's.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mikaelfr edited a comment on pull request #9204: Add filtering tasks/task instances by tags

[GitBox]

mikaelfr edited a comment on pull request #9204:
URL: https://github.com/apache/airflow/pull/9204#issuecomment-648749495


   > Please provide screenshots of the new UI functionality.
   
   I edited my original comment to include some.
   
   > I'm not sure storing this against TI is actually the right thing to do -- 
the task tag is a property of the Task, not the TaskInstance. What about 
instead if we just looked at the tags property of `dag.tasks`.
   > 
   > i.e. something like `tasks = [task for dag.tasks if task.has_tag(val)]`
   > 
   > (With the planned "DAG versioning" where we store a different version of 
the serialized dag model when it changes any concern about this being out of 
date with latest definition goes away)
   
   I'm not sure how difficult it would be to extract tasks from serialized dags 
in FAB TaskInstanceModelView and use those tasks to somehow filter task 
instances by tags. What I can gather from my limited experience with FAB, it 
seems that it's not very kind to doing things in ways differing from the norm.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mikaelfr commented on pull request #9204: Add filtering tasks/task instances by tags

[GitBox]

mikaelfr commented on pull request #9204:
URL: https://github.com/apache/airflow/pull/9204#issuecomment-648749495


   > Please provide screenshots of the new UI functionality.
   I edited my original comment to include some.
   
   > I'm not sure storing this against TI is actually the right thing to do -- 
the task tag is a property of the Task, not the TaskInstance. What about 
instead if we just looked at the tags property of `dag.tasks`.
   > 
   > i.e. something like `tasks = [task for dag.tasks if task.has_tag(val)]`
   > 
   > (With the planned "DAG versioning" where we store a different version of 
the serialized dag model when it changes any concern about this being out of 
date with latest definition goes away)
   
   I'm not sure how difficult it would be to extract tasks from serialized dags 
in FAB TaskInstanceModelView and use those tasks to somehow filter task 
instances by tags. What I can gather from my limited experience with FAB, it 
seems that it's not very kind to doing things in ways differing from the norm.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] feluelle commented on a change in pull request #9394: Fix PythonVirtualenvOperator not working with Airflow context

[GitBox]

feluelle commented on a change in pull request #9394:
URL: https://github.com/apache/airflow/pull/9394#discussion_r444811541



##
File path: airflow/operators/python.py
##
@@ -269,140 +281,132 @@ def __init__(  # pylint: disable=too-many-arguments
 templates_exts=templates_exts,
 *args,
 **kwargs)
-self.requirements = requirements or []
+self.requirements = list(requirements or [])
 self.string_args = string_args or []
 self.python_version = python_version
 self.use_dill = use_dill
 self.system_site_packages = system_site_packages
-# check that dill is present if needed
-dill_in_requirements = map(lambda x: x.lower().startswith('dill'),
-   self.requirements)
-if (not system_site_packages) and use_dill and not 
any(dill_in_requirements):
-raise AirflowException('If using dill, dill must be in the 
environment ' +
-   'either via system_site_packages or 
requirements')
-# check that a function is passed, and that it is not a lambda
-if (not isinstance(self.python_callable,
-   types.FunctionType) or 
(self.python_callable.__name__ ==
-   (lambda x: 0).__name__)):
-raise AirflowException('{} only supports functions for 
python_callable arg'.format(
-self.__class__.__name__))
-# check that args are passed iff python major version matches
-if (python_version is not None and
-   str(python_version)[0] != str(sys.version_info[0]) and
-   self._pass_op_args()):
-raise AirflowException("Passing op_args or op_kwargs is not 
supported across "
-   "different Python major versions "
-   "for PythonVirtualenvOperator. "
-   "Please use string_args.")
+if not self.system_site_packages and self.use_dill and 'dill' not in 
self.requirements:
+self.requirements.append('dill')
+self.pickling_library = dill if self.use_dill else pickle
 
 def execute_callable(self):
 with TemporaryDirectory(prefix='venv') as tmp_dir:
 if self.templates_dict:
 self.op_kwargs['templates_dict'] = self.templates_dict
-# generate filenames
+
 input_filename = os.path.join(tmp_dir, 'script.in')
 output_filename = os.path.join(tmp_dir, 'script.out')
 string_args_filename = os.path.join(tmp_dir, 'string_args.txt')
 script_filename = os.path.join(tmp_dir, 'script.py')
 
-# set up virtualenv
-python_bin = 'python' + str(self.python_version) if 
self.python_version else None
 prepare_virtualenv(
 venv_directory=tmp_dir,
-python_bin=python_bin,
+python_bin=f'python{self.python_version}' if 
self.python_version else None,
 system_site_packages=self.system_site_packages,
-requirements=self.requirements,
+requirements=self.requirements
 )
 
 self._write_args(input_filename)
-self._write_script(script_filename)
 self._write_string_args(string_args_filename)
+self._write_script(script_filename)
+
+execute_in_subprocess(cmd=[
+f'{tmp_dir}/bin/python',
+script_filename,
+input_filename,
+output_filename,
+string_args_filename
+])
 
-# execute command in virtualenv
-execute_in_subprocess(
-self._generate_python_cmd(tmp_dir,
-  script_filename,
-  input_filename,
-  output_filename,
-  string_args_filename))
 return self._read_result(output_filename)
 
-def _pass_op_args(self):
-# we should only pass op_args if any are given to us
-return len(self.op_args) + len(self.op_kwargs) > 0
+def _write_args(self, filename):
+if self.op_args or self.op_kwargs:
+if self.op_kwargs:
+# some items from context can't be loaded in virtual env
+self._keep_serializable_op_kwargs()
+print(self.op_kwargs)
+with open(filename, 'wb') as file:
+self.pickling_library.dump({'args': self.op_args, 'kwargs': 
self.op_kwargs}, file)
+
+def _keep_serializable_op_kwargs(self):
+# Remove unserializable objects
+# otherwise "KeyError: 'Variable __getstate__ does not exist'" would 
be raised.
+self.op_kwargs.pop('var', None)
+# otherwise "TypeError: cannot serialize '_io.

[GitHub] [airflow] mik-laj commented on a change in pull request #9331: add log endpoint

[GitBox]

mik-laj commented on a change in pull request #9331:
URL: https://github.com/apache/airflow/pull/9331#discussion_r444799752



##
File path: tests/api_connexion/endpoints/test_log_endpoint.py
##
@@ -14,25 +14,254 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+import copy
+import logging.config
+import os
+import shutil
+import sys
+import tempfile
 import unittest
+from unittest import mock
+from unittest.mock import PropertyMock
 
-import pytest
+from itsdangerous.url_safe import URLSafeSerializer
 
+from airflow import DAG, settings
+from airflow.config_templates.airflow_local_settings import 
DEFAULT_LOGGING_CONFIG
+from airflow.models import DagRun, TaskInstance
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.utils import timezone
+from airflow.utils.session import create_session, provide_session
+from airflow.utils.types import DagRunType
 from airflow.www import app
+from tests.test_utils.config import conf_vars
+from tests.test_utils.db import clear_db_runs
 
 
 class TestGetLog(unittest.TestCase):
+DAG_ID = 'dag_for_testing_log_endpoint'
+DAG_ID_REMOVED = 'removed_dag_for_testing_log_endpoint'
+TASK_ID = 'task_for_testing_log_endpoint'
+TRY_NUMBER = 1
+
 @classmethod
-def setUpClass(cls) -> None:
-super().setUpClass()
-cls.app = app.create_app(testing=True)  # type:ignore
+def setUpClass(cls):
+settings.configure_orm()
+cls.session = settings.Session
+cls.app = app.create_app(testing=True)
 
 def setUp(self) -> None:
-self.client = self.app.test_client()  # type:ignore
+self.default_time = "2020-06-10T20:00:00+00:00"
+self.client = self.app.test_client()
+self.log_dir = tempfile.mkdtemp()
+# Make sure that the configure_logging is not cached
+self.old_modules = dict(sys.modules)
+self._prepare_log_files()
+self._configure_loggers()
+self._prepare_db()
+
+def _create_dagrun(self, session):
+dagrun_model = DagRun(
+dag_id=self.DAG_ID,
+run_id='TEST_DAG_RUN_ID',
+run_type=DagRunType.MANUAL.value,
+execution_date=timezone.parse(self.default_time),
+start_date=timezone.parse(self.default_time),
+external_trigger=True,
+)
+session.add(dagrun_model)
+session.commit()
+
+def _configure_loggers(self):
+# Create a custom logging configuration
+logging_config = copy.deepcopy(DEFAULT_LOGGING_CONFIG)
+logging_config['handlers']['task']['base_log_folder'] = self.log_dir
+
+logging_config['handlers']['task']['filename_template'] = \
+'{{ ti.dag_id }}/{{ ti.task_id }}/' \
+'{{ ts | replace(":", ".") }}/{{ try_number }}.log'
+
+# Write the custom logging configuration to a file
+self.settings_folder = tempfile.mkdtemp()
+settings_file = os.path.join(self.settings_folder, 
"airflow_local_settings.py")
+new_logging_file = "LOGGING_CONFIG = {}".format(logging_config)
+with open(settings_file, 'w') as handle:
+handle.writelines(new_logging_file)
+sys.path.append(self.settings_folder)
+
+with conf_vars({('logging', 'logging_config_class'): 
'airflow_local_settings.LOGGING_CONFIG'}):
+self.app = app.create_app(testing=True)
+self.client = self.app.test_client()
+settings.configure_logging()
+
+def _prepare_db(self):
+dagbag = self.app.dag_bag  # pylint: disable=no-member
+dag = DAG(self.DAG_ID, start_date=timezone.parse(self.default_time))
+dag.sync_to_db()
+dag_removed = DAG(self.DAG_ID_REMOVED, 
start_date=timezone.parse(self.default_time))
+dag_removed.sync_to_db()
+dagbag.bag_dag(dag, parent_dag=dag, root_dag=dag)
+with create_session() as session:
+self.ti = TaskInstance(
+task=DummyOperator(task_id=self.TASK_ID, dag=dag),
+execution_date=timezone.parse(self.default_time)
+)
+self.ti.try_number = 1
+self.ti_removed_dag = TaskInstance(
+task=DummyOperator(task_id=self.TASK_ID, dag=dag_removed),
+execution_date=timezone.parse(self.default_time)
+)
+self.ti_removed_dag.try_number = 1
+
+session.merge(self.ti)
+session.merge(self.ti_removed_dag)
+
+def _prepare_log_files(self):
+dir_path = f"{self.log_dir}/{self.DAG_ID}/{self.TASK_ID}/" \
+   f"{self.default_time.replace(':', '.')}/"
+os.makedirs(dir_path)
+with open(f"{dir_path}/1.log", "w+") as file:
+file.write("Log for testing.")
+file.flush()
 
-@pytest.mark.skip(reason="Not implemented yet")
-def test_should_response_200(self

[airflow] branch master updated (c703ce2 -> 2b61912)

[kamilbregula]

This is an automated email from the ASF dual-hosted git repository.

kamilbregula pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from c703ce2  Move python import path from operationId into 
x-openapi-router-controller (#9495)
 add 2b61912  Add extra links endpoint (#9475)

No new revisions were added by this update.

Summary of changes:
 .../api_connexion/endpoints/extra_link_endpoint.py |  37 +++-
 .../endpoints/test_extra_link_endpoint.py  | 190 -
 tests/cli/commands/test_plugins_command.py |  56 +-
 tests/test_utils/db.py |   7 +-
 .../mock_plugins.py}   |  61 ++-
 5 files changed, 238 insertions(+), 113 deletions(-)
 copy tests/{cli/commands/test_plugins_command.py => 
test_utils/mock_plugins.py} (56%)

[airflow] branch master updated (c703ce2 -> 2b61912)

[kamilbregula]

This is an automated email from the ASF dual-hosted git repository.

kamilbregula pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from c703ce2  Move python import path from operationId into 
x-openapi-router-controller (#9495)
 add 2b61912  Add extra links endpoint (#9475)

No new revisions were added by this update.

Summary of changes:
 .../api_connexion/endpoints/extra_link_endpoint.py |  37 +++-
 .../endpoints/test_extra_link_endpoint.py  | 190 -
 tests/cli/commands/test_plugins_command.py |  56 +-
 tests/test_utils/db.py |   7 +-
 .../mock_plugins.py}   |  61 ++-
 5 files changed, 238 insertions(+), 113 deletions(-)
 copy tests/{cli/commands/test_plugins_command.py => 
test_utils/mock_plugins.py} (56%)

[GitHub] [airflow] mik-laj merged pull request #9475: Add extra links endpoint

[GitBox]

mik-laj merged pull request #9475:
URL: https://github.com/apache/airflow/pull/9475


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj closed issue #8140: API Endpoint - Extra Links

[GitBox]

mik-laj closed issue #8140:
URL: https://github.com/apache/airflow/issues/8140


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vaddisrinivas commented on issue #9474: Airflow support for S3 compatible storages

[GitBox]

vaddisrinivas commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-648724411


   hi @dispensable , will try this and update the ticket asap!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dispensable edited a comment on issue #9474: Airflow support for S3 compatible storages

[GitBox]

dispensable edited a comment on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-648721448


   We are currently using Ceph RGW as our airflow cluster logging backend. It 
works perfect. But setting the connection is a little bit tricky U should not 
set host in the host form field otherwise set host in the `Extra` fields with 
`{"host": "http://YOUR_S3_URL:PORT"}`, just leave the `host`/`port`/`schema` 
form fields blank.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dispensable commented on issue #9474: Airflow support for S3 compatible storages

[GitBox]

dispensable commented on issue #9474:
URL: https://github.com/apache/airflow/issues/9474#issuecomment-648721448


   We are currently using Minio as our airflow cluster logging backend. It 
works perfect. But setting the connection is a little bit tricky U should not 
set host in the host form field otherwise set host in the `Extra` fields with 
`{"host": "http://YOUR_S3_URL:PORT"}`, just leave the `host`/`port`/`schema` 
form fields blank.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] auvipy opened a new pull request #9496: update celery and kombu

[GitBox]

auvipy opened a new pull request #9496:
URL: https://github.com/apache/airflow/pull/9496


   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Target Github ISSUE in description if exists
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #9413: Support airflowignore for plugins

[GitBox]

mik-laj commented on issue #9413:
URL: https://github.com/apache/airflow/issues/9413#issuecomment-648708277


   @j-y-matsubara  This seems like a good place. I assigned you to this ticket. 
:-)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] evgenyshulman commented on pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

[GitBox]

evgenyshulman commented on pull request #8962:
URL: https://github.com/apache/airflow/pull/8962#issuecomment-648701073


   @casassg  great initiative and awesome implementation! 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #9330: Add read-only Task endpoint

[GitBox]

turbaszek commented on a change in pull request #9330:
URL: https://github.com/apache/airflow/pull/9330#discussion_r444756955



##
File path: tests/api_connexion/endpoints/test_dag_endpoint.py
##
@@ -14,35 +14,109 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+import os
 import unittest
+from datetime import datetime
 
 import pytest
 
+from airflow import DAG
+from airflow.models import DagBag
+from airflow.models.serialized_dag import SerializedDagModel
+from airflow.operators.dummy_operator import DummyOperator
 from airflow.www import app
+from tests.test_utils.db import clear_db_dags, clear_db_runs, 
clear_db_serialized_dags
 
 
 class TestDagEndpoint(unittest.TestCase):
+dag_id = "test_dag"
+task_id = "op1"
+
+@staticmethod
+def clean_db():
+clear_db_runs()
+clear_db_dags()
+clear_db_serialized_dags()
+
 @classmethod
 def setUpClass(cls) -> None:
 super().setUpClass()
 cls.app = app.create_app(testing=True)  # type:ignore
+cls.app_serialized = app.create_app(testing=True)  # type:ignore
+
+with DAG(
+cls.dag_id, start_date=datetime(2020, 6, 15), doc_md="details"
+) as dag:
+DummyOperator(task_id=cls.task_id)
+
+cls.dag = dag  # type:ignore
+
+dag_bag = DagBag(os.devnull, include_examples=False)
+dag_bag.dags = {dag.dag_id: dag}
+cls.app.dag_bag = dag_bag  # type:ignore
+
+dag_bag = DagBag(os.devnull, include_examples=False, 
store_serialized_dags=True)
+cls.app_serialized.dag_bag = dag_bag  # type:ignore
 
 def setUp(self) -> None:
+self.clean_db()
 self.client = self.app.test_client()  # type:ignore
+self.client_serialized = self.app_serialized.test_client()  # 
type:ignore

Review comment:
   I was thinking about creating a new app in test with serialization, WDYT?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org