date:20200915

[GitHub] [airflow] mariuszliksza-kinesso commented on issue #10435: Negsignal.SIGKILL error on macOS

2020-09-15 Thread GitBox



mariuszliksza-kinesso commented on issue #10435:
URL: https://github.com/apache/airflow/issues/10435#issuecomment-693199779


   One update: 
   My process does not always work, even with Sequential Executor. There are 
1-2 tasks that usually fail and return code Negsignal.SIGKILL. Please help as I 
can't run this process on my local machine due to that error. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] khyurri commented on issue #8284: Add support for Microsoft Azure Blob Storage in Google Cloud Storage Transfer Service operators

2020-09-15 Thread GitBox



khyurri commented on issue #8284:
URL: https://github.com/apache/airflow/issues/8284#issuecomment-693177910


   @turbaszek 
   Sure, https://github.com/apache/airflow/pull/8818



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on pull request #10951: Add example dag and system test for S3ToGCSOperator

2020-09-15 Thread GitBox



ephraimbuddy commented on pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#issuecomment-693171348


   Hi @potiuk , I can't seem to figure out this pylint error. I will appreciate 
your assistance 🙂. Thanks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] yuqian90 closed pull request #10153: [AIP-34] TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

2020-09-15 Thread GitBox



yuqian90 closed pull request #10153:
URL: https://github.com/apache/airflow/pull/10153


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jhtimmins edited a comment on pull request #10594: Add permissions for stable API

2020-09-15 Thread GitBox



jhtimmins edited a comment on pull request #10594:
URL: https://github.com/apache/airflow/pull/10594#issuecomment-693154312


   Good catch @mik-laj. I've updated the authorization code to properly handle 
cases where a user has access to some dags but not all. The endpoint is also 
updated to only return the proper results. Tests are included as well. 
https://github.com/apache/airflow/pull/10594/files#diff-9982141783adf996be12cffcf84d6fe5R59
   
   @ashb you may be curious as well



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jhtimmins edited a comment on pull request #10594: Add permissions for stable API

2020-09-15 Thread GitBox



jhtimmins edited a comment on pull request #10594:
URL: https://github.com/apache/airflow/pull/10594#issuecomment-693154312


   Good catch @mik-laj. I've updated the authorization code to properly handle 
cases where a user has access to some dags but not all. The endpoint is also 
updated to only return the proper results. 
https://github.com/apache/airflow/pull/10594/files#diff-9982141783adf996be12cffcf84d6fe5R59
   
   @ashb you may be curious as well



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jhtimmins commented on pull request #10594: Add permissions for stable API

2020-09-15 Thread GitBox



jhtimmins commented on pull request #10594:
URL: https://github.com/apache/airflow/pull/10594#issuecomment-693154312


   Good catch @mik-laj. I've updated the authorization code to properly handle 
cases where a user has access to some dags but not all. The endpoint is also 
updated to only return the proper results. 
https://github.com/apache/airflow/pull/10594/files#diff-9982141783adf996be12cffcf84d6fe5R59



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10970: Add environment variables documentation to cli-ref.rst

2020-09-15 Thread GitBox

boring-cyborg[bot] commented on pull request #10970:
URL: https://github.com/apache/airflow/pull/10970#issuecomment-693146665

Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our Contribution Guide
(https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type
annotations). Our [pre-commits](
https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
will help you with that.
- In case of a new feature add useful documentation (in docstrings or in
`docs/` directory). Adding a new operator? Check this short
[guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst)
Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze
environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for
testing locally, it’s a heavy docker but it ships with a working Airflow and a
lot of integrations.
- Be patient and persistent. It might take some time to get a review or get
the final approval from Committers.
- Please follow [ASF Code of
Conduct](https://www.apache.org/foundation/policies/conduct) for all
communication including (but not limited to) comments on Pull Requests, Mailing
list and Slack.
- Be sure to read the [Airflow Coding style](
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it
better 🚀.
In case of doubts contact the developers at:
Mailing List: d...@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] faihegberg opened a new pull request #10970: Add environment variables documentation to cli-ref.rst

2020-09-15 Thread GitBox



faihegberg opened a new pull request #10970:
URL: https://github.com/apache/airflow/pull/10970


   This PR documents environment variables in cli-ref.rst, along with a brief 
description of each. This enables the user to refer to these variables from 
other places in the documentation.
   
   Related: #9414 
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] yuqian90 commented on a change in pull request #10153: [AIP-34] TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

2020-09-15 Thread GitBox



yuqian90 commented on a change in pull request #10153:
URL: https://github.com/apache/airflow/pull/10153#discussion_r489137031



##
File path: tests/utils/test_task_group.py
##
@@ -0,0 +1,529 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pendulum
+import pytest
+
+from airflow.models import DAG
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.utils.task_group import TaskGroup
+from airflow.www.views import dag_edges, task_group_to_dict
+
+EXPECTED_JSON = {
+'id': None,
+'value': {
+'label': None,
+'labelStyle': 'fill:#000;',
+'style': 'fill:CornflowerBlue',
+'rx': 5,
+'ry': 5,
+'clusterLabelPos': 'top',
+},
+'tooltip': '',
+'children': [
+{
+'id': 'group234',
+'value': {
+'label': 'group234',
+'labelStyle': 'fill:#000;',
+'style': 'fill:CornflowerBlue',
+'rx': 5,
+'ry': 5,
+'clusterLabelPos': 'top',
+},
+'tooltip': '',
+'children': [
+{
+'id': 'group234.group34',
+'value': {
+'label': 'group34',
+'labelStyle': 'fill:#000;',
+'style': 'fill:CornflowerBlue',
+'rx': 5,
+'ry': 5,
+'clusterLabelPos': 'top',
+},
+'tooltip': '',
+'children': [
+{
+'id': 'group234.group34.task3',
+'value': {
+'label': 'task3',
+'labelStyle': 'fill:#000;',
+'style': 'fill:#e8f7e4;',
+'rx': 5,
+'ry': 5,
+},
+},
+{
+'id': 'group234.group34.task4',
+'value': {
+'label': 'task4',
+'labelStyle': 'fill:#000;',
+'style': 'fill:#e8f7e4;',
+'rx': 5,
+'ry': 5,
+},
+},
+{
+'id': 'group234.group34.downstream_join_id',
+'value': {
+'label': '',
+'labelStyle': 'fill:#000;',
+'style': 'fill:CornflowerBlue;',
+'shape': 'circle',
+},
+},
+],
+},
+{
+'id': 'group234.task2',
+'value': {
+'label': 'task2',
+'labelStyle': 'fill:#000;',
+'style': 'fill:#e8f7e4;',
+'rx': 5,
+'ry': 5,
+},
+},
+{
+'id': 'group234.upstream_join_id',
+'value': {
+'label': '',
+'labelStyle': 'fill:#000;',
+'style': 'fill:CornflowerBlue;',
+'shape': 'circle',
+},
+},
+],
+},
+{
+'id': 'task1',
+'value': {
+'label': 'task1',
+'labelStyle': 'fill:#000;',
+'style': 'fill:#e8f7e4;',
+'rx': 5,
+'ry': 5,
+},
+},
+{
+'id': 'task5',
+'value': {
+'label': 'task5',
+'labelStyle': 'fill:#000;',
+'style': 'fill:#e8f7e4;',
+'rx': 5,
+'ry': 5,
+},
+},
+],
+}
+
+
+def test

[GitHub] [airflow] dimberman commented on issue #9532: Cannot run KubernetesPodOperator tasks.

2020-09-15 Thread GitBox



dimberman commented on issue #9532:
URL: https://github.com/apache/airflow/issues/9532#issuecomment-693137662


   @tuan-cs wait, what? The KubernetesExecutor can absolutely work with the 
KubernetesPodOperator...



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] tag nightly-master updated (87e7ad9 -> 22c6316)

2020-09-15 Thread github-bot

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


*** WARNING: tag nightly-master was modified! ***

from 87e7ad9  (commit)
  to 22c6316  (commit)
from 87e7ad9  fix bug where multiple volume mounts created (#10915)
 add 791f904  Adds the maintain-heart-rate to quarantine. (#10922)
 add d26830f  Make vrious scripts Google Shell Guide compatible (#10812)
 add f696322  Make Clients Google Shell guide compatible (#10810)
 add b2dc346  Make breeeze-complete Google Shell Guide compatible (#10708)
 add ad4f232  Remove unknown pytest.ini setting (#10923)
 add 9e42a97  Mark task as failed when it fails sending in Celery (#10881)
 add 383a118  Add more type annotations to AWS hooks (#10671)
 add 7b4dc63  Add Formatted Stacktrace for Spelling Error (#10919)
 add e9add79  Fix Failing static tests on Master (#10927)
 add eaa49b2  Fix chain methods for XComArg (#10827)
 add 67402b7  Fix grammar in Bug Report Template (#10936)
 add 2832c95  Stop running Doc tests with Spelling tests (#10938)
 add 4e1f3a6  [AIRFLOW-10645] Add AWS Secrets Manager Hook (#10655)
 add 5d6d5a2  Allow to specify path to kubeconfig in KubernetesHook (#10453)
 add d24b8f6  Add Examples and documentation for pod_template_file feature 
(#10916)
 add aab9599  Fix 'Tasks State' stats  UI Overlap (#10939)
 add b628067  Minor refactor of the login methods in tests.www.test_views 
(#10918)
 add 83ed6bd  Cache for kubernetes tests is updateable (#10945)
 add 9616518  Add CeleryKubernetesExecutor (#10901)
 add 23768f6  Proposal: remove -serviceaccount suffix from KSA names in 
helm chart (#10892)
 add 791a175  Fix the method defaults for _construct_volume (#10948)
 add d43bb75  Remove test dependency from TestApiKerberos (#10950)
 add 14f2763  Fixes retrieval of correct branch in non-master related 
builds (#10912)
 add 45272a8  Github repository can be overridden in command line by Breeze 
(#10943)
 add e4e316a  Fix Kubernetes Executor logs for long dag names (#10942)
 add ce19657  Fix case of GitHub. (#10955)
 add f7da7d9  Fix ExternalTaskMarker serialized fields (#10924)
 add 905cdd5  Add a default for DagModel.default_view (#10897)
 add 22c6316  Fix more docs spellings (#10965)

No new revisions were added by this update.

Summary of changes:
 .github/ISSUE_TEMPLATE/bug_report.md   |   2 +-
 .github/SECURITY.rst   |   2 +-
 .github/workflows/build-images-workflow-run.yml|  36 ++-
 .github/workflows/ci.yml   |   2 +-
 .pre-commit-config.yaml|   4 +-
 BREEZE.rst |   2 +-
 CHANGELOG.txt  |   6 +-
 CI.rst |  10 +-
 CONTRIBUTING.rst   |  16 +-
 IMAGES.rst |  10 +-
 STATIC_CODE_CHECKS.rst |   2 +-
 airflow/config_templates/config.yml|  18 +-
 airflow/config_templates/default_airflow.cfg   |  13 +-
 airflow/example_dags/example_xcomargs.py   |  18 +-
 airflow/executors/celery_executor.py   |   6 +-
 airflow/executors/celery_kubernetes_executor.py| 158 +++
 airflow/executors/executor_loader.py   |  30 ++-
 .../dags_in_image_template.yaml|  84 ++
 .../dags_in_volume_template.yaml   |  81 ++
 .../git_sync_template.yaml | 107 
 airflow/kubernetes/worker_configuration.py |   2 +-
 airflow/models/dag.py  |  15 ++
 airflow/models/taskinstance.py |   2 +-
 airflow/models/xcom_arg.py |  18 +-
 airflow/providers/amazon/aws/hooks/aws_dynamodb.py |  13 +-
 airflow/providers/amazon/aws/hooks/base_aws.py |  36 ++-
 airflow/providers/amazon/aws/hooks/ec2.py  |   5 +-
 airflow/providers/amazon/aws/hooks/emr.py  |  13 +-
 airflow/providers/amazon/aws/hooks/kinesis.py  |   9 +-
 .../providers/amazon/aws/hooks/lambda_function.py  |  15 +-
 airflow/providers/amazon/aws/hooks/logs.py |  19 +-
 airflow/providers/amazon/aws/hooks/redshift.py |   5 +-
 .../providers/amazon/aws/hooks/secrets_manager.py  |  68 +
 airflow/providers/amazon/aws/hooks/sqs.py  |  17 +-
 .../providers/amazon/aws/hooks/step_function.py|   5 +-
 .../providers/cncf/kubernetes/hooks/kubernetes.py  |  55 +++-
 .../dingding/example_dags/example_dingding.py  |   2 +-
 airflow/providers/google/cloud/hooks/bigquery.py   |   2 +-
 .../providers/google/cloud/hooks/datacatalog.py|   2 +-
 airflow/providers/google/cloud/hooks/gdm.py|   2 +-
 .../google/cloud/operators/da

[airflow] tag nightly-master updated (87e7ad9 -> 22c6316)

2020-09-15 Thread github-bot

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


*** WARNING: tag nightly-master was modified! ***

from 87e7ad9  (commit)
  to 22c6316  (commit)
from 87e7ad9  fix bug where multiple volume mounts created (#10915)
 add 791f904  Adds the maintain-heart-rate to quarantine. (#10922)
 add d26830f  Make vrious scripts Google Shell Guide compatible (#10812)
 add f696322  Make Clients Google Shell guide compatible (#10810)
 add b2dc346  Make breeeze-complete Google Shell Guide compatible (#10708)
 add ad4f232  Remove unknown pytest.ini setting (#10923)
 add 9e42a97  Mark task as failed when it fails sending in Celery (#10881)
 add 383a118  Add more type annotations to AWS hooks (#10671)
 add 7b4dc63  Add Formatted Stacktrace for Spelling Error (#10919)
 add e9add79  Fix Failing static tests on Master (#10927)
 add eaa49b2  Fix chain methods for XComArg (#10827)
 add 67402b7  Fix grammar in Bug Report Template (#10936)
 add 2832c95  Stop running Doc tests with Spelling tests (#10938)
 add 4e1f3a6  [AIRFLOW-10645] Add AWS Secrets Manager Hook (#10655)
 add 5d6d5a2  Allow to specify path to kubeconfig in KubernetesHook (#10453)
 add d24b8f6  Add Examples and documentation for pod_template_file feature 
(#10916)
 add aab9599  Fix 'Tasks State' stats  UI Overlap (#10939)
 add b628067  Minor refactor of the login methods in tests.www.test_views 
(#10918)
 add 83ed6bd  Cache for kubernetes tests is updateable (#10945)
 add 9616518  Add CeleryKubernetesExecutor (#10901)
 add 23768f6  Proposal: remove -serviceaccount suffix from KSA names in 
helm chart (#10892)
 add 791a175  Fix the method defaults for _construct_volume (#10948)
 add d43bb75  Remove test dependency from TestApiKerberos (#10950)
 add 14f2763  Fixes retrieval of correct branch in non-master related 
builds (#10912)
 add 45272a8  Github repository can be overridden in command line by Breeze 
(#10943)
 add e4e316a  Fix Kubernetes Executor logs for long dag names (#10942)
 add ce19657  Fix case of GitHub. (#10955)
 add f7da7d9  Fix ExternalTaskMarker serialized fields (#10924)
 add 905cdd5  Add a default for DagModel.default_view (#10897)
 add 22c6316  Fix more docs spellings (#10965)

No new revisions were added by this update.

Summary of changes:
 .github/ISSUE_TEMPLATE/bug_report.md   |   2 +-
 .github/SECURITY.rst   |   2 +-
 .github/workflows/build-images-workflow-run.yml|  36 ++-
 .github/workflows/ci.yml   |   2 +-
 .pre-commit-config.yaml|   4 +-
 BREEZE.rst |   2 +-
 CHANGELOG.txt  |   6 +-
 CI.rst |  10 +-
 CONTRIBUTING.rst   |  16 +-
 IMAGES.rst |  10 +-
 STATIC_CODE_CHECKS.rst |   2 +-
 airflow/config_templates/config.yml|  18 +-
 airflow/config_templates/default_airflow.cfg   |  13 +-
 airflow/example_dags/example_xcomargs.py   |  18 +-
 airflow/executors/celery_executor.py   |   6 +-
 airflow/executors/celery_kubernetes_executor.py| 158 +++
 airflow/executors/executor_loader.py   |  30 ++-
 .../dags_in_image_template.yaml|  84 ++
 .../dags_in_volume_template.yaml   |  81 ++
 .../git_sync_template.yaml | 107 
 airflow/kubernetes/worker_configuration.py |   2 +-
 airflow/models/dag.py  |  15 ++
 airflow/models/taskinstance.py |   2 +-
 airflow/models/xcom_arg.py |  18 +-
 airflow/providers/amazon/aws/hooks/aws_dynamodb.py |  13 +-
 airflow/providers/amazon/aws/hooks/base_aws.py |  36 ++-
 airflow/providers/amazon/aws/hooks/ec2.py  |   5 +-
 airflow/providers/amazon/aws/hooks/emr.py  |  13 +-
 airflow/providers/amazon/aws/hooks/kinesis.py  |   9 +-
 .../providers/amazon/aws/hooks/lambda_function.py  |  15 +-
 airflow/providers/amazon/aws/hooks/logs.py |  19 +-
 airflow/providers/amazon/aws/hooks/redshift.py |   5 +-
 .../providers/amazon/aws/hooks/secrets_manager.py  |  68 +
 airflow/providers/amazon/aws/hooks/sqs.py  |  17 +-
 .../providers/amazon/aws/hooks/step_function.py|   5 +-
 .../providers/cncf/kubernetes/hooks/kubernetes.py  |  55 +++-
 .../dingding/example_dags/example_dingding.py  |   2 +-
 airflow/providers/google/cloud/hooks/bigquery.py   |   2 +-
 .../providers/google/cloud/hooks/datacatalog.py|   2 +-
 airflow/providers/google/cloud/hooks/gdm.py|   2 +-
 .../google/cloud/operators/da

[GitHub] [airflow] tuan-cs closed issue #9532: Cannot run KubernetesPodOperator tasks.

2020-09-15 Thread GitBox



tuan-cs closed issue #9532:
URL: https://github.com/apache/airflow/issues/9532


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] tuan-cs commented on issue #9532: Cannot run KubernetesPodOperator tasks.

2020-09-15 Thread GitBox



tuan-cs commented on issue #9532:
URL: https://github.com/apache/airflow/issues/9532#issuecomment-693123229


   I found the reason is KubernetesExecutor cannot work with 
KubernetesPodOperator.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jaketf opened a new pull request #10969: [WIP] sketch import error alert emails

2020-09-15 Thread GitBox



jaketf opened a new pull request #10969:
URL: https://github.com/apache/airflow/pull/10969


   This PR is a conversation piece that came from pondering how to support 
import error alert emails in #10968.
   This is a feature will be especially useful for admins who use 
`ClusterPolicyViolation` or those who have pesky folks that push bad DAGs 
cluttering the UI with notifications that should be triaged to the on-call w/o 
human intervention.
   
   I believe that the `import_errors` is a sad place in the airflow codebase 
and now might be the time to refactor this.
   I've left TODO comments with questions about how to best keep track of 
import_error emails we have sent (as to not spam the recipients with emails 
every scheduler loop).
   
   It seems to me we can either keep history of import errors in the existing 
table or create a new table (e.g. `import_error_history` which has a bool field 
for email_sent) for this purpose.
   
   cc: @mik-laj 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jaketf opened a new issue #10968: import error alert emails

2020-09-15 Thread GitBox



jaketf opened a new issue #10968:
URL: https://github.com/apache/airflow/issues/10968


   **Description**
   
   Send alert emails on import errors.
   
   **Use case / motivation**
   This is a feature will be especially useful for admins who use 
`ClusterPolicyViolation` or those who have pesky folks that push bad DAGs 
cluttering the UI with notifications that should be triaged to the on-call.
   
   **Related Issues**
   I believe that the `import_errors` is a sad place in the airflow codebase 
and now might be the time to refactor this.
   I have questions about how to best keep track of import error emails we have 
sent (as to not spam the recipients with emails every scheduler loop).
   
   It seems to me we can either keep history of import errors in the existing 
table (this would require a fundamental change to the scheduler logic which 
clears this table on each loop and a change to how the UI uses this table) or 
create a new table (e.g. `import_error_history` which has a bool field for 
email_sent) for this purpose.
   
   I'm curious to hear from community if there are other use cases / 
improvements we should consider in refactoring the import errors.
   
   cc: @mik-laj 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #10035: Security context KubernetesPodOperator

2020-09-15 Thread GitBox



dimberman commented on issue #10035:
URL: https://github.com/apache/airflow/issues/10035#issuecomment-693108795


   @gardnerdev can you try doing this using a pod_template_file or a pod object 
and see if that remedies the issue?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #8275: Environment variable ordering for refrencing secrets with KubernetesPodOperator

2020-09-15 Thread GitBox



dimberman commented on issue #8275:
URL: https://github.com/apache/airflow/issues/8275#issuecomment-693108401


   @Rested It's out in 1.10.12, though some of the legacy stuff will be fixed 
in 1.10.13 :)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] yuqian90 commented on a change in pull request #10930: Introduce TaskMixin

2020-09-15 Thread GitBox



yuqian90 commented on a change in pull request #10930:
URL: https://github.com/apache/airflow/pull/10930#discussion_r489093794



##
File path: airflow/models/baseoperator.py
##
@@ -1146,27 +1115,22 @@ def add_only_new(self, item_set: Set[str], item: str) 
-> None:
 else:
 item_set.add(item)
 
+@property
+def operator(self) -> "BaseOperator":

Review comment:
   One request. Is it possible to make this property a `List[BaseOperator]` 
instead? If we want `TaskGroup` to implement `TaskMixin`, it'll be difficult to 
decide what `operator` should be if it's a singular operator.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jbampton opened a new pull request #10967: Remove redundant parentheses from Python files

2020-09-15 Thread GitBox



jbampton opened a new pull request #10967:
URL: https://github.com/apache/airflow/pull/10967


   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #10966: Remove redundant parentheses in serialized_objects.py

2020-09-15 Thread GitBox



kaxil opened a new pull request #10966:
URL: https://github.com/apache/airflow/pull/10966


   While working on another DAG Serialization issue, I found these redundant 
parentheses 
   
   `elif isinstance(var, (Timezone)):` -> `elif isinstance(var, Timezone):`
   
   
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (905cdd5 -> 22c6316)

2020-09-15 Thread kaxilnaik

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 905cdd5  Add a default for DagModel.default_view (#10897)
 add 22c6316  Fix more docs spellings (#10965)

No new revisions were added by this update.

Summary of changes:
 CHANGELOG.txt   | 2 +-
 airflow/providers/google/cloud/hooks/bigquery.py| 2 +-
 airflow/providers/google/cloud/hooks/datacatalog.py | 2 +-
 airflow/providers/google/cloud/hooks/gdm.py | 2 +-
 airflow/providers/google/cloud/operators/datacatalog.py | 2 +-
 docs/spelling_wordlist.txt  | 4 
 6 files changed, 5 insertions(+), 9 deletions(-)

[airflow] branch master updated (905cdd5 -> 22c6316)

2020-09-15 Thread kaxilnaik

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 905cdd5  Add a default for DagModel.default_view (#10897)
 add 22c6316  Fix more docs spellings (#10965)

No new revisions were added by this update.

Summary of changes:
 CHANGELOG.txt   | 2 +-
 airflow/providers/google/cloud/hooks/bigquery.py| 2 +-
 airflow/providers/google/cloud/hooks/datacatalog.py | 2 +-
 airflow/providers/google/cloud/hooks/gdm.py | 2 +-
 airflow/providers/google/cloud/operators/datacatalog.py | 2 +-
 docs/spelling_wordlist.txt  | 4 
 6 files changed, 5 insertions(+), 9 deletions(-)

[GitHub] [airflow] kaxil merged pull request #10965: Fix more docs spellings

2020-09-15 Thread GitBox



kaxil merged pull request #10965:
URL: https://github.com/apache/airflow/pull/10965


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #10965: Fix more docs spellings

2020-09-15 Thread GitBox



kaxil opened a new pull request #10965:
URL: https://github.com/apache/airflow/pull/10965


   `desecending` -> `descending`
   `Retrives` -> `Retrieves`
   `programatic` -> `programmatic`
   `subproces` -> `subprocess`
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10951: Add example dag and system test for S3ToGCSOperator

2020-09-15 Thread GitBox



ephraimbuddy commented on a change in pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#discussion_r488910813



##
File path: docs/operators-and-hooks-ref.rst
##
@@ -934,11 +934,15 @@ These integrations allow you to copy data from/to Google 
Cloud.
  - :doc:`How to use `
  - :mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3`
 
-   * - `Amazon Simple Storage Service (S3) `__
+   * - Any Data Storage service

Review comment:
   I'm not sure the best way to word this but I see that Google Transfer 
service is not only for the S3. 
https://cloud.google.com/storage-transfer-service.
   
   Also, having it as S3 to GCS transfer will overshadow the S3ToGCSOperator. 
That's the reason I changed it and added the S3 to GCS below for S3ToGCSOperator

##
File path: docs/operators-and-hooks-ref.rst
##
@@ -934,11 +934,15 @@ These integrations allow you to copy data from/to Google 
Cloud.
  - :doc:`How to use `
  - :mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3`
 
-   * - `Amazon Simple Storage Service (S3) `__
+   * - Any Data Storage service

Review comment:
   I have removed this entirely. It's a duplicate. It's covered in google 
cloud operators as Storage Transfer Service and in Cloud Transfer Operators as 
s3 to gcs.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (f7da7d9 -> 905cdd5)

2020-09-15 Thread kaxilnaik

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from f7da7d9  Fix ExternalTaskMarker serialized fields (#10924)
 add 905cdd5  Add a default for DagModel.default_view (#10897)

No new revisions were added by this update.

Summary of changes:
 airflow/models/dag.py   | 15 +++
 airflow/www/templates/airflow/dag.html  |  4 ++--
 airflow/www/templates/airflow/dags.html |  2 +-
 tests/models/test_dag.py| 11 +++
 4 files changed, 29 insertions(+), 3 deletions(-)

[airflow] branch master updated (f7da7d9 -> 905cdd5)

2020-09-15 Thread kaxilnaik

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from f7da7d9  Fix ExternalTaskMarker serialized fields (#10924)
 add 905cdd5  Add a default for DagModel.default_view (#10897)

No new revisions were added by this update.

Summary of changes:
 airflow/models/dag.py   | 15 +++
 airflow/www/templates/airflow/dag.html  |  4 ++--
 airflow/www/templates/airflow/dags.html |  2 +-
 tests/models/test_dag.py| 11 +++
 4 files changed, 29 insertions(+), 3 deletions(-)

[GitHub] [airflow] n-oden closed pull request #10958: Allow customization of navigation bar title text

2020-09-15 Thread GitBox



n-oden closed pull request #10958:
URL: https://github.com/apache/airflow/pull/10958


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: Add a default for DagModel.default_view (#10897)

2020-09-15 Thread kaxilnaik

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 905cdd5  Add a default for DagModel.default_view (#10897)
905cdd5 is described below

commit 905cdd502a541e666a2bf80aa590b07b423e700f
Author: Kaxil Naik 
AuthorDate: Wed Sep 16 00:23:47 2020 +0100

Add a default for DagModel.default_view (#10897)

fixes https://github.com/apache/airflow/issues/10283
---
 airflow/models/dag.py   | 15 +++
 airflow/www/templates/airflow/dag.html  |  4 ++--
 airflow/www/templates/airflow/dags.html |  2 +-
 tests/models/test_dag.py| 11 +++
 4 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/airflow/models/dag.py b/airflow/models/dag.py
index 1374006..eecf6b4 100644
--- a/airflow/models/dag.py
+++ b/airflow/models/dag.py
@@ -1621,6 +1621,13 @@ class DAG(BaseDag, LoggingMixin):
 """
 self.bulk_sync_to_db([self], sync_time, session)
 
+def get_default_view(self):
+"""This is only there for backward compatible jinja2 templates"""
+if self.default_view is None:
+return conf.get('webserver', 'dag_default_view').lower()
+else:
+return self.default_view
+
 @staticmethod
 @provide_session
 def deactivate_unknown_dags(active_dag_ids, session=None):
@@ -1820,6 +1827,14 @@ class DagModel(Base):
 paused_dag_ids = set(paused_dag_id for paused_dag_id, in 
paused_dag_ids)
 return paused_dag_ids
 
+def get_default_view(self) -> str:
+"""
+Get the Default DAG View, returns the default config value if DagModel 
does not
+have a value
+"""
+# This is for backwards-compatibility with old dags that don't have 
None as default_view
+return self.default_view or conf.get('webserver', 
'dag_default_view').lower()
+
 @property
 def safe_dag_id(self):
 return self.dag_id.replace('.', '__dot__')
diff --git a/airflow/www/templates/airflow/dag.html 
b/airflow/www/templates/airflow/dag.html
index bc03cae..c33519c 100644
--- a/airflow/www/templates/airflow/dag.html
+++ b/airflow/www/templates/airflow/dag.html
@@ -92,7 +92,7 @@
   
   
 
-  
+  
 
   
   
@@ -125,7 +125,7 @@
 
 
   
-
+
   Zoom into Sub DAG
 
 
diff --git a/airflow/www/templates/airflow/dags.html 
b/airflow/www/templates/airflow/dags.html
index 3157bdc..f7697fe 100644
--- a/airflow/www/templates/airflow/dags.html
+++ b/airflow/www/templates/airflow/dags.html
@@ -108,7 +108,7 @@
 
 
   
-
   {{ dag.dag_id }}
 
diff --git a/tests/models/test_dag.py b/tests/models/test_dag.py
index a348b71..eac4f1e 100644
--- a/tests/models/test_dag.py
+++ b/tests/models/test_dag.py
@@ -58,9 +58,11 @@ class TestDag(unittest.TestCase):
 
 def setUp(self) -> None:
 clear_db_runs()
+clear_db_dags()
 
 def tearDown(self) -> None:
 clear_db_runs()
+clear_db_dags()
 
 @staticmethod
 def _clean_up(dag_id: str):
@@ -895,6 +897,15 @@ class TestDag(unittest.TestCase):
 self.assertTrue(orm_dag.is_paused)
 session.close()
 
+def test_existing_dag_default_view(self):
+
+with create_session() as session:
+session.add(DagModel(dag_id='dag_default_view_old', 
default_view=None))
+session.commit()
+orm_dag = session.query(DagModel).filter(DagModel.dag_id == 
'dag_default_view_old').one()
+self.assertIsNone(orm_dag.default_view)
+self.assertEqual(orm_dag.get_default_view(), conf.get('webserver', 
'dag_default_view').lower())
+
 def test_dag_is_deactivated_upon_dagfile_deletion(self):
 dag_id = 'old_existing_dag'
 dag_fileloc = "/usr/local/airflow/dags/non_existing_path.py"

[GitHub] [airflow] kaxil merged pull request #10897: Add a default for DagModel.default_view

2020-09-15 Thread GitBox



kaxil merged pull request #10897:
URL: https://github.com/apache/airflow/pull/10897


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil closed issue #10283: Airflow Homepage is not able to render due to TypeError: must be str, not NoneType

2020-09-15 Thread GitBox



kaxil closed issue #10283:
URL: https://github.com/apache/airflow/issues/10283


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #10897: Add a default for DagModel.default_view

2020-09-15 Thread GitBox



mik-laj commented on a change in pull request #10897:
URL: https://github.com/apache/airflow/pull/10897#discussion_r489070805



##
File path: airflow/models/dag.py
##
@@ -1621,6 +1621,13 @@ def sync_to_db(self, sync_time=None, session=None):
 """
 self.bulk_sync_to_db([self], sync_time, session)
 
+def get_default_view(self):
+"""This is only there for backward compatible jinja2 templates"""
+if self.default_view is None:
+return conf.get('webserver', 'dag_default_view').lower()
+else:
+return self.default_view

Review comment:
   Okey. Now I understand the whole change. It looks good.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #10963: KubernetesPodOperator template fix

2020-09-15 Thread GitBox



kaxil commented on a change in pull request #10963:
URL: https://github.com/apache/airflow/pull/10963#discussion_r489057956



##
File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
##
@@ -355,24 +359,12 @@ def _set_name(self, name):
 validate_key(name, max_length=220)
 return re.sub(r'[^a-z0-9.-]+', '-', name.lower())
 
-def create_new_pod_for_operator(self, labels, launcher) -> Tuple[State, 
k8s.V1Pod, Optional[str]]:
+def create_pod_request_obj(self) -> k8s.V1Pod:
 """
-Creates a new pod and monitors for duration of task
+Creates a V1Pod based on user parameters. Note that a `pod` or 
`pod_template_file`
+will supercede all other values.

Review comment:
   ```suggestion
   will supersede all other values.
   ```
   
   Spelling error:
   ```
    Error   2 
   
_api/airflow/providers/cncf/kubernetes/operators/kubernetes_pod/index.rst:190: 
(supercede)  will supercede all other values.
   
   File path: 
_api/airflow/providers/cncf/kubernetes/operators/kubernetes_pod/index.rst
   Incorrect Spelling: 'supercede'
   Line with Error: 'will supercede all other values.'
   Line Number: 190
186 | 
187 |
188 |.. method:: create_pod_request_obj(self)
189 | 
190 |   Creates a V1Pod based on user parameters. Note that a `pod` or 
`pod_template_file`
191 |   will supercede all other values.
192 | 
193 | 
194 | 
195 |
   ==
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pcandoalmeida commented on pull request #10162: Add Airflow UI site_title configuration option

2020-09-15 Thread GitBox



pcandoalmeida commented on pull request #10162:
URL: https://github.com/apache/airflow/pull/10162#issuecomment-693019071


   Hi @mik-laj I've resolved about 16 failing tests in `www/`. I'm having a 
strange Github issue when trying to rebase, but will update the PR soon. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Rested commented on issue #8275: Environment variable ordering for refrencing secrets with KubernetesPodOperator

2020-09-15 Thread GitBox



Rested commented on issue #8275:
URL: https://github.com/apache/airflow/issues/8275#issuecomment-693018138


   Has that been added to a 1.x release @dimberman I remember reading 
https://github.com/apache/airflow/pull/6230#issuecomment-597176715 and assuming 
it would not? The solution I went with in the end was to write a lib adding an 
operator to make pods from deployments which allows me to manage the image 
versions with gitops https://github.com/rekon-oss/airkupofrod



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (ce19657 -> f7da7d9)

2020-09-15 Thread kaxilnaik

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from ce19657  Fix case of GitHub. (#10955)
 add f7da7d9  Fix ExternalTaskMarker serialized fields (#10924)

No new revisions were added by this update.

Summary of changes:
 airflow/models/taskinstance.py |  2 +-
 airflow/sensors/external_task_sensor.py| 16 +++-
 tests/models/test_taskinstance.py  | 18 ++
 tests/sensors/test_external_task_sensor.py | 21 +
 4 files changed, 55 insertions(+), 2 deletions(-)

[airflow] branch master updated: Fix ExternalTaskMarker serialized fields (#10924)

2020-09-15 Thread kaxilnaik

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new f7da7d9  Fix ExternalTaskMarker serialized fields (#10924)
f7da7d9 is described below

commit f7da7d94b4ac6dc59fb50a4f4abba69776aac798
Author: Denis Evseev 
AuthorDate: Wed Sep 16 01:40:41 2020 +0300

Fix ExternalTaskMarker serialized fields (#10924)

Co-authored-by: Denis Evseev 
Co-authored-by: Kaxil Naik 
---
 airflow/models/taskinstance.py |  2 +-
 airflow/sensors/external_task_sensor.py| 16 +++-
 tests/models/test_taskinstance.py  | 18 ++
 tests/sensors/test_external_task_sensor.py | 21 +
 4 files changed, 55 insertions(+), 2 deletions(-)

diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
index 76f3134..5280cb4 100644
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -572,7 +572,7 @@ class TaskInstance(Base, LoggingMixin): # pylint: 
disable=R0902,R0904
 self.run_as_user = task.run_as_user
 self.max_tries = task.retries
 self.executor_config = task.executor_config
-self.operator = task.__class__.__name__
+self.operator = task.task_type
 
 @provide_session
 def clear_xcom_data(self, session=None):
diff --git a/airflow/sensors/external_task_sensor.py 
b/airflow/sensors/external_task_sensor.py
index ab2fb20..365761e 100644
--- a/airflow/sensors/external_task_sensor.py
+++ b/airflow/sensors/external_task_sensor.py
@@ -18,7 +18,7 @@
 
 import datetime
 import os
-from typing import Optional, Union
+from typing import FrozenSet, Optional, Union
 
 from sqlalchemy import func
 
@@ -242,6 +242,9 @@ class ExternalTaskMarker(DummyOperator):
 template_fields = ['external_dag_id', 'external_task_id', 'execution_date']
 ui_color = '#19647e'
 
+# The _serialized_fields are lazily loaded when get_serialized_fields() 
method is called
+__serialized_fields: Optional[FrozenSet[str]] = None
+
 @apply_defaults
 def __init__(self, *,
  external_dag_id,
@@ -262,3 +265,14 @@ class ExternalTaskMarker(DummyOperator):
 if recursion_depth <= 0:
 raise ValueError("recursion_depth should be a positive integer")
 self.recursion_depth = recursion_depth
+
+@classmethod
+def get_serialized_fields(cls):
+"""Serialized ExternalTaskMarker contain exactly these fields + 
templated_fields ."""
+if not cls.__serialized_fields:
+cls.__serialized_fields = frozenset(
+super().get_serialized_fields() | {
+"recursion_depth"
+}
+)
+return cls.__serialized_fields
diff --git a/tests/models/test_taskinstance.py 
b/tests/models/test_taskinstance.py
index 4b30e09..21838e7 100644
--- a/tests/models/test_taskinstance.py
+++ b/tests/models/test_taskinstance.py
@@ -40,6 +40,7 @@ from airflow.operators.dummy_operator import DummyOperator
 from airflow.operators.python import PythonOperator
 from airflow.sensors.base_sensor_operator import BaseSensorOperator
 from airflow.sensors.python import PythonSensor
+from airflow.serialization.serialized_objects import SerializedBaseOperator
 from airflow.stats import Stats
 from airflow.ti_deps.dependencies_deps import REQUEUEABLE_DEPS, RUNNING_DEPS
 from airflow.ti_deps.dependencies_states import RUNNABLE_STATES
@@ -1751,3 +1752,20 @@ class TestRunRawTaskQueriesCount(unittest.TestCase):
 "airflow.models.taskinstance.STORE_SERIALIZED_DAGS", True
 ):
 ti._run_raw_task()
+
+def test_operator_field_with_serialization(self):
+
+dag = DAG('test_queries', start_date=DEFAULT_DATE)
+task = DummyOperator(task_id='op', dag=dag)
+self.assertEqual(task.task_type, 'DummyOperator')
+
+# Verify that ti.operator field renders correctly "without" 
Serialization
+ti = TI(task=task, execution_date=datetime.datetime.now())
+self.assertEqual(ti.operator, "DummyOperator")
+
+serialized_op = SerializedBaseOperator.serialize_operator(task)
+deserialized_op = 
SerializedBaseOperator.deserialize_operator(serialized_op)
+self.assertEqual(deserialized_op.task_type, 'DummyOperator')
+# Verify that ti.operator field renders correctly "with" Serialization
+ser_ti = TI(task=deserialized_op, 
execution_date=datetime.datetime.now())
+self.assertEqual(ser_ti.operator, "DummyOperator")
diff --git a/tests/sensors/test_external_task_sensor.py 
b/tests/sensors/test_external_task_sensor.py
index 8335e7b..a25023f 100644
--- a/tests/sensors/test_external_task_sensor.py
+++ b/tests/sensors/test_external_task_sensor.py
@@ -28,6 +28,7 @@ from airflow.operators.bash import BashOperator
 from airflow.operators.dummy_op

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10924: Fix operator field update for SerializedBaseOperator

2020-09-15 Thread GitBox



boring-cyborg[bot] commented on pull request #10924:
URL: https://github.com/apache/airflow/pull/10924#issuecomment-693016982


   Awesome work, congrats on your first merged pull request!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil merged pull request #10924: Fix operator field update for SerializedBaseOperator

2020-09-15 Thread GitBox



kaxil merged pull request #10924:
URL: https://github.com/apache/airflow/pull/10924


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil closed issue #10794: ExternalTaskMarker don't work with store_serialized_dags

2020-09-15 Thread GitBox



kaxil closed issue #10794:
URL: https://github.com/apache/airflow/issues/10794


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on a change in pull request #10963: K8s operator pod template fix

2020-09-15 Thread GitBox



dimberman commented on a change in pull request #10963:
URL: https://github.com/apache/airflow/pull/10963#discussion_r489019882



##
File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
##
@@ -240,7 +240,9 @@ def __init__(self,  # pylint: 
disable=too-many-arguments,too-many-locals
 self.pod_template_file = pod_template_file
 self.name = self._set_name(name)
 self.termination_grace_period = termination_grace_period
-self.client = None
+self.client: CoreV1Api = None
+
+self.pod: k8s.V1Pod = self.create_pod()

Review comment:
   @kaxil added





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #10963: K8s operator pod template fix

2020-09-15 Thread GitBox



kaxil commented on a change in pull request #10963:
URL: https://github.com/apache/airflow/pull/10963#discussion_r489018068



##
File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
##
@@ -240,7 +240,9 @@ def __init__(self,  # pylint: 
disable=too-many-arguments,too-many-locals
 self.pod_template_file = pod_template_file
 self.name = self._set_name(name)
 self.termination_grace_period = termination_grace_period
-self.client = None
+self.client: CoreV1Api = None
+
+self.pod: k8s.V1Pod = self.create_pod()

Review comment:
   Can you also add a Description to the PR please describing the bug that 
is being fixed





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on a change in pull request #10963: K8s operator pod template fix

2020-09-15 Thread GitBox



dimberman commented on a change in pull request #10963:
URL: https://github.com/apache/airflow/pull/10963#discussion_r489016649



##
File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
##
@@ -240,7 +240,9 @@ def __init__(self,  # pylint: 
disable=too-many-arguments,too-many-locals
 self.pod_template_file = pod_template_file
 self.name = self._set_name(name)
 self.termination_grace_period = termination_grace_period
-self.client = None
+self.client: CoreV1Api = None
+
+self.pod: k8s.V1Pod = self.create_pod()

Review comment:
   @kaxil fixed





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #10897: Add a default for DagModel.default_view

2020-09-15 Thread GitBox



kaxil commented on a change in pull request #10897:
URL: https://github.com/apache/airflow/pull/10897#discussion_r489009722



##
File path: airflow/models/dag.py
##
@@ -1621,6 +1621,13 @@ def sync_to_db(self, sync_time=None, session=None):
 """
 self.bulk_sync_to_db([self], sync_time, session)
 
+def get_default_view(self):
+"""This is only there for backward compatible jinja2 templates"""
+if self.default_view is None:
+return conf.get('webserver', 'dag_default_view').lower()
+else:
+return self.default_view

Review comment:
   @mik-laj Any comments?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #10963: K8s operator pod template fix

2020-09-15 Thread GitBox



kaxil commented on a change in pull request #10963:
URL: https://github.com/apache/airflow/pull/10963#discussion_r489009282



##
File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
##
@@ -240,7 +240,9 @@ def __init__(self,  # pylint: 
disable=too-many-arguments,too-many-locals
 self.pod_template_file = pod_template_file
 self.name = self._set_name(name)
 self.termination_grace_period = termination_grace_period
-self.client = None
+self.client: CoreV1Api = None
+
+self.pod: k8s.V1Pod = self.create_pod()

Review comment:
   This will run `self.create_pod()` whenever `KubernetesPodOperator` is 
initialized, let's keep it None and initialize it separately.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] adamo57 opened a new issue #10964: No response from gunicorn master within 120 seconds After Changing Worker Class

2020-09-15 Thread GitBox



adamo57 opened a new issue #10964:
URL: https://github.com/apache/airflow/issues/10964


   
   
   
   
   **Apache Airflow version**:
   1.10.10
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
   ```
   Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", 
GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", 
BuildDate:"2020-07-15T23:30:39Z", GoVersion:"go1.14.4", Compiler:"gc", 
Platform:"darwin/amd64"}
   Server Version: version.Info{Major:"1", Minor:"16+", 
GitVersion:"v1.16.13-eks-2ba888", 
GitCommit:"2ba888155c7f8093a1bc06e3336333fbdb27b3da", GitTreeState:"clean", 
BuildDate:"2020-07-17T18:48:53Z", GoVersion:"go1.13.9", Compiler:"gc", 
Platform:"linux/amd64"}
   ```
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**:
   AWS using EKS with ELB
   - **OS** (e.g. from /etc/os-release):
   - **Kernel** (e.g. `uname -a`):
   - **Install tools**:
   - **Others**:
   
   **What happened**:
   
   webserver stayed up for the 120ish seconds then crashed.
   
   
   
   **What you expected to happen**:
   
   I expected the webserver to not crash after the 120ish seconds
   
   
   
   I think the issue might be with the gunicorn webserver code not handling the 
other worker classes properly.
   
   **How to reproduce it**:
   
   
   Use the stable/airflow helm chart to install airflow at version 7.6.0
   In values.yaml, set `AIRFLOW__WEBSERVER__WORKER_CLASS: "eventlet"`
   
   **Anything else we need to know**:
   
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #10959: All versions in CI yamls are not hard-coded any more

2020-09-15 Thread GitBox



potiuk commented on pull request #10959:
URL: https://github.com/apache/airflow/pull/10959#issuecomment-693001277


   All versions in the "CI Build" and "Image Build" are now taken from the 
scripts - single source of truth that we have for versions:
   
   Build Image:
   
   ![Screenshot from 2020-09-15 
23-56-29](https://user-images.githubusercontent.com/595491/93269365-20cf1080-f7af-11ea-9461-8542c7896612.png)
   
   CI Image:
   
   
   ![Screenshot from 2020-09-15 
23-51-40](https://user-images.githubusercontent.com/595491/93269009-7a830b00-f7ae-11ea-9d03-e22ee234b123.png)
   
   
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #10924: Fix operator field update for SerializedBaseOperator

2020-09-15 Thread GitBox



kaxil commented on pull request #10924:
URL: https://github.com/apache/airflow/pull/10924#issuecomment-693001303


   I have updated the PR (template_fields were already written and there was a 
bug in last commit -- hence tests were failing) and added some tests too 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] OmairK commented on issue #10020: Re-Add try_number as identify label on a POD when using KubernetesPodOperator

2020-09-15 Thread GitBox



OmairK commented on issue #10020:
URL: https://github.com/apache/airflow/issues/10020#issuecomment-692998441


   > Hi @kaxil @OmairK
   > 
   > I don't think this is the right way to solve this issue.
   > 
   > In the case of pod failures, we WANT to reattach the first time so the 
failure can be reported.
   > 
   > Instead, we should add an annotation to that failed pod of 
"already_checked=True". This way the first time the task re-runs it would 
re-attach and fail, but the second time it would see the annotation and create 
a new pod instead.
   
   Hi @dimberman, should I open a new issue for this implementation? I would 
like to take this issue.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #10963: K8s operator pod template fix

2020-09-15 Thread GitBox



mik-laj commented on a change in pull request #10963:
URL: https://github.com/apache/airflow/pull/10963#discussion_r488973557



##
File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
##
@@ -410,12 +403,32 @@ def create_new_pod_for_operator(self, labels, launcher) 
-> Tuple[State, k8s.V1Po
 self.volumes +  # type: ignore
 self.volume_mounts  # type: ignore
 )
+return pod
 
-self.pod = pod
+def create_new_pod_for_operator(self, labels, launcher) -> Tuple[State, 
k8s.V1Pod, Optional[str]]:
+"""
+Creates a new pod and monitors for duration of task
+
+:param labels: labels used to track pod
+:param launcher: pod launcher that will manage launching and 
monitoring pods
+:return:
+"""
+if not (self.full_pod_spec or self.pod_template_file):
+# Add Airflow Version to the label
+# And a label to identify that pod is launched by 
KubernetesPodOperator
+self.labels.update(
+{
+'airflow_version': airflow_version.replace('+', '-'),
+'kubernetes_pod_operator': 'True',
+}
+)
+self.labels.update(labels)
+self.pod.metadata.labels = self.labels
+pod = self.pod

Review comment:
   Should we remove tthis variable?  This is a bit confusing because in one 
function we refer to the same object in two ways - `pod`, `self.pod`.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #10963: K8s operator pod template fix

2020-09-15 Thread GitBox



mik-laj commented on a change in pull request #10963:
URL: https://github.com/apache/airflow/pull/10963#discussion_r488972693



##
File path: airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
##
@@ -355,24 +359,13 @@ def _set_name(self, name):
 validate_key(name, max_length=220)
 return re.sub(r'[^a-z0-9.-]+', '-', name.lower())
 
-def create_new_pod_for_operator(self, labels, launcher) -> Tuple[State, 
k8s.V1Pod, Optional[str]]:
+def create_pod(self) -> k8s.V1Pod:
 """
-Creates a new pod and monitors for duration of task
+Creates a V1Pod based on user parameters. Note that a `pod` or 
`pod_template_file`
+will supercede all other values.
 
-:param labels: labels used to track pod
-:param launcher: pod launcher that will manage launching and 
monitoring pods
-:return:
+@return:

Review comment:
   ```suggestion
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #8275: Environment variable ordering for refrencing secrets with KubernetesPodOperator

2020-09-15 Thread GitBox



dimberman commented on issue #8275:
URL: https://github.com/apache/airflow/issues/8275#issuecomment-692977054


   @Rested you should be able to get around this by using the `pod` or 
`pod_template_file` parameters.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #9532: Cannot run KubernetesPodOperator tasks.

2020-09-15 Thread GitBox



dimberman commented on issue #9532:
URL: https://github.com/apache/airflow/issues/9532#issuecomment-692975590


   @tuan-cs your volume and volume_mount don't match (your volume is named 
`intput`). Could this be the root of your problem ? can you please post task 
logs?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pcandoalmeida commented on pull request #10162: Add Airflow UI site_title configuration option

2020-09-15 Thread GitBox



pcandoalmeida commented on pull request #10162:
URL: https://github.com/apache/airflow/pull/10162#issuecomment-692975873


   Sure @mik-laj doing that now.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pcandoalmeida commented on pull request #10958: Allow customization of navigation bar title text

2020-09-15 Thread GitBox



pcandoalmeida commented on pull request #10958:
URL: https://github.com/apache/airflow/pull/10958#issuecomment-692975108


   No worries at all @n-oden I'm currently reviewing the latest updates by 
@mik-laj so I'm hoping to get this functionality reviewed and merged in soon, 
I'll give you a shout if for some reason I'm not able to progress it! 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #10162: Add Airflow UI site_title configuration option

2020-09-15 Thread GitBox



mik-laj commented on pull request #10162:
URL: https://github.com/apache/airflow/pull/10162#issuecomment-692973693


   @pcandoalmeida I support this change, but it is helpful if you rebase your 
branch to the newest master and fix all failed tests.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] n-oden commented on pull request #10958: Allow customization of navigation bar title text

2020-09-15 Thread GitBox



n-oden commented on pull request #10958:
URL: https://github.com/apache/airflow/pull/10958#issuecomment-692972732


   oh, sorry, didn't realize that there was another change in-flight with the 
same goal.  my bad for not scrutinizing the open PRs more closely!  
@pcandoalmeida I'm happy to close this and let yours proceed if it's in closer 
to submittable shape which I imagine it probably is. :)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] pcandoalmeida commented on pull request #10162: Add Airflow UI site_title configuration option

2020-09-15 Thread GitBox



pcandoalmeida commented on pull request #10162:
URL: https://github.com/apache/airflow/pull/10162#issuecomment-692970956


   Thanks @mik-laj I'm reviewing your updates now; let me know how you want to 
proceed. Can close this issue if you want to go with the other update. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on pull request #10062: [WIP] Try_number as identifying label on a POD when using KubernetesPodOperator

2020-09-15 Thread GitBox



dimberman commented on pull request #10062:
URL: https://github.com/apache/airflow/pull/10062#issuecomment-692969705


   Hi @kaxil @OmairK
   
   I don't think this is the right way to solve this issue.
   
   In the case of pod failures, we WANT to reattach the first time so the 
failure can be reported.
   
   Instead, we should add an annotation to that failed pod of 
"already_checked=True". This way the first time the task re-runs it would 
re-attach and fail, but the second time it would see the annotation and create 
a new pod instead.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #10020: Re-Add try_number as identify label on a POD when using KubernetesPodOperator

2020-09-15 Thread GitBox



dimberman commented on issue #10020:
URL: https://github.com/apache/airflow/issues/10020#issuecomment-692969606


   Hi @kaxil @OmairK 
   
   I don't think this is the right way to solve this issue.
   
   In the case of pod failures, we WANT to reattach the first time so the 
failure can be reported.
   
   Instead, we should add an annotation to that failed pod of 
"already_checked=True". This way the first time the task re-runs it would 
re-attach and fail, but the second time it would see the annotation and create 
a new pod instead.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file

2020-09-15 Thread GitBox



dimberman commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-692967644


   Also @SilvrDuck heads up that template won't work. The K8sPodOperator only 
takes a pod as a template (it would be bad to have two levels of retry logic so 
we explicitly do not allow jobs). Also the container would need to be named 
"base" for logs to work.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file

2020-09-15 Thread GitBox



dimberman commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-692966901


   https://github.com/apache/airflow/pull/10963 PR for a fix. This should go 
out in 1.10.13.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman opened a new pull request #10963: K8s operator pod template fix

2020-09-15 Thread GitBox



dimberman opened a new pull request #10963:
URL: https://github.com/apache/airflow/pull/10963


   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #10958: Allow customization of navigation bar title text

2020-09-15 Thread GitBox



mik-laj commented on pull request #10958:
URL: https://github.com/apache/airflow/pull/10958#issuecomment-692964406


   Similar: https://github.com/apache/airflow/pull/10162
   CC: @pcandoalmeida 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy opened a new pull request #10962: Fix doc preview error in editor for google operators doc

2020-09-15 Thread GitBox



ephraimbuddy opened a new pull request #10962:
URL: https://github.com/apache/airflow/pull/10962


   When I open the documentation for google operators in my editor, I receive 
an error instead of a preview. 
   Here's the error:
   ```
   No preview available.
   
   
   Error output:
   :34: (SEVERE/4) Problems with "include" directive path: InputError: [Errno 
2] No such file or directory: 
'/howto/operator/google/_partials/prerequisite_tasks.rst'. Exiting due to 
level-4 (SEVERE) system message.
   ```
   This PR addresses it.
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] iseessel opened a new issue #10961: airflow initdb: undefined symbol: Py_GetArgcArgv

2020-09-15 Thread GitBox



iseessel opened a new issue #10961:
URL: https://github.com/apache/airflow/issues/10961


   **Apache Airflow version**: 1.10.12
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl 
version`): N/A
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**: Local Env
   - **OS** (e.g. from /etc/os-release): macOS Catalina, Version 10.15.6
   - **Kernel** (e.g. `uname -a`): Hybrid (XNU)
   - **Install tools**: pip3
   - **Python Version**: Python 3.7.3 (natively installed on macOS Catalina).
   -**Pip Version**: pip 20.2.3
   
   **What happened**:
   
   https://user-images.githubusercontent.com/25669348/93260197-0b39f680-f76f-11ea-8ade-708a8303a3dd.png";>
   
   
   **What you expected to happen**:
   
   Airflow initdb passes. Airflow 1.10.12 should support Python version 3.7.3. 
Solution is to upgrade python version, however, my airflow code is not 
compatible with higher python versions. See 
https://stackoverflow.com/questions/60684146/airflow-initdb-undefined-symbol-py-getargcargv
   
   **How to reproduce it**:
   
   See above. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #10961: airflow initdb: undefined symbol: Py_GetArgcArgv

2020-09-15 Thread GitBox



boring-cyborg[bot] commented on issue #10961:
URL: https://github.com/apache/airflow/issues/10961#issuecomment-692960941


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jbampton opened a new pull request #10960: Fix case of Python.

2020-09-15 Thread GitBox



jbampton opened a new pull request #10960:
URL: https://github.com/apache/airflow/pull/10960


   Changed `python` to `Python`.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] nasir-bhanpuri commented on issue #10435: Negsignal.SIGKILL error on macOS

2020-09-15 Thread GitBox



nasir-bhanpuri commented on issue #10435:
URL: https://github.com/apache/airflow/issues/10435#issuecomment-692949804


   I am running into a similar issue with Airflow 1.10.10 using Docker on macOS.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #10949: Allow CeleryExecutor to "adopt" an orphaned queued or running task

2020-09-15 Thread GitBox



XD-DENG commented on a change in pull request #10949:
URL: https://github.com/apache/airflow/pull/10949#discussion_r488939428



##
File path: 
airflow/migrations/versions/e1a11ece99cc_add_external_executor_id_to_ti.py
##
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Add external executor ID to TI
+
+Revision ID: e1a11ece99cc
+Revises: b247b1e3d1ed
+Create Date: 2020-09-12 08:23:45.698865
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = 'e1a11ece99cc'
+down_revision = 'b247b1e3d1ed'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+"""Apply Add external executor ID to TI"""
+with op.batch_alter_table('task_instance', schema=None) as batch_op:
+batch_op.add_column(sa.Column('external_executor_id', 
sa.String(length=250), nullable=True))

Review comment:
   Get your point. 
   
   IMHO possibly somewhere around 100 would suffice given its nature as `id`. 
But feel free to ignore this one.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a change in pull request #10949: Allow CeleryExecutor to "adopt" an orphaned queued or running task

2020-09-15 Thread GitBox



ashb commented on a change in pull request #10949:
URL: https://github.com/apache/airflow/pull/10949#discussion_r488937065



##
File path: airflow/executors/celery_executor.py
##
@@ -274,6 +330,61 @@ def execute_async(self,
 def terminate(self):
 pass
 
+def try_adopt_task_instances(self, tis: List[TaskInstance]) -> 
List[TaskInstance]:
+# See which if the tis are still alive (or have finished even!)

Review comment:
   Yup, `which of` it should be





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a change in pull request #10949: Allow CeleryExecutor to "adopt" an orphaned queued or running task

2020-09-15 Thread GitBox



ashb commented on a change in pull request #10949:
URL: https://github.com/apache/airflow/pull/10949#discussion_r488937450



##
File path: 
airflow/migrations/versions/e1a11ece99cc_add_external_executor_id_to_ti.py
##
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Add external executor ID to TI
+
+Revision ID: e1a11ece99cc
+Revises: b247b1e3d1ed
+Create Date: 2020-09-12 08:23:45.698865
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = 'e1a11ece99cc'
+down_revision = 'b247b1e3d1ed'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+"""Apply Add external executor ID to TI"""
+with op.batch_alter_table('task_instance', schema=None) as batch_op:
+batch_op.add_column(sa.Column('external_executor_id', 
sa.String(length=250), nullable=True))

Review comment:
   To big for Celery, but this gives a bit more freedom for other executors 
to store things in there.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request #10959: All versions in CI scripts are not hard-coded any more

2020-09-15 Thread GitBox

potiuk opened a new pull request #10959:
URL: https://github.com/apache/airflow/pull/10959

GitHub Actions allow to use `fromJson` method to read arrays
or even more complex json objects into the CI workflow yaml files.

This, connected with set::output commands, allows to read the
list of allowed versions as well as default ones from the
environment variables configured in
./scripts/ci/libraries/initialization.sh

This means that we can have one plece in which versions are
configured. We also need to do it in "breeze-complete" as this is
a standalone script that should not source anything we added
BATS tests to verify if the versions in breeze-complete
correspond with those defined in the initialization.sh

Also we do not limit tests any more in regular PRs now - we run
all combinations of available versions. Our tests run quite a
bit faster now so we should be able to run more complete
matrixes. We can still exclude individual values of the matrixes
if this is too much.

MySQL 8 is disabled from breeze for now. I plan a separate follow
up PR where we will run MySQL 8 tests (they were not run so far)

---
**^ Add meaningful description above**

Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10951: Add example dag and system test for S3ToGCSOperator

2020-09-15 Thread GitBox



ephraimbuddy commented on a change in pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#discussion_r488935277



##
File path: docs/operators-and-hooks-ref.rst
##
@@ -934,11 +934,15 @@ These integrations allow you to copy data from/to Google 
Cloud.
  - :doc:`How to use `
  - :mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3`
 
-   * - `Amazon Simple Storage Service (S3) `__
+   * - Any Data Storage service

Review comment:
   
https://github.com/apache/airflow/pull/10951/files#diff-471d3adab9e55d525a899f46345f3b2fL880-R884
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #10949: Allow CeleryExecutor to "adopt" an orphaned queued or running task

2020-09-15 Thread GitBox



XD-DENG commented on a change in pull request #10949:
URL: https://github.com/apache/airflow/pull/10949#discussion_r488925717



##
File path: airflow/executors/celery_executor.py
##
@@ -274,6 +330,61 @@ def execute_async(self,
 def terminate(self):
 pass
 
+def try_adopt_task_instances(self, tis: List[TaskInstance]) -> 
List[TaskInstance]:
+# See which if the tis are still alive (or have finished even!)
+#
+# Since Celery doesn't store "SENT" state for queued commands (if we 
create an AsyncResult with a made
+# up id it just returns PENDING state for it), we have to store 
Celery's task_id against the TI row to
+# look at in future.
+#
+# This process is not perfect -- we could have sent the task to 
celery, and crashed before we were
+# able to record the AsyncResult.task_id in the TaskInstance table, in 
which case we won't adopt the
+# task (it'll either run and update the TI state, or the scheduler 
will clear and re-queue it. Either
+# way it won't get executed more than once)
+#
+# (If we swapped it around, and generated a task_id for Celery, stored 
that in TI and enqueued that
+# there is also still a race condition where we could generate and 
store the task_id, but die before
+# we managed to enqueue the command. Since neither way is perfect we 
always have to deal with this
+# process not being perfect.)
+
+celery_tasks = {
+ti.external_executor_id: AsyncResult(ti.external_executor_id)
+for ti in tis
+if ti.external_executor_id is not None
+}
+
+if not celery_tasks:
+# Nothing to adopt
+return tis
+
+tis_by_celery_task_id = {ti.external_executor_id: ti for ti in tis}
+
+states_by_celery_task_id = 
self.bulk_state_fetcher.get_many(celery_tasks.values())
+
+not_adopted = [ti for ti in tis if ti.external_executor_id is None]
+adopted = []
+cached_celery_backend = next(iter(celery_tasks.values())).backend
+
+for celery_task_id, (state, info) in states_by_celery_task_id.items():
+ti = tis_by_celery_task_id[celery_task_id]
+result = celery_tasks[celery_task_id]
+result.backend = cached_celery_backend

Review comment:
   Maybe we don't need to have `tis_by_celery_task_id = 
{ti.external_executor_id: ti for ti in tis}` (line 360) separately, instead, we 
can have 
   ```python
   celery_tasks = {
   ti.external_executor_id: (AsyncResult(ti.external_executor_id), ti)
   for ti in tis
   if ti.external_executor_id is not None
   }
   ```
   
   Then lines 369-370 can also be merged into
   ```python
   result, ti = celery_tasks[celery_task_id]
   ```
   
   By doing this, we can avoid traversing `tis` for two times (only one time 
would suffice), also save one query on the dict.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #10949: Allow CeleryExecutor to "adopt" an orphaned queued or running task

2020-09-15 Thread GitBox



XD-DENG commented on a change in pull request #10949:
URL: https://github.com/apache/airflow/pull/10949#discussion_r488925717



##
File path: airflow/executors/celery_executor.py
##
@@ -274,6 +330,61 @@ def execute_async(self,
 def terminate(self):
 pass
 
+def try_adopt_task_instances(self, tis: List[TaskInstance]) -> 
List[TaskInstance]:
+# See which if the tis are still alive (or have finished even!)
+#
+# Since Celery doesn't store "SENT" state for queued commands (if we 
create an AsyncResult with a made
+# up id it just returns PENDING state for it), we have to store 
Celery's task_id against the TI row to
+# look at in future.
+#
+# This process is not perfect -- we could have sent the task to 
celery, and crashed before we were
+# able to record the AsyncResult.task_id in the TaskInstance table, in 
which case we won't adopt the
+# task (it'll either run and update the TI state, or the scheduler 
will clear and re-queue it. Either
+# way it won't get executed more than once)
+#
+# (If we swapped it around, and generated a task_id for Celery, stored 
that in TI and enqueued that
+# there is also still a race condition where we could generate and 
store the task_id, but die before
+# we managed to enqueue the command. Since neither way is perfect we 
always have to deal with this
+# process not being perfect.)
+
+celery_tasks = {
+ti.external_executor_id: AsyncResult(ti.external_executor_id)
+for ti in tis
+if ti.external_executor_id is not None
+}
+
+if not celery_tasks:
+# Nothing to adopt
+return tis
+
+tis_by_celery_task_id = {ti.external_executor_id: ti for ti in tis}
+
+states_by_celery_task_id = 
self.bulk_state_fetcher.get_many(celery_tasks.values())
+
+not_adopted = [ti for ti in tis if ti.external_executor_id is None]
+adopted = []
+cached_celery_backend = next(iter(celery_tasks.values())).backend
+
+for celery_task_id, (state, info) in states_by_celery_task_id.items():
+ti = tis_by_celery_task_id[celery_task_id]
+result = celery_tasks[celery_task_id]
+result.backend = cached_celery_backend

Review comment:
   Maybe we do not need to have `tis_by_celery_task_id = 
{ti.external_executor_id: ti for ti in tis}` (line 360) separately. Instead, we 
can have 
   ```python
   celery_tasks = {
   ti.external_executor_id: (AsyncResult(ti.external_executor_id), ti)
   for ti in tis
   if ti.external_executor_id is not None
   }
   ```
   
   Then lines 369-370 can also be merged into
   ```python
   result, ti = celery_tasks[celery_task_id]
   ```
   
   By doing this, we can avoid traversing `tis` for two times (only one time 
would suffice), also save one query on the dict.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10951: Add example dag and system test for S3ToGCSOperator

2020-09-15 Thread GitBox



ephraimbuddy commented on a change in pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#discussion_r488933951



##
File path: docs/operators-and-hooks-ref.rst
##
@@ -934,11 +934,15 @@ These integrations allow you to copy data from/to Google 
Cloud.
  - :doc:`How to use `
  - :mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3`
 
-   * - `Amazon Simple Storage Service (S3) `__
+   * - Any Data Storage service

Review comment:
   I have removed this entirely. It's a duplicate. It's covered in google 
cloud operators as Storage Transfer Service and in Cloud Transfer Operators as 
s3 to gcs.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #10949: Allow CeleryExecutor to "adopt" an orphaned queued or running task

2020-09-15 Thread GitBox



XD-DENG commented on a change in pull request #10949:
URL: https://github.com/apache/airflow/pull/10949#discussion_r488925717



##
File path: airflow/executors/celery_executor.py
##
@@ -274,6 +330,61 @@ def execute_async(self,
 def terminate(self):
 pass
 
+def try_adopt_task_instances(self, tis: List[TaskInstance]) -> 
List[TaskInstance]:
+# See which if the tis are still alive (or have finished even!)
+#
+# Since Celery doesn't store "SENT" state for queued commands (if we 
create an AsyncResult with a made
+# up id it just returns PENDING state for it), we have to store 
Celery's task_id against the TI row to
+# look at in future.
+#
+# This process is not perfect -- we could have sent the task to 
celery, and crashed before we were
+# able to record the AsyncResult.task_id in the TaskInstance table, in 
which case we won't adopt the
+# task (it'll either run and update the TI state, or the scheduler 
will clear and re-queue it. Either
+# way it won't get executed more than once)
+#
+# (If we swapped it around, and generated a task_id for Celery, stored 
that in TI and enqueued that
+# there is also still a race condition where we could generate and 
store the task_id, but die before
+# we managed to enqueue the command. Since neither way is perfect we 
always have to deal with this
+# process not being perfect.)
+
+celery_tasks = {
+ti.external_executor_id: AsyncResult(ti.external_executor_id)
+for ti in tis
+if ti.external_executor_id is not None
+}
+
+if not celery_tasks:
+# Nothing to adopt
+return tis
+
+tis_by_celery_task_id = {ti.external_executor_id: ti for ti in tis}
+
+states_by_celery_task_id = 
self.bulk_state_fetcher.get_many(celery_tasks.values())
+
+not_adopted = [ti for ti in tis if ti.external_executor_id is None]
+adopted = []
+cached_celery_backend = next(iter(celery_tasks.values())).backend
+
+for celery_task_id, (state, info) in states_by_celery_task_id.items():
+ti = tis_by_celery_task_id[celery_task_id]
+result = celery_tasks[celery_task_id]
+result.backend = cached_celery_backend

Review comment:
   Maybe we don't need to have `tis_by_celery_task_id = 
{ti.external_executor_id: ti for ti in tis}` (line 360) separately, instead, we 
can have 
   ```python
   celery_tasks = {
   ti.external_executor_id: (AsyncResult(ti.external_executor_id), ti)
   for ti in tis
   if ti.external_executor_id is not None
   }
   ```
   
   Then lines 369-370 can also be merged into `result, ti = 
celery_tasks[celery_task_id]`
   
   By doing this, we can avoid traversing `tis` for two times (only one time 
would suffice), also save one query on the dict.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #10949: Allow CeleryExecutor to "adopt" an orphaned queued or running task

2020-09-15 Thread GitBox



XD-DENG commented on a change in pull request #10949:
URL: https://github.com/apache/airflow/pull/10949#discussion_r488932380



##
File path: 
airflow/migrations/versions/e1a11ece99cc_add_external_executor_id_to_ti.py
##
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Add external executor ID to TI
+
+Revision ID: e1a11ece99cc
+Revises: b247b1e3d1ed
+Create Date: 2020-09-12 08:23:45.698865
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = 'e1a11ece99cc'
+down_revision = 'b247b1e3d1ed'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+"""Apply Add external executor ID to TI"""
+with op.batch_alter_table('task_instance', schema=None) as batch_op:
+batch_op.add_column(sa.Column('external_executor_id', 
sa.String(length=250), nullable=True))

Review comment:
   `external_executor_id` is the [Celery task's 
UUID](https://docs.celeryproject.org/en/stable/reference/celery.result.html#celery.result.AsyncResult.id)
 if I'm not wrong. Is 250 too big for it?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #10949: Allow CeleryExecutor to "adopt" an orphaned queued or running task

2020-09-15 Thread GitBox



XD-DENG commented on a change in pull request #10949:
URL: https://github.com/apache/airflow/pull/10949#discussion_r488929900



##
File path: airflow/executors/celery_executor.py
##
@@ -274,6 +330,61 @@ def execute_async(self,
 def terminate(self):
 pass
 
+def try_adopt_task_instances(self, tis: List[TaskInstance]) -> 
List[TaskInstance]:
+# See which if the tis are still alive (or have finished even!)

Review comment:
   `see which if the tis...` may contain a typo?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #10949: Allow CeleryExecutor to "adopt" an orphaned queued or running task

2020-09-15 Thread GitBox



XD-DENG commented on a change in pull request #10949:
URL: https://github.com/apache/airflow/pull/10949#discussion_r488928231



##
File path: airflow/executors/celery_executor.py
##
@@ -274,6 +330,61 @@ def execute_async(self,
 def terminate(self):
 pass
 
+def try_adopt_task_instances(self, tis: List[TaskInstance]) -> 
List[TaskInstance]:
+# See which if the tis are still alive (or have finished even!)
+#
+# Since Celery doesn't store "SENT" state for queued commands (if we 
create an AsyncResult with a made
+# up id it just returns PENDING state for it), we have to store 
Celery's task_id against the TI row to
+# look at in future.
+#
+# This process is not perfect -- we could have sent the task to 
celery, and crashed before we were
+# able to record the AsyncResult.task_id in the TaskInstance table, in 
which case we won't adopt the
+# task (it'll either run and update the TI state, or the scheduler 
will clear and re-queue it. Either
+# way it won't get executed more than once)
+#
+# (If we swapped it around, and generated a task_id for Celery, stored 
that in TI and enqueued that
+# there is also still a race condition where we could generate and 
store the task_id, but die before
+# we managed to enqueue the command. Since neither way is perfect we 
always have to deal with this
+# process not being perfect.)
+
+celery_tasks = {
+ti.external_executor_id: AsyncResult(ti.external_executor_id)
+for ti in tis
+if ti.external_executor_id is not None
+}
+
+if not celery_tasks:
+# Nothing to adopt
+return tis
+
+tis_by_celery_task_id = {ti.external_executor_id: ti for ti in tis}
+
+states_by_celery_task_id = 
self.bulk_state_fetcher.get_many(celery_tasks.values())
+
+not_adopted = [ti for ti in tis if ti.external_executor_id is None]
+adopted = []
+cached_celery_backend = next(iter(celery_tasks.values())).backend
+
+for celery_task_id, (state, info) in states_by_celery_task_id.items():
+ti = tis_by_celery_task_id[celery_task_id]
+result = celery_tasks[celery_task_id]
+result.backend = cached_celery_backend

Review comment:
   In addition, `tis_by_celery_task_id` covers all `tis`, which is not 
necessary, because it will only be used to query `tis` which present in 
`celery_tasks` (essentially means `tis` whose `external_executor_id` is not 
`None`).
   
   So my suggested revision above may save a little memory as well ;-) 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #10949: Allow CeleryExecutor to "adopt" an orphaned queued or running task

2020-09-15 Thread GitBox



XD-DENG commented on a change in pull request #10949:
URL: https://github.com/apache/airflow/pull/10949#discussion_r488925717



##
File path: airflow/executors/celery_executor.py
##
@@ -274,6 +330,61 @@ def execute_async(self,
 def terminate(self):
 pass
 
+def try_adopt_task_instances(self, tis: List[TaskInstance]) -> 
List[TaskInstance]:
+# See which if the tis are still alive (or have finished even!)
+#
+# Since Celery doesn't store "SENT" state for queued commands (if we 
create an AsyncResult with a made
+# up id it just returns PENDING state for it), we have to store 
Celery's task_id against the TI row to
+# look at in future.
+#
+# This process is not perfect -- we could have sent the task to 
celery, and crashed before we were
+# able to record the AsyncResult.task_id in the TaskInstance table, in 
which case we won't adopt the
+# task (it'll either run and update the TI state, or the scheduler 
will clear and re-queue it. Either
+# way it won't get executed more than once)
+#
+# (If we swapped it around, and generated a task_id for Celery, stored 
that in TI and enqueued that
+# there is also still a race condition where we could generate and 
store the task_id, but die before
+# we managed to enqueue the command. Since neither way is perfect we 
always have to deal with this
+# process not being perfect.)
+
+celery_tasks = {
+ti.external_executor_id: AsyncResult(ti.external_executor_id)
+for ti in tis
+if ti.external_executor_id is not None
+}
+
+if not celery_tasks:
+# Nothing to adopt
+return tis
+
+tis_by_celery_task_id = {ti.external_executor_id: ti for ti in tis}
+
+states_by_celery_task_id = 
self.bulk_state_fetcher.get_many(celery_tasks.values())
+
+not_adopted = [ti for ti in tis if ti.external_executor_id is None]
+adopted = []
+cached_celery_backend = next(iter(celery_tasks.values())).backend
+
+for celery_task_id, (state, info) in states_by_celery_task_id.items():
+ti = tis_by_celery_task_id[celery_task_id]
+result = celery_tasks[celery_task_id]
+result.backend = cached_celery_backend

Review comment:
   Maybe we don't need to have `tis_by_celery_task_id = 
{ti.external_executor_id: ti for ti in tis}` (line 360) separately, instead, we 
can have 
   ```python
   celery_tasks = {
   ti.external_executor_id: (AsyncResult(ti.external_executor_id), ti)
   for ti in tis
   if ti.external_executor_id is not None
   }
   ```
   
   Then lines 369-370 can also be merged into `result, ti = 
celery_tasks[celery_task_id]`
   
   By doing this, we can avoid traversing `tis` two times, also save one query 
on the dict.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10951: Add example dag and system test for S3ToGCSOperator

2020-09-15 Thread GitBox



ephraimbuddy commented on a change in pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#discussion_r488910813



##
File path: docs/operators-and-hooks-ref.rst
##
@@ -934,11 +934,15 @@ These integrations allow you to copy data from/to Google 
Cloud.
  - :doc:`How to use `
  - :mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3`
 
-   * - `Amazon Simple Storage Service (S3) `__
+   * - Any Data Storage service

Review comment:
   I'm not sure the best way to word this but I see that Google Transfer 
service is not only for the S3. 
https://cloud.google.com/storage-transfer-service.
   
   Also, having it as S3 to GCS transfer will overshadow the S3ToGCSOperator. 
That's the reason I changed it and added the S3 to GCS below for S3ToGCSOperator





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil edited a comment on pull request #10924: fix operator field update for SerializedBaseOperator

2020-09-15 Thread GitBox



kaxil edited a comment on pull request #10924:
URL: https://github.com/apache/airflow/pull/10924#issuecomment-692922474


   I have rebased to the Master, do you need any help @xOnelinx



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #10924: fix operator field update for SerializedBaseOperator

2020-09-15 Thread GitBox



kaxil commented on pull request #10924:
URL: https://github.com/apache/airflow/pull/10924#issuecomment-692922474


   I have rebased to the Master



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on pull request #10958: Allow customization of navigation bar title text

2020-09-15 Thread GitBox



turbaszek commented on pull request #10958:
URL: https://github.com/apache/airflow/pull/10958#issuecomment-692921275


   @n-oden can you provide a screen shot?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10958: Allow customization of navigation bar title text

2020-09-15 Thread GitBox

boring-cyborg[bot] commented on pull request #10958:
URL: https://github.com/apache/airflow/pull/10958#issuecomment-692920697

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] n-oden opened a new pull request #10958: Allow customization of navigation bar title text

2020-09-15 Thread GitBox



n-oden opened a new pull request #10958:
URL: https://github.com/apache/airflow/pull/10958


   For organizations who may run more than one airflow instance,
   this will allow for quick visual distinguishing between for
   example a "production" and "test" airflow instance.
   
   Signed-off-by: Nathan J. Mehl 
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-09-15 Thread GitBox



turbaszek commented on a change in pull request #10956:
URL: https://github.com/apache/airflow/pull/10956#discussion_r488904165



##
File path: airflow/jobs/scheduler_job.py
##
@@ -1707,53 +1343,119 @@ def _run_scheduler_loop(self) -> None:
 )
 break
 
-def _validate_and_run_task_instances(self, simple_dag_bag: SimpleDagBag) 
-> bool:
-if simple_dag_bag.serialized_dags:
-try:
-self._process_and_execute_tasks(simple_dag_bag)
-except Exception as e:  # pylint: disable=broad-except
-self.log.error("Error queuing tasks")
-self.log.exception(e)
-return False
-
-# Call heartbeats
-self.log.debug("Heartbeating the executor")
-self.executor.heartbeat()
-
-self._change_state_for_tasks_failed_to_execute()
-
-# Process events from the executor
-self._process_executor_events(simple_dag_bag)
-return True
-
-def _process_and_execute_tasks(self, simple_dag_bag: SimpleDagBag) -> None:
-# Handle cases where a DAG run state is set (perhaps manually) to
-# a non-running state. Handle task instances that belong to
-# DAG runs in those states
-# If a task instance is up for retry but the corresponding DAG run
-# isn't running, mark the task instance as FAILED so we don't try
-# to re-run it.
-self._change_state_for_tis_without_dagrun(
-simple_dag_bag=simple_dag_bag,
-old_states=[State.UP_FOR_RETRY],
-new_state=State.FAILED
-)
-# If a task instance is scheduled or queued or up for reschedule,
-# but the corresponding DAG run isn't running, set the state to
-# NONE so we don't try to re-run it.
-self._change_state_for_tis_without_dagrun(
-simple_dag_bag=simple_dag_bag,
-old_states=[State.QUEUED,
-State.SCHEDULED,
-State.UP_FOR_RESCHEDULE,
-State.SENSING],
-new_state=State.NONE
-)
-self._execute_task_instances(simple_dag_bag)
+def _scheduler_loop_critical_section(self, dag_bag, session) -> Union[int, 
_NoLockObtained]:
+"""

Review comment:
   Can you please add a description so people in the future will be able to 
understand the purpose of this method?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] SalAlba commented on pull request #10864: Increase typing coverage for postgres provider

2020-09-15 Thread GitBox



SalAlba commented on pull request #10864:
URL: https://github.com/apache/airflow/pull/10864#issuecomment-692919402


   > Please rebase to latest master - we had some breaking change a commit ago 
and the PR will need to be rebased.
   
   @potiuk  I will do that according to this tips [How to rebase 
PR](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#id9)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] michalslowikowski00 commented on a change in pull request #10947: WIP Add transfer operator amazon gcs

2020-09-15 Thread GitBox



michalslowikowski00 commented on a change in pull request #10947:
URL: https://github.com/apache/airflow/pull/10947#discussion_r488902837



##
File path: tests/providers/amazon/aws/hooks/test_glacier.py
##
@@ -0,0 +1,180 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import mock
+import pytest
+from botocore.exceptions import ClientError
+from testfixtures import LogCapture
+
+from airflow.providers.amazon.aws.hooks.glacier import GlacierHook
+
+VAULT_NAME = "airflow"
+JOB_ID = "1234abcd"
+REQUEST_RESULT = {"jobId": "1234abcd"}
+RESPONSE_BODY = {"body": "data"}
+JOB_STATUS = {"Action": "", "StatusCode": "Succeeded"}
+
+
+class TestAmazonGlacierHook(unittest.TestCase):
+def setUp(self):
+with 
mock.patch("airflow.providers.amazon.aws.hooks.glacier.GlacierHook.__init__", 
return_value=None):
+self.hook = GlacierHook(aws_conn_id="aws_default")
+
+
@mock.patch("airflow.providers.amazon.aws.hooks.glacier.GlacierHook.get_conn")
+def test_retrieve_inventory_should_be_called_once_with(self, mock_conn):
+# when
+self.hook.retrieve_inventory(VAULT_NAME)
+# then
+mock_conn.assert_called_once_with()
+
+
@mock.patch("airflow.providers.amazon.aws.hooks.glacier.GlacierHook.get_conn")
+def test_retrieve_inventory_should_return_job_id(self, mock_conn):
+# Given
+job_id = {"jobId": "1234abcd"}
+# when
+mock_conn.return_value.initiate_job.return_value = job_id
+result = self.hook.retrieve_inventory(VAULT_NAME)
+# then
+self.assertEqual(job_id, result)
+
+@mock.patch(
+"airflow.providers.amazon.aws.hooks.glacier.GlacierHook.get_conn", 
side_effect=[ClientError({}, {})]
+)
+def 
test_retrieve_inventory_should_raise_client_error_when_error_occurred(self, 
mock_conn):
+# when
+with pytest.raises(ClientError) as err:
+self.hook.retrieve_inventory(VAULT_NAME)
+# then
+assert "ClientError" in str(err)

Review comment:
   Ok then, but it looks... naked without this assertion. 😉
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] michalslowikowski00 commented on a change in pull request #10947: WIP Add transfer operator amazon gcs

2020-09-15 Thread GitBox



michalslowikowski00 commented on a change in pull request #10947:
URL: https://github.com/apache/airflow/pull/10947#discussion_r488901592



##
File path: tests/providers/amazon/aws/hooks/test_glacier.py
##
@@ -0,0 +1,180 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import mock
+import pytest
+from botocore.exceptions import ClientError
+from testfixtures import LogCapture
+
+from airflow.providers.amazon.aws.hooks.glacier import GlacierHook
+
+VAULT_NAME = "airflow"
+JOB_ID = "1234abcd"
+REQUEST_RESULT = {"jobId": "1234abcd"}
+RESPONSE_BODY = {"body": "data"}
+JOB_STATUS = {"Action": "", "StatusCode": "Succeeded"}
+
+
+class TestAmazonGlacierHook(unittest.TestCase):
+def setUp(self):
+with 
mock.patch("airflow.providers.amazon.aws.hooks.glacier.GlacierHook.__init__", 
return_value=None):
+self.hook = GlacierHook(aws_conn_id="aws_default")
+
+
@mock.patch("airflow.providers.amazon.aws.hooks.glacier.GlacierHook.get_conn")

Review comment:
   Thanks :D





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-09-15 Thread GitBox



turbaszek commented on a change in pull request #10956:
URL: https://github.com/apache/airflow/pull/10956#discussion_r488900780



##
File path: airflow/jobs/scheduler_job.py
##
@@ -303,10 +303,10 @@ def done(self) -> bool:
 return False
 
 @property
-def result(self) -> Optional[Tuple[List[dict], int]]:
+def result(self) -> Optional[int]:
 """
 :return: result of running SchedulerJob.process_file()
-:rtype: Optional[Tuple[List[dict], int]]
+:rtype: int or None

Review comment:
   ```suggestion
   :rtype: Optional[None]
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-09-15 Thread GitBox



turbaszek commented on a change in pull request #10956:
URL: https://github.com/apache/airflow/pull/10956#discussion_r488899697



##
File path: airflow/executors/base_executor.py
##
@@ -69,16 +70,16 @@ def start(self):  # pragma: no cover
 """
 
 def queue_command(self,
-  simple_task_instance: SimpleTaskInstance,

Review comment:
   Should we be able to remove `SimpleTaskInstance` totally from Airflow?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #10897: Add a default for DagModel.default_view

2020-09-15 Thread GitBox



kaxil commented on a change in pull request #10897:
URL: https://github.com/apache/airflow/pull/10897#discussion_r488895516



##
File path: airflow/models/dag.py
##
@@ -1621,6 +1621,13 @@ def sync_to_db(self, sync_time=None, session=None):
 """
 self.bulk_sync_to_db([self], sync_time, session)
 
+def get_default_view(self):
+"""This is only there for backward compatible jinja2 templates"""
+if self.default_view is None:
+return conf.get('webserver', 'dag_default_view').lower()
+else:
+return self.default_view

Review comment:
   I kept it there for parity between Dag and DagModel, otherwise dag.html 
and dags.html might use or the other depending on if we are using Dag or 
DagModel
   
   The other option is to create a Database migration to fill all the `null` 
values, WDYT? @mik-laj 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #10897: Add a default for DagModel.default_view

2020-09-15 Thread GitBox



kaxil commented on a change in pull request #10897:
URL: https://github.com/apache/airflow/pull/10897#discussion_r488895516



##
File path: airflow/models/dag.py
##
@@ -1621,6 +1621,13 @@ def sync_to_db(self, sync_time=None, session=None):
 """
 self.bulk_sync_to_db([self], sync_time, session)
 
+def get_default_view(self):
+"""This is only there for backward compatible jinja2 templates"""
+if self.default_view is None:
+return conf.get('webserver', 'dag_default_view').lower()
+else:
+return self.default_view

Review comment:
   I kept it there for parity between Dag and DagModel.
   
   The other option is to create a Database migration to fill all the `null` 
values, WDYT? @mik-laj 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #10947: WIP Add transfer operator amazon gcs

2020-09-15 Thread GitBox



turbaszek commented on a change in pull request #10947:
URL: https://github.com/apache/airflow/pull/10947#discussion_r488895252



##
File path: tests/providers/amazon/aws/hooks/test_glacier.py
##
@@ -0,0 +1,180 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import mock
+import pytest
+from botocore.exceptions import ClientError
+from testfixtures import LogCapture
+
+from airflow.providers.amazon.aws.hooks.glacier import GlacierHook
+
+VAULT_NAME = "airflow"
+JOB_ID = "1234abcd"
+REQUEST_RESULT = {"jobId": "1234abcd"}
+RESPONSE_BODY = {"body": "data"}
+JOB_STATUS = {"Action": "", "StatusCode": "Succeeded"}
+
+
+class TestAmazonGlacierHook(unittest.TestCase):
+def setUp(self):
+with 
mock.patch("airflow.providers.amazon.aws.hooks.glacier.GlacierHook.__init__", 
return_value=None):
+self.hook = GlacierHook(aws_conn_id="aws_default")
+
+
@mock.patch("airflow.providers.amazon.aws.hooks.glacier.GlacierHook.get_conn")

Review comment:
   Examples:
   
https://github.com/apache/airflow/blob/ce19657ec685abff5871df80c8d47f8585eeed99/tests/providers/google/cloud/hooks/test_automl.py#L58-L59
   
   But no strong opinion here, as long as the mocking works I'm fine





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ryw edited a comment on pull request #10956: Officially support running more than one scheduler concurrently.

2020-09-15 Thread GitBox



ryw edited a comment on pull request #10956:
URL: https://github.com/apache/airflow/pull/10956#issuecomment-692908582


   We're working on a short doc that will provide an easy way to test all the 
components/PRs of AIP-15 together, and some preliminary benchmarks for 
performance improvement, hopefully have that out tomorrow.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

1 2 3 >

1 - 100 of 225 matches

Mail list logo