[GitHub] [airflow] potix2 commented on pull request #11531: Add type hints to aws provider

[GitBox]

potix2 commented on pull request #11531:
URL: https://github.com/apache/airflow/pull/11531#issuecomment-710753161


   I checked failed tests in local but it's successful. I try to trigger 
actions in CI.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mlgruby opened a new pull request #11609: strict type checking for provider Google

[GitBox]

mlgruby opened a new pull request #11609:
URL: https://github.com/apache/airflow/pull/11609


   Strict type checks for provider google cloud. Part of: #9708
   @kaxil a quick one to review - This should complete strict type checking for 
provider google!
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] tag nightly-master updated (d305876 -> ee9bdff)

[github-bot]

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


*** WARNING: tag nightly-master was modified! ***

from d305876  (commit)
  to ee9bdff  (commit)
from d305876  Remove redundant None provided as default to dict.get() 
(#11448)
 add 1eca73c  Fix spelling (#11453)
 add 02ce45c  Refactor celery worker command (#11336)
 add 358e61d  Move the test_process_dags_queries_count test to quarantine 
(#11455)
 add 06141d6  Google cloud operator strict type check (#11450)
 add 697465d  Increase timeout for waiting for images (#11460)
 add b1fcac2  Add more testing methods to dev/README.md (#11458)
 add 9142eed  Adds missing schema for kerberos sidecar configuration 
(#11413)
 add 32f2a45  Rename backport packages to provider packages (#11459)
 add da565c9  Add option to enable TCP keepalive for communication with 
Kubernetes API (#11406)
 add f124d3f  Enables back duplicate cancelling on push/schedule (#11471)
 add 81a977b  Fix typo in docker-context-files/README.md (#11473)
 add d38a0a7  added type hints for aws cloud formation (#11470)
 add 4e32546  Mask Password in Log table when using the CLI (#11468)
 add 045d68d  Mount volumes and volumemounts into scheduler and workers 
(#11426)
 add c47383d  Bump FAB to 3.1 (#11475)
 add 7b3a00d  Allow multiple schedulers in helm chart (#11330)
 add 2345cd1  Fix Harcoded Airflow version (#11483)
 add b8cecf5  Add link on External Task Sensor to navigate to target dag 
(#11481)
 add 623d5cd  Spend less time waiting for LocalTaskJob's subprocss process 
to finish (#11373)
 add 5772d4d  Add endpoints for task instances (#9597)
 add 2c8d203  Enable serialization by default (#11491)
 add cb4cd91  Add missing values entries to Parameters in chart/README.md 
(#11477)
 add 760cd14  Rename "functional DAGs" to "Decorated Flows" (#11497)
 add f43d855  Prevent text-selection of scheduler interval when selecting 
DAG ID (#11503)
 add e3e8fd8  Mark Smart Sensor as an early-access feature (#11499)
 add 42b979d  Fix spelling for Airbnb (#11505)
 add 16e7129  Added support for provider packages for Airflow 2.0 (#11487)
 add 095756c  Airflow tutorial to use Decorated Flows (#11308)
 add 56b761b  Bump to Airflow 2.0.0a1 (#11507)
 add bf9ece7  Update CONTRIBUTING.rst (#11461)
 add c604891  Change Airflow version to 2.0.0a1 in Updating.md (#11508)
 add 4551a5e  Updated tutorial_decorated_flows.rst to add links (#11510)
 add 4297aba  Combine back multiple test types into single jobs (#11504)
 add e9f7bdd  Fix typo in scripts/ci/libraries/_initialization.sh (#11517)
 add d963467  Fix example in UPDATING.md (#11518)
 add bcf0557  Fixes remaining test-type strategy problems (#11522)
 add 6c8cf6a  Add reset_dag_run option on dagrun_operator to clear existing 
dag run (#11484)
 add 2509d13  Minor improvements to dev/README.md (#11525)
 add 3163912  Remove flask-admin based Plugins (#11515)
 add 545ba8e  Upgrade to pymssql 2.1.5 for Py 3.8 support (#11523)
 add 0646849  Add protocol_version to conn_config for Cassandrahook (#11036)
 add 7b7cc3c  backport for add_xcom_sidecar (#11478)
 add c099e0a  Fix documentation errors (#11536)
 add 765d29e  Pymssql is maintained again (#11537)
 add 3447b55  More stable kubernetes port forwarding (#11538)
 add 03a632e  Resolve MSSQL DAG serialization bug by changing datatype for 
execution date (#11512)
 add 1ddeddc  Rename (confusing) dag.sub_dag to dag.partial_subset (#11542)
 add e7dc964  Adds capability of installing wheel packages in CI image 
(#11527)
 add 13959df  Feature: Auto-refresh Graph view chart (#11534)
 add eee4e30  Add better debug logging to K8sexec and K8sPodOp (#11502)
 add 81a5b03  Visual tweaking of TI swatches, legend refactoring (#11550)
 add f7e01ab  Create job for airflow migrations (#11533)
 add 89d4dd8  Auto-refresh default state (#11559)
 add 7ab6210  Prepend `DAG:` to dag permissions (#11189)
 add 8372ab1  Add/implement Webpack plugin to minify CSS files on-compile 
(#11564)
 add 5d4fbce  Clarify breeze docs --install-airflow-version/-reference 
(#11570)
 add 6733f2d  The scripts fixing ownership and cleaning tmp use docker run 
(#11569)
 add 3cddc11  Updated template_fields_rendereds for PostgresOperator and 
SimpleHttpOperator (#11555)
 add 399b81d  Prevent pop-over elements from being cut off by hidden 
overflow (#11574)
 add df75610  Fix broken backtick usage in Timezone docs (#11575)
 add 8865d14  Strict type checking for provider google cloud  (#11548)
 add 6f0bc0d  Fixes dependencies to pre-release versions of apache-airflow 
(#11578)
 add 3163016  Guard against kubernetes not being installed (#11558)
 add 45d6083  Utilize the state 

[airflow] branch master updated: Remove redundant code to serialized k8s.V1Pod (#11602)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 16ed974  Remove redundant code to serialized k8s.V1Pod (#11602)
16ed974 is described below

commit 16ed9740d766f1b2aaebd5a00da78d2cdf4d2cb9
Author: Kaxil Naik 
AuthorDate: Sat Oct 17 02:29:18 2020 +0100

Remove redundant code to serialized k8s.V1Pod (#11602)
---
 airflow/serialization/serialized_objects.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/airflow/serialization/serialized_objects.py 
b/airflow/serialization/serialized_objects.py
index 8caff33..8d1eb34 100644
--- a/airflow/serialization/serialized_objects.py
+++ b/airflow/serialization/serialized_objects.py
@@ -190,9 +190,6 @@ class BaseSerialization:
 {str(k): cls._serialize(v) for k, v in var.items()},
 type_=DAT.DICT
 )
-elif HAS_KUBERNETES and isinstance(var, k8s.V1Pod):
-json_pod = PodGenerator.serialize_pod(var)
-return cls._encode(json_pod, type_=DAT.POD)
 elif isinstance(var, list):
 return [cls._serialize(v) for v in var]
 elif HAS_KUBERNETES and isinstance(var, k8s.V1Pod):

[airflow] branch master updated: Remove redundant code to serialized k8s.V1Pod (#11602)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 16ed974  Remove redundant code to serialized k8s.V1Pod (#11602)
16ed974 is described below

commit 16ed9740d766f1b2aaebd5a00da78d2cdf4d2cb9
Author: Kaxil Naik 
AuthorDate: Sat Oct 17 02:29:18 2020 +0100

Remove redundant code to serialized k8s.V1Pod (#11602)
---
 airflow/serialization/serialized_objects.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/airflow/serialization/serialized_objects.py 
b/airflow/serialization/serialized_objects.py
index 8caff33..8d1eb34 100644
--- a/airflow/serialization/serialized_objects.py
+++ b/airflow/serialization/serialized_objects.py
@@ -190,9 +190,6 @@ class BaseSerialization:
 {str(k): cls._serialize(v) for k, v in var.items()},
 type_=DAT.DICT
 )
-elif HAS_KUBERNETES and isinstance(var, k8s.V1Pod):
-json_pod = PodGenerator.serialize_pod(var)
-return cls._encode(json_pod, type_=DAT.POD)
 elif isinstance(var, list):
 return [cls._serialize(v) for v in var]
 elif HAS_KUBERNETES and isinstance(var, k8s.V1Pod):

[GitHub] [airflow] kaxil merged pull request #11602: Remove redundant code to serialized k8s.V1Pod

[GitBox]

kaxil merged pull request #11602:
URL: https://github.com/apache/airflow/pull/11602


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #10587: Add @dag decorator

[GitBox]

github-actions[bot] commented on pull request #10587:
URL: https://github.com/apache/airflow/pull/10587#issuecomment-710716958


   [The Workflow run](https://github.com/apache/airflow/actions/runs/311720324) 
is cancelling this PR. Building images for the PR has failed. Follow the the 
workflow link to check the reason.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ad-m commented on pull request #8550: Add DataflowStartFlexTemplateOperator

[GitBox]

ad-m commented on pull request #8550:
URL: https://github.com/apache/airflow/pull/8550#issuecomment-710715682


   Ja także dziękuje @TobKed za rozwój projektu Airflow!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #11608: Replace old screenshots for managing Connections with new

[GitBox]

kaxil opened a new pull request #11608:
URL: https://github.com/apache/airflow/pull/11608


   Replace old screenshots for managing Connections with new screenshots
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: Minor doc improvements in blob_storage_to_gcs.rst (#11607)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new ee9bdff  Minor doc improvements in blob_storage_to_gcs.rst (#11607)
ee9bdff is described below

commit ee9bdff6f76906dc39d398951f99db0768e714cd
Author: Kaxil Naik 
AuthorDate: Sat Oct 17 01:45:04 2020 +0100

Minor doc improvements in blob_storage_to_gcs.rst (#11607)

- Sentence completion
- code-block rendering
- Link to english docs for Azure instructions
---
 docs/howto/operator/microsoft/transfer/blob_storage_to_gcs.rst | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/docs/howto/operator/microsoft/transfer/blob_storage_to_gcs.rst 
b/docs/howto/operator/microsoft/transfer/blob_storage_to_gcs.rst
index b125f8e..12a7aac 100644
--- a/docs/howto/operator/microsoft/transfer/blob_storage_to_gcs.rst
+++ b/docs/howto/operator/microsoft/transfer/blob_storage_to_gcs.rst
@@ -21,18 +21,21 @@ Azure Blob Storage Transfer Operator
 The Blob service stores text and binary data as objects in the cloud.
 The Blob service offers the following three resources: the storage account, 
containers, and blobs.
 Within your storage account, containers provide a way to organize sets of 
blobs.
-For more information about the service visit `Azure Blob Storage API 
documentation 
`_
+For more information about the service visit `Azure Blob Storage API 
documentation 
`_.
 
 Before you begin
 
 Before using Blob Storage within Airflow you need to authenticate your account 
with Token, Login and Password.
-Please follow Azure `instructions 
`_
 to do it.
+Please follow Azure
+`instructions 
`_
+to do it.
 
 TOKEN should be added to the Connection in Airflow in JSON format, Login and 
Password as plain text.
 You can check `how to do such connection 
`_.
 
 See following example.
 Set values for these fields:
+
 .. code-block::
 
   Conn Id: wasb_default

[GitHub] [airflow] kaxil merged pull request #11607: Minor doc improvements in blob_storage_to_gcs.rst

[GitBox]

kaxil merged pull request #11607:
URL: https://github.com/apache/airflow/pull/11607


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil merged pull request #11601: Fix Task definition in docs

[GitBox]

kaxil merged pull request #11601:
URL: https://github.com/apache/airflow/pull/11601


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on pull request #11592: Fix #11591 lost redirection

[GitBox]

dimberman commented on pull request #11592:
URL: https://github.com/apache/airflow/pull/11592#issuecomment-710711023


   @mpermana I know that we know eachother but that doesn't mean I'm going to 
merge untested code :). Especially since this fixes a bug, can you please write 
a unit test mimicking the failure so someone else doesn't break it in the 
future. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (b6ab0b5 -> 673386d)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from b6ab0b5  Fix rendering of code-block in operator/kubernetes.rst 
(#11606)
 add 673386d  Fix Task definition in docs (#11601)

No new revisions were added by this update.

Summary of changes:
 docs/tutorial.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[airflow] branch master updated (210a948 -> b6ab0b5)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 210a948  Fix tcp keepalive parameters parsing (#11594)
 add b6ab0b5  Fix rendering of code-block in operator/kubernetes.rst 
(#11606)

No new revisions were added by this update.

Summary of changes:
 docs/howto/operator/kubernetes.rst | 1 +
 1 file changed, 1 insertion(+)

[airflow] branch master updated: Fix tcp keepalive parameters parsing (#11594)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 210a948  Fix tcp keepalive parameters parsing (#11594)
210a948 is described below

commit 210a948658b96ef596e3d6f068be033e7fa87b73
Author: Michał Misiewicz 
AuthorDate: Sat Oct 17 02:34:11 2020 +0200

Fix tcp keepalive parameters parsing (#11594)
---
 airflow/kubernetes/kube_client.py |  6 +++---
 tests/kubernetes/test_client.py   | 19 ++-
 2 files changed, 21 insertions(+), 4 deletions(-)

diff --git a/airflow/kubernetes/kube_client.py 
b/airflow/kubernetes/kube_client.py
index 3c30807..e51274f 100644
--- a/airflow/kubernetes/kube_client.py
+++ b/airflow/kubernetes/kube_client.py
@@ -78,9 +78,9 @@ def _enable_tcp_keepalive() -> None:
 
 from urllib3.connection import HTTPConnection, HTTPSConnection
 
-tcp_keep_idle = conf.get('kubernetes', 'tcp_keep_idle', fallback=120)
-tcp_keep_intvl = conf.get('kubernetes', 'tcp_keep_intvl', fallback=30)
-tcp_keep_cnt = conf.get('kubernetes', 'tcp_keep_cnt', fallback=6)
+tcp_keep_idle = conf.getint('kubernetes', 'tcp_keep_idle', fallback=120)
+tcp_keep_intvl = conf.getint('kubernetes', 'tcp_keep_intvl', fallback=30)
+tcp_keep_cnt = conf.getint('kubernetes', 'tcp_keep_cnt', fallback=6)
 
 socket_options = [
 (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
diff --git a/tests/kubernetes/test_client.py b/tests/kubernetes/test_client.py
index 8aa83a2..a67efae 100644
--- a/tests/kubernetes/test_client.py
+++ b/tests/kubernetes/test_client.py
@@ -15,11 +15,13 @@
 # specific language governing permissions and limitations
 # under the License.
 
+import socket
 import unittest
 
 import mock
+from urllib3.connection import HTTPConnection, HTTPSConnection
 
-from airflow.kubernetes.kube_client import RefreshConfiguration, 
get_kube_client
+from airflow.kubernetes.kube_client import RefreshConfiguration, 
_enable_tcp_keepalive, get_kube_client
 
 
 class TestClient(unittest.TestCase):
@@ -34,3 +36,18 @@ class TestClient(unittest.TestCase):
 def test_load_file_config(self, _, _2):
 client = get_kube_client(in_cluster=False)
 assert isinstance(client.api_client.configuration, 
RefreshConfiguration)
+
+def test_enable_tcp_keepalive(self):
+socket_options = [
+(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
+(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 120),
+(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 30),
+(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 6),
+]
+expected_http_connection_options = 
HTTPConnection.default_socket_options + socket_options
+expected_https_connection_options = 
HTTPSConnection.default_socket_options + socket_options
+
+_enable_tcp_keepalive()
+
+self.assertEqual(HTTPConnection.default_socket_options, 
expected_http_connection_options)
+self.assertEqual(HTTPSConnection.default_socket_options, 
expected_https_connection_options)

[GitHub] [airflow] dimberman merged pull request #11594: Fix tcp keepalive parameters parsing

[GitBox]

dimberman merged pull request #11594:
URL: https://github.com/apache/airflow/pull/11594


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil merged pull request #11606: Fix rendering of code-block in operator/kubernetes.rst

[GitBox]

kaxil merged pull request #11606:
URL: https://github.com/apache/airflow/pull/11606


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #11589: WIP: Perform "mini scheduling run" after task has finished

[GitBox]

mik-laj commented on a change in pull request #11589:
URL: https://github.com/apache/airflow/pull/11589#discussion_r506771453



##
File path: airflow/models/dagrun.py
##
@@ -43,6 +44,20 @@
 from airflow.utils.types import DagRunType
 
 
+class _TISchedulingDecision(TypedDict):

Review comment:
   Is there a reason we need to use TypedDict? NamedTuple is much easier to 
use in many cases.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #11589: WIP: Perform "mini scheduling run" after task has finished

[GitBox]

mik-laj commented on a change in pull request #11589:
URL: https://github.com/apache/airflow/pull/11589#discussion_r506771250



##
File path: airflow/jobs/scheduler_job.py
##
@@ -1664,42 +1664,11 @@ def _schedule_dag_run(self, dag_run: DagRun, 
currently_active_runs: int, session
 
 self._send_dag_callbacks_to_processor(dag_run, callback_to_run)
 
-# Get list of TIs that do not need to executed, these are
-# tasks using DummyOperator and without on_execute_callback / 
on_success_callback
-dummy_tis = [
-ti for ti in schedulable_tis
-if
-(
-ti.task.task_type == "DummyOperator"
-and not ti.task.on_execute_callback
-and not ti.task.on_success_callback
-)
-]
-
 # This will do one query per dag run. We "could" build up a complex

Review comment:
   It seems to me that we should also move the comment. Now it has lost 
context.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (b718b51 -> 55a4d8c)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from b718b51  Minor improvements to docs/tutorial.rst (#11604)
 add 55a4d8c  Remove unnecessary use of comprehension in 
setup_provider_packages.py (#11605)

No new revisions were added by this update.

Summary of changes:
 provider_packages/setup_provider_packages.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[GitHub] [airflow] kaxil merged pull request #11605: Remove unnecessary use of comprehension in setup_provider_packages.py

[GitBox]

kaxil merged pull request #11605:
URL: https://github.com/apache/airflow/pull/11605


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (bb615f2 -> b718b51)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from bb615f2  Cross Reference XCom in tutorial (#11600)
 add b718b51  Minor improvements to docs/tutorial.rst (#11604)

No new revisions were added by this update.

Summary of changes:
 docs/tutorial.rst | 16 
 1 file changed, 8 insertions(+), 8 deletions(-)

[GitHub] [airflow] kaxil merged pull request #11604: Minor improvements to docs/tutorial.rst

[GitBox]

kaxil merged pull request #11604:
URL: https://github.com/apache/airflow/pull/11604


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (84dc2fb -> bb615f2)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 84dc2fb  Set doc_md when using task decorator and function has __doc__ 
(#11598)
 add bb615f2  Cross Reference XCom in tutorial (#11600)

No new revisions were added by this update.

Summary of changes:
 docs/tutorial.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[GitHub] [airflow] kaxil merged pull request #11600: Cross Reference XCom in tutorial

[GitBox]

kaxil merged pull request #11600:
URL: https://github.com/apache/airflow/pull/11600


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #11607: Minor doc improvements in blob_storage_to_gcs.rst

[GitBox]

kaxil opened a new pull request #11607:
URL: https://github.com/apache/airflow/pull/11607


   - Sentence completion
   - code-block rendering
   
![image](https://user-images.githubusercontent.com/8811558/96323709-4487af80-1016-11eb-8ec3-477df39b75d9.png)
   
   - Link to English docs for Azure instructions (before it was Polish -- I 
think)
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (00dd758 -> 84dc2fb)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 00dd758  Raises a warning for provide_context instead of killing the 
task (#11597)
 add 84dc2fb  Set doc_md when using task decorator and function has __doc__ 
(#11598)

No new revisions were added by this update.

Summary of changes:
 airflow/example_dags/tutorial_decorated_etl_dag.py | 30 +++---
 airflow/operators/python.py|  2 ++
 tests/operators/test_python.py | 16 
 3 files changed, 33 insertions(+), 15 deletions(-)

[airflow] branch master updated (00dd758 -> 84dc2fb)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 00dd758  Raises a warning for provide_context instead of killing the 
task (#11597)
 add 84dc2fb  Set doc_md when using task decorator and function has __doc__ 
(#11598)

No new revisions were added by this update.

Summary of changes:
 airflow/example_dags/tutorial_decorated_etl_dag.py | 30 +++---
 airflow/operators/python.py|  2 ++
 tests/operators/test_python.py | 16 
 3 files changed, 33 insertions(+), 15 deletions(-)

[GitHub] [airflow] kaxil merged pull request #11598: Set doc_md when using task decorator and function has __doc__

[GitBox]

kaxil merged pull request #11598:
URL: https://github.com/apache/airflow/pull/11598


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #11606: Fix rendering of code-block in operator/kubernetes.rst

[GitBox]

kaxil opened a new pull request #11606:
URL: https://github.com/apache/airflow/pull/11606


   Docs were rendered incorrectly because of a missing blank line
   
![image](https://user-images.githubusercontent.com/8811558/96323468-25d4e900-1015-11eb-9e79-1a277672569d.png)
   
   
https://airflow.readthedocs.io/en/latest/howto/operator/kubernetes.html#difference-between-kubernetespodoperator-and-kubernetes-object-spec
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #11605: Remove unnecessary use of comprehension in setup_provider_packages.py

[GitBox]

kaxil opened a new pull request #11605:
URL: https://github.com/apache/airflow/pull/11605


   Since we are just iterating without modification, it is simpler to just use 
`list(PROVIDERS_REQUIREMENTS.keys())`
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #11604: Minor improvements to docs/tutorial.rst

[GitBox]

kaxil opened a new pull request #11604:
URL: https://github.com/apache/airflow/pull/11604


   Sentence completions by adding periods 
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #11603: KubernetesExecutor with volumes and volume_mounts specified does not launch tasks

[GitBox]

boring-cyborg[bot] commented on issue #11603:
URL: https://github.com/apache/airflow/issues/11603#issuecomment-710699877


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] yunduz opened a new issue #11603: KubernetesExecutor with volumes and volume_mounts specified does not launch tasks

[GitBox]

yunduz opened a new issue #11603:
URL: https://github.com/apache/airflow/issues/11603


   
   **Apache Airflow version**: 1.10.12
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl 
version`): Openshift 3.11 (Kubernetes 1.11)
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**: Openshift 3.11
   - **OS** (e.g. from /etc/os-release): Red Hat 7.7
   
   **What happened**:
   Switched from Airflow 1.10.7 to 1.10.12 and one of the tasks fails to 
launch, scheduler shows an error and Airflow becomes unusable (ex can't trigger 
dag) and has to be restarted. Operator is a PythonOperator, executor is 
KubernetesExecutor, `executor_config` is specified which includes "volumes" and 
"volume_mounts". If "volumes" and "volume_mounts" are not specified in the 
`executor_config`, the issue does not occur.
   
   **What you expected to happen**: Pod for a task is launched, Airflow 
continues to work.
   
   **How to reproduce it**: I modified one of the example dags included with 
Airflow which reproduces the problem.
   ```
   import os
   
   from airflow.models import DAG
   from airflow.operators.python_operator import PythonOperator
   from airflow.utils.dates import days_ago
   
   
   default_args = {
   'owner': 'Airflow',
   'start_date': days_ago(2)
   }
   
   with DAG(
   dag_id='test_yunduz',
   default_args=default_args,
   schedule_interval=None
   ) as dag:
   
   def test_volume_mount():
   with open('/foo/volume_mount_test.txt', 'w') as foo:
   foo.write('Hello')
   
   return_code = os.system("cat /foo/volume_mount_test.txt")
   assert return_code == 0
   
   second_task = PythonOperator(
   task_id="four_task",
   python_callable=test_volume_mount,
   executor_config={
   "KubernetesExecutor": {
   "volumes": [
   {
   "name": "example-kubernetes-test-volume",
   "hostPath": {"path": "/tmp/"},
   },
   ],
   "volume_mounts": [
   {
   "mountPath": "/foo/",
   "name": "example-kubernetes-test-volume",
   },
   ]
   }
   }
   )
   
   second_task
   
   ```
   
   
   **Anything else we need to know**: 
   error in airflow scheduler log
   
   ```
   
   [2020-10-16 19:03:54,972] {scheduler_job.py:1401} ERROR - Exception when 
executing execute_helper
   Traceback (most recent call last):
 File 
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py",
 line 105, in _mutate_pod_backcompat
   new_pod = PodGenerator.reconcile_pods(pod, dummy_pod)
 File 
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py",
 line 410, in reconcile_pods
   client_pod_cp.spec = PodGenerator.reconcile_specs(base_pod.spec, 
client_pod_cp.spec)
 File 
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py",
 line 459, in reconcile_specs
   merged_spec = extend_object_field(base_spec, client_spec, 'volumes')
 File 
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py",
 line 657, in extend_object_field
   appended_fields = _merge_list_of_objects(base_obj_set, client_obj_set)
 File 
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py",
 line 668, in _merge_list_of_objects
   client_obj_set[k] = merge_objects(v, client_obj_set[k])
 File 
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py",
 line 621, in merge_objects
   for base_key in base_obj.to_dict().keys():
   AttributeError: 'dict' object has no attribute 'to_dict'
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
 File 
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py",
 line 108, in _mutate_pod_backcompat
   settings.pod_mutation_hook(pod)
 File "/app/airflow/config/airflow_local_settings.py", line 62, in 
pod_mutation_hook
   log.info(f"Running command: {pod.cmds} ")
   AttributeError: 'V1Pod' object has no attribute 'cmds'
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
 File 
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py",
 line 1399, in _execute
   self._execute_helper()
 File 
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py",
 line 1470, in _execute_helper
   if not 
self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag):
 File 

[GitHub] [airflow] kaxil opened a new pull request #11602: Remove redundant code to serialized k8s.V1Pod

[GitBox]

kaxil opened a new pull request #11602:
URL: https://github.com/apache/airflow/pull/11602


   https://github.com/apache/airflow/pull/11558/files#r506763359
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #11558: Guard against kubernetes not being installed

[GitBox]

kaxil commented on a change in pull request #11558:
URL: https://github.com/apache/airflow/pull/11558#discussion_r506764891



##
File path: airflow/serialization/serialized_objects.py
##
@@ -187,12 +190,12 @@ def _serialize(cls, var: Any) -> Any:  # Unfortunately 
there is no support for r
 {str(k): cls._serialize(v) for k, v in var.items()},
 type_=DAT.DICT
 )
-elif isinstance(var, k8s.V1Pod):
+elif HAS_KUBERNETES and isinstance(var, k8s.V1Pod):
 json_pod = PodGenerator.serialize_pod(var)
 return cls._encode(json_pod, type_=DAT.POD)
 elif isinstance(var, list):
 return [cls._serialize(v) for v in var]
-elif isinstance(var, k8s.V1Pod):
+elif HAS_KUBERNETES and isinstance(var, k8s.V1Pod):

Review comment:
   PR: https://github.com/apache/airflow/pull/11602





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #11601: Fix Task definition in docs

[GitBox]

kaxil opened a new pull request #11601:
URL: https://github.com/apache/airflow/pull/11601


   I don't know why this was called constructor, am I missing something here??
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mjpieters commented on a change in pull request #11558: Guard against kubernetes not being installed

[GitBox]

mjpieters commented on a change in pull request #11558:
URL: https://github.com/apache/airflow/pull/11558#discussion_r506763359



##
File path: airflow/serialization/serialized_objects.py
##
@@ -187,12 +190,12 @@ def _serialize(cls, var: Any) -> Any:  # Unfortunately 
there is no support for r
 {str(k): cls._serialize(v) for k, v in var.items()},
 type_=DAT.DICT
 )
-elif isinstance(var, k8s.V1Pod):
+elif HAS_KUBERNETES and isinstance(var, k8s.V1Pod):
 json_pod = PodGenerator.serialize_pod(var)
 return cls._encode(json_pod, type_=DAT.POD)
 elif isinstance(var, list):
 return [cls._serialize(v) for v in var]
-elif isinstance(var, k8s.V1Pod):
+elif HAS_KUBERNETES and isinstance(var, k8s.V1Pod):

Review comment:
   This block is redundant actually; no need to test for k8s.V1Pod _twice_.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #10587: Add @dag decorator

[GitBox]

github-actions[bot] commented on pull request #10587:
URL: https://github.com/apache/airflow/pull/10587#issuecomment-710697734


   [The Workflow run](https://github.com/apache/airflow/actions/runs/311631410) 
is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static 
checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm 
tests$,^Test OpenAPI*.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #11600: Cross Reference XCom in tutorial

[GitBox]

kaxil opened a new pull request #11600:
URL: https://github.com/apache/airflow/pull/11600


   Cross Reference XCom in tutorial
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] casassg commented on pull request #10587: Add @dag decorator

[GitBox]

casassg commented on pull request #10587:
URL: https://github.com/apache/airflow/pull/10587#issuecomment-710688343


   Rolledback a change as I didn't understand well how DagRun could be changed 
it's configuration from trigger interfaces. I proposed #11599 to fill the UI 
gap I see here. 
   
   Added a few more unit tests.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] casassg opened a new issue #11599: Default DagRun configuration

[GitBox]

casassg opened a new issue #11599:
URL: https://github.com/apache/airflow/issues/11599


   
   **Description**
   
   It would nice to have the trigger UI pre-populated with default values for 
your dagrun configuration: 
https://airflow.apache.org/docs/stable/dag-run.html?highlight=params#using-ui
   
   For that we need to add a new field to DAG that keeps the default conf for 
DAGRuns, serialize it and have it available on the Trigger UI.
   
   This should allow user to more easily know what parameters/keys in config 
dictionary to expect.
   
   **Use case / motivation**
   
   Parameterize your DAG and change values easily on Trigger DAG UI.
   
   
   **Related Issues**
   
   #10587
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11598: Set doc_md when using task decorator and function has __doc__

[GitBox]

github-actions[bot] commented on pull request #11598:
URL: https://github.com/apache/airflow/pull/11598#issuecomment-710681132


   [The Workflow run](https://github.com/apache/airflow/actions/runs/311543264) 
is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static 
checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm 
tests$,^Test OpenAPI*.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman merged pull request #11597: Raises a warning for provide_context instead of killing the task

[GitBox]

dimberman merged pull request #11597:
URL: https://github.com/apache/airflow/pull/11597


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: Raises a warning for provide_context instead of killing the task (#11597)

[dimberman]

This is an automated email from the ASF dual-hosted git repository.

dimberman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 00dd758  Raises a warning for provide_context instead of killing the 
task (#11597)
00dd758 is described below

commit 00dd7586fb6b41bff2b89b9008742c7b34cd4fdd
Author: Daniel Imberman 
AuthorDate: Fri Oct 16 15:18:55 2020 -0700

Raises a warning for provide_context instead of killing the task (#11597)

* raises a warning for provide_context instead of killing the task

* Update airflow/operators/python.py

Co-authored-by: Kaxil Naik 

* static checks

Co-authored-by: Kaxil Naik 
---
 airflow/operators/python.py|  5 +
 tests/operators/test_python.py | 25 +
 2 files changed, 30 insertions(+)

diff --git a/airflow/operators/python.py b/airflow/operators/python.py
index 58a1095..1be50e2 100644
--- a/airflow/operators/python.py
+++ b/airflow/operators/python.py
@@ -22,6 +22,7 @@ import pickle
 import re
 import sys
 import types
+import warnings
 from inspect import signature
 from itertools import islice
 from tempfile import TemporaryDirectory
@@ -85,6 +86,10 @@ class PythonOperator(BaseOperator):
 templates_exts: Optional[List[str]] = None,
 **kwargs
 ) -> None:
+if kwargs.get("provide_context"):
+warnings.warn("provide_context is deprecated as of 2.0 and is no 
longer required",
+  DeprecationWarning, stacklevel=2)
+kwargs.pop('provide_context', None)
 super().__init__(**kwargs)
 if not callable(python_callable):
 raise AirflowException('`python_callable` param must be callable')
diff --git a/tests/operators/test_python.py b/tests/operators/test_python.py
index 91529ffb..a61b9a1 100644
--- a/tests/operators/test_python.py
+++ b/tests/operators/test_python.py
@@ -275,6 +275,31 @@ class TestPythonOperator(TestPythonBase):
 python_operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
 self.assertTrue('dag' in context.exception, "'dag' not found in 
the exception")
 
+def test_provide_context_does_not_fail(self):
+"""
+ensures that provide_context doesn't break dags in 2.0
+"""
+self.dag.create_dagrun(
+run_type=DagRunType.MANUAL,
+execution_date=DEFAULT_DATE,
+start_date=DEFAULT_DATE,
+state=State.RUNNING,
+external_trigger=False,
+)
+
+def func(custom, dag):
+self.assertEqual(1, custom, "custom should be 1")
+self.assertIsNotNone(dag, "dag should be set")
+
+python_operator = PythonOperator(
+task_id='python_operator',
+op_kwargs={'custom': 1},
+python_callable=func,
+provide_context=True,
+dag=self.dag
+)
+python_operator.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+
 def test_context_with_conflicting_op_args(self):
 self.dag.create_dagrun(
 run_type=DagRunType.MANUAL,

[airflow] branch master updated (be72817 -> 4582c80)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from be72817  Allow loading plugins on Airflow start-up (#11596)
 add 4582c80  UX Enhancement: Add button to clear search query from DAG 
search (#11583)

No new revisions were added by this update.

Summary of changes:
 airflow/www/static/css/main.css | 15 +++
 airflow/www/templates/airflow/dags.html | 14 --
 2 files changed, 27 insertions(+), 2 deletions(-)

[airflow] branch master updated (be72817 -> 4582c80)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from be72817  Allow loading plugins on Airflow start-up (#11596)
 add 4582c80  UX Enhancement: Add button to clear search query from DAG 
search (#11583)

No new revisions were added by this update.

Summary of changes:
 airflow/www/static/css/main.css | 15 +++
 airflow/www/templates/airflow/dags.html | 14 --
 2 files changed, 27 insertions(+), 2 deletions(-)

[airflow] branch master updated (0c5bbe8 -> be72817)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 0c5bbe8  Replace methods on state with frozenset properties (#11576)
 add be72817  Allow loading plugins on Airflow start-up (#11596)

No new revisions were added by this update.

Summary of changes:
 airflow/__init__.py  | 5 +
 airflow/config_templates/config.yml  | 8 
 airflow/config_templates/default_airflow.cfg | 4 
 airflow/settings.py  | 4 
 docs/plugins.rst | 4 +++-
 5 files changed, 24 insertions(+), 1 deletion(-)

[airflow] branch master updated (be72817 -> 4582c80)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from be72817  Allow loading plugins on Airflow start-up (#11596)
 add 4582c80  UX Enhancement: Add button to clear search query from DAG 
search (#11583)

No new revisions were added by this update.

Summary of changes:
 airflow/www/static/css/main.css | 15 +++
 airflow/www/templates/airflow/dags.html | 14 --
 2 files changed, 27 insertions(+), 2 deletions(-)

[airflow] branch master updated (0c5bbe8 -> be72817)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 0c5bbe8  Replace methods on state with frozenset properties (#11576)
 add be72817  Allow loading plugins on Airflow start-up (#11596)

No new revisions were added by this update.

Summary of changes:
 airflow/__init__.py  | 5 +
 airflow/config_templates/config.yml  | 8 
 airflow/config_templates/default_airflow.cfg | 4 
 airflow/settings.py  | 4 
 docs/plugins.rst | 4 +++-
 5 files changed, 24 insertions(+), 1 deletion(-)

[GitHub] [airflow] kaxil commented on a change in pull request #11396: Add creating_job_id to DagRun table

[GitBox]

kaxil commented on a change in pull request #11396:
URL: https://github.com/apache/airflow/pull/11396#discussion_r506739203



##
File path: 
airflow/migrations/versions/364159666cbd_add_job_id_to_dagrun_table.py
##
@@ -0,0 +1,44 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Add creating_job_id to DagRun table
+
+Revision ID: 364159666cbd
+Revises: 98271e7606e2

Review comment:
   ```suggestion
   Revises: 849da589634d
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil merged pull request #11583: UX Enhancement: Add button to clear search query from DAG search

[GitBox]

kaxil merged pull request #11583:
URL: https://github.com/apache/airflow/pull/11583


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil merged pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

kaxil merged pull request #11596:
URL: https://github.com/apache/airflow/pull/11596


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #10587: Add @dag decorator

[GitBox]

github-actions[bot] commented on pull request #10587:
URL: https://github.com/apache/airflow/pull/10587#issuecomment-710670236


   [The Workflow run](https://github.com/apache/airflow/actions/runs/311510535) 
is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static 
checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm 
tests$,^Test OpenAPI*.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] casassg opened a new pull request #11598: Set doc_md when using task decorator and function has __doc__

[GitBox]

casassg opened a new pull request #11598:
URL: https://github.com/apache/airflow/pull/11598


   This is a nice to have addition to `@task` decorator. The idea is that we 
can use function documentation to populate documentation of operator. This 
simplifies development and encourages users to add documentation to their adhoc 
function tasks as well.
   
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11526: Adds automated installation of dependent packages

[GitBox]

github-actions[bot] commented on pull request #11526:
URL: https://github.com/apache/airflow/pull/11526#issuecomment-710664002


   [The Workflow run](https://github.com/apache/airflow/actions/runs/311489636) 
is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static 
checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm 
tests$,^Test OpenAPI*.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11597: Raises a warning for provide_context instead of killing the task

[GitBox]

kaxil commented on pull request #11597:
URL: https://github.com/apache/airflow/pull/11597#issuecomment-710645488


   Just an FYI for future reviewers, we already have the following line in 
`Updating.md`:
   
   > The change is backwards compatible, setting provide_context will add the 
provide_context variable to the kwargs (but won't do anything).
   
   
https://github.com/apache/airflow/blob/master/UPDATING.md#airflowoperatorspythonpythonoperator
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #11597: Raises a warning for provide_context instead of killing the task

[GitBox]

mik-laj commented on pull request #11597:
URL: https://github.com/apache/airflow/pull/11597#issuecomment-710644860


   I don't know if using kwargs is the best idea in this case. I have to check 
if the IDE supports it.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11597: Raises a warning for provide_context instead of killing the task

[GitBox]

github-actions[bot] commented on pull request #11597:
URL: https://github.com/apache/airflow/pull/11597#issuecomment-710609083


   [The Workflow run](https://github.com/apache/airflow/actions/runs/311403614) 
is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static 
checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm 
tests$,^Test OpenAPI*.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ryanahamilton commented on a change in pull request #11583: UX Enhancement: Add button to clear search query from DAG search

[GitBox]

ryanahamilton commented on a change in pull request #11583:
URL: https://github.com/apache/airflow/pull/11583#discussion_r506714600



##
File path: airflow/www/templates/airflow/dags.html
##
@@ -313,6 +316,12 @@ DAGs
   }
 });
 
+$('#search_form').on('reset', function() {
+  var query = new URLSearchParams(window.location.search);
+  query.delete('search');

Review comment:
   I take that back… I was confusing the page param with how the 
search/tags/status params were being handled. Updated to delete the page param 
completely.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

ashb commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710607306


   > >  allow creating more tables when running airflow db upgrade via plugins 
for which it is needed that the plugin is loaded when Scheduler starts.
   > 
   > I still don't understand the use cases for this option. Why do we want to 
load plugins always and we can't do it on a specific case? If this is due to 
database migrations, then we can add loading plugins when the database is 
initialized.
   
   There is more cases then just db migrations - plugins wanting to do start up 
initialization, or preloading modules used in all dags as 
https://github.com/apache/airflow/pull/11596#issuecomment-710474026 was hinting 
it (he mentioned more explicitly in slack).
   
   This is a workaround until we add a richer plugin system to Airflow, but it 
is not onerous to support.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #8707: FIX Entrypoint and _CMD config variables (#8705)

[GitBox]

potiuk commented on pull request #8707:
URL: https://github.com/apache/airflow/pull/8707#issuecomment-710598955


   @NBardelot - are you going to complete that one?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] nullhack commented on pull request #9464: Fix DockerOperator xcom

[GitBox]

nullhack commented on pull request #9464:
URL: https://github.com/apache/airflow/pull/9464#issuecomment-710595586


   Ok, I'll change the code then to include deprecation warning and the new 
flag with default value as binary



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ryanahamilton commented on pull request #11583: UX Enhancement: Add button to clear search query from DAG search

[GitBox]

ryanahamilton commented on pull request #11583:
URL: https://github.com/apache/airflow/pull/11583#issuecomment-710591573


   I've got a follow-up to this PR in the works that will apply this same 
pattern to the tag filter as well:
   https://user-images.githubusercontent.com/3267/96306651-14b4b980-0fce-11eb-908a-ffd6d7d075e7.png;>
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #11543: Scheduler Deadlock in tests for MySQL 5.7

[GitBox]

potiuk commented on issue #11543:
URL: https://github.com/apache/airflow/issues/11543#issuecomment-710587828


   I'd love to get to the bottom of the problem. Retrying is really a 
workaround for the real problem in those cases, I believe.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mpermana commented on pull request #11592: Fix #11591 lost redirection

[GitBox]

mpermana commented on pull request #11592:
URL: https://github.com/apache/airflow/pull/11592#issuecomment-710583352


   @dimberman Added elaborate  description , no we don't need test , you know 
me, thank you



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11587: #10472: Add D200 pydocstyle check

[GitBox]

potiuk commented on pull request #11587:
URL: https://github.com/apache/airflow/pull/11587#issuecomment-710567148


   Can you please rebase @Samira-g-js  ? there are some conflicts :(



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (AIRFLOW-6829) Auto apply apply_default decorator to all sub-classess

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215639#comment-17215639
 ] 

ASF GitHub Bot commented on AIRFLOW-6829:
-

github-actions[bot] commented on pull request #7450:
URL: https://github.com/apache/airflow/pull/7450#issuecomment-710559724


   [The Workflow run](https://github.com/apache/airflow/actions/runs/311248383) 
is cancelling this PR. Building images for the PR has failed. Follow the the 
workflow link to check the reason.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Auto apply apply_default decorator to all sub-classess
> --
>
> Key: AIRFLOW-6829
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6829
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.9
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] github-actions[bot] commented on pull request #7450: [AIRFLOW-6829][WIP] Auto-apply apply_default

[GitBox]

github-actions[bot] commented on pull request #7450:
URL: https://github.com/apache/airflow/pull/7450#issuecomment-710559724


   [The Workflow run](https://github.com/apache/airflow/actions/runs/311248383) 
is cancelling this PR. Building images for the PR has failed. Follow the the 
workflow link to check the reason.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (44031bf -> 0c5bbe8)

[ash]

This is an automated email from the ASF dual-hosted git repository.

ash pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 44031bf  Fix broken migration for Sqlite3 (#11573)
 add 0c5bbe8  Replace methods on state with frozenset properties (#11576)

No new revisions were added by this update.

Summary of changes:
 airflow/api/common/experimental/mark_tasks.py|  2 +-
 airflow/jobs/backfill_job.py |  4 +-
 airflow/jobs/scheduler_job.py|  4 +-
 airflow/models/dagrun.py |  6 +--
 airflow/models/sensorinstance.py |  2 +-
 airflow/models/taskinstance.py   |  4 +-
 airflow/sensors/smart_sensor_operator.py |  2 +-
 airflow/ti_deps/dep_context.py   |  2 +-
 airflow/utils/state.py   | 68 +++-
 tests/api/common/experimental/test_mark_tasks.py |  2 +-
 tests/jobs/test_scheduler_job.py |  2 +-
 tests/ti_deps/deps/test_trigger_rule_dep.py  |  2 +-
 12 files changed, 47 insertions(+), 53 deletions(-)

[airflow] branch master updated (44031bf -> 0c5bbe8)

[ash]

This is an automated email from the ASF dual-hosted git repository.

ash pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 44031bf  Fix broken migration for Sqlite3 (#11573)
 add 0c5bbe8  Replace methods on state with frozenset properties (#11576)

No new revisions were added by this update.

Summary of changes:
 airflow/api/common/experimental/mark_tasks.py|  2 +-
 airflow/jobs/backfill_job.py |  4 +-
 airflow/jobs/scheduler_job.py|  4 +-
 airflow/models/dagrun.py |  6 +--
 airflow/models/sensorinstance.py |  2 +-
 airflow/models/taskinstance.py   |  4 +-
 airflow/sensors/smart_sensor_operator.py |  2 +-
 airflow/ti_deps/dep_context.py   |  2 +-
 airflow/utils/state.py   | 68 +++-
 tests/api/common/experimental/test_mark_tasks.py |  2 +-
 tests/jobs/test_scheduler_job.py |  2 +-
 tests/ti_deps/deps/test_trigger_rule_dep.py  |  2 +-
 12 files changed, 47 insertions(+), 53 deletions(-)

[GitHub] [airflow] mik-laj commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

mik-laj commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710545254


   >  allow creating more tables when running airflow db upgrade via plugins 
for which it is needed that the plugin is loaded when Scheduler starts.
   
   I still don't understand the use cases for this option. Why do we want to 
load plugins always and we can't do it on a specific case? If this is due to 
database migrations, then we can add loading plugins when the database is 
initialized.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb merged pull request #11576: Replace collection methods on State with frozenset properties

[GitBox]

ashb merged pull request #11576:
URL: https://github.com/apache/airflow/pull/11576


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

kaxil commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710538257


   > > to any DAG that needs to have access to plugin-provided operators, 
sensors or hooks, I suppose.
   > 
   > We plan to end support for operators and Hooks plugins in the near future.
   > See:
   > #9506
   > #9500
   
   https://github.com/apache/airflow/pull/11596#issuecomment-710481212 -- Yup



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

mik-laj commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710535998


   > to any DAG that needs to have access to plugin-provided operators, sensors 
or hooks, I suppose.
   
   We plan to end support for operators and Hooks plugins in the near future.
   See: 
   https://github.com/apache/airflow/issues/9506
   https://github.com/apache/airflow/issues/9500



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil edited a comment on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

kaxil edited a comment on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710532498


   > > allow creating more tables when running airflow db upgrade via plugins 
for which it is needed that the plugin is loaded when Scheduler starts.
   > 
   > Can we not load plugins in this one case? It makes sense to me that 
plugins can have separate migrations, but then I think it's worth documenting 
too.
   
   This is an optional feature, this does not change the current behavior. I 
have documented when users can load the plugin at startup of each Airflow 
process in 
https://github.com/apache/airflow/pull/11596/commits/5d9f709dc4231b7a8e238a27330644cae77fe4f5.
 Do you think we need more docs? What specifically?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil edited a comment on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

kaxil edited a comment on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710532498


   > > allow creating more tables when running airflow db upgrade via plugins 
for which it is needed that the plugin is loaded when Scheduler starts.
   > 
   > Can we not load plugins in this one case? It makes sense to me that 
plugins can have separate migrations, but then I think it's worth documenting 
too.
   
   This is an optional feature, this does not change the behavior. I have 
documented when users can load plugin at startup of each Airflow process in 
https://github.com/apache/airflow/pull/11596/commits/5d9f709dc4231b7a8e238a27330644cae77fe4f5.
 Do you think we need more docs? What specifically?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed pull request #11420: Using new version of setup-kind using new add-path mechanism

[GitBox]

potiuk closed pull request #11420:
URL: https://github.com/apache/airflow/pull/11420


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #10587: Add @dag decorator

[GitBox]

github-actions[bot] commented on pull request #10587:
URL: https://github.com/apache/airflow/pull/10587#issuecomment-710533432


   [The Workflow run](https://github.com/apache/airflow/actions/runs/311351853) 
is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static 
checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm 
tests$,^Test OpenAPI*.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

kaxil commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710532498


   > > allow creating more tables when running airflow db upgrade via plugins 
for which it is needed that the plugin is loaded when Scheduler starts.
   > 
   > Can we not load plugins in this one case? It makes sense to me that 
plugins can have separate migrations, but then I think it's worth documenting 
too.
   
   This is an optional feature, this does not change the behavior. I have 
documented when users can load plugin at startup of each Airflow process



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

mik-laj commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710530941


   > allow creating more tables when running airflow db upgrade via plugins for 
which it is needed that the plugin is loaded when Scheduler starts.
   
   Can we not load plugins in this one case? It makes sense to me that plugins 
can have separate migrations, but then I think it's worth documenting too.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil edited a comment on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

kaxil edited a comment on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710489207


   > What are the use cases for this option? Why would the user want to set 
these options to true?
   
   For a good number of examples listed here 
([link1](https://airflow.readthedocs.io/en/latest/plugins.html#what-for) and 
[link2](https://airflow.readthedocs.io/en/latest/plugins.html#why-build-on-top-of-airflow)),
 it is ideal to load plugins at the startup. One of the other use-cases is to 
allow creating more tables when running `airflow db upgrade` via plugins for 
which it is needed that the plugin is loaded when Scheduler starts.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #10587: Add @dag decorator

[GitBox]

github-actions[bot] commented on pull request #10587:
URL: https://github.com/apache/airflow/pull/10587#issuecomment-710489888


   [The Workflow run](https://github.com/apache/airflow/actions/runs/311319137) 
is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static 
checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm 
tests$,^Test OpenAPI*.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

kaxil commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710489207


   > What are the use cases for this option? Why would the user want to set 
these options to true?
   
   For a good number of examples listed here, it is ideal to load plugins at 
the startup. One of the other use-cases is to allow creating more tables when 
running `airflow db upgrade` via plugins for which it is needed that the plugin 
is loaded when Scheduler starts.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (4d611f2 -> 44031bf)

[msumit]

This is an automated email from the ASF dual-hosted git repository.

msumit pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 4d611f2  Clean up _trigger_dag function (#11584)
 add 44031bf  Fix broken migration for Sqlite3 (#11573)

No new revisions were added by this update.

Summary of changes:
 .../versions/98271e7606e2_add_scheduling_decision_to_dagrun_and_.py  | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

[GitHub] [airflow] msumit merged pull request #11573: Fix migration for Sqlite3

[GitBox]

msumit merged pull request #11573:
URL: https://github.com/apache/airflow/pull/11573


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

kaxil commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710481212


   > Another workaround is to just import these items directly, not from the 
`airflow.(hooks|operators|sensors).` namespace.
   
   For hooks, operators and sensors I would recommend not using Plugin, instead 
treat them as Python modules and import them directly
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mjpieters commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

mjpieters commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710478877


   Another workaround is to just import these items directly, not from the 
`airflow.(hooks|operators|sensors).` namespace.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

kaxil commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710476213


   > TBH, the lack of plugin loading in the scheduler _annoying_. I can also add
   > 
   > ```python
   > from airflow import plugins_manager
   > plugins_manager.ensure_plugins_loaded()
   > ```
   > 
   > to any DAG that needs to have access to plugin-provided operators, sensors 
or hooks, I suppose.
   
   Yeah, the flag in this PR should help you, having to add those import lines 
in all the DAGs is annoying.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

kaxil commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710476478


   > Can you add some docs to `docs/plugins.rst`? 
https://airflow.readthedocs.io/en/latest/plugins.html
   
   Added docs



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mjpieters commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

mjpieters commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710474026


   TBH, the lack of plugin loading in the scheduler _annoying_. I can also add
   
   ```python
   from airflow import plugins_manager
   plugins_manager.ensure_plugins_loaded()
   ```
   
   to any DAG that needs to have access to plugin-provided operators, sensors 
or hooks, I suppose.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (84c7028 -> 4d611f2)

[ash]

This is an automated email from the ASF dual-hosted git repository.

ash pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 84c7028  Fix tooltip typo (#11593)
 add 4d611f2  Clean up _trigger_dag function (#11584)

No new revisions were added by this update.

Summary of changes:
 airflow/api/common/experimental/trigger_dag.py|  18 ++--
 tests/api/common/experimental/test_trigger_dag.py | 108 +-
 2 files changed, 29 insertions(+), 97 deletions(-)

[GitHub] [airflow] ashb commented on a change in pull request #11566: Update bash_operator's doc_string

[GitBox]

ashb commented on a change in pull request #11566:
URL: https://github.com/apache/airflow/pull/11566#discussion_r506678353



##
File path: airflow/operators/bash.py
##
@@ -62,6 +62,13 @@ class BashOperator(BaseOperator):
 
 bash_command = "set -e; python3 script.py '{{ next_execution_date }}'"
 
+.. note::
+
+Add a space after the script name when directly calling a a ``.sh`` 
script with the

Review comment:
   Double `a a`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (84c7028 -> 4d611f2)

[ash]

This is an automated email from the ASF dual-hosted git repository.

ash pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 84c7028  Fix tooltip typo (#11593)
 add 4d611f2  Clean up _trigger_dag function (#11584)

No new revisions were added by this update.

Summary of changes:
 airflow/api/common/experimental/trigger_dag.py|  18 ++--
 tests/api/common/experimental/test_trigger_dag.py | 108 +-
 2 files changed, 29 insertions(+), 97 deletions(-)

[GitHub] [airflow] potix2 commented on a change in pull request #11434: Type annotation for aws operators in transfers

[GitBox]

potix2 commented on a change in pull request #11434:
URL: https://github.com/apache/airflow/pull/11434#discussion_r506678114



##
File path: airflow/providers/google/cloud/operators/bigquery.py
##
@@ -947,9 +947,7 @@ def execute(self, context) -> None:
 delegate_to=self.delegate_to,
 impersonation_chain=self.impersonation_chain,
 )
-schema_fields = json.loads(
-gcs_hook.download(gcs_bucket, gcs_object).decode("utf-8")  # 
type: ignore[attr-defined]
-)  # type: ignore[attr-defined]
+schema_fields = json.loads(gcs_hook.download(gcs_bucket, 
gcs_object))

Review comment:
   Thank you for your review. Static type checking is really cool because 
we can find like above the issues.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb merged pull request #11584: Clean up _trigger_dag function

[GitBox]

ashb merged pull request #11584:
URL: https://github.com/apache/airflow/pull/11584


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #11597: Raises a warning for provide_context instead of killing the task

[GitBox]

kaxil commented on a change in pull request #11597:
URL: https://github.com/apache/airflow/pull/11597#discussion_r506677382



##
File path: airflow/operators/python.py
##
@@ -85,6 +86,9 @@ def __init__(
 templates_exts: Optional[List[str]] = None,
 **kwargs
 ) -> None:
+if kwargs.get("provide_context"):
+warnings.warn("provide_context is deprecated as of 2.0 and is no 
longer required")
+kwargs.pop('provide_context', None)

Review comment:
   ```suggestion
   if kwargs.get("provide_context"):
   warnings.warn("provide_context is deprecated as of 2.0 and is no 
longer required", DeprecationWarning, stacklevel=2)
   kwargs.pop('provide_context', None)
   ```
   
   This should take care of @ashb 's suggestion





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mjpieters commented on pull request #11596: Allow loading plugins on Airflow start-up

[GitBox]

mjpieters commented on pull request #11596:
URL: https://github.com/apache/airflow/pull/11596#issuecomment-710463154


   Looks like I'll need this switch?
   
   We have dags that use plugin operators & sensors, and the scheduler fails to 
load our dags as the plugin is not loaded when processing each file.
   
   So `airflow dags list` works, `airflow dags show `  works, the 
webserver is able to show all the dags, but the `import_error` table shows that 
the _scheduler_ (or rather, the `DagFileProcessorProcess` job), can't load our 
dags due to the missing plugins.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] casassg commented on a change in pull request #10587: Add @dag decorator

[GitBox]

casassg commented on a change in pull request #10587:
URL: https://github.com/apache/airflow/pull/10587#discussion_r506673787



##
File path: docs/concepts.rst
##
@@ -162,6 +162,31 @@ Example DAG with decorated style
 html_content=email_info['body']
 )
 
+DAG decorator
+-
+
+.. versionadded:: 2.0.0
+
+In addition to creating DAGs using context managed, in Airflow 2.0 you can 
also create DAGs from a function.
+DAG decorator creates a DAG generator function. This function when called 
returns a DAG.
+
+DAG decorator also sets up the parameters you have in the function as DAG 
params. This allows you to parametrize
+your DAGs and set the parameters when triggering the DAG manually. See
+:ref:`Passing Parameters when triggering dags ` to learn 
how to pass parameters when triggering DAGs.
+
+Example DAG with decorator:
+
+.. exampleinclude:: /../airflow/example_dags/example_dag_decorator.py
+:language: python
+:dedent: 4
+:start-after: [START dag_decorator_usage]
+:end-before: [END dag_decorator_usage]
+
+.. note:: Note that Airflow will only load DAGs that appear in``globals()``. 
Which means you need to make sure to have
+  a variable for your returned DAG in the module scope. Otherwise Airflow 
won't detect your decorated DAG. In addition,
+  you may want to make your dag variable named ``DAG`` such that Airflow 
doesn't skip this module when the
+  ``DAG_DISCOVERY_SAFE_MODE`` is activated.

Review comment:
   Thoughts? I'm not sure how to run the system dags to make sure it runs 
CC @turbaszek 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org