[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17506007#comment-17506007
 ] 

ASF GitHub Bot commented on AIRFLOW-5071:
-

ghostbody commented on issue #10790:
URL: https://github.com/apache/airflow/issues/10790#issuecomment-1066406484


   @turbaszek Let me make a PR later~ We are doing pressure tests these days 
and this problem had appeared often.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Thousand os Executor reports task instance X finished (success) although the 
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png, 
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands 
> of daily messages like the following in the logs:
>  
> ```
>  {{__init__.py:1580}} ERROR - Executor reports task instance  2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says 
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance  2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says 
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the 
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [airflow] ghostbody commented on issue #10790: Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the task ki

[GitBox]

ghostbody commented on issue #10790:
URL: https://github.com/apache/airflow/issues/10790#issuecomment-1066406484


   @turbaszek Let me make a PR later~ We are doing pressure tests these days 
and this problem had appeared often.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #22102: Enhance `db upgrade` args

[GitBox]

dstandish commented on a change in pull request #22102:
URL: https://github.com/apache/airflow/pull/22102#discussion_r825605446



##
File path: airflow/cli/cli_parser.py
##
@@ -516,18 +516,32 @@ def string_list_type(val):
 type=int,
 default=60,
 )
-ARG_DB_VERSION = Arg(
+ARG_DB_UG_VERSION = Arg(
 (
 "-n",
 "--version",
 ),
-help="The airflow version to downgrade to. Note: must provide either 
`--revision` or `--version`.",
+help="(Optional) The airflow version to upgrade to. Note: must provide 
either `--revision` or `--version`.",
+)
+ARG_DB_UG_REVISION = Arg(

Review comment:
   Yup you're right that's better




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] nitinmahawadiwar edited a comment on issue #22007: Add Variable, Connection "description" fields available in the API

[GitBox]

nitinmahawadiwar edited a comment on issue #22007:
URL: https://github.com/apache/airflow/issues/22007#issuecomment-1066142969


   Hello @potiuk @eladkal , I am able to return the description field from get 
variable api, also found the places to update documentation  
(/api_connexion/openapi/v1.yaml - verifying locally) but somehow not able to 
understand the Test cases & its structure.
   Do you have any document to go through for test case implementation.
   I am referring https://github.com/apache/airflow/blob/main/TESTING.rst &
   updated  test cases in a) tests/models/test_variable.py & b) 
tests/api_connexion/endpoints/test_variable_endpoint.py 
   
   Thanks
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #22184: WIP - Add run_id and map_index to SlaMiss

[GitBox]

dstandish commented on a change in pull request #22184:
URL: https://github.com/apache/airflow/pull/22184#discussion_r825594340



##
File path: airflow/dag_processing/processor.py
##
@@ -375,21 +376,16 @@ def manage_slas(self, dag: DAG, session: Session = None) 
-> None:
 .group_by(TI.task_id)

Review comment:
   @ashb @uranusjr  i think we can add a `filter(TI.map_index==-1)` to 
exclude mapped tasks.  this, combined with a check when initializing a mapped 
operator  (check that no SLA  / warn) should effectively disable SLAs for 
mapped tasks i think 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #22184: WIP - Add run_id and map_index to SlaMiss

[GitBox]

dstandish commented on a change in pull request #22184:
URL: https://github.com/apache/airflow/pull/22184#discussion_r825593332



##
File path: airflow/models/taskinstance.py
##
@@ -425,6 +425,16 @@ class TaskInstance(Base, LoggingMixin):
 
 dag_run = relationship("DagRun", back_populates="task_instances", 
lazy='joined', innerjoin=True)
 rendered_task_instance_fields = relationship("RenderedTaskInstanceFields", 
lazy='noload', uselist=False)
+sla_miss = relationship(

Review comment:
   @ashb @uranusjr this is what i came up with for defining a "FK" 
relationship without actually defining a db-level FK constraint.  does this 
look right?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Bowrna commented on a change in pull request #22217: added docs about detailed network and constraints information

[GitBox]

Bowrna commented on a change in pull request #22217:
URL: https://github.com/apache/airflow/pull/22217#discussion_r825570397



##
File path: airflow/providers/docker/operators/docker_swarm.py
##
@@ -86,8 +86,19 @@ class DockerSwarmOperator(DockerOperator):
 :param mode: Indicate whether a service should be deployed as a replicated 
or global service,
 and associated parameters
 :param networks: List of network names or IDs or NetworkAttachmentConfig 
to attach the service to.
-:param placement: Placement instructions for the scheduler. If a list is 
passed instead,
-it is assumed to be a list of constraints as part of a Placement 
object.
+NetworkAttachmentConfig object accepts following params:

Review comment:
   yes, it makes sense. I have raised this PR in response to this issue 
#15974
   It looked like the constraints and network option is overlooked. therefore I 
have added this to the documentation. What do you suggest wrt this issue? 
@mik-laj 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] fatmumuhomer opened a new issue #22238: Option to display Last Run and Next Run based on data_interval_end

[GitBox]

fatmumuhomer opened a new issue #22238:
URL: https://github.com/apache/airflow/issues/22238


   ### Description
   
   Currently the DAGs view shows the Last Run and Next Run values using the 
data_interval_start. The full data interval information is available if the 
user hovers over the field. Airflow should allow these two fields to display 
the data_interval_end instead. This could be configurable at a system level via 
an environment variable or airflow.cfg or, if possible, per user.
   
   ### Use case/motivation
   
   There are two use case:
   1. To support users who are interested in monitoring data loads but who are 
not familiar with Airflow and may be confused by the concept of the data 
interval and, specifically, why the UI is showing the data_interval_start.
   2. The majority of the DAGs my team manages care more about the 
data_interval_end than the start and so we would, at a glance, prefer to be 
able to see the DAG information in relation to that.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] edithturn edited a comment on issue #19971: CI: Rewrite selective check script in Python

[GitBox]

edithturn edited a comment on issue #19971:
URL: https://github.com/apache/airflow/issues/19971#issuecomment-1066266872


   Thank you @potiuk , I will ping you Tuesday, Poland Time.  ⏲️ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] edithturn commented on issue #19971: CI: Rewrite selective check script in Python

[GitBox]

edithturn commented on issue #19971:
URL: https://github.com/apache/airflow/issues/19971#issuecomment-1066266872


   Thank you Jarek, I will ping you Tuesday, Poland Time.  ⏲️ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #19971: CI: Rewrite selective check script in Python

[GitBox]

potiuk commented on issue #19971:
URL: https://github.com/apache/airflow/issues/19971#issuecomment-1066264966


   Tuesday is fine. I will be mostly out since Wed though.
   
   On Mon, Mar 14, 2022 at 2:59 AM Edith Puclla ***@***.***>
   wrote:
   
   > @potiuk  do you have 10 minutes to discuss
   > this a little bit? Maybe Tuesday? I will bring some ideas and we can split
   > to start to work on it. 
   >
   > —
   > Reply to this email directly, view it on GitHub
   > ,
   > or unsubscribe
   > 

   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #22236: Addressed some issues in the tutorial mentioned in discussion #22233

[GitBox]

boring-cyborg[bot] commented on pull request #22236:
URL: https://github.com/apache/airflow/pull/22236#issuecomment-1066260971


   Congratulations on your first Pull Request and welcome to the Apache Airflow 
community! If you have any issues or are unsure about any anything please check 
our Contribution Guide 
(https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (flake8, mypy and type 
annotations). Our [pre-commits]( 
https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
 will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in 
`docs/` directory). Adding a new operator? Check this short 
[guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst)
 Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze 
environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for 
testing locally, it’s a heavy docker but it ships with a working Airflow and a 
lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get 
the final approval from Committers.
   - Please follow [ASF Code of 
Conduct](https://www.apache.org/foundation/policies/conduct) for all 
communication including (but not limited to) comments on Pull Requests, Mailing 
list and Slack.
   - Be sure to read the [Airflow Coding style]( 
https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
   Apache Airflow is a community-driven project and together we are making it 
better .
   In case of doubts contact the developers at:
   Mailing List: d...@airflow.apache.org
   Slack: https://s.apache.org/airflow-slack
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] MattTriano opened a new pull request #22236: Addressed some issues in the tutorial mentioned in discussion #22233

[GitBox]

MattTriano opened a new pull request #22236:
URL: https://github.com/apache/airflow/pull/22236


   While working through the tutorial, some parts seemed a bit unclear or 
incorrect, which took me to issue #18950 ( converted into discussion #22233 
about 15 minutes ago). I had already resolved some of the open concerns and saw 
potiuk encouraging people to improve the documentation, so I took it upon 
myself.
   
   Main Changes:
   * Wrote tasks to handle db-table-creation in the DAG rather than leave that 
as an exercise for the reader.
   * Added a step to ensure the existence of the data output directory in the 
`get_data()` task.
   * Fixed the logic in the `merge_data()` task so that its behavior matches 
the stated description.
   
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] edithturn commented on issue #19971: CI: Rewrite selective check script in Python

[GitBox]

edithturn commented on issue #19971:
URL: https://github.com/apache/airflow/issues/19971#issuecomment-1066260099


   @potiuk do you have 10 minutes to discuss this a little bit? Maybe Tuesday? 
I will bring some ideas and we can split to start to work on it.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Issue 20453 - Fixes the test_http and test_sheets assert calls only (#22104)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 3931394  Issue 20453 - Fixes the test_http and test_sheets assert 
calls only (#22104)
3931394 is described below

commit 393139449a54bd3afe6f59448ea8c63510830f76
Author: Harish Kesava Rao 
AuthorDate: Sun Mar 13 18:57:09 2022 -0700

Issue 20453 - Fixes the test_http and test_sheets assert calls only (#22104)
---
 tests/providers/google/suite/operators/test_sheets.py | 9 +++--
 tests/providers/http/operators/test_http.py   | 6 ++
 2 files changed, 5 insertions(+), 10 deletions(-)

diff --git a/tests/providers/google/suite/operators/test_sheets.py 
b/tests/providers/google/suite/operators/test_sheets.py
index 0bf2da4..c67ef41 100644
--- a/tests/providers/google/suite/operators/test_sheets.py
+++ b/tests/providers/google/suite/operators/test_sheets.py
@@ -39,12 +39,9 @@ class TestGoogleSheetsCreateSpreadsheet:
 op = GoogleSheetsCreateSpreadsheetOperator(
 task_id="test_task", spreadsheet=spreadsheet, 
gcp_conn_id=GCP_CONN_ID
 )
-op.execute(context)
+op_execute_result = op.execute(context)
 
 
mock_hook.return_value.create_spreadsheet.assert_called_once_with(spreadsheet=spreadsheet)
 
-calls = [
-mock.call(context, "spreadsheet_id", SPREADSHEET_ID),
-mock.call(context, "spreadsheet_url", SPREADSHEET_URL),
-]
-mock_xcom.has_calls(calls)
+assert op_execute_result['spreadsheetId'] == '1234567890'
+assert op_execute_result['spreadsheetUrl'] == 'https://example/sheets'
diff --git a/tests/providers/http/operators/test_http.py 
b/tests/providers/http/operators/test_http.py
index d4c622c..c4e342c 100644
--- a/tests/providers/http/operators/test_http.py
+++ b/tests/providers/http/operators/test_http.py
@@ -44,10 +44,8 @@ class TestSimpleHttpOp(unittest.TestCase):
 log_response=True,
 )
 
-with mock.patch.object(operator.log, 'info') as mock_info:
-operator.execute(None)
-calls = [mock.call('Example.com fake response'), 
mock.call('Example.com fake response')]
-mock_info.has_calls(calls)
+result = operator.execute('Example.com fake response')
+assert result == 'Example.com fake response'
 
 @requests_mock.mock()
 def test_response_in_logs_after_failed_check(self, m):

[GitHub] [airflow] potiuk merged pull request #22104: Issue 20453 - Fixes the test_http and test_sheets assert calls only

[GitBox]

potiuk merged pull request #22104:
URL: https://github.com/apache/airflow/pull/22104


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #20147: clean up temp files correctly when using impersonation

[GitBox]

potiuk commented on pull request #20147:
URL: https://github.com/apache/airflow/pull/20147#issuecomment-1066257530


   Checks + docs are failing though


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #20377: Dataplex operators

[GitBox]

potiuk commented on pull request #20377:
URL: https://github.com/apache/airflow/pull/20377#issuecomment-1066248962


   It needs at least rebase and checking if the errors were accidental.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #22160: Directly submitting task to executor doesn't set `queued_dttm` which can cause scheduler crashloop

[GitBox]

potiuk commented on issue #22160:
URL: https://github.com/apache/airflow/issues/22160#issuecomment-1066234187


   Agree. This is a valid case and solution 3. seems to be best.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #21539: Filter out default configs when overrides exist.

[GitBox]

potiuk edited a comment on pull request #21539:
URL: https://github.com/apache/airflow/pull/21539#issuecomment-1066233262


   After looking agian - I think what is also missing here is a bit "user" 
description in the 
https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html 
   
   I think we might have a lot of questions from users about the behaviour of 
config values as this is a "user" thing. I doubt our users will look in detail 
in "docstring" - they will mostly look at the documentation to understand how 
configuration precedence works, and especially - there should be a 
documentation explaining that it has changed in 2.3.0.
   
   Would it be possible tha tyou add such "user" facing documentation @xyu ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #21539: Filter out default configs when overrides exist.

[GitBox]

potiuk commented on pull request #21539:
URL: https://github.com/apache/airflow/pull/21539#issuecomment-1066233262


   After looking agian - I think what is also missing here is a bit "user" 
description in the 
https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html 
   
   I think we might have a lot of questions from users about the behaviour of 
config values as this is a "user" thing. I doubt our users will look in detail 
in "docstring" - they will mostly look at the documentation to understand how 
configuration precedence works, and especially - there should be a 
documentation explaining that it has changed in 2.3.0.
   
   Would it be possible tha tyou add such "user" facing documentation @xyu 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #20884: JSONDecodeError caused by incomplete JSON created by cat /airflow/xcom/return.json when using KubernetePodOperator

[GitBox]

uranusjr commented on issue #20884:
URL: https://github.com/apache/airflow/issues/20884#issuecomment-1066225166


   They say you need to file to https://github.com/kubernetes-client/python 
instead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #22227: Masking extras in GET /connections/ endpoint

[GitBox]

uranusjr commented on a change in pull request #7:
URL: https://github.com/apache/airflow/pull/7#discussion_r825528294



##
File path: airflow/api_connexion/schemas/connection_schema.py
##
@@ -44,7 +46,30 @@ class ConnectionSchema(ConnectionCollectionItemSchema):
 """Connection schema"""
 
 password = auto_field(load_only=True)
-extra = auto_field()
+extra = fields.Method('get_extra', deserialize='set_extra')
+
+@staticmethod
+def get_extra(obj: Connection):
+if obj.extra is None:
+return
+from airflow.utils.log.secrets_masker import redact
+
+try:
+extra = json.loads(obj.extra)
+return str(redact(extra))

Review comment:
   Should this use `json.dumps`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #22231: S3 list key filter

[GitBox]

uranusjr commented on a change in pull request #22231:
URL: https://github.com/apache/airflow/pull/22231#discussion_r825527820



##
File path: airflow/providers/amazon/aws/hooks/s3.py
##
@@ -255,6 +256,23 @@ def list_prefixes(
 
 return prefixes
 
+def _list_key_object_filter(
+self, keys: list, from_datetime: Optional[DateTime] = None, 
to_datetime: Optional[DateTime] = None
+) -> list:
+if from_datetime is None and to_datetime is None:
+return [k['Key'] for k in keys]
+elif to_datetime is None:
+return [k['Key'] for k in keys if k['LastModified'] >= 
from_datetime]
+elif from_datetime is None:
+return [k['Key'] for k in keys if k['LastModified'] < to_datetime]
+else:
+return [
+k['Key']
+for k in keys
+if k['LastModified'] >= from_datetime and k['LastModified'] < 
to_datetime
+]
+return [k['Key'] for k in keys]

Review comment:
   How about
   
   ```suggestion
   def _list_key_object_filter(
   self, keys: list, from_datetime: Optional[DateTime] = None, 
to_datetime: Optional[DateTime] = None
   ) -> list:
   def _is_in_period(dt: datetime) -> bool:
   if from_datetime is not None and dt < from_datetime:
   return False
   if to_datetime is not None and dt > to_datetime:
   return False
   return True
   
   return [k['Key'] for k in keys if _is_in_period(k['LastModified'])]
   ```

##
File path: airflow/providers/amazon/aws/hooks/s3.py
##
@@ -263,6 +281,10 @@ def list_keys(
 delimiter: Optional[str] = None,
 page_size: Optional[int] = None,
 max_items: Optional[int] = None,
+start_after_key: Optional[str] = None,
+from_datetime: Optional[DateTime] = None,
+to_datetime: Optional[DateTime] = None,

Review comment:
   I don’t think this needs to take `pendulum.DateTime`. Normal 
`datetime.datetime` works equally well (and is compatible with 
`pendulum.DateTime`).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #17975: Backfill blocks all tasks from being queued when a single pool is full.

[GitBox]

github-actions[bot] commented on issue #17975:
URL: https://github.com/apache/airflow/issues/17975#issuecomment-1066214696


   This issue has been automatically marked as stale because it has been open 
for 30 days with no response from the author. It will be closed in next 7 days 
if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] sunank200 opened a new pull request #22231: S3 list key filter

[GitBox]

sunank200 opened a new pull request #22231:
URL: https://github.com/apache/airflow/pull/22231


   Implemented as discussed in [closed 
PR](https://github.com/apache/airflow/pull/19018).
   
   Add more filter options to list_keys of S3Hook
   - `start_after_key`: should return only keys greater than this key
   - `from_datetime`: should return only keys with LastModified attr greater 
than this equal `from_datetime`.
   - `to_datetime`: should return only keys with LastModified attr less than 
this `to_datetime`.
   - `object_filter`: Function callable that receives the list of the S3 
objects, `from_datetime` and `to_datetime` and returns the List of the matched 
key.
   
   Add test for the added argument to `list_keys`.
   
   closes: #16627
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #22102: Enhance `db upgrade` args

[GitBox]

potiuk commented on a change in pull request #22102:
URL: https://github.com/apache/airflow/pull/22102#discussion_r825520725



##
File path: airflow/cli/cli_parser.py
##
@@ -516,18 +516,32 @@ def string_list_type(val):
 type=int,
 default=60,
 )
-ARG_DB_VERSION = Arg(
+ARG_DB_UG_VERSION = Arg(
 (
 "-n",
 "--version",
 ),
-help="The airflow version to downgrade to. Note: must provide either 
`--revision` or `--version`.",
+help="(Optional) The airflow version to upgrade to. Note: must provide 
either `--revision` or `--version`.",
+)
+ARG_DB_UG_REVISION = Arg(

Review comment:
   NIT: Why not UPGRADE/DOWNGRADE verbatim?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (12e9e2c -> 6f8f535)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 12e9e2c  Databricks hook - retry on HTTP Status 429 as well (#21852)
 add 6f8f535  Add sample dags and update doc for RedshiftClusterSensor, 
RedshiftPauseClusterOperator and RedshiftResumeClusterOperator (#22128)

No new revisions were added by this update.

Summary of changes:
 .../aws/example_dags/example_redshift_cluster.py   | 71 ++
 .../amazon/aws/sensors/redshift_cluster.py |  4 ++
 .../operators/redshift_cluster.rst | 67 +++-
 3 files changed, 127 insertions(+), 15 deletions(-)
 create mode 100644 
airflow/providers/amazon/aws/example_dags/example_redshift_cluster.py

[GitHub] [airflow] potiuk merged pull request #22128: Add sample dags and update doc for RedshiftClusterSensor, RedshiftPau…

[GitBox]

potiuk merged pull request #22128:
URL: https://github.com/apache/airflow/pull/22128


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #22152: render_template_as_native_obj=True in DAG constructor prevents failure e-mail from sending

[GitBox]

uranusjr commented on issue #22152:
URL: https://github.com/apache/airflow/issues/22152#issuecomment-1066206647


   Please also see #22218 for some context.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #22226: Add documentation for RC2 package release for Feb 2022

[GitBox]

github-actions[bot] commented on pull request #6:
URL: https://github.com/apache/airflow/pull/6#issuecomment-1066206473


   The PR is likely OK to be merged with just subset of tests for default 
Python and Database versions without running the full matrix of tests, because 
it does not modify the core of Airflow. If the committers decide that the full 
tests matrix is needed, they will add the label 'full tests needed'. Then you 
should rebase to the latest main or amend the last commit of the PR, and push 
it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #19736: Add Databricks Deferrable Operators

[GitBox]

potiuk commented on pull request #19736:
URL: https://github.com/apache/airflow/pull/19736#issuecomment-1066204353


   Some changes are merged so the conflicts need to be resolved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Databricks hook - retry on HTTP Status 429 as well (#21852)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 12e9e2c  Databricks hook - retry on HTTP Status 429 as well (#21852)
12e9e2c is described below

commit 12e9e2c695f9ebb9d3dde9c0f7dfaa112654f0d6
Author: Alex Ott 
AuthorDate: Mon Mar 14 00:19:01 2022 +0100

Databricks hook - retry on HTTP Status 429 as well (#21852)

* Databricks hook - retry on HTTP Status 429 as well

this fixes #21559

* Reimplement retries using tenacity

it's now uses exponential backoff by default
---
 airflow/providers/databricks/hooks/databricks.py   |   4 +-
 .../providers/databricks/hooks/databricks_base.py  | 196 +++--
 .../providers/databricks/operators/databricks.py   |   8 +
 .../operators/run_now.rst  |   2 +
 .../operators/submit_run.rst   |   2 +
 .../providers/databricks/hooks/test_databricks.py  |  61 ---
 .../databricks/operators/test_databricks.py|  45 -
 7 files changed, 195 insertions(+), 123 deletions(-)

diff --git a/airflow/providers/databricks/hooks/databricks.py 
b/airflow/providers/databricks/hooks/databricks.py
index 20255e3..cdab4d0 100644
--- a/airflow/providers/databricks/hooks/databricks.py
+++ b/airflow/providers/databricks/hooks/databricks.py
@@ -100,6 +100,7 @@ class DatabricksHook(BaseDatabricksHook):
 service outages.
 :param retry_delay: The number of seconds to wait between retries (it
 might be a floating point number).
+:param retry_args: An optional dictionary with arguments passed to 
``tenacity.Retrying`` class.
 """
 
 hook_name = 'Databricks'
@@ -110,8 +111,9 @@ class DatabricksHook(BaseDatabricksHook):
 timeout_seconds: int = 180,
 retry_limit: int = 3,
 retry_delay: float = 1.0,
+retry_args: Optional[Dict[Any, Any]] = None,
 ) -> None:
-super().__init__(databricks_conn_id, timeout_seconds, retry_limit, 
retry_delay)
+super().__init__(databricks_conn_id, timeout_seconds, retry_limit, 
retry_delay, retry_args)
 
 def run_now(self, json: dict) -> int:
 """
diff --git a/airflow/providers/databricks/hooks/databricks_base.py 
b/airflow/providers/databricks/hooks/databricks_base.py
index 1a0d639..f545f26 100644
--- a/airflow/providers/databricks/hooks/databricks_base.py
+++ b/airflow/providers/databricks/hooks/databricks_base.py
@@ -22,15 +22,16 @@ This hook enable the submitting and running of jobs to the 
Databricks platform.
 operators talk to the ``api/2.0/jobs/runs/submit``
 `endpoint `_.
 """
+import copy
 import sys
 import time
-from time import sleep
 from typing import Any, Dict, Optional, Tuple
 from urllib.parse import urlparse
 
 import requests
 from requests import PreparedRequest, exceptions as requests_exceptions
 from requests.auth import AuthBase, HTTPBasicAuth
+from tenacity import RetryError, Retrying, retry_if_exception, 
stop_after_attempt, wait_exponential
 
 from airflow import __version__
 from airflow.exceptions import AirflowException
@@ -68,6 +69,7 @@ class BaseDatabricksHook(BaseHook):
 service outages.
 :param retry_delay: The number of seconds to wait between retries (it
 might be a floating point number).
+:param retry_args: An optional dictionary with arguments passed to 
``tenacity.Retrying`` class.
 """
 
 conn_name_attr = 'databricks_conn_id'
@@ -89,17 +91,33 @@ class BaseDatabricksHook(BaseHook):
 timeout_seconds: int = 180,
 retry_limit: int = 3,
 retry_delay: float = 1.0,
+retry_args: Optional[Dict[Any, Any]] = None,
 ) -> None:
 super().__init__()
 self.databricks_conn_id = databricks_conn_id
 self.timeout_seconds = timeout_seconds
 if retry_limit < 1:
-raise ValueError('Retry limit must be greater than equal to 1')
+raise ValueError('Retry limit must be greater than or equal to 1')
 self.retry_limit = retry_limit
 self.retry_delay = retry_delay
 self.aad_tokens: Dict[str, dict] = {}
 self.aad_timeout_seconds = 10
 
+def my_after_func(retry_state):
+self._log_request_error(retry_state.attempt_number, 
retry_state.outcome)
+
+if retry_args:
+self.retry_args = copy.copy(retry_args)
+self.retry_args['retry'] = 
retry_if_exception(self._retryable_error)
+self.retry_args['after'] = my_after_func
+else:
+self.retry_args = dict(
+stop=stop_after_attempt(self.retry_limit),
+wait=wait_exponential(min=self.retry_delay, max=(2 ** 
retry_limit)),
+retry=retry_if_exception(self._retryable_error),
+

[GitHub] [airflow] potiuk merged pull request #21852: Databricks hook - retry on HTTP Status 429 as well

[GitBox]

potiuk merged pull request #21852:
URL: https://github.com/apache/airflow/pull/21852


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed issue #21559: Databricks hook: Retry also on HTTP Status 429 - rate limit exceeded

[GitBox]

potiuk closed issue #21559:
URL: https://github.com/apache/airflow/issues/21559


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #22228: Installing Apache Hive provider package results in sasl error

[GitBox]

boring-cyborg[bot] commented on issue #8:
URL: https://github.com/apache/airflow/issues/8#issuecomment-1066197295


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] klazaj opened a new issue #22228: Installing Apache Hive provider package results in sasl error

[GitBox]

klazaj opened a new issue #8:
URL: https://github.com/apache/airflow/issues/8


   ### Apache Airflow version
   
   2.2.4 (latest released)
   
   ### What happened
   
   I'm trying to add Hive Connection using the apache-hive provider from 
airflow.
   
   `RUN pip3 install --no-cache-dir apache-airflow-providers-apache-hive==2.3.0`
   
it results in an error:
   ```
   Building wheel for sasl (setup.py): started
 Building wheel for sasl (setup.py): finished with status 'error'
 ERROR: Command errored out with exit status 1:
  command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, 
tokenize; sys.argv[0] = 
'"'"'/tmp/pip-install-hz7mix5i/sasl_6bf120bbc77044dc8d7584de06a0a0b1/setup.py'"'"';
 
__file__='"'"'/tmp/pip-install-hz7mix5i/sasl_6bf120bbc77044dc8d7584de06a0a0b1/setup.py'"'"';f
 = getattr(tokenize, '"'"'open'"'"', open)(__file__) if 
os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; 
setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', 
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' 
bdist_wheel -d /tmp/pip-wheel-ohtsw15a
  cwd: /tmp/pip-install-hz7mix5i/sasl_6bf120bbc77044dc8d7584de06a0a0b1/
 Complete output (26 lines):
 running bdist_wheel
 running build
 running build_py
 creating build
 creating build/lib.linux-x86_64-3.7
 creating build/lib.linux-x86_64-3.7/sasl
 copying sasl/__init__.py -> build/lib.linux-x86_64-3.7/sasl
 running egg_info
 writing sasl.egg-info/PKG-INFO
 writing dependency_links to sasl.egg-info/dependency_links.txt
 writing requirements to sasl.egg-info/requires.txt
 writing top-level names to sasl.egg-info/top_level.txt
 reading manifest file 'sasl.egg-info/SOURCES.txt'
 reading manifest template 'MANIFEST.in'
 adding license file 'LICENSE.txt'
 writing manifest file 'sasl.egg-info/SOURCES.txt'
 copying sasl/saslwrapper.cpp -> build/lib.linux-x86_64-3.7/sasl
 copying sasl/saslwrapper.h -> build/lib.linux-x86_64-3.7/sasl
 copying sasl/saslwrapper.pyx -> build/lib.linux-x86_64-3.7/sasl
 running build_ext
 building 'sasl.saslwrapper' extension
 creating build/temp.linux-x86_64-3.7
 creating build/temp.linux-x86_64-3.7/sasl
 gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 
-Wall -fPIC -Isasl -I/usr/local/include/python3.7m -c sasl/saslwrapper.cpp -o 
build/temp.linux-x86_64-3.7/sasl/saslwrapper.o
 unable to execute 'gcc': No such file or directory
 error: command 'gcc' failed with exit status 1
 
 ERROR: Failed building wheel for sasl
 Running setup.py clean for sasl
 Building wheel for pyhive (setup.py): started
 Building wheel for pyhive (setup.py): finished with status 'done'
 Created wheel for pyhive: filename=PyHive-0.6.5-py3-none-any.whl 
size=51574 
sha256=968710edee6b9cf14452f8cb8afae35a192d3d2d4b1fec8afcc89cc3e9cd0347
 Stored in directory: 
/tmp/pip-ephem-wheel-cache-ih9zbbs8/wheels/5e/89/eb/d5871485edc5db37f651909a8713501b2ba4d400b52816f271
 Building wheel for pure-sasl (setup.py): started
 Building wheel for pure-sasl (setup.py): finished with status 'done'
 Created wheel for pure-sasl: filename=pure_sasl-0.6.2-py3-none-any.whl 
size=11443 
sha256=5237949f406e0cf98060d806cc7f2c05aeaae665bdb519eb5be8d17a2cb6a015
 Stored in directory: 
/tmp/pip-ephem-wheel-cache-ih9zbbs8/wheels/60/6d/c4/eeb9f5eae120dce84399f49fc92b447f5b9c2788217aee3209
   Successfully built thrift pyhive pure-sasl
   Failed to build sasl
   Installing collected packages: thrift, pure-sasl, thrift-sasl, sasl, pyhive, 
hmsclient, apache-airflow-providers-apache-hive
   Running setup.py install for sasl: started
   Running setup.py install for sasl: finished with status 'error'
   ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, 
tokenize; sys.argv[0] = 
'"'"'/tmp/pip-install-hz7mix5i/sasl_6bf120bbc77044dc8d7584de06a0a0b1/setup.py'"'"';
 
__file__='"'"'/tmp/pip-install-hz7mix5i/sasl_6bf120bbc77044dc8d7584de06a0a0b1/setup.py'"'"';f
 = getattr(tokenize, '"'"'open'"'"', open)(__file__) if 
os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; 
setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', 
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install 
--record /tmp/pip-record-zzc1wx3h/install-record.txt 
--single-version-externally-managed --user --prefix= --compile 
--install-headers /home/airflow/.local/include/python3.7m/sasl
cwd: 
/tmp/pip-install-hz7mix5i/sasl_6bf120bbc77044dc8d7584de06a0a0b1/
   Complete output (28 lines):
   running install
   
/home/airflow/.local/lib/python3.7/site-packages/setuptools/command/install.py:37:
 SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and 
pip and other standards-based 

[GitHub] [airflow] SamWheating opened a new pull request #22227: Masking extras in GET /connections/ endpoint

[GitBox]

SamWheating opened a new pull request #7:
URL: https://github.com/apache/airflow/pull/7


   Applying secrets masking when reading the `extras` field of a connection via 
the REST API.
   
   Users can still update the extras with the `patch` or `post` endpoints, but 
when reading them they will be recursively redacted.
   
   Also adding some commonly used sensitive `extras` keys to the default 
sensitive values so that they will be properly redacted.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #22226: Add documentation for RC2 package release for Feb 2022

[GitBox]

potiuk commented on pull request #6:
URL: https://github.com/apache/airflow/pull/6#issuecomment-1066192737


   Docs for RC2 of Feb providers:
   
   * amazon and dbt.cloud - bug fixes
   * snowflake and databricks- Upcoming Python 3.10-related changes
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request #22226: Add documentation for RC2 package release for Feb 2022

[GitBox]

potiuk opened a new pull request #6:
URL: https://github.com/apache/airflow/pull/6


   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ulsc commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

ulsc commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825503425



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = (
+'local_paths',
+'drive_folder',
+)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+delegate_to: Optional[str] = None,
+impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.local_paths = local_paths
+self.drive_folder = drive_folder
+self.gcp_conn_id = gcp_conn_id
+self.delete = delete
+self.chunk_size = chunk_size
+self.resumable = resumable
+self.delegate_to = delegate_to
+self.impersonation_chain = impersonation_chain
+
+def execute(self, context: Any) -> Sequence[str]:

Review comment:
   sure




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #22204: Add description on the vendoring process we use

[GitBox]

potiuk commented on pull request #22204:
URL: https://github.com/apache/airflow/pull/22204#issuecomment-1066180414


   First in-line to get finally Python 3.10 support :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request #22225: Optimize Multiplatform cache builds

[GitBox]

potiuk opened a new pull request #5:
URL: https://github.com/apache/airflow/pull/5


   This PR starts an ARM EC2 instance and forwards socket via SSH
   so that it can be used by docker buildx build with airflow_cache
   multi-platform builders.
   
   The instance is created only when "main" build reaches the cache
   build job/step. The instances run for maximum 50 minutes and
   then self-terminate, also the instance is killed when the job
   either succeeds or fails.
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on issue #21951: Tasks stuck in scheduled state from different dags waiting for other dag tasks to complete first

[GitBox]

ephraimbuddy commented on issue #21951:
URL: https://github.com/apache/airflow/issues/21951#issuecomment-1066179851


   > > max_active_tasks_per_dag == dag_concurrency.
   > > There was a rename of dag_concurrency to max_active_tasks_per_dag. Can 
you set this correctly?
   > 
   > @ephraimbuddy I do have the "max_active_tasks_per_dag" set as well, are 
you saying i should remove "dag_concurrency" setting itself and just have the 
"max_active_tasks_per_dag" and "parallelism" defined?
   
   Yes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on issue #22059: DagBag import timeout while retrieving variable

[GitBox]

ephraimbuddy commented on issue #22059:
URL: https://github.com/apache/airflow/issues/22059#issuecomment-1066179734


   > Yes variable is there. This is happening intermittently.
   
   Can you follow this best practice for variables: 
https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#best-practices-airflow-variables
   
   If you are not able to resolve it with the above, please explain how to 
reproduce it
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ulsc commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

ulsc commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825502086



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = (
+'local_paths',
+'drive_folder',
+)
+
+def __init__(
+self,
+local_paths,

Review comment:
   sure




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ulsc commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

ulsc commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825502032



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = (
+'local_paths',
+'drive_folder',
+)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+delegate_to: Optional[str] = None,
+impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.local_paths = local_paths
+self.drive_folder = drive_folder
+self.gcp_conn_id = gcp_conn_id
+self.delete = delete
+self.chunk_size = chunk_size
+self.resumable = resumable
+self.delegate_to = delegate_to
+self.impersonation_chain = impersonation_chain
+
+def execute(self, context: Any) -> Sequence[str]:
+hook = GoogleDriveHook(
+gcp_conn_id=self.gcp_conn_id,
+delegate_to=self.delegate_to,
+impersonation_chain=self.impersonation_chain,
+)
+
+remote_file_ids = []
+
+for local_path in self.local_paths:
+self.log.info(f'Uploading file to Google Drive: {local_path}')
+file_name = local_path.split('/')[-1]

Review comment:
   How about changing both local_paths and drive_folder to use `Path`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ulsc commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

ulsc commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825501469



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = (
+'local_paths',
+'drive_folder',
+)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+delegate_to: Optional[str] = None,
+impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.local_paths = local_paths
+self.drive_folder = drive_folder
+self.gcp_conn_id = gcp_conn_id
+self.delete = delete
+self.chunk_size = chunk_size
+self.resumable = resumable
+self.delegate_to = delegate_to
+self.impersonation_chain = impersonation_chain
+
+def execute(self, context: Any) -> Sequence[str]:
+hook = GoogleDriveHook(
+gcp_conn_id=self.gcp_conn_id,
+delegate_to=self.delegate_to,
+impersonation_chain=self.impersonation_chain,
+)
+
+remote_file_ids = []
+
+for local_path in self.local_paths:
+self.log.info(f'Uploading file to Google Drive: {local_path}')
+file_name = local_path.split('/')[-1]
+
+try:
+remote_file_id = hook.upload_file(
+local_location=local_path,
+remote_location=os.path.join(self.drive_folder, file_name),
+chunk_size=self.chunk_size,
+resumable=self.resumable,
+)
+
+remote_file_ids.append(remote_file_id)
+
+if self.delete:
+os.remove(local_path)
+self.log.info(f'Deleted local file: {local_path}')
+except FileNotFoundError:
+

[GitHub] [airflow] ephraimbuddy closed issue #22096: Status of testing of Apache Airflow Helm Chart 1.5.0rc2

[GitBox]

ephraimbuddy closed issue #22096:
URL: https://github.com/apache/airflow/issues/22096


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on issue #22096: Status of testing of Apache Airflow Helm Chart 1.5.0rc2

[GitBox]

ephraimbuddy commented on issue #22096:
URL: https://github.com/apache/airflow/issues/22096#issuecomment-1066177479


   Thanks, everyone, 1.5.0 has been released


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ulsc commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

ulsc commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825500406



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = (
+'local_paths',
+'drive_folder',
+)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,

Review comment:
   sure




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Updating constraints. Build id:1977305453

[github-bot]

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 83ddc73  Updating constraints. Build id:1977305453
83ddc73 is described below

commit 83ddc736a9df8c7f77949e2ba8944eb79fc0d520
Author: Automated GitHub Actions commit 
AuthorDate: Sun Mar 13 20:30:44 2022 +

Updating constraints. Build id:1977305453

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
HEAD of 'refs/heads/main' in 'apache/airflow'
with commit sha c108f264abde68e8f458a401296a53ccbe7a47f6.

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.7.txt  | 4 ++--
 constraints-3.8.txt  | 4 ++--
 constraints-3.9.txt  | 4 ++--
 constraints-no-providers-3.7.txt | 4 ++--
 constraints-no-providers-3.8.txt | 4 ++--
 constraints-no-providers-3.9.txt | 4 ++--
 constraints-source-providers-3.7.txt | 4 ++--
 constraints-source-providers-3.8.txt | 4 ++--
 constraints-source-providers-3.9.txt | 4 ++--
 9 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 2e47713..8ac08f0 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-03-13T17:43:30Z
+# This constraints file was automatically generated on 2022-03-13T20:25:20Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -301,7 +301,7 @@ identify==2.4.11
 idna==3.3
 ijson==3.1.4
 imagesize==1.3.0
-importlib-metadata==4.11.2
+importlib-metadata==4.11.3
 importlib-resources==5.4.0
 inflection==0.5.1
 influxdb-client==1.26.0
diff --git a/constraints-3.8.txt b/constraints-3.8.txt
index 4749529..39f894f 100644
--- a/constraints-3.8.txt
+++ b/constraints-3.8.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-03-13T17:43:03Z
+# This constraints file was automatically generated on 2022-03-13T20:13:01Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -302,7 +302,7 @@ identify==2.4.11
 idna==3.3
 ijson==3.1.4
 imagesize==1.3.0
-importlib-metadata==4.11.2
+importlib-metadata==4.11.3
 importlib-resources==5.4.0
 inflection==0.5.1
 influxdb-client==1.26.0
diff --git a/constraints-3.9.txt b/constraints-3.9.txt
index f65faef..1361bcf 100644
--- a/constraints-3.9.txt
+++ b/constraints-3.9.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-03-13T17:43:31Z
+# This constraints file was automatically generated on 2022-03-13T20:13:00Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -300,7 +300,7 @@ identify==2.4.11
 idna==3.3
 ijson==3.1.4
 imagesize==1.3.0
-importlib-metadata==4.11.2
+importlib-metadata==4.11.3
 inflection==0.5.1
 influxdb-client==1.26.0
 iniconfig==1.1.1
diff --git a/constraints-no-providers-3.7.txt b/constraints-no-providers-3.7.txt
index 70b5683..e496067 100644
--- a/constraints-no-providers-3.7.txt
+++ b/constraints-no-providers-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-03-13T17:47:13Z
+# This constraints file was automatically generated on 2022-03-13T20:30:35Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install just the 'bare' 'apache-airflow' package 
build from the HEAD of
 # the branch, without installing any of the providers.
@@ -95,7 +95,7 @@ httpcore==0.14.7
 httpx==0.22.0
 humanize==4.0.0
 idna==3.3
-importlib-metadata==4.11.2
+importlib-metadata==4.11.3
 importlib-resources==5.4.0
 inflection==0.5.1
 itsdangerous==1.1.0
diff --git a/constraints-no-providers-3.8.txt b/constraints-no-providers-3.8.txt
index a1763fa..169bc14 100644
--- a/constraints-no-providers-3.8.txt
+++ b/constraints-no-providers-3.8.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-03-13T17:47:08Z
+# This constraints file was automatically generated on 2022-03-13T20:30:25Z
 # via "eager-upgrade" mechanism of PIP. 

[GitHub] [airflow] kyan113 commented on issue #22152: render_template_as_native_obj=True in DAG constructor prevents failure e-mail from sending

[GitBox]

kyan113 commented on issue #22152:
URL: https://github.com/apache/airflow/issues/22152#issuecomment-1066172041


   If you'd like.  I don't think I have the expertise to put together a pull 
request, so someone else will need to take a crack at it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (af9d85c -> c108f26)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from af9d85c  Skip some tests for Databricks from running on Python 3.10 
(#1)
 add c108f26  Change the default `chunk_size` to a clear representation & 
add documentation (#2)

No new revisions were added by this update.

Summary of changes:
 airflow/providers/google/suite/hooks/drive.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

[GitHub] [airflow] potiuk merged pull request #22222: Make `chunk_size` in `GoogleDriveHook` standard

[GitBox]

potiuk merged pull request #2:
URL: https://github.com/apache/airflow/pull/2


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #22222: Make `chunk_size` in `GoogleDriveHook` standard

[GitBox]

github-actions[bot] commented on pull request #2:
URL: https://github.com/apache/airflow/pull/2#issuecomment-1066163363


   The PR is likely OK to be merged with just subset of tests for default 
Python and Database versions without running the full matrix of tests, because 
it does not modify the core of Airflow. If the committers decide that the full 
tests matrix is needed, they will add the label 'full tests needed'. Then you 
should rebase to the latest main or amend the last commit of the PR, and push 
it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

turbaszek commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825490843



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = (
+'local_paths',
+'drive_folder',
+)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+delegate_to: Optional[str] = None,
+impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.local_paths = local_paths
+self.drive_folder = drive_folder
+self.gcp_conn_id = gcp_conn_id
+self.delete = delete
+self.chunk_size = chunk_size
+self.resumable = resumable
+self.delegate_to = delegate_to
+self.impersonation_chain = impersonation_chain
+
+def execute(self, context: Any) -> Sequence[str]:
+hook = GoogleDriveHook(
+gcp_conn_id=self.gcp_conn_id,
+delegate_to=self.delegate_to,
+impersonation_chain=self.impersonation_chain,
+)
+
+remote_file_ids = []
+
+for local_path in self.local_paths:
+self.log.info(f'Uploading file to Google Drive: {local_path}')
+file_name = local_path.split('/')[-1]

Review comment:
   I would suggest using `Path` from 
[pathlib](https://docs.python.org/3/library/pathlib.html) so you can do 
`local_path.name` instead of splitting string.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

turbaszek commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825490555



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = (
+'local_paths',
+'drive_folder',
+)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+delegate_to: Optional[str] = None,
+impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.local_paths = local_paths
+self.drive_folder = drive_folder
+self.gcp_conn_id = gcp_conn_id
+self.delete = delete
+self.chunk_size = chunk_size
+self.resumable = resumable
+self.delegate_to = delegate_to
+self.impersonation_chain = impersonation_chain
+
+def execute(self, context: Any) -> Sequence[str]:
+hook = GoogleDriveHook(
+gcp_conn_id=self.gcp_conn_id,
+delegate_to=self.delegate_to,
+impersonation_chain=self.impersonation_chain,
+)
+
+remote_file_ids = []
+
+for local_path in self.local_paths:
+self.log.info(f'Uploading file to Google Drive: {local_path}')
+file_name = local_path.split('/')[-1]
+
+try:
+remote_file_id = hook.upload_file(
+local_location=local_path,
+remote_location=os.path.join(self.drive_folder, file_name),
+chunk_size=self.chunk_size,
+resumable=self.resumable,
+)
+
+remote_file_ids.append(remote_file_id)
+
+if self.delete:
+os.remove(local_path)
+self.log.info(f'Deleted local file: {local_path}')
+except FileNotFoundError:
+

[GitHub] [airflow] turbaszek commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

turbaszek commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825490457



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = (
+'local_paths',
+'drive_folder',
+)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+delegate_to: Optional[str] = None,
+impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.local_paths = local_paths
+self.drive_folder = drive_folder
+self.gcp_conn_id = gcp_conn_id
+self.delete = delete
+self.chunk_size = chunk_size
+self.resumable = resumable
+self.delegate_to = delegate_to
+self.impersonation_chain = impersonation_chain
+
+def execute(self, context: Any) -> Sequence[str]:

Review comment:
   ```suggestion
   def execute(self, context: "Context") -> Sequence[str]:
   ```
   
   You will need:
   ```
   from typing import TYPE_CHECKING
   
   if TYPE_CHECKING:
   from airflow.utils.context import Context
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

turbaszek commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825490206



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = (
+'local_paths',
+'drive_folder',
+)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,

Review comment:
   Can we use default conn name as we do in other Google operators?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

turbaszek commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825490158



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = (
+'local_paths',
+'drive_folder',
+)
+
+def __init__(
+self,
+local_paths,

Review comment:
   Please add type hint




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Tonkonozhenko opened a new issue #22224: Task runs second time after success

[GitBox]

Tonkonozhenko opened a new issue #4:
URL: https://github.com/apache/airflow/issues/4


   ### Apache Airflow version
   
   2.2.4 (latest released)
   
   ### What happened
   
   Task runs a second time after a first successful run.
   In logs I see suspicious line: 
   ```
   [2022-03-13, 09:10:59 UTC] {local_task_job.py:99} INFO - Task is not able to 
be run
   ```
   
   Full log:
   ```*** Reading remote log from 
s3://.../airflow/dag_id/task_id/2022-03-13T08:10:00+00:00/1.log.
   [2022-03-13, 09:10:59 UTC] {taskinstance.py:1027} INFO - Dependencies not 
met for , dependency 'Task Instance State' FAILED: Task is in the 
'scheduled' state.
   [2022-03-13, 09:10:59 UTC] {local_task_job.py:99} INFO - Task is not able to 
be run
   
   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1037} INFO - Dependencies all 
met for 
   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1037} INFO - Dependencies all 
met for 
   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1243} INFO -
   

   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1244} INFO - Starting attempt 1 
of 3
   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1245} INFO -
   

   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1264} INFO - Executing 
 on 2022-03-13 08:10:00+00:00
   [2022-03-13, 09:11:16 UTC] {standard_task_runner.py:52} INFO - Started 
process 17071 to run task
   [2022-03-13, 09:11:16 UTC] {standard_task_runner.py:76} INFO - Running: 
['airflow', 'tasks', 'run', 'dag_id', 'task_id', 
'scheduled__2022-03-13T08:10:00+00:00', '--job-id', '1331338', '--raw', 
'--subdir', 'DAGS_FOLDER/dag_id.py', '--cfg-path', '/tmp/tmp6jzxb7ar', 
'--error-file', '/tmp/tmpru98mcz9']
   [2022-03-13, 09:11:16 UTC] {standard_task_runner.py:77} INFO - Job 1331338: 
Subtask task_id
   [2022-03-13, 09:11:16 UTC] {logging_mixin.py:109} INFO - Running 
 
on host airflow-official-worker-7d594658ff-9d4nf
   [2022-03-13, 09:11:17 UTC] {taskinstance.py:1429} INFO - Exporting the 
following env vars:
   AIRFLOW_CTX_DAG_EMAIL=...
   AIRFLOW_CTX_DAG_OWNER=airflow
   AIRFLOW_CTX_DAG_ID=dag_id
   AIRFLOW_CTX_TASK_ID=task_id
   AIRFLOW_CTX_EXECUTION_DATE=2022-03-13T08:10:00+00:00
   AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-03-13T08:10:00+00:00
   [2022-03-13, 09:11:17 UTC] {python.py:175} INFO - Done. Returned value was: 
None
   [2022-03-13, 09:11:17 UTC] {taskinstance.py:1272} INFO - Marking task as 
SUCCESS. dag_id=dag_id, task_id=task_id, execution_date=20220313T081000, 
start_date=20220313T091116, end_date=20220313T091117
   [2022-03-13, 09:11:17 UTC] {local_task_job.py:154} INFO - Task exited with 
return code 0
   [2022-03-13, 09:11:17 UTC] {local_task_job.py:264} INFO - 0 downstream tasks 
scheduled from follow-on schedule check
   ```
   
   ### What you expected to happen
   
   _No response_
   
   ### How to reproduce
   
   _No response_
   
   ### Operating System
   
   Debian 10, extended official airflow image
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   We have some hourly dags and it happens once per few days.
   Full Logs: 
   ```*** Reading remote log from 
s3://.../airflow/dag_id/task_id/2022-03-13T08:10:00+00:00/1.log.
   [2022-03-13, 09:10:59 UTC] {taskinstance.py:1027} INFO - Dependencies not 
met for , dependency 'Task Instance State' FAILED: Task is in the 
'scheduled' state.
   [2022-03-13, 09:10:59 UTC] {local_task_job.py:99} INFO - Task is not able to 
be run
   
   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1037} INFO - Dependencies all 
met for 
   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1037} INFO - Dependencies all 
met for 
   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1243} INFO -
   

   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1244} INFO - Starting attempt 1 
of 3
   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1245} INFO -
   

   [2022-03-13, 09:11:16 UTC] {taskinstance.py:1264} INFO - Executing 
 on 2022-03-13 08:10:00+00:00
   [2022-03-13, 09:11:16 UTC] {standard_task_runner.py:52} INFO - Started 
process 17071 to run task
   [2022-03-13, 09:11:16 UTC] {standard_task_runner.py:76} INFO - Running: 
['airflow', 'tasks', 'run', 'dag_id', 'task_id', 
'scheduled__2022-03-13T08:10:00+00:00', '--job-id', '1331338', '--raw', 
'--subdir', 'DAGS_FOLDER/aws_transforms.py', '--cfg-path', '/tmp/tmp6jzxb7ar', 
'--error-file', '/tmp/tmpru98mcz9']
   [2022-03-13, 09:11:16 UTC] {standard_task_runner.py:77} INFO - Job 1331338: 
Subtask task_id
   [2022-03-13, 09:11:16 UTC] {logging_mixin.py:109} INFO - Running 
 
on host airflow-official-worker-7d594658ff-9d4nf
   

[GitHub] [airflow] potiuk closed pull request #22050: Add Python 3.10 support

[GitBox]

potiuk closed pull request #22050:
URL: https://github.com/apache/airflow/pull/22050


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] likecodingloveproblems edited a comment on issue #22152: render_template_as_native_obj=True in DAG constructor prevents failure e-mail from sending

[GitBox]

likecodingloveproblems edited a comment on issue #22152:
URL: https://github.com/apache/airflow/issues/22152#issuecomment-1066153902


   Would I work on this issue please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] likecodingloveproblems commented on issue #22152: render_template_as_native_obj=True in DAG constructor prevents failure e-mail from sending

[GitBox]

likecodingloveproblems commented on issue #22152:
URL: https://github.com/apache/airflow/issues/22152#issuecomment-1066153902


   Would I work on this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Updating constraints. Build id:1976939596

[github-bot]

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 0cf1688  Updating constraints. Build id:1976939596
0cf1688 is described below

commit 0cf1688125636bcbb8c8236a33de3258772745c8
Author: Automated GitHub Actions commit 
AuthorDate: Sun Mar 13 17:47:22 2022 +

Updating constraints. Build id:1976939596

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
HEAD of 'refs/heads/main' in 'apache/airflow'
with commit sha af9d85ccd8abdc3c252c19764d3ea16970ae0f20.

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.7.txt  | 10 +-
 constraints-3.8.txt  | 10 +-
 constraints-3.9.txt  | 10 +-
 constraints-no-providers-3.7.txt |  2 +-
 constraints-no-providers-3.8.txt |  2 +-
 constraints-no-providers-3.9.txt |  2 +-
 constraints-source-providers-3.7.txt |  2 +-
 constraints-source-providers-3.8.txt |  2 +-
 constraints-source-providers-3.9.txt |  2 +-
 9 files changed, 9 insertions(+), 33 deletions(-)

diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 006afa0..2e47713 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-03-13T14:19:39Z
+# This constraints file was automatically generated on 2022-03-13T17:43:30Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -86,20 +86,14 @@ apache-airflow-providers-docker==2.5.0
 apache-airflow-providers-elasticsearch==3.0.0
 apache-airflow-providers-exasol==2.1.1
 apache-airflow-providers-facebook==2.2.1
-apache-airflow-providers-ftp==2.1.0
-apache-airflow-providers-github==1.0.1
 apache-airflow-providers-google==6.5.0
 apache-airflow-providers-grpc==2.0.2
 apache-airflow-providers-hashicorp==2.1.2
-apache-airflow-providers-http==2.1.0
-apache-airflow-providers-imap==2.2.1
-apache-airflow-providers-influxdb==1.1.1
 apache-airflow-providers-jdbc==2.1.1
 apache-airflow-providers-jenkins==2.0.5
 apache-airflow-providers-jira==2.0.2
 apache-airflow-providers-microsoft-azure==3.7.0
 apache-airflow-providers-microsoft-mssql==2.1.1
-apache-airflow-providers-microsoft-psrp==1.1.1
 apache-airflow-providers-microsoft-winrm==2.0.3
 apache-airflow-providers-mongo==2.3.1
 apache-airflow-providers-mysql==2.2.1
@@ -118,12 +112,10 @@ apache-airflow-providers-redis==2.0.2
 apache-airflow-providers-salesforce==3.4.1
 apache-airflow-providers-samba==3.0.2
 apache-airflow-providers-segment==2.0.2
-apache-airflow-providers-sendgrid==2.0.2
 apache-airflow-providers-sftp==2.5.0
 apache-airflow-providers-singularity==2.0.2
 apache-airflow-providers-slack==4.2.1
 apache-airflow-providers-snowflake==2.5.1
-apache-airflow-providers-sqlite==2.1.1
 apache-airflow-providers-ssh==2.4.1
 apache-airflow-providers-tableau==2.1.5
 apache-airflow-providers-telegram==2.0.2
diff --git a/constraints-3.8.txt b/constraints-3.8.txt
index c687846..4749529 100644
--- a/constraints-3.8.txt
+++ b/constraints-3.8.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-03-13T14:19:36Z
+# This constraints file was automatically generated on 2022-03-13T17:43:03Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -86,20 +86,14 @@ apache-airflow-providers-docker==2.5.0
 apache-airflow-providers-elasticsearch==3.0.0
 apache-airflow-providers-exasol==2.1.1
 apache-airflow-providers-facebook==2.2.1
-apache-airflow-providers-ftp==2.1.0
-apache-airflow-providers-github==1.0.1
 apache-airflow-providers-google==6.5.0
 apache-airflow-providers-grpc==2.0.2
 apache-airflow-providers-hashicorp==2.1.2
-apache-airflow-providers-http==2.1.0
-apache-airflow-providers-imap==2.2.1
-apache-airflow-providers-influxdb==1.1.1
 apache-airflow-providers-jdbc==2.1.1
 apache-airflow-providers-jenkins==2.0.5
 apache-airflow-providers-jira==2.0.2
 apache-airflow-providers-microsoft-azure==3.7.0
 apache-airflow-providers-microsoft-mssql==2.1.1
-apache-airflow-providers-microsoft-psrp==1.1.1
 apache-airflow-providers-microsoft-winrm==2.0.3
 apache-airflow-providers-mongo==2.3.1
 apache-airflow-providers-mysql==2.2.1
@@ -118,12 +112,10 @@ 

[GitHub] [airflow] ddelange commented on issue #20884: JSONDecodeError caused by incomplete JSON created by cat /airflow/xcom/return.json when using KubernetePodOperator

[GitBox]

ddelange commented on issue #20884:
URL: https://github.com/apache/airflow/issues/20884#issuecomment-1066146827


   FYI, the upstream issue has been closed, as the repo was archived


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] nitinmahawadiwar commented on issue #22007: Add Variable, Connection "description" fields available in the API

[GitBox]

nitinmahawadiwar commented on issue #22007:
URL: https://github.com/apache/airflow/issues/22007#issuecomment-1066142969


   Hello @potiuk @eladkal , I am able to return the description field from get 
api, also found the places to update documentation (verifying locally) but 
somehow not able to understand the Test cases & its structure.
   Do you have any reference material to go through on test cases 
   I am referring https://github.com/apache/airflow/blob/main/TESTING.rst &
   updated  test cases in a) tests/models/test_variable.py & b) 
tests/api_connexion/endpoints/test_variable_endpoint.py 
   
   Thanks
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ulsc commented on pull request #22222: Make `chunk_size` in `GoogleDriveHook` standard

[GitBox]

ulsc commented on pull request #2:
URL: https://github.com/apache/airflow/pull/2#issuecomment-1066141472


   @potiuk this is the mentioned change from the "chunk size" discussion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Skip some tests for Databricks from running on Python 3.10 (#22221)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new af9d85c  Skip some tests for Databricks from running on Python 3.10 
(#1)
af9d85c is described below

commit af9d85ccd8abdc3c252c19764d3ea16970ae0f20
Author: Jarek Potiuk 
AuthorDate: Sun Mar 13 17:53:20 2022 +0100

Skip some tests for Databricks from running on Python 3.10 (#1)

This is a temporary measure until the Databricks SQL library
is released in Python 3.10 - compatible version.

Related to: #0
---
 airflow/providers/databricks/provider.yaml  | 3 +++
 tests/providers/databricks/operators/test_databricks_sql.py | 8 +++-
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/airflow/providers/databricks/provider.yaml 
b/airflow/providers/databricks/provider.yaml
index c4a2aec..2bf7f99 100644
--- a/airflow/providers/databricks/provider.yaml
+++ b/airflow/providers/databricks/provider.yaml
@@ -34,6 +34,9 @@ versions:
 additional-dependencies:
   - apache-airflow>=2.1.0
 
+excluded-python-versions:
+  - "3.10"
+
 integrations:
   - integration-name: Databricks
 external-doc-url: https://databricks.com/
diff --git a/tests/providers/databricks/operators/test_databricks_sql.py 
b/tests/providers/databricks/operators/test_databricks_sql.py
index 6b9fb43..08e24d72 100644
--- a/tests/providers/databricks/operators/test_databricks_sql.py
+++ b/tests/providers/databricks/operators/test_databricks_sql.py
@@ -24,7 +24,7 @@ from unittest import mock
 import pytest
 from databricks.sql.types import Row
 
-from airflow import AirflowException
+from airflow import PY310, AirflowException
 from airflow.providers.databricks.operators.databricks_sql import (
 DatabricksCopyIntoOperator,
 DatabricksSqlOperator,
@@ -83,6 +83,12 @@ class TestDatabricksSqlOperator(unittest.TestCase):
 db_mock.run.assert_called_once_with(sql, parameters=None)
 
 
+@pytest.mark.skipif(
+PY310,
+reason="Databricks SQL tests not run on Python 3.10 because there is 
direct Iterable import from"
+" collections in the databricks SQL library, where it should be imported 
from collections.abc."
+" This could be removed when 
https://github.com/apache/airflow/issues/0 is solved",
+)
 class TestDatabricksSqlCopyIntoOperator(unittest.TestCase):
 def test_copy_with_files(self):
 op = DatabricksCopyIntoOperator(

[GitHub] [airflow] potiuk merged pull request #22221: Skip some tests for Databricks from running on Python 3.10

[GitBox]

potiuk merged pull request #1:
URL: https://github.com/apache/airflow/pull/1


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] alexott opened a new pull request #22223: DatabricksSqlOperator - switch to databricks-sql-connector 2.x

[GitBox]

alexott opened a new pull request #3:
URL: https://github.com/apache/airflow/pull/3


   Initial work to switch to databricks-sql-connector 2.x in Databricks SQL 
Operators


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ulsc opened a new pull request #22222: Make `chunk_size` in `GoogleDriveHook` standard

[GitBox]

ulsc opened a new pull request #2:
URL: https://github.com/apache/airflow/pull/2


   Change the default `chunk_size` to a clear representation & add documentation


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ulsc commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

ulsc commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825468633



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param api_version: version of the Google Drive API
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = ('local_paths', 'drive_folder',)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+api_version: str = 'v3',

Review comment:
   I understand, but anyways, if the user wants to change the api version 
directly from the operator, this may require a breaking change on the 
underlying hook. So if api_version will change, it should be changed in the 
hook level.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

eladkal commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825468164



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param api_version: version of the Google Drive API
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = ('local_paths', 'drive_folder',)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+api_version: str = 'v3',

Review comment:
   i don't think removing it is the way to go.
   if there are multiple versions of the API users might want to set what they 
wish.
   My question was only about the default value of the parameter.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

eladkal commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825468164



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param api_version: version of the Google Drive API
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = ('local_paths', 'drive_folder',)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+api_version: str = 'v3',

Review comment:
   i dont think removing it is the way to go.
   if there are multipule versions of the API users might want to set what they 
wish.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #22221: Skip some tests for Databricks from running on Python 3.10

[GitBox]

github-actions[bot] commented on pull request #1:
URL: https://github.com/apache/airflow/pull/1#issuecomment-1066126192


   The PR is likely OK to be merged with just subset of tests for default 
Python and Database versions without running the full matrix of tests, because 
it does not modify the core of Airflow. If the committers decide that the full 
tests matrix is needed, they will add the label 'full tests needed'. Then you 
should rebase to the latest main or amend the last commit of the PR, and push 
it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed pull request #22206: Add cgroupspy to _vendor folder

[GitBox]

potiuk closed pull request #22206:
URL: https://github.com/apache/airflow/pull/22206


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ulsc commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

ulsc commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825462049



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param api_version: version of the Google Drive API
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = ('local_paths', 'drive_folder',)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+api_version: str = 'v3',

Review comment:
   actually, we can get rid of it altogether, because it's passed to the 
hook and the hook already has the version default. But, 

##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files 

[GitHub] [airflow] eladkal commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

eladkal commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825461523



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param api_version: version of the Google Drive API
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = ('local_paths', 'drive_folder',)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+api_version: str = 'v3',

Review comment:
   Must we default a specific version?
   
   The google provider is full of specific versions
   
https://github.com/apache/airflow/blob/a840561e1f00ce026a813039e430a085d2cfe35e/airflow/providers/google/cloud/operators/datafusion.py#L166
   
https://github.com/apache/airflow/blob/602abe8394fafe7de54df7e73af56de848cdf617/airflow/providers/google/marketing_platform/operators/display_video.py#L76
   
   
   This means that we need to remember to bump it when the version is 
deprecated/old and make it as a breaking change.
   Does Google offer default version that comes within its own lib?
   like facebook has (for example)
   
https://github.com/apache/airflow/blob/602abe8394fafe7de54df7e73af56de848cdf617/airflow/providers/google/cloud/transfers/facebook_ads_to_gcs.py#L65-L66
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on a change in pull request #22219: Gdrive upload operator

[GitBox]

eladkal commented on a change in pull request #22219:
URL: https://github.com/apache/airflow/pull/22219#discussion_r825461523



##
File path: airflow/providers/google/suite/operators/drive.py
##
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This file contains Google Drive operators"""
+
+import os
+from typing import Any, Optional, Sequence, Union
+
+from airflow.models import BaseOperator
+from airflow.providers.google.suite.hooks.drive import GoogleDriveHook
+
+
+class GoogleDriveUploadOperator(BaseOperator):
+"""
+Upload a list of files to a Google Drive folder.
+This operator uploads a list of local files to a Google Drive folder.
+The local files can be deleted after upload (optional)
+
+:param local_paths: Python list of local file path strings
+:param drive_folder: path of the Drive folder
+:param gcp_conn_id: Airflow Connection ID for GCP
+:param delete: should the local files be deleted after upload?
+:param chunk_size: File will be uploaded in chunks of this many bytes. Only
+used if resumable=True. Pass in a value of -1 if the file is to be
+uploaded as a single chunk. Note that Google App Engine has a 5MB limit
+on request size, so you should never set your chunk size larger than 
5MB,
+or to -1.
+:param resumable: True if this is a resumable upload. False means upload
+in a single request.
+:param api_version: version of the Google Drive API
+:param delegate_to: The account to impersonate using domain-wide 
delegation of authority,
+if any. For this to work, the service account making the request must 
have
+domain-wide delegation enabled.
+:param impersonation_chain: Optional service account to impersonate using 
short-term
+credentials, or chained list of accounts required to get the 
access_token
+of the last account in the list, which will be impersonated in the 
request.
+If set as a string, the account must grant the originating account
+the Service Account Token Creator IAM role.
+If set as a sequence, the identities from the list must grant
+Service Account Token Creator IAM role to the directly preceding 
identity, with first
+account from the list granting this role to the originating account
+:return: Remote file ids after upload
+:rtype: Sequence[str]
+"""
+
+template_fields = ('local_paths', 'drive_folder',)
+
+def __init__(
+self,
+local_paths,
+drive_folder: str,
+gcp_conn_id: str,
+delete: bool = False,
+chunk_size: int = 100 * 1024 * 1024,
+resumable: bool = False,
+api_version: str = 'v3',

Review comment:
   Must we pin the version?
   
   The google provider is full of pinning versions
   
https://github.com/apache/airflow/blob/a840561e1f00ce026a813039e430a085d2cfe35e/airflow/providers/google/cloud/operators/datafusion.py#L166
   
https://github.com/apache/airflow/blob/602abe8394fafe7de54df7e73af56de848cdf617/airflow/providers/google/marketing_platform/operators/display_video.py#L76
   
   
   This means that we need to remember to bump it and make it as a breaking 
change.
   Does Google offer default version that comes within its own lib?
   like facebook has (for example)
   
https://github.com/apache/airflow/blob/602abe8394fafe7de54df7e73af56de848cdf617/airflow/providers/google/cloud/transfers/facebook_ads_to_gcs.py#L65-L66
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #22221: Skip some tests for Databricks from running on Python 3.10

[GitBox]

potiuk commented on pull request #1:
URL: https://github.com/apache/airflow/pull/1#issuecomment-1066119933


   cc: @alexott 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request #22221: Skip some tests for Databricks from running on Python 3.10

[GitBox]

potiuk opened a new pull request #1:
URL: https://github.com/apache/airflow/pull/1


   This is a temporary measure until the Databricks SQL library
   is released in Python 3.10 - compatible version.
   
   Related to: #0
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #22220: Databricks SQL fails on Python 3.10

[GitBox]

potiuk commented on issue #0:
URL: https://github.com/apache/airflow/issues/0#issuecomment-1066119096


   cc: @alexott  - I created an issue to track it (and proceed with excluding 
the failing test in Python 3.10 for now so that it does not block us from 
enabling 3.10 for Airflow main CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new issue #22220: Databricks SQL fails on Python 3.10

[GitBox]

potiuk opened a new issue #0:
URL: https://github.com/apache/airflow/issues/0


   ### Apache Airflow Provider(s)
   
   databricks
   
   ### Versions of Apache Airflow Providers
   
   The databricks SQL does not work on Python 3.10 due to "from collections 
import Iterable" in the `databricks-sql-connector`
   
   * https://pypi.org/project/databricks-sql-connector/
   
   Details of this issue dicussed in 
https://github.com/apache/airflow/pull/22050
   
   For now we will likely just exclude the tests (and mark databricks provider 
as non-python 3.10 compatible). But once this is fixed (in either 1.0.2 or 
upcoming 2.0.0 version of the library, we wil restore it back). 
   
   ### Apache Airflow version
   
   main (development)
   
   ### Operating System
   
   All
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   Just Breeze with Python 3.10
   
   ### What happened
   
   The tests are failing:
   
   ```
 self = 
 item = ['file1', 'file2', 'file3']
 
 def escape_item(self, item):
 if item is None:
 return 'NULL'
 elif isinstance(item, (int, float)):
 return self.escape_number(item)
 elif isinstance(item, basestring):
 return self.escape_string(item)
 >   elif isinstance(item, collections.Iterable):
 E   AttributeError: module 'collections' has no attribute 'Iterable'
 ```
   
   
https://github.com/apache/airflow/runs/5523057543?check_suite_focus=true#step:8:16781
 
   
   
   ### What you expected to happen
   
   Test succeed :)
   
   ### How to reproduce
   
   Run `TestDatabricksSqlCopyIntoOperator` in Python 3.10 environment.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ulsc opened a new pull request #22219: Gdrive upload operator

[GitBox]

ulsc opened a new pull request #22219:
URL: https://github.com/apache/airflow/pull/22219


   Operator to upload a list of files to a Google Drive folder, alongside a 
file which can be used to have more Google Drive operators.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] tag constraints-2.2.0 updated (ac1121b -> 7cae736)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to tag constraints-2.2.0
in repository https://gitbox.apache.org/repos/asf/airflow.git.


*** WARNING: tag constraints-2.2.0 was modified! ***

from ac1121b  (commit)
  to 7cae736  (commit)
from ac1121b  Fix importlib-metadata for flake compatibility
 new 7cae736  Manually fix constraints for Flake8 for python 3.6/7

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 constraints-3.6.txt  | 6 +++---
 constraints-3.7.txt  | 4 ++--
 constraints-source-providers-3.6.txt | 2 +-
 constraints-source-providers-3.7.txt | 6 +++---
 4 files changed, 9 insertions(+), 9 deletions(-)

[airflow] 01/01: Manually fix constraints for Flake8 for python 3.6/7

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to tag constraints-2.2.0
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 7cae73639009611f5de000f579da290b7fe63941
Author: Jarek Potiuk 
AuthorDate: Sun Mar 13 15:29:25 2022 +0100

Manually fix constraints for Flake8 for python 3.6/7

Apparently `pip` detects more conflicts and we have to
downgrade flake and related libraries to avoid conflicts with
importlib.
---
 constraints-3.6.txt  | 6 +++---
 constraints-3.7.txt  | 4 ++--
 constraints-source-providers-3.6.txt | 2 +-
 constraints-source-providers-3.7.txt | 6 +++---
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/constraints-3.6.txt b/constraints-3.6.txt
index 99c0b1f..85b9b77 100644
--- a/constraints-3.6.txt
+++ b/constraints-3.6.txt
@@ -229,7 +229,7 @@ fastavro==1.4.5
 filelock==3.3.0
 fissix==21.6.6
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -404,7 +404,7 @@ pyOpenSSL==20.0.1
 pyarrow==4.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.20
 pycryptodome==3.11.0
@@ -414,7 +414,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.21.1
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.0
 pymssql==2.2.2
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 7679d10..e9603b7 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -226,7 +226,7 @@ fastavro==1.4.5
 filelock==3.3.0
 fissix==21.6.6
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -411,7 +411,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.21.1
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.0
 pymssql==2.2.2
diff --git a/constraints-source-providers-3.6.txt 
b/constraints-source-providers-3.6.txt
index 5790bee..cbd9dcd 100644
--- a/constraints-source-providers-3.6.txt
+++ b/constraints-source-providers-3.6.txt
@@ -333,7 +333,7 @@ pyOpenSSL==20.0.1
 pyarrow==4.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.20
 pycryptodome==3.11.0
diff --git a/constraints-source-providers-3.7.txt 
b/constraints-source-providers-3.7.txt
index 04b4195..6237fe9 100644
--- a/constraints-source-providers-3.7.txt
+++ b/constraints-source-providers-3.7.txt
@@ -155,7 +155,7 @@ fastavro==1.4.5
 filelock==3.3.0
 fissix==21.6.6
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -330,7 +330,7 @@ pyOpenSSL==20.0.1
 pyarrow==4.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.20
 pycryptodome==3.11.0
@@ -340,7 +340,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.21.1
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.0
 pymssql==2.2.2

[airflow] 01/01: Manually fix constraints for Flake8 for python 3.6/7

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to tag constraints-2.2.1
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 411957b8188db3deeb9fd81f785bad858c319586
Author: Jarek Potiuk 
AuthorDate: Sun Mar 13 15:29:25 2022 +0100

Manually fix constraints for Flake8 for python 3.6/7

Apparently `pip` detects more conflicts and we have to
downgrade flake and related libraries to avoid conflicts with
importlib.
---
 constraints-3.6.txt  | 6 +++---
 constraints-3.7.txt  | 4 ++--
 constraints-source-providers-3.6.txt | 2 +-
 constraints-source-providers-3.7.txt | 6 +++---
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/constraints-3.6.txt b/constraints-3.6.txt
index 99c0b1f..85b9b77 100644
--- a/constraints-3.6.txt
+++ b/constraints-3.6.txt
@@ -229,7 +229,7 @@ fastavro==1.4.5
 filelock==3.3.0
 fissix==21.6.6
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -404,7 +404,7 @@ pyOpenSSL==20.0.1
 pyarrow==4.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.20
 pycryptodome==3.11.0
@@ -414,7 +414,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.21.1
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.0
 pymssql==2.2.2
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 7679d10..e9603b7 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -226,7 +226,7 @@ fastavro==1.4.5
 filelock==3.3.0
 fissix==21.6.6
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -411,7 +411,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.21.1
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.0
 pymssql==2.2.2
diff --git a/constraints-source-providers-3.6.txt 
b/constraints-source-providers-3.6.txt
index 5790bee..cbd9dcd 100644
--- a/constraints-source-providers-3.6.txt
+++ b/constraints-source-providers-3.6.txt
@@ -333,7 +333,7 @@ pyOpenSSL==20.0.1
 pyarrow==4.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.20
 pycryptodome==3.11.0
diff --git a/constraints-source-providers-3.7.txt 
b/constraints-source-providers-3.7.txt
index 04b4195..6237fe9 100644
--- a/constraints-source-providers-3.7.txt
+++ b/constraints-source-providers-3.7.txt
@@ -155,7 +155,7 @@ fastavro==1.4.5
 filelock==3.3.0
 fissix==21.6.6
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -330,7 +330,7 @@ pyOpenSSL==20.0.1
 pyarrow==4.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.20
 pycryptodome==3.11.0
@@ -340,7 +340,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.21.1
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.0
 pymssql==2.2.2

[airflow] tag constraints-2.2.1 updated (84e6faa -> 411957b)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to tag constraints-2.2.1
in repository https://gitbox.apache.org/repos/asf/airflow.git.


*** WARNING: tag constraints-2.2.1 was modified! ***

from 84e6faa  (tag)
  to 411957b  (commit)
from ac1121b  Fix importlib-metadata for flake compatibility
 new 411957b  Manually fix constraints for Flake8 for python 3.6/7

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 constraints-3.6.txt  | 6 +++---
 constraints-3.7.txt  | 4 ++--
 constraints-source-providers-3.6.txt | 2 +-
 constraints-source-providers-3.7.txt | 6 +++---
 4 files changed, 9 insertions(+), 9 deletions(-)

[airflow] tag constraints-2.2.2 updated (9b206b8 -> 67c26e8)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to tag constraints-2.2.2
in repository https://gitbox.apache.org/repos/asf/airflow.git.


*** WARNING: tag constraints-2.2.2 was modified! ***

from 9b206b8  (commit)
  to 67c26e8  (commit)
from 9b206b8  Fixed typo
 add 67c26e8  Manually fix constraints for Flake8 for python 3.6/7

No new revisions were added by this update.

Summary of changes:
 constraints-3.6.txt  | 6 +++---
 constraints-3.7.txt  | 4 ++--
 constraints-source-providers-3.6.txt | 2 +-
 constraints-source-providers-3.7.txt | 6 +++---
 4 files changed, 9 insertions(+), 9 deletions(-)

[airflow] branch constraints-2-2-2-fixed updated: Manually fix constraints for Flake8 for python 3.6/7

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch constraints-2-2-2-fixed
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-2-2-2-fixed by 
this push:
 new 67c26e8  Manually fix constraints for Flake8 for python 3.6/7
67c26e8 is described below

commit 67c26e85b94f18b3327b555499705bc71c8ebfb7
Author: Jarek Potiuk 
AuthorDate: Sun Mar 13 15:29:25 2022 +0100

Manually fix constraints for Flake8 for python 3.6/7

Apparently `pip` detects more conflicts and we have to
downgrade flake and related libraries to avoid conflicts with
importlib.
---
 constraints-3.6.txt  | 6 +++---
 constraints-3.7.txt  | 4 ++--
 constraints-source-providers-3.6.txt | 2 +-
 constraints-source-providers-3.7.txt | 6 +++---
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/constraints-3.6.txt b/constraints-3.6.txt
index 44e67cb..cbc1975 100644
--- a/constraints-3.6.txt
+++ b/constraints-3.6.txt
@@ -230,7 +230,7 @@ fastavro==1.4.7
 filelock==3.3.2
 fissix==21.6.6
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -408,7 +408,7 @@ pyOpenSSL==20.0.1
 pyarrow==4.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.21
 pycryptodome==3.11.0
@@ -418,7 +418,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.21.1
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.1
 pymssql==2.2.2
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 9281e5c..180a705 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -227,7 +227,7 @@ fastavro==1.4.7
 filelock==3.3.2
 fissix==21.6.6
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -415,7 +415,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.21.1
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.1
 pymssql==2.2.2
diff --git a/constraints-source-providers-3.6.txt 
b/constraints-source-providers-3.6.txt
index 3e44c0c..9a0c83c 100644
--- a/constraints-source-providers-3.6.txt
+++ b/constraints-source-providers-3.6.txt
@@ -337,7 +337,7 @@ pyOpenSSL==20.0.1
 pyarrow==4.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.21
 pycryptodome==3.11.0
diff --git a/constraints-source-providers-3.7.txt 
b/constraints-source-providers-3.7.txt
index b23e763..79e28fa 100644
--- a/constraints-source-providers-3.7.txt
+++ b/constraints-source-providers-3.7.txt
@@ -156,7 +156,7 @@ fastavro==1.4.7
 filelock==3.3.2
 fissix==21.6.6
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -334,7 +334,7 @@ pyOpenSSL==20.0.1
 pyarrow==4.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.21
 pycryptodome==3.11.0
@@ -344,7 +344,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.21.1
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.1
 pymssql==2.2.2

[airflow] 01/01: Manually fix constraints for Flake8 for python 3.6/7

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to tag constraints-2.2.4
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 201326c4817331149fd69d8870381eff348280a0
Author: Jarek Potiuk 
AuthorDate: Sun Mar 13 15:29:25 2022 +0100

Manually fix constraints for Flake8 for python 3.6/7

Apparently `pip` detects more conflicts and we have to
downgrade flake and related libraries to avoid conflicts with
importlib.
---
 constraints-3.6.txt  | 6 +++---
 constraints-3.7.txt  | 4 ++--
 constraints-source-providers-3.6.txt | 2 +-
 constraints-source-providers-3.7.txt | 6 +++---
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/constraints-3.6.txt b/constraints-3.6.txt
index 949c5cc..d30fb87 100644
--- a/constraints-3.6.txt
+++ b/constraints-3.6.txt
@@ -232,7 +232,7 @@ fastavro==1.4.7
 filelock==3.4.1
 fissix==21.11.13
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -415,7 +415,7 @@ pyOpenSSL==20.0.1
 pyarrow==6.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==22.1.10
 pycparser==2.21
 pycryptodome==3.14.1
@@ -425,7 +425,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.24.0
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.3
 pymssql==2.2.4
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 0449500..e350004 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -229,7 +229,7 @@ fastavro==1.4.9
 filelock==3.6.0
 fissix==21.11.13
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -422,7 +422,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.24.0
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.3
 pymssql==2.2.4
diff --git a/constraints-source-providers-3.6.txt 
b/constraints-source-providers-3.6.txt
index a0a58c1..4f87487 100644
--- a/constraints-source-providers-3.6.txt
+++ b/constraints-source-providers-3.6.txt
@@ -341,7 +341,7 @@ pyOpenSSL==20.0.1
 pyarrow==6.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==22.1.10
 pycparser==2.21
 pycryptodome==3.14.1
diff --git a/constraints-source-providers-3.7.txt 
b/constraints-source-providers-3.7.txt
index 25a81bf..e3db182 100644
--- a/constraints-source-providers-3.7.txt
+++ b/constraints-source-providers-3.7.txt
@@ -158,7 +158,7 @@ fastavro==1.4.9
 filelock==3.6.0
 fissix==21.11.13
 flake8-colors==0.1.9
-flake8==4.0.1
+flake8==3.9.2
 flaky==3.7.0
 flower==1.0.0
 freezegun==1.1.0
@@ -338,7 +338,7 @@ pyOpenSSL==20.0.1
 pyarrow==6.0.1
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==22.1.10
 pycparser==2.21
 pycryptodome==3.14.1
@@ -348,7 +348,7 @@ pydot==1.4.2
 pydruid==0.6.2
 pyenchant==3.2.2
 pyexasol==0.24.0
-pyflakes==2.4.0
+pyflakes==2.3.1
 pykerberos==1.2.1
 pymongo==3.12.3
 pymssql==2.2.4

[airflow] tag constraints-2.2.4 updated (3c76d72 -> 201326c)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to tag constraints-2.2.4
in repository https://gitbox.apache.org/repos/asf/airflow.git.


*** WARNING: tag constraints-2.2.4 was modified! ***

from 3c76d72  (tag)
  to 201326c  (commit)
from 8dde99b  Update constraints after RC3 providers relese for Jan 2022
 new 201326c  Manually fix constraints for Flake8 for python 3.6/7

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 constraints-3.6.txt  | 6 +++---
 constraints-3.7.txt  | 4 ++--
 constraints-source-providers-3.6.txt | 2 +-
 constraints-source-providers-3.7.txt | 6 +++---
 4 files changed, 9 insertions(+), 9 deletions(-)

[GitHub] [airflow] mik-laj commented on a change in pull request #22217: added docs about detailed network and constraints information

[GitBox]

mik-laj commented on a change in pull request #22217:
URL: https://github.com/apache/airflow/pull/22217#discussion_r825457299



##
File path: airflow/providers/docker/operators/docker_swarm.py
##
@@ -86,8 +86,19 @@ class DockerSwarmOperator(DockerOperator):
 :param mode: Indicate whether a service should be deployed as a replicated 
or global service,
 and associated parameters
 :param networks: List of network names or IDs or NetworkAttachmentConfig 
to attach the service to.
-:param placement: Placement instructions for the scheduler. If a list is 
passed instead,
-it is assumed to be a list of constraints as part of a Placement 
object.
+NetworkAttachmentConfig object accepts following params:

Review comment:
   We already have a link to Docker API reference.
   https://user-images.githubusercontent.com/12058428/158064360-0a96c238-527d-4a15-a54f-d66cd705d4d8.png;>
   I am not sure if we need to duplicate this documentation as it can be 
maintenance burden




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (4543539 -> a840561)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 4543539  Support Uploading Bigger Files to Google Drive (#22179)
 add a840561  AWS RDS integration fixes (#22125)

No new revisions were added by this update.

Summary of changes:
 .../amazon/aws/example_dags/example_rds.py | 10 +++---
 airflow/providers/amazon/aws/operators/rds.py  | 20 ++-
 airflow/providers/amazon/aws/sensors/rds.py| 14 +---
 setup.py   |  2 +-
 tests/providers/amazon/aws/operators/test_rds.py   | 42 +++---
 tests/providers/amazon/aws/sensors/test_rds.py | 20 +--
 6 files changed, 57 insertions(+), 51 deletions(-)

[GitHub] [airflow] potiuk merged pull request #22125: AWS RDS integration fixes

[GitBox]

potiuk merged pull request #22125:
URL: https://github.com/apache/airflow/pull/22125


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Update constraints after moto fixes

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 9dbf8bb  Update constraints after moto fixes
9dbf8bb is described below

commit 9dbf8bb3db51e95330602cb30f2312b7dc45894c
Author: Jarek Potiuk 
AuthorDate: Sun Mar 13 15:26:08 2022 +0100

Update constraints after moto fixes
---
 constraints-3.7.txt   | 22 +++---
 constraints-3.8.txt   | 22 +++---
 constraints-3.9.txt   | 22 +++---
 constraints-no-providers-3.10.txt |  4 ++--
 constraints-no-providers-3.7.txt  |  4 ++--
 constraints-no-providers-3.8.txt  |  4 ++--
 constraints-no-providers-3.9.txt  |  4 ++--
 constraints-source-providers-3.10.txt | 22 +++---
 constraints-source-providers-3.7.txt  | 16 
 constraints-source-providers-3.8.txt  | 16 
 constraints-source-providers-3.9.txt  | 16 
 11 files changed, 88 insertions(+), 64 deletions(-)

diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 3dbf540..006afa0 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-03-11T21:25:43Z
+# This constraints file was automatically generated on 2022-03-13T14:19:39Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -86,14 +86,20 @@ apache-airflow-providers-docker==2.5.0
 apache-airflow-providers-elasticsearch==3.0.0
 apache-airflow-providers-exasol==2.1.1
 apache-airflow-providers-facebook==2.2.1
+apache-airflow-providers-ftp==2.1.0
+apache-airflow-providers-github==1.0.1
 apache-airflow-providers-google==6.5.0
 apache-airflow-providers-grpc==2.0.2
 apache-airflow-providers-hashicorp==2.1.2
+apache-airflow-providers-http==2.1.0
+apache-airflow-providers-imap==2.2.1
+apache-airflow-providers-influxdb==1.1.1
 apache-airflow-providers-jdbc==2.1.1
 apache-airflow-providers-jenkins==2.0.5
 apache-airflow-providers-jira==2.0.2
 apache-airflow-providers-microsoft-azure==3.7.0
 apache-airflow-providers-microsoft-mssql==2.1.1
+apache-airflow-providers-microsoft-psrp==1.1.1
 apache-airflow-providers-microsoft-winrm==2.0.3
 apache-airflow-providers-mongo==2.3.1
 apache-airflow-providers-mysql==2.2.1
@@ -112,10 +118,12 @@ apache-airflow-providers-redis==2.0.2
 apache-airflow-providers-salesforce==3.4.1
 apache-airflow-providers-samba==3.0.2
 apache-airflow-providers-segment==2.0.2
+apache-airflow-providers-sendgrid==2.0.2
 apache-airflow-providers-sftp==2.5.0
 apache-airflow-providers-singularity==2.0.2
 apache-airflow-providers-slack==4.2.1
 apache-airflow-providers-snowflake==2.5.1
+apache-airflow-providers-sqlite==2.1.1
 apache-airflow-providers-ssh==2.4.1
 apache-airflow-providers-tableau==2.1.5
 apache-airflow-providers-telegram==2.0.2
@@ -130,7 +138,7 @@ argcomplete==1.12.3
 arrow==1.2.2
 asana==0.10.3
 asn1crypto==1.5.0
-astroid==2.10.0
+astroid==2.11.0
 async-timeout==4.0.2
 asynctest==0.13.0
 atlasclient==1.0.0
@@ -335,13 +343,13 @@ lxml==4.8.0
 marshmallow-enum==1.5.1
 marshmallow-oneofschema==3.0.1
 marshmallow-sqlalchemy==0.26.1
-marshmallow==3.14.1
+marshmallow==3.15.0
 matplotlib-inline==0.1.3
 mccabe==0.6.1
 mongomock==4.0.0
 monotonic==1.6
 moreorless==0.4.0
-moto==3.0.7
+moto==3.1.0
 msal-extensions==0.3.1
 msal==1.17.0
 msgpack==1.0.3
@@ -374,7 +382,7 @@ pandas-gbq==0.17.3
 pandas==1.3.5
 papermill==2.3.4
 parameterized==0.8.1
-paramiko==2.9.2
+paramiko==2.10.1
 parso==0.8.3
 partd==1.2.0
 pathspec==0.9.0
@@ -549,13 +557,13 @@ types-python-dateutil==2.8.9
 types-python-slugify==5.0.3
 types-pytz==2021.3.5
 types-redis==4.1.17
-types-requests==2.27.11
+types-requests==2.27.12
 types-setuptools==57.4.10
 types-six==1.16.12
 types-tabulate==0.8.5
 types-termcolor==1.1.3
 types-toml==0.10.4
-types-urllib3==1.26.10
+types-urllib3==1.26.11
 typing_extensions==4.1.1
 tzdata==2021.5
 tzlocal==4.1
diff --git a/constraints-3.8.txt b/constraints-3.8.txt
index 4bdd6fb..c687846 100644
--- a/constraints-3.8.txt
+++ b/constraints-3.8.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-03-11T21:25:43Z
+# This constraints file was automatically generated on 2022-03-13T14:19:36Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -86,14 +86,20 @@ apache-airflow-providers-docker==2.5.0
 

[GitHub] [airflow] potiuk commented on pull request #22125: AWS RDS integration fixes

[GitBox]

potiuk commented on pull request #22125:
URL: https://github.com/apache/airflow/pull/22125#issuecomment-1066113634


   Updated constraints manually. Merging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] 01/01: Also downgrade pycodestyle (dependency party!)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to tag constraints-2.2.3
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 38f73659dc90fa7a3c50dcd0c62b470a601188fb
Author: Jarek Potiuk 
AuthorDate: Sun Mar 13 15:15:02 2022 +0100

Also downgrade pycodestyle (dependency party!)
---
 constraints-3.6.txt  | 2 +-
 constraints-3.7.txt  | 2 +-
 constraints-source-providers-3.6.txt | 2 +-
 constraints-source-providers-3.7.txt | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/constraints-3.6.txt b/constraints-3.6.txt
index cf295a2..87966dc 100644
--- a/constraints-3.6.txt
+++ b/constraints-3.6.txt
@@ -409,7 +409,7 @@ pyOpenSSL==20.0.1
 pyarrow==5.0.0
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.21
 pycryptodome==3.12.0
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 3b1e46f..f06afb8 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -406,7 +406,7 @@ pyOpenSSL==20.0.1
 pyarrow==5.0.0
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.21
 pycryptodome==3.12.0
diff --git a/constraints-source-providers-3.6.txt 
b/constraints-source-providers-3.6.txt
index 43c606e..8f79ce6 100644
--- a/constraints-source-providers-3.6.txt
+++ b/constraints-source-providers-3.6.txt
@@ -336,7 +336,7 @@ pyOpenSSL==20.0.1
 pyarrow==5.0.0
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.21
 pycryptodome==3.12.0
diff --git a/constraints-source-providers-3.7.txt 
b/constraints-source-providers-3.7.txt
index 9858a5a..6f400a5 100644
--- a/constraints-source-providers-3.7.txt
+++ b/constraints-source-providers-3.7.txt
@@ -333,7 +333,7 @@ pyOpenSSL==20.0.1
 pyarrow==5.0.0
 pyasn1-modules==0.2.8
 pyasn1==0.4.8
-pycodestyle==2.8.0
+pycodestyle==2.7.0
 pycountry==20.7.3
 pycparser==2.21
 pycryptodome==3.12.0