[GitHub] [airflow] kaxil commented on issue #14396: Make context less nebulous

[GitBox]

kaxil commented on issue #14396:
URL: https://github.com/apache/airflow/issues/14396#issuecomment-937472485


   Yup we would not close it for sure :) . It's just that it hasn't fallen into 
priorities as yet.
   
   Anyone who is willing to work on the issue should take a stab at it and we 
are happy to help if needed.
   
   Even if there are no thumbs up, you can still work on it :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on pull request #18201: Add check for 403 code

[GitBox]

eladkal commented on pull request #18201:
URL: https://github.com/apache/airflow/pull/18201#issuecomment-937461447


   > Could you provide some reference and rationale behind this change?
   
   Both me and @xinbinhuang discussed it on the issue 
https://github.com/apache/airflow/issues/18111 we both think this is a won't fix


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] r-richmond edited a comment on issue #14396: Make context less nebulous

[GitBox]

r-richmond edited a comment on issue #14396:
URL: https://github.com/apache/airflow/issues/14396#issuecomment-937445404


   >I believe so far simply no-one else found it really useful or needed. 
   
   What is the bar to clear here?.. i.e. 5 thumbs up on opening issue doesn't 
meet it so I'm curious at what point your opinion would change.
   
   >We usually do not plan work which is not "urgent", "important" or "part of 
bigger feature we work on".
   
   Makes sense, ty for info
   
   >why don't you create a Draft PR with it and share it here to see how it can 
look like so that you could reiterate your points showing them with a real 
code. If you convince others (including myself), we will merge it. If not - we 
will drop it.
   
   I'm interested in seeing it tackled and am willing to take a stab at it. I'm 
just really busy for the next couple months but will give it a go after that. 
I.e. please don't close the issue yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] r-richmond commented on issue #14396: Make context less nebulous

[GitBox]

r-richmond commented on issue #14396:
URL: https://github.com/apache/airflow/issues/14396#issuecomment-937445404


   >I believe so far simply no-one else found it really useful or needed. 
   
   What is the bar to clear here?.. i.e. 5 thumbs up on opening issue doesn't 
meet it so I'm curious at what point your opinion would change.
   
   >We usually do not plan work which is not "urgent", "important" or "part of 
bigger feature we work on".
   
   Makes sense, ty for info
   
   >why don't you create a Draft PR with it and share it here to see how it can 
look like so that you could reiterate your points showing them with a real 
code. If you convince others (including myself), we will merge it. If not - we 
will drop it.
   
   I'm interested in seeing it tackled and am willing to take a stab at it. I'm 
just really busy for the next couple months. I.e. please don't close the issue 
yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] xinbinhuang edited a comment on issue #18111: HeadObject error when trying to upload file to S3

[GitBox]

xinbinhuang edited a comment on issue #18111:
URL: https://github.com/apache/airflow/issues/18111#issuecomment-936953895


   > I want it to be able to upload a file in any case (file already exists or 
not), I know about the `replace` flag, but I also want to cover the case when 
the file doesn't exists.
   > 
   > My suggestion is to catch the error and allow the upload, without adding 
another error. It will be something like, updating:
   > 
   > ```python
   > except ClientError as e:
   > if e.response["ResponseMetadata"]["HTTPStatusCode"] == 404:
   > return False
   > else:
   > raise e
   > ```
   > 
   > to:
   > 
   > ```python
   > except ClientError as e:
   > if e.response["ResponseMetadata"]["HTTPStatusCode"] IN (404, 
403):
   > return False
   > else:
   > raise e
   > ```
   > 
   > shortest PR ever  I will try to submit one next week.
   
   I believe the `replace=True` flag will just upload the file regardless if 
the key/file exists or not.  We don't call the `check_for_key` function when 
`replace=True`. Have you tried it?
   
   IMO, It's not reasonable for us to swallow the permission error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #18201: Add check for 403 code

[GitBox]

uranusjr commented on pull request #18201:
URL: https://github.com/apache/airflow/pull/18201#issuecomment-937422763


   Could you provide some reference and rationale behind this change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #18199: Create docker-operator-with-localexecutor-in-docker-compose.md

[GitBox]

uranusjr commented on pull request #18199:
URL: https://github.com/apache/airflow/pull/18199#issuecomment-937422424


   Gentle ping on this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] chinwobble commented on issue #8125: Conditionally symlink 'latest' directory

[GitBox]

chinwobble commented on issue #8125:
URL: https://github.com/apache/airflow/issues/8125#issuecomment-937421078


   We are experiencing the same issue. Is it possible to turn off the symlink 
behaviour?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] michaelkhan3 commented on issue #15279: Error on logging empty line to Cloudwatch

[GitBox]

michaelkhan3 commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-937418131


   > i've gotten around it by creating a `log_config.py` with the following:
   > 
   > ```
   > from copy import deepcopy
   > import logging
   > 
   > from airflow.config_templates.airflow_local_settings import 
DEFAULT_LOGGING_CONFIG
   > 
   > LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   > class EmptyLogFilter(logging.Filter):
   > def filter(self, record: logging.LogRecord) -> bool:
   > if not record.getMessage():
   > return False
   > else:
   > return True
   > 
   > 
   > LOGGING_CONFIG['filters']['remove_blanks'] = {'()': 
'log_config.EmptyLogFilter'}
   > 
   > CLOUDWATCH_HANDLER = 
'airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler'
   > 
   > TASK_HANDLER = LOGGING_CONFIG['handlers']['task']
   > if TASK_HANDLER['class'] == CLOUDWATCH_HANDLER:
   > # This thing blows up on empty log messages so apply filter
   > filters = TASK_HANDLER.get('filters', [])
   > filters.append('remove_blanks')
   > TASK_HANDLER['filters'] = filters
   > ```
   > 
   > don't forget to enable this logger in `airflow.cfg` with 
`logging_config_class = log_config.LOGGING_CONFIG`
   
   Where did you add the log_config.py file? In the airflow docker image or 
with the dags?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr opened a new pull request #18788: Accept custom run ID in TriggerDagRunOperator

[GitBox]

uranusjr opened a new pull request #18788:
URL: https://github.com/apache/airflow/pull/18788


   Fix #17438.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #12851: Find a way to inform our users about deprecated extras

[GitBox]

uranusjr commented on issue #12851:
URL: https://github.com/apache/airflow/issues/12851#issuecomment-937416195


   A built-in way to handle extra deprecation could be a good feature addition 
to the packaging system though. Someone should raise it on discuss.python.org  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #12851: Find a way to inform our users about deprecated extras

[GitBox]

uranusjr commented on issue #12851:
URL: https://github.com/apache/airflow/issues/12851#issuecomment-937415897


   If the deprecated extras are merged into the main dependency list, they can 
simply be removed since unrecognised extras are automatically ignored (with a 
warning). But there's not a good way to express that deprecation if the extra 
is renamed.
   
   One hack to this would be to introduce various empty 
`apache-airflow-deprecated-extra-foo` packages, and organise dependencies like 
this:
   
   ```python
   foo = ["dependency"]
   
   deprecated_foo = ["dependency", "apache-airflow-deprecated-extra-foo"]
   ```
   
   And detect the presence of `apache-airflow-deprecated-extra-foo` at runtime 
(with `importlib.metadata`) to show the deprecation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #17305: [Bug] Backfill job fails to run when there are tasks run into rescheduling state.

[GitBox]

github-actions[bot] commented on pull request #17305:
URL: https://github.com/apache/airflow/pull/17305#issuecomment-937402224


   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest main at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #18304: User induced deadlock for DAGs with `depends_on_past`

[GitBox]

uranusjr commented on issue #18304:
URL: https://github.com/apache/airflow/issues/18304#issuecomment-937399559


   What's the actionable here? Since the deadlock is induced by the user, 
should we prevent this from happening by not allowing Task B to be cleared? Or 
do we somehow resolve that deadlock—how?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #17473: New view "Operator arguments" between "Task Instance Details" and "Rendered"

[GitBox]

uranusjr commented on issue #17473:
URL: https://github.com/apache/airflow/issues/17473#issuecomment-937398844


   Do you want `file_format` to show 
`globals_variables['file_format_csv']['es']` literally? I don't think it's 
possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #16475: Add the ability to rerun success state task_instance in the Backfill API(#16452)

[GitBox]

uranusjr commented on pull request #16475:
URL: https://github.com/apache/airflow/pull/16475#issuecomment-937394114


   A couple of nitpicks, lgtm in general.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #16475: Add the ability to rerun success state task_instance in the Backfill API(#16452)

[GitBox]

uranusjr commented on a change in pull request #16475:
URL: https://github.com/apache/airflow/pull/16475#discussion_r723801733



##
File path: airflow/cli/commands/dag_command.py
##
@@ -122,13 +123,15 @@ def dag_backfill(args, dag=None):
 verbose=args.verbose,
 conf=run_conf,
 rerun_failed_tasks=args.rerun_failed_tasks,
+rerun_succeeded_task=args.rerun_succeeded_tasks,
 run_backwards=args.run_backwards,
 )
 except ValueError as vr:
 print(str(vr))
 sys.exit(1)
 
 
+

Review comment:
   I think linter would complain about this extra empty line?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #16475: Add the ability to rerun success state task_instance in the Backfill API(#16452)

[GitBox]

uranusjr commented on a change in pull request #16475:
URL: https://github.com/apache/airflow/pull/16475#discussion_r723801595



##
File path: airflow/cli/cli_parser.py
##
@@ -909,7 +917,9 @@ class GroupCommand(NamedTuple):
 "Run subsections of a DAG for a specified date range. If 
reset_dag_run option is used, "
 "backfill will first prompt users whether airflow should clear all 
the previous dag_run and "
 "task_instances within the backfill date range. If 
rerun_failed_tasks is used, backfill "
-"will auto re-run the previous failed task instances  within the 
backfill date range"
+"will auto re-run the previous failed task instances  within the 
backfill date range. If "

Review comment:
   ```suggestion
   "will auto re-run the previous failed task instances within the 
backfill date range. If "
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18622: Use seconds for schedule_delay metric

[GitBox]

uranusjr commented on a change in pull request #18622:
URL: https://github.com/apache/airflow/pull/18622#discussion_r723799273



##
File path: airflow/stats.py
##
@@ -331,10 +332,12 @@ def gauge(self, stat, value, rate=1, delta=False, 
tags=None):
 return None
 
 @validate_stat
-def timing(self, stat, dt, tags=None):
+def timing(self, stat: str, dt: Union[float, datetime.timedelta], tags: 
Optional[List[str]] = None):

Review comment:
   Also needed in `StatsLogger` (because that's what's actually gets 
inspected, not the concrete classes).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18622: Use seconds for schedule_delay metric

[GitBox]

uranusjr commented on a change in pull request #18622:
URL: https://github.com/apache/airflow/pull/18622#discussion_r723799273



##
File path: airflow/stats.py
##
@@ -331,10 +332,12 @@ def gauge(self, stat, value, rate=1, delta=False, 
tags=None):
 return None
 
 @validate_stat
-def timing(self, stat, dt, tags=None):
+def timing(self, stat: str, dt: Union[float, datetime.timedelta], tags: 
Optional[List[str]] = None):

Review comment:
   Also needed in `TimerProtocol` (because that's what's actually gets 
inspected, not the concrete classes).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #18568: [ISSUE-18468] add container option for SparkKubernetesSensor

[GitBox]

uranusjr commented on pull request #18568:
URL: https://github.com/apache/airflow/pull/18568#issuecomment-937390238


   Also, why is the current default value chosen? (The answer may be obvious 
but I know next to nothing about Spark and Kubernetes.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Updating constraints. Build id:1314305756

[github-bot]

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 7213d0c  Updating constraints. Build id:1314305756
7213d0c is described below

commit 7213d0cadf603e563ec8963ffe7e1482b8b42b30
Author: Automated GitHub Actions commit 
AuthorDate: Thu Oct 7 02:13:20 2021 +

Updating constraints. Build id:1314305756

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
HEAD of 'refs/heads/main' in 'apache/airflow'
with commit sha 63ca12c0f2b4b6db1c46dd55405f7055dcb1c02c.

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.6.txt  | 125 ++-
 constraints-3.7.txt  | 125 ++-
 constraints-3.8.txt  | 125 ++-
 constraints-3.9.txt  | 125 ++-
 constraints-no-providers-3.6.txt |  37 +++
 constraints-no-providers-3.7.txt |  39 ++-
 constraints-no-providers-3.8.txt |  39 ++-
 constraints-no-providers-3.9.txt |  37 +++
 constraints-source-providers-3.6.txt |  94 +-
 constraints-source-providers-3.7.txt |  94 +-
 constraints-source-providers-3.8.txt |  94 +-
 constraints-source-providers-3.9.txt |  94 +-
 12 files changed, 542 insertions(+), 486 deletions(-)

diff --git a/constraints-3.6.txt b/constraints-3.6.txt
index 5d165c3..0f34b6f 100644
--- a/constraints-3.6.txt
+++ b/constraints-3.6.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2021-09-27T11:33:07Z
+# This constraints file was automatically generated on 2021-10-07T02:10:45Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -37,6 +37,7 @@ PyJWT==1.7.1
 PyNaCl==1.4.0
 PyYAML==5.4.1
 Pygments==2.10.0
+Rx==3.2.0
 SQLAlchemy-JSONField==1.0.0
 SQLAlchemy-Utils==0.37.8
 SQLAlchemy==1.3.24
@@ -48,7 +49,7 @@ Werkzeug==1.0.1
 adal==1.2.7
 aiohttp==3.7.4.post0
 alabaster==0.7.12
-alembic==1.7.3
+alembic==1.7.4
 aliyun-python-sdk-core==2.13.35
 aliyun-python-sdk-kms==2.15.0
 amqp==5.0.6
@@ -59,10 +60,10 @@ apache-airflow-providers-airbyte==2.1.1
 apache-airflow-providers-alibaba==1.0.0
 apache-airflow-providers-amazon==2.2.0
 apache-airflow-providers-apache-beam==3.0.1
-apache-airflow-providers-apache-cassandra==2.0.1
+apache-airflow-providers-apache-cassandra==2.1.0
 apache-airflow-providers-apache-drill==1.0.1
 apache-airflow-providers-apache-druid==2.0.2
-apache-airflow-providers-apache-hdfs==2.1.0
+apache-airflow-providers-apache-hdfs==2.1.1
 apache-airflow-providers-apache-hive==2.0.2
 apache-airflow-providers-apache-kylin==2.0.1
 apache-airflow-providers-apache-livy==2.1.0
@@ -73,39 +74,40 @@ apache-airflow-providers-apache-sqoop==2.0.2
 apache-airflow-providers-asana==1.1.0
 apache-airflow-providers-celery==2.1.0
 apache-airflow-providers-cloudant==2.0.1
-apache-airflow-providers-cncf-kubernetes==2.0.2
-apache-airflow-providers-databricks==2.0.1
+apache-airflow-providers-cncf-kubernetes==2.0.3
+apache-airflow-providers-databricks==2.0.2
 apache-airflow-providers-datadog==2.0.1
 apache-airflow-providers-dingding==2.0.1
 apache-airflow-providers-discord==2.0.1
-apache-airflow-providers-docker==2.1.1
+apache-airflow-providers-docker==2.2.0
 apache-airflow-providers-elasticsearch==2.0.3
 apache-airflow-providers-exasol==2.0.1
 apache-airflow-providers-facebook==2.0.1
 apache-airflow-providers-ftp==2.0.1
-apache-airflow-providers-google==5.1.0
+apache-airflow-providers-google==6.0.0
 apache-airflow-providers-grpc==2.0.1
-apache-airflow-providers-hashicorp==2.1.0
+apache-airflow-providers-hashicorp==2.1.1
 apache-airflow-providers-http==2.0.1
 apache-airflow-providers-imap==2.0.1
+apache-airflow-providers-influxdb==1.0.0
 apache-airflow-providers-jdbc==2.0.1
 apache-airflow-providers-jenkins==2.0.2
 apache-airflow-providers-jira==2.0.1
-apache-airflow-providers-microsoft-azure==3.1.1
+apache-airflow-providers-microsoft-azure==3.2.0
 apache-airflow-providers-microsoft-mssql==2.0.1
 apache-airflow-providers-microsoft-psrp==1.0.1
 apache-airflow-providers-microsoft-winrm==2.0.1
 apache-airflow-providers-mongo==2.1.0
 apache-airflow-providers-mysql==2.1.1
-apache-airflow-providers-neo4j==2.0.1
+apache-airflow-providers-neo4j==2.0.2
 

[GitHub] [airflow] uranusjr commented on pull request #18622: Use seconds for schedule_delay metric

[GitBox]

uranusjr commented on pull request #18622:
URL: https://github.com/apache/airflow/pull/18622#issuecomment-937380123


   [Function parameter 
annotations](https://www.python.org/dev/peps/pep-3107/#parameters). The 
functions are already partially annotated, but we should annotate `dt` with 
`Union[float, datetime.timedelta]`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] coinzerge commented on pull request #18622: Use seconds for schedule_delay metric

[GitBox]

coinzerge commented on pull request #18622:
URL: https://github.com/apache/airflow/pull/18622#issuecomment-937378491


   added test. what do you mean by `we should also add annotations`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18563: Test kubernetes refresh config

[GitBox]

uranusjr commented on a change in pull request #18563:
URL: https://github.com/apache/airflow/pull/18563#discussion_r723789246



##
File path: tests/kubernetes/test_refresh_config.py
##
@@ -35,3 +41,60 @@ def 
test_parse_timestamp_should_convert_regular_timezone_to_unix_timestamp(self)
 def test_parse_timestamp_should_throw_exception(self):
 with pytest.raises(ParserError):
 _parse_timestamp("foobar")
+
+def test_get_kube_config_loader_for_yaml_file(self):
+refresh_kube_config_loader = 
_get_kube_config_loader_for_yaml_file('./kube_config')
+
+assert refresh_kube_config_loader is not None
+
+assert refresh_kube_config_loader.current_context['name'] == 
'federal-context'
+
+context = refresh_kube_config_loader.current_context['context']
+assert context is not None
+assert context['cluster'] == 'horse-cluster'
+assert context['namespace'] == 'chisel-ns'
+assert context['user'] == 'green-user'
+
+def test_get_api_key_with_prefix(self):
+
+refresh_config = RefreshConfiguration()
+refresh_config.api_key['key'] = '1234'
+assert refresh_config is not None
+
+api_key = refresh_config.get_api_key_with_prefix("key")
+
+assert api_key == '1234'
+
+@mock.patch('kubernetes.config.exec_provider.ExecProvider.__init__')
+@mock.patch('kubernetes.config.exec_provider.ExecProvider.run')
+def test_refresh_kube_config_loader(self, exec_provider_run, 
exec_provider_init):
+current_context = 
_get_kube_config_loader_for_yaml_file('./kube_config').current_context
+
+config_dict = {}
+config_dict['current-context'] = 'federal-context'
+config_dict['contexts'] = []
+config_dict['contexts'].append(current_context)
+
+config_dict['clusters'] = []
+
+cluster_config = {}
+cluster_config['api-version'] = 'v1'
+cluster_config['server'] = 'http://cow.org:8080'
+cluster_config['name'] = 'horse-cluster'
+cluster_root_config = {}
+cluster_root_config['cluster'] = cluster_config
+cluster_root_config['name'] = 'horse-cluster'
+config_dict['clusters'].append(cluster_root_config)
+
+refresh_kube_config_loader = 
RefreshKubeConfigLoader(config_dict=config_dict)
+refresh_kube_config_loader._user = {}
+
+config_node = ConfigNode('command', 'test')
+config_node.__dict__['apiVersion'] = '2.0'
+config_node.__dict__['command'] = 'test'
+
+exec_provider_init.return_value = None

Review comment:
   You can do this in the `patch` call by adding `return_value=None`. (Same 
for `exec_provider_run.return_value` below.)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #17651: Create CMDExecutor for raising an exception in case of error during cmd execution

[GitBox]

uranusjr commented on a change in pull request #17651:
URL: https://github.com/apache/airflow/pull/17651#discussion_r723788778



##
File path: tests/test_utils/gcp_system_helpers.py
##
@@ -123,6 +123,11 @@ def provide_gcs_bucket(bucket_name: str):
 
 @pytest.mark.system("google")
 class GoogleSystemTest(SystemTest):
+@staticmethod
+def execute_cmd(*args, **kwargs):
+executor = CMDExecutor()
+return executor.execute_cmd(*args, **kwargs)

Review comment:
   What's the need of this new function if it's only used once? What's 
wrong with instantiating the executor inline (like in the old code)?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #17651: Create CMDExecutor for raising an exception in case of error during cmd execution

[GitBox]

uranusjr commented on a change in pull request #17651:
URL: https://github.com/apache/airflow/pull/17651#discussion_r723788007



##
File path: tests/test_utils/logging_command_executor.py
##
@@ -62,5 +62,39 @@ def check_output(self, cmd):
 return output
 
 
+class CMDExecutor(LoggingCommandExecutor):

Review comment:
   I think we should spell out *Command* instead; there's no advantage to 
save the four characters. Same goes for other additions.

##
File path: airflow/exceptions.py
##
@@ -261,3 +261,7 @@ def __repr__(self) -> str:
 
 class TaskDeferralError(AirflowException):
 """Raised when a task failed during deferral for some reason."""
+
+
+class CMDExecutionError(AirflowException):
+"""Raise in case of error during cmd execution"""

Review comment:
   Since this exception is only ever raised in tests, it should not live in 
the main Airflow code base.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #18584: WIP: Pickling fix

[GitBox]

uranusjr commented on pull request #18584:
URL: https://github.com/apache/airflow/pull/18584#issuecomment-937374645


   How should the PR progress? From the discussion in the linked issue it seems 
we are not going down this route.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on pull request #18447: Add RedshiftSQLHook, RedshiftSQLOperator

[GitBox]

dstandish commented on pull request #18447:
URL: https://github.com/apache/airflow/pull/18447#issuecomment-937374301


   This looks good to me.  Anyone else want to give it a second look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #17446: Add support of `path` parameter for GCloud Storage Transfer Service operators

[GitBox]

github-actions[bot] commented on pull request #17446:
URL: https://github.com/apache/airflow/pull/17446#issuecomment-937372966


   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest main at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #18657: Cleanup of start_date and default arg use for Apache example DAGs

[GitBox]

github-actions[bot] commented on pull request #18657:
URL: https://github.com/apache/airflow/pull/18657#issuecomment-937372117


   The PR is likely OK to be merged with just subset of tests for default 
Python and Database versions without running the full matrix of tests, because 
it does not modify the core of Airflow. If the committers decide that the full 
tests matrix is needed, they will add the label 'full tests needed'. Then you 
should rebase to the latest main or amend the last commit of the PR, and push 
it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18657: Cleanup of start_date and default arg use for Apache example DAGs

[GitBox]

uranusjr commented on a change in pull request #18657:
URL: https://github.com/apache/airflow/pull/18657#discussion_r723784424



##
File path: airflow/providers/apache/druid/example_dags/example_druid_dag.py
##
@@ -19,22 +19,19 @@
 """
 Example Airflow DAG to submit Apache Druid json index file using 
`DruidOperator`
 """
+from datetime import datetime
+
 from airflow.models import DAG
 from airflow.providers.apache.druid.operators.druid import DruidOperator
-from airflow.utils.dates import days_ago
 
 with DAG(
 dag_id='example_druid_operator',
 schedule_interval=None,
-start_date=days_ago(2),
+start_date=datetime(2021, 1, 1),
 tags=['example'],
 ) as dag:
 # [START howto_operator_druid_submit]
-submit_job = DruidOperator(
-task_id='spark_submit_job',
-json_index_file='json_index.json',
-druid_ingest_conn_id='druid_ingest_default',
-)
+submit_job = DruidOperator(task_id='spark_submit_job', 
json_index_file='json_index.json')

Review comment:
   No, just making sure.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #18777: [Quarantine] test_no_orphan_process_will_be_left

[GitBox]

uranusjr commented on issue #18777:
URL: https://github.com/apache/airflow/issues/18777#issuecomment-937365630


   Should this be closed since the PR is merged?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (dc45f97 -> 63ca12c)

[uranusjr]

This is an automated email from the ASF dual-hosted git repository.

uranusjr pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from dc45f97  Adds CeleryKubernetesExecutor check to AirflowBaseView.run 
(#18441)
 add 63ca12c  Open src and dst in binary for samba copy (#18752)

No new revisions were added by this update.

Summary of changes:
 airflow/providers/samba/hooks/samba.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[GitHub] [airflow] uranusjr merged pull request #18752: Open src and dst in binary for samba copy

[GitBox]

uranusjr merged pull request #18752:
URL: https://github.com/apache/airflow/pull/18752


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18439: Check and run migration in commands if necessary

[GitBox]

uranusjr commented on a change in pull request #18439:
URL: https://github.com/apache/airflow/pull/18439#discussion_r723779357



##
File path: airflow/utils/db.py
##
@@ -626,27 +629,88 @@ def check_migrations(timeout):
 :param timeout: Timeout for the migration in seconds
 :return: None
 """
-from alembic.runtime.migration import MigrationContext
+ticker = 0
+while True:
+source_heads = get_source_heads()
+db_heads = get_db_heads()
+if source_heads == db_heads:
+break
+if ticker >= timeout:
+raise TimeoutError(
+f"There are still unapplied migrations after {ticker} seconds. 
"
+f"Migration Head(s) in DB: {db_heads} | Migration Head(s) in 
Source Code: {source_heads}"
+)
+ticker += 1
+time.sleep(1)
+log.info('Waiting for migrations... %s second(s)', ticker)
+
+
+def get_source_heads():
+"""
+Function to get the current migration head in the source code.
+
+:return: List of migration head(s)

Review comment:
   But the return value is actually a set? :p




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18439: Check and run migration in commands if necessary

[GitBox]

uranusjr commented on a change in pull request #18439:
URL: https://github.com/apache/airflow/pull/18439#discussion_r723779104



##
File path: airflow/utils/db.py
##
@@ -626,27 +629,88 @@ def check_migrations(timeout):
 :param timeout: Timeout for the migration in seconds
 :return: None
 """
-from alembic.runtime.migration import MigrationContext
+ticker = 0
+while True:
+source_heads = get_source_heads()
+db_heads = get_db_heads()
+if source_heads == db_heads:
+break
+if ticker >= timeout:
+raise TimeoutError(
+f"There are still unapplied migrations after {ticker} seconds. 
"
+f"Migration Head(s) in DB: {db_heads} | Migration Head(s) in 
Source Code: {source_heads}"
+)
+ticker += 1
+time.sleep(1)
+log.info('Waiting for migrations... %s second(s)', ticker)

Review comment:
   Maybe?
   
   ```suggestion
   for ticker in range(timeout):
   source_heads = get_source_heads()
   db_heads = get_db_heads()
   if source_heads == db_heads:
   return
   time.sleep(1)
   log.info('Waiting for migrations... %s second(s)', ticker)
   raise TimeoutError(
   f"There are still unapplied migrations after {timeout} seconds. "
   f"Migration Head(s) in DB: {db_heads} | Migration Head(s) in Source 
Code: {source_heads}"
   )
   ```
   
   (This emits one extra *Waiting for migrations* line; we can eliminate that 
with an additional `if` check but I don't feel it's worthwhile.)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18787: Show if an executor supports /run via duck-typing

[GitBox]

uranusjr commented on a change in pull request #18787:
URL: https://github.com/apache/airflow/pull/18787#discussion_r723773701



##
File path: airflow/executors/kubernetes_executor.py
##
@@ -419,9 +419,11 @@ def get_base_pod_from_template(pod_template_file: 
Optional[str], kube_config: An
 return 
PodGenerator.deserialize_model_file(kube_config.pod_template_file)
 
 
-class KubernetesExecutor(BaseExecutor, LoggingMixin):
+class KubernetesExecutor(BaseExecutor):

Review comment:
   BaseExecutor already inherits LoggingMixin so this is not needed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr opened a new pull request #18787: Show if an executor supports /run via duck-typing

[GitBox]

uranusjr opened a new pull request #18787:
URL: https://github.com/apache/airflow/pull/18787


   A follow-up to #18441 making the check more extendable in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #18786: Docs: Move part of timetable guide to concepts

[GitBox]

github-actions[bot] commented on pull request #18786:
URL: https://github.com/apache/airflow/pull/18786#issuecomment-937354638


   The PR is likely ready to be merged. No tests are needed as no important 
environment files, nor python files were modified by it. However, committers 
might decide that full test matrix is needed and add the 'full tests needed' 
label. Then you should rebase it to the latest main or amend the last commit of 
the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham commented on a change in pull request #18786: Docs: Move part of timetable guide to concepts

[GitBox]

jedcunningham commented on a change in pull request #18786:
URL: https://github.com/apache/airflow/pull/18786#discussion_r723771612



##
File path: docs/apache-airflow/dag-run.rst
##
@@ -22,7 +22,8 @@ A DAG Run is an object representing an instantiation of the 
DAG in time.
 Each DAG may or may not have a schedule, which informs how DAG Runs are
 created. ``schedule_interval`` is defined as a DAG argument, which can be 
passed a
 `cron expression `_ as
-a ``str``, a ``datetime.timedelta`` object, or one of the following cron 
"presets".
+a ``str``, a ``datetime.timedelta`` object, a ``Timetable`` object,
+or one of the following cron "presets".

Review comment:
   oops 臘‍♂️




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18786: Docs: Move part of timetable guide to concepts

[GitBox]

uranusjr commented on a change in pull request #18786:
URL: https://github.com/apache/airflow/pull/18786#discussion_r723768339



##
File path: docs/apache-airflow/dag-run.rst
##
@@ -22,7 +22,8 @@ A DAG Run is an object representing an instantiation of the 
DAG in time.
 Each DAG may or may not have a schedule, which informs how DAG Runs are
 created. ``schedule_interval`` is defined as a DAG argument, which can be 
passed a
 `cron expression `_ as
-a ``str``, a ``datetime.timedelta`` object, or one of the following cron 
"presets".
+a ``str``, a ``datetime.timedelta`` object, a ``Timetable`` object,
+or one of the following cron "presets".

Review comment:
   `schedule_interval` only accepts `str` (cron expression), `timedelta`, 
and `None`. A timetable object needs to be passed to the `timetable` argument 
instead.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (32947a4 -> dc45f97)

[uranusjr]

This is an automated email from the ASF dual-hosted git repository.

uranusjr pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 32947a4  Fix deprecated default for ``fab_logging_level`` to 
``WARNING`` (#18783)
 add dc45f97  Adds CeleryKubernetesExecutor check to AirflowBaseView.run 
(#18441)

No new revisions were added by this update.

Summary of changes:
 airflow/www/views.py | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #18441: Adds CeleryKubernetesExecutor check to AirflowBaseView.run

[GitBox]

boring-cyborg[bot] commented on pull request #18441:
URL: https://github.com/apache/airflow/pull/18441#issuecomment-937346192


   Awesome work, congrats on your first merged pull request!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr merged pull request #18441: Adds CeleryKubernetesExecutor check to AirflowBaseView.run

[GitBox]

uranusjr merged pull request #18441:
URL: https://github.com/apache/airflow/pull/18441


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr closed issue #18440: Can't Run Tasks from UI when using CeleryKubernetesExecutor

[GitBox]

uranusjr closed issue #18440:
URL: https://github.com/apache/airflow/issues/18440


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jedcunningham opened a new pull request #18786: Docs: Move part of timetable guide to concepts

[GitBox]

jedcunningham opened a new pull request #18786:
URL: https://github.com/apache/airflow/pull/18786


   It took me a while to find the docs for timetables. This proposes moving 
part of the guide into concepts.
   
   I considered moving part of the top level dag run page here also.
   
   @uranusjr, @ashb, @kaxil - feel free to keep iterating on this while I'm 
offline.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18755: Support mysql to s3 in parquet format

[GitBox]

uranusjr commented on a change in pull request #18755:
URL: https://github.com/apache/airflow/pull/18755#discussion_r723766341



##
File path: airflow/providers/amazon/aws/transfers/mysql_to_s3.py
##
@@ -92,15 +117,40 @@ def __init__(
 self.aws_conn_id = aws_conn_id
 self.verify = verify
 
-self.pd_csv_kwargs = pd_csv_kwargs or {}
-if "path_or_buf" in self.pd_csv_kwargs:
-raise AirflowException('The argument path_or_buf is not allowed, 
please remove it')
-if "index" not in self.pd_csv_kwargs:
-self.pd_csv_kwargs["index"] = index
-if "header" not in self.pd_csv_kwargs:
-self.pd_csv_kwargs["header"] = header
+if file_format == "csv":
+self.file_format = FILE_FORMAT.CSV
+else:
+self.file_format = FILE_FORMAT.PARQUET
+
+if pd_csv_kwargs:
+warnings.warn(
+"pd_csv_kwargs is deprecated. Please use pd_kwargs.",
+DeprecationWarning,
+stacklevel=2,
+)
+if index or header:
+warnings.warn(
+"index and header are deprecated. Please pass them via 
pd_kwargs.",
+DeprecationWarning,
+stacklevel=2,
+)

Review comment:
   > Please pass them via pd_kwargs.
   
   This should explain what exactly should set in `pd_kwargs`, e.g. *Please set 
`"index": True` in `pd_kwargs`*.
   
   But why do we need to change this in the first place? Why not just keep the 
current `header` and `index` function arguments and *not* add `header` and 
`index` keys in `pd_kwargs`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies

[GitBox]

uranusjr commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r723765288



##
File path: docs/apache-airflow/concepts/tasks.rst
##
@@ -36,16 +36,30 @@ Relationships
 
 The key part of using Tasks is defining how they relate to each other - their 
*dependencies*, or as we say in Airflow, their *upstream* and *downstream* 
tasks. You declare your Tasks first, and then you declare their dependencies 
second.
 
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` 
(bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
 
 first_task >> second_task >> [third_task, fourth_task]
 
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
 
 first_task.set_downstream(second_task)
 third_task.set_upstream(second_task)
 
-These both do exactly the same thing, but in general we recommend you use the 
bitshift operators, as they are easier to read in most cases.
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow 
`::
+
+@task
+def second_task(some_parameter):
+...
+@task
+def first_task():
+...
+
+second_task(first_task())
+
+
+The former two do exactly the same thing and work on any type of Operator, but 
in general we recommend you use the bitshift operators, as they are easier to 
read in most cases. The TaskFlow API likewise connects dependencies between 
tasks, but is tailored to python functions written with the @task-decorator, 
note that the functions here are not executed inline, rather they are converted 
to tasks to be scheduled and sent to Executors, for more information see 
:ref:`TaskFlow `.

Review comment:
   ```suggestion
   The former two do exactly the same thing and work on any type of Operator, 
but in general we recommend you use the bitshift operators, as they are easier 
to read in most cases. The TaskFlow API likewise connects dependencies between 
tasks, but is tailored to python functions written with the 
``@task-decorator``, note that the functions here are not executed inline, 
rather they are converted to tasks to be scheduled and sent to Executors, for 
more information see :ref:`TaskFlow `.
   ```
   
   Also the CI says ``concepts:taskflow` is not a defined reference. Where do 
you intend to link this to? (There's another occurrence of this above.)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #17927: Add example of TaskFlow to declare task dependencies

[GitBox]

uranusjr commented on a change in pull request #17927:
URL: https://github.com/apache/airflow/pull/17927#discussion_r723764445



##
File path: docs/apache-airflow/concepts/tasks.rst
##
@@ -36,16 +36,30 @@ Relationships
 
 The key part of using Tasks is defining how they relate to each other - their 
*dependencies*, or as we say in Airflow, their *upstream* and *downstream* 
tasks. You declare your Tasks first, and then you declare their dependencies 
second.
 
-There are two ways of declaring dependencies - using the ``>>`` and ``<<`` 
(bitshift) operators::
+There are three ways of declaring dependencies:
+
+Option 1: using the ``>>`` and ``<<`` (bitshift) operators::
 
 first_task >> second_task >> [third_task, fourth_task]
 
-Or the more explicit ``set_upstream`` and ``set_downstream`` methods::
+Option 2: The more explicit ``set_upstream`` and ``set_downstream`` methods::
 
 first_task.set_downstream(second_task)
 third_task.set_upstream(second_task)
 
-These both do exactly the same thing, but in general we recommend you use the 
bitshift operators, as they are easier to read in most cases.
+Option 3: The @task decorator provided by TaskFlow API, see :ref:`TaskFlow 
`::
+
+@task
+def second_task(some_parameter):
+...
+@task
+def first_task():
+...

Review comment:
   ```suggestion
   @task
   def second_task(some_parameter):
   ...
   
   @task
   def first_task():
   ...
   ```
   
   (I hope the linter won't complain this needs two empty lines.)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #17178: Amazon EMR on Amazon EKS

[GitBox]

github-actions[bot] commented on pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#issuecomment-937336230


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #17600: Fix wrong Postgres search_path set up instructions

[GitBox]

github-actions[bot] commented on pull request #17600:
URL: https://github.com/apache/airflow/pull/17600#issuecomment-937336213


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] guotongfei commented on a change in pull request #18755: Support mysql to s3 in parquet format

[GitBox]

guotongfei commented on a change in pull request #18755:
URL: https://github.com/apache/airflow/pull/18755#discussion_r723759306



##
File path: airflow/providers/amazon/aws/transfers/mysql_to_s3.py
##
@@ -92,15 +117,34 @@ def __init__(
 self.aws_conn_id = aws_conn_id
 self.verify = verify
 
-self.pd_csv_kwargs = pd_csv_kwargs or {}
-if "path_or_buf" in self.pd_csv_kwargs:
-raise AirflowException('The argument path_or_buf is not allowed, 
please remove it')
-if "index" not in self.pd_csv_kwargs:
-self.pd_csv_kwargs["index"] = index
-if "header" not in self.pd_csv_kwargs:
-self.pd_csv_kwargs["header"] = header
+if file_format == "csv":
+self.file_format = FILE_FORMAT.CSV
+else:
+self.file_format = FILE_FORMAT.PARQUET
+
+if pd_csv_kwargs:
+warnings.warn(
+"pd_csv_kwargs is deprecated. Please use pd_kwargs.",
+DeprecationWarning,
+stacklevel=2,
+)
+
+self.pd_kwargs = pd_kwargs or pd_csv_kwargs or {}
+if self.file_format == FILE_FORMAT.CSV:
+if "path_or_buf" in self.pd_kwargs:
+raise AirflowException('The argument path_or_buf is not 
allowed, please remove it')
+if "index" not in self.pd_kwargs:
+self.pd_kwargs["index"] = index
+if "header" not in self.pd_kwargs:
+self.pd_kwargs["header"] = header

Review comment:
   Create a warning, but it is not perfect.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] guotongfei commented on a change in pull request #18755: Support mysql to s3 in parquet format

[GitBox]

guotongfei commented on a change in pull request #18755:
URL: https://github.com/apache/airflow/pull/18755#discussion_r723757667



##
File path: airflow/providers/amazon/aws/transfers/mysql_to_s3.py
##
@@ -92,15 +117,34 @@ def __init__(
 self.aws_conn_id = aws_conn_id
 self.verify = verify
 
-self.pd_csv_kwargs = pd_csv_kwargs or {}
-if "path_or_buf" in self.pd_csv_kwargs:
-raise AirflowException('The argument path_or_buf is not allowed, 
please remove it')
-if "index" not in self.pd_csv_kwargs:
-self.pd_csv_kwargs["index"] = index
-if "header" not in self.pd_csv_kwargs:
-self.pd_csv_kwargs["header"] = header
+if file_format == "csv":
+self.file_format = FILE_FORMAT.CSV
+else:
+self.file_format = FILE_FORMAT.PARQUET
+
+if pd_csv_kwargs:
+warnings.warn(
+"pd_csv_kwargs is deprecated. Please use pd_kwargs.",
+DeprecationWarning,
+stacklevel=2,
+)
+
+self.pd_kwargs = pd_kwargs or pd_csv_kwargs or {}
+if self.file_format == FILE_FORMAT.CSV:
+if "path_or_buf" in self.pd_kwargs:
+raise AirflowException('The argument path_or_buf is not 
allowed, please remove it')
+if "index" not in self.pd_kwargs:
+self.pd_kwargs["index"] = index
+if "header" not in self.pd_kwargs:
+self.pd_kwargs["header"] = header

Review comment:
   Default values are provided for index and header. Any suggestion to 
check if they are set by users?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18784: Make a separate hook for interacting with the Pagerduty Events API

[GitBox]

uranusjr commented on a change in pull request #18784:
URL: https://github.com/apache/airflow/pull/18784#discussion_r723756897



##
File path: airflow/providers/pagerduty/hooks/pagerduty.py
##
@@ -26,14 +27,31 @@
 
 class PagerdutyHook(BaseHook):
 """
-Takes both PagerDuty API token directly and connection that has PagerDuty 
API token.
+The PagerdutyHook can be used to interact with both the PagerDuty API and 
the PagerDuty Events API.
 
+Takes both PagerDuty API token directly and connection that has PagerDuty 
API token.
 If both supplied, PagerDuty API token will be used.
+In these cases, the PagerDuty API token refers to an account token:
+
https://support.pagerduty.com/docs/generating-api-keys#generating-a-general-access-rest-api-key
+
https://support.pagerduty.com/docs/generating-api-keys#generating-a-personal-rest-api-key
+
+In order to send events (with the Pagerduty Events API), you will also 
need to specify the
+routing_key in the `extra` field

Review comment:
   ```suggestion
   routing_key in the ``extra`` field
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18690: Fixing tests that leave traces (users)

[GitBox]

uranusjr commented on a change in pull request #18690:
URL: https://github.com/apache/airflow/pull/18690#discussion_r723747637



##
File path: tests/test_utils/api_connexion_utils.py
##
@@ -14,26 +14,43 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+from contextlib import contextmanager
+
 from airflow.api_connexion.exceptions import EXCEPTIONS_LINK_MAP
 from airflow.www.security import EXISTING_ROLES
 
 
-def create_user(app, *, username, role_name, email=None, permissions=None):
+@contextmanager
+def create_user_scope(app, **kwargs) -> None:
+"""
+Helper function designed to be used with pytest fixture mainly.
+It will create a user and provide it for the fixture via YIELD (generator)
+then will tidy up once test is complete
+"""
+test_user = create_user(app, **kwargs)
+if kwargs.get('no_roles', None):
+test_user.roles = []
+try:
+yield test_user
+finally:
+delete_user(app, kwargs.get('username'))
+
+
+def create_user(app, **kwargs):
 appbuilder = app.appbuilder
+username = kwargs.get('username')
+role_name = kwargs.get('role_name')

Review comment:
   Why not make these appear directly in the argument list? Also the use of 
`get` here is likely wrong; `username` and `role_name` cannot be `None` (which 
is what `get` returns if the key is missing), so those arguments should be 
required in the first place.
   
   Instead of adding a `no_roles` argument in the We should change `role_name` 
in the original function signature to optional, and clear the roles if 
`role_name` is None.
   
   Something like
   
   ```python
   def create_user(app, *, username, role_name=None, email=None, 
permissions=None):
   ...
   if role_name is None:
   # This automatically reuses the built-in role.
   role = appbuilder.sm.find_role("Public")
   else:
   delete_role(app, role_name)
   role = create_role(app, role_name, kwargs.get('permissions'))
   return appbuilder.sm.add_user(...)
   
   @contextmanager
   def setup_user(app, *, username, role_name=None, email=None, 
permissions=None):
   user = create_user(
   app,
   username=username,
   role_name=role_name,
   email=email,
   permissions=permissions,
   )
   if role_name is None:
   user.roles = []
   try:
   yield user
   finally:
   delete_user(app, username)
   ```

##
File path: tests/test_utils/api_connexion_utils.py
##
@@ -14,26 +14,43 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+from contextlib import contextmanager
+
 from airflow.api_connexion.exceptions import EXCEPTIONS_LINK_MAP
 from airflow.www.security import EXISTING_ROLES
 
 
-def create_user(app, *, username, role_name, email=None, permissions=None):
+@contextmanager
+def create_user_scope(app, **kwargs) -> None:

Review comment:
   Same here; `kwargs` should be an explicit argument list. Also the return 
type is wrong.
   
   ```suggestion
   def create_user_scope(app, *, username, role_name=None, email=None, 
permissions=None):
   ```

##
File path: tests/test_utils/api_connexion_utils.py
##
@@ -14,26 +14,43 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+from contextlib import contextmanager
+
 from airflow.api_connexion.exceptions import EXCEPTIONS_LINK_MAP
 from airflow.www.security import EXISTING_ROLES
 
 
-def create_user(app, *, username, role_name, email=None, permissions=None):
+@contextmanager
+def create_user_scope(app, **kwargs) -> None:
+"""
+Helper function designed to be used with pytest fixture mainly.
+It will create a user and provide it for the fixture via YIELD (generator)
+then will tidy up once test is complete
+"""
+test_user = create_user(app, **kwargs)
+if kwargs.get('no_roles', None):
+test_user.roles = []
+try:
+yield test_user
+finally:
+delete_user(app, kwargs.get('username'))
+
+
+def create_user(app, **kwargs):
 appbuilder = app.appbuilder
+username = kwargs.get('username')
+role_name = kwargs.get('role_name')

Review comment:
   Also, should we just turn `create_user` into a context manager and 
always use it? Or are there instances where we _don't_ want the user to be 
deleted? That feels wrong to me.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi edited a comment on pull request #18645: Add AWS Fargate profile support

[GitBox]

ferruzzi edited a comment on pull request #18645:
URL: https://github.com/apache/airflow/pull/18645#issuecomment-937326775


   CI is failing due to the deprecation warnings that @mik-laj  added, but I'm 
not sure why.  The error message says pod name needs to be explicitly passed 
and lists three example dags that violate that rule, but those three dags were 
updated in 
https://github.com/apache/airflow/pull/18645/commits/b51151c446136ec8743699c3cff507e80ffa6844
 to pass the name in, so it shouldn't be an issue.
   
   What have I missed?
   
   [EDIT: Looks like it restarted and corrected itself.  I may have just been 
too impatient.]
   [EDIT 2:  Currently shows "all checks have passed".ferruzzi_patience++]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] guotongfei commented on a change in pull request #18755: Support mysql to s3 in parquet format

[GitBox]

guotongfei commented on a change in pull request #18755:
URL: https://github.com/apache/airflow/pull/18755#discussion_r723755406



##
File path: airflow/providers/amazon/aws/transfers/mysql_to_s3.py
##
@@ -92,15 +117,34 @@ def __init__(
 self.aws_conn_id = aws_conn_id
 self.verify = verify
 
-self.pd_csv_kwargs = pd_csv_kwargs or {}
-if "path_or_buf" in self.pd_csv_kwargs:
-raise AirflowException('The argument path_or_buf is not allowed, 
please remove it')
-if "index" not in self.pd_csv_kwargs:
-self.pd_csv_kwargs["index"] = index
-if "header" not in self.pd_csv_kwargs:
-self.pd_csv_kwargs["header"] = header
+if file_format == "csv":
+self.file_format = FILE_FORMAT.CSV
+else:
+self.file_format = FILE_FORMAT.PARQUET
+
+if pd_csv_kwargs:
+warnings.warn(
+"pd_csv_kwargs is deprecated. Please use pd_kwargs.",
+DeprecationWarning,
+stacklevel=2,
+)
+
+self.pd_kwargs = pd_kwargs or pd_csv_kwargs or {}
+if self.file_format == FILE_FORMAT.CSV:
+if "path_or_buf" in self.pd_kwargs:
+raise AirflowException('The argument path_or_buf is not 
allowed, please remove it')
+if "index" not in self.pd_kwargs:
+self.pd_kwargs["index"] = index
+if "header" not in self.pd_kwargs:
+self.pd_kwargs["header"] = header

Review comment:
   They have value as parameters, I guess it is just for convenience. We 
could deprecate them.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi edited a comment on pull request #18645: Add AWS Fargate profile support

[GitBox]

ferruzzi edited a comment on pull request #18645:
URL: https://github.com/apache/airflow/pull/18645#issuecomment-937326775


   CI is failing due to the deprecation warnings that @mik-laj  added, but I'm 
not sure why.  The error message says pod name needs to be explicitly passed 
and lists three example dags that violate that rule, but those three dags were 
updated in 
https://github.com/apache/airflow/pull/18645/commits/b51151c446136ec8743699c3cff507e80ffa6844
 to pass the name in, so it shouldn't be an issue.
   
   What have I missed?
   
   [Looks like it restarted and corrected itself.  I may have just been too 
impatient.]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on pull request #18645: Add AWS Fargate profile support

[GitBox]

ferruzzi commented on pull request #18645:
URL: https://github.com/apache/airflow/pull/18645#issuecomment-937326775


   CI is failing due to the deprecation warnings that @mik-laj  added, but I'm 
not sure why.  The error message says pod name needs to be explicitly passed 
and lists three example dags that violate that rule, but those three dags were 
updated in 
https://github.com/apache/airflow/pull/18645/commits/b51151c446136ec8743699c3cff507e80ffa6844
 to pass the name in, so it shouldn't be an issue.
   
   What have I missed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr closed issue #18779: Task status is not updating during run within TaskGroup on UI

[GitBox]

uranusjr closed issue #18779:
URL: https://github.com/apache/airflow/issues/18779


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #18779: Task status is not updating during run within TaskGroup on UI

[GitBox]

uranusjr commented on issue #18779:
URL: https://github.com/apache/airflow/issues/18779#issuecomment-937320978


   Seems to be the case, yes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a change in pull request #18718: Add hook_params in BaseSqlOperator

[GitBox]

uranusjr commented on a change in pull request #18718:
URL: https://github.com/apache/airflow/pull/18718#discussion_r723745553



##
File path: airflow/models/connection.py
##
@@ -289,8 +289,11 @@ def rotate_fernet_key(self):
 if self._extra and self.is_extra_encrypted:
 self._extra = fernet.rotate(self._extra.encode('utf-8')).decode()
 
-def get_hook(self):
-"""Return hook based on conn_type."""
+def get_hook(self, **kwargs):
+"""
+Return hook based on conn_type
+:param hook_params: dictionary of keyword arguments to be passed to 
the hook constructor
+"""

Review comment:
   This docstring does not seem to match the actual argument?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #18622: Use seconds for schedule_delay metric

[GitBox]

uranusjr commented on pull request #18622:
URL: https://github.com/apache/airflow/pull/18622#issuecomment-937318611


   Could you add a test for this in `tests/core/test_stats.py`? And while we're 
at it, we should also add annotations to `dt` in the `stats` module to make it 
explicit it can be either float or timedelta.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] alex-astronomer commented on issue #18714: Make it possible to delete a permission from a role using the stable REST API

[GitBox]

alex-astronomer commented on issue #18714:
URL: https://github.com/apache/airflow/issues/18714#issuecomment-937292186


   Heard that.  I'll make that a part of this PR that I started working on for 
this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #18714: Make it possible to delete a permission from a role using the stable REST API

[GitBox]

uranusjr commented on issue #18714:
URL: https://github.com/apache/airflow/issues/18714#issuecomment-937289868


   We can, although the current PATCH behaviour is still somehow unintuitive 
and should at least be documented.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] zachliu commented on pull request #17626: ECSOperator realtime logging

[GitBox]

zachliu commented on pull request #17626:
URL: https://github.com/apache/airflow/pull/17626#issuecomment-937257582


   @codenamestif i'm testing out this feature in our integration env right now. 
IT IS AWESOME! 
   i'll soon move it to production so that we can really see 
https://github.com/apache/airflow/pull/18733 in action!
   thank you! :+1: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] john-jac opened a new pull request #18785: Amazon Athena Example

[GitBox]

john-jac opened a new pull request #18785:
URL: https://github.com/apache/airflow/pull/18785


   This example, and associated documentation, shows how to use Amazon Athena 
to read a table from a CSV file in an S3 bucket, read from that table, and 
clean up all resources.
   
   The S3 functions use decorators and Task Flow API, and the only prerequisite 
is an AWS account and an existing Athena database.  The rest of the sample data 
is created and cleaned up within the example.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] alex-astronomer commented on issue #18714: Make it possible to delete a permission from a role using the stable REST API

[GitBox]

alex-astronomer commented on issue #18714:
URL: https://github.com/apache/airflow/issues/18714#issuecomment-937177641


   Maybe we make a new `PUT` endpoint that has the behavior that we expect?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi edited a comment on pull request #18645: Add AWS Fargate profile support

[GitBox]

ferruzzi edited a comment on pull request #18645:
URL: https://github.com/apache/airflow/pull/18645#issuecomment-937150276


   Rebased and added terminal state checks to the Sensors and sensor unit tests.
   
   I believe that all above concerns/comments have been addressed at this point.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on a change in pull request #18645: Add AWS Fargate profile support

[GitBox]

ferruzzi commented on a change in pull request #18645:
URL: https://github.com/apache/airflow/pull/18645#discussion_r723681321



##
File path: airflow/providers/amazon/aws/sensors/eks.py
##
@@ -77,6 +82,65 @@ def poke(self, context):
 return cluster_state == self.target_state
 
 
+class EKSFargateProfileStateSensor(BaseSensorOperator):
+"""
+Check the state of an AWS Fargate profile until the state of the profile 
equals the target state.
+
+:param cluster_name: The name of the Cluster which the AWS Fargate profile 
is attached to. (templated)
+:type cluster_name: str
+:param fargate_profile_name: The name of the Fargate profile to watch. 
(templated)
+:type fargate_profile_name: str
+:param target_state: Target state of the Fargate profile. (templated)
+:type target_state: FargateProfileStates
+:param region: Which AWS region the connection should use. (templated)
+If this is None or empty then the default boto3 behaviour is used.
+:type region: str
+:param aws_conn_id: The Airflow connection used for AWS credentials. 
(templated)
+ If this is None or empty then the default boto3 behaviour is used. If
+ running Airflow in a distributed manner and aws_conn_id is None or
+ empty, then the default boto3 configuration would be used (and must be
+ maintained on each worker node).
+:type aws_conn_id: str
+"""
+
+template_fields = ("cluster_name", "fargate_profile_name", "target_state", 
"aws_conn_id", "region")
+ui_color = "#ff9900"
+ui_fgcolor = "#232F3E"
+
+def __init__(
+self,
+*,
+cluster_name: str,
+fargate_profile_name: str,
+target_state: FargateProfileStates = FargateProfileStates.ACTIVE,
+aws_conn_id: str = DEFAULT_CONN_ID,
+region: Optional[str] = None,
+**kwargs,
+):
+self.cluster_name = cluster_name
+self.fargate_profile_name = fargate_profile_name
+self.target_state = (
+target_state
+if isinstance(target_state, FargateProfileStates)
+else FargateProfileStates(str(target_state).upper())
+)
+self.aws_conn_id = aws_conn_id
+self.region = region
+super().__init__(**kwargs)
+
+def poke(self, context):
+eks_hook = EKSHook(
+aws_conn_id=self.aws_conn_id,
+region_name=self.region,
+)
+
+fargate_profile_state = eks_hook.get_fargate_profile_state(
+clusterName=self.cluster_name, 
fargateProfileName=self.fargate_profile_name
+)
+self.log.info("Fargate profile state: %s", fargate_profile_state)
+return fargate_profile_state == self.target_state

Review comment:
   Addressed in 
https://github.com/apache/airflow/pull/18645/commits/575dffa2b252a004efa4676e98dbb6add2f686a7




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ferruzzi commented on pull request #18645: Add AWS Fargate profile support

[GitBox]

ferruzzi commented on pull request #18645:
URL: https://github.com/apache/airflow/pull/18645#issuecomment-937150276


   Rebased and added terminal state checks to the Sensors and sensor unit tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bbovenzi commented on issue #18732: Tooltip element is not removed and overlays another clickable elements

[GitBox]

bbovenzi commented on issue #18732:
URL: https://github.com/apache/airflow/issues/18732#issuecomment-937143372


   This is a weird issue. I can confirm it happens with "All, Paused, None". 
But at least most of our tooltips do not experience this behavior. 
   I'm going to need to look into this more.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] coinzerge edited a comment on pull request #18622: Use seconds for schedule_delay metric

[GitBox]

coinzerge edited a comment on pull request #18622:
URL: https://github.com/apache/airflow/pull/18622#issuecomment-937142337


   @ashb @uranusjr updated PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] coinzerge commented on pull request #18622: Use seconds for schedule_delay metric

[GitBox]

coinzerge commented on pull request #18622:
URL: https://github.com/apache/airflow/pull/18622#issuecomment-937142337


   @uranusjr updated PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bbovenzi commented on issue #18779: Task status is not updating during run within TaskGroup on UI

[GitBox]

bbovenzi commented on issue #18779:
URL: https://github.com/apache/airflow/issues/18779#issuecomment-937102128


   I believe this is a duplicate of #18369


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (e351ead -> 32947a4)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from e351ead  Change the color for deferred status to mediumpurple (#18414)
 add 32947a4  Fix deprecated default for ``fab_logging_level`` to 
``WARNING`` (#18783)

No new revisions were added by this update.

Summary of changes:
 airflow/config_templates/config.yml  | 2 +-
 airflow/config_templates/default_airflow.cfg | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

[GitHub] [airflow] kaxil merged pull request #18783: Fix deprecated default for ``fab_logging_level`` to ``WARNING``

[GitBox]

kaxil merged pull request #18783:
URL: https://github.com/apache/airflow/pull/18783


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] matsavage commented on a change in pull request #18773: updates pipeline_timeout CloudDataFusionStartPipelineOperator

[GitBox]

matsavage commented on a change in pull request #18773:
URL: https://github.com/apache/airflow/pull/18773#discussion_r723626576



##
File path: airflow/providers/google/cloud/operators/datafusion.py
##
@@ -826,10 +826,8 @@ def __init__(
 
 if success_states:
 self.success_states = success_states
-self.pipeline_timeout = pipeline_timeout

Review comment:
   I think I fixed this with my change on line 826 :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] matsavage commented on a change in pull request #18773: updates pipeline_timeout CloudDataFusionStartPipelineOperator

[GitBox]

matsavage commented on a change in pull request #18773:
URL: https://github.com/apache/airflow/pull/18773#discussion_r723625899



##
File path: airflow/providers/google/cloud/operators/datafusion.py
##
@@ -823,13 +823,12 @@ def __init__(
 self.delegate_to = delegate_to
 self.impersonation_chain = impersonation_chain
 self.asynchronous = asynchronous
+self.pipeline_timeout = pipeline_timeout

Review comment:
   @ephraimbuddy does this solve the issue you meant, or am I still missing 
something?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Change the color for deferred status to mediumpurple (#18414)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new e351ead  Change the color for deferred status to mediumpurple (#18414)
e351ead is described below

commit e351eada1189ed50abef8facb1036599ae96399d
Author: Collin McNulty 
AuthorDate: Wed Oct 6 14:45:12 2021 -0500

Change the color for deferred status to mediumpurple (#18414)

Changes the color for the deferred status to something more distinct, 
mediumpurple.


![image](https://user-images.githubusercontent.com/13177948/134234580-11d1cec0-d9ad-45e4-9577-e9066b829363.png)


![image](https://user-images.githubusercontent.com/13177948/134234619-04851b97-5a26-4ecb-baa8-dbf3cd7270b0.png)


closes: #18245
---
 airflow/settings.py| 2 +-
 airflow/utils/state.py | 4 ++--
 docs/apache-airflow/howto/customize-ui.rst | 2 +-
 tests/www/views/test_views_home.py | 4 ++--
 4 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/airflow/settings.py b/airflow/settings.py
index 1403759..fc2c6bb 100644
--- a/airflow/settings.py
+++ b/airflow/settings.py
@@ -93,7 +93,7 @@ STATE_COLORS = {
 "upstream_failed": "orange",
 "skipped": "pink",
 "scheduled": "tan",
-"deferred": "lightseagreen",
+"deferred": "mediumpurple",
 }
 
 
diff --git a/airflow/utils/state.py b/airflow/utils/state.py
index 39e89b5..99fe392 100644
--- a/airflow/utils/state.py
+++ b/airflow/utils/state.py
@@ -120,9 +120,9 @@ class State:
 TaskInstanceState.SKIPPED: 'pink',
 TaskInstanceState.REMOVED: 'lightgrey',
 TaskInstanceState.SCHEDULED: 'tan',
-TaskInstanceState.SENSING: 'lightseagreen',
-TaskInstanceState.DEFERRED: 'lightseagreen',
+TaskInstanceState.DEFERRED: 'mediumpurple',
 }
+state_color[TaskInstanceState.SENSING] = 
state_color[TaskInstanceState.DEFERRED]
 state_color.update(STATE_COLORS)  # type: ignore
 
 @classmethod
diff --git a/docs/apache-airflow/howto/customize-ui.rst 
b/docs/apache-airflow/howto/customize-ui.rst
index 1d52e36..179c223 100644
--- a/docs/apache-airflow/howto/customize-ui.rst
+++ b/docs/apache-airflow/howto/customize-ui.rst
@@ -45,7 +45,7 @@ following steps:
   "upstream_failed": "orange",
   "skipped": "darkorchid",
   "scheduled": "tan",
-  "deferred": "lightseagreen",
+  "deferred": "mediumpurple",
   }
 
 
diff --git a/tests/www/views/test_views_home.py 
b/tests/www/views/test_views_home.py
index 1225df7..748f5ee 100644
--- a/tests/www/views/test_views_home.py
+++ b/tests/www/views/test_views_home.py
@@ -51,10 +51,10 @@ def test_home(capture_templates, admin_client):
 check_content_in_response('DAGs', resp)
 val_state_color_mapping = (
 'const STATE_COLOR = {'
-'"deferred": "lightseagreen", "failed": "red", '
+'"deferred": "mediumpurple", "failed": "red", '
 '"null": "lightblue", "queued": "gray", '
 '"removed": "lightgrey", "restarting": "violet", "running": 
"lime", '
-'"scheduled": "tan", "sensing": "lightseagreen", '
+'"scheduled": "tan", "sensing": "mediumpurple", '
 '"shutdown": "blue", "skipped": "pink", '
 '"success": "green", "up_for_reschedule": "turquoise", '
 '"up_for_retry": "gold", "upstream_failed": "orange"};'

[GitHub] [airflow] kaxil merged pull request #18414: Change the color for deferred status to mediumpurple

[GitBox]

kaxil merged pull request #18414:
URL: https://github.com/apache/airflow/pull/18414


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil closed issue #18245: Deferred status color not distinct enough

[GitBox]

kaxil closed issue #18245:
URL: https://github.com/apache/airflow/issues/18245


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #18359: Session propagation for task_failed_deps command

[GitBox]

kaxil commented on pull request #18359:
URL: https://github.com/apache/airflow/pull/18359#issuecomment-936994977


   closed by https://github.com/apache/airflow/pull/18499


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil closed pull request #18359: Session propagation for task_failed_deps command

[GitBox]

kaxil closed pull request #18359:
URL: https://github.com/apache/airflow/pull/18359


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil closed issue #16896: Deleting task instance from UI deletes Dag Run too

[GitBox]

kaxil closed issue #16896:
URL: https://github.com/apache/airflow/issues/16896


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #16896: Deleting task instance from UI deletes Dag Run too

[GitBox]

kaxil commented on issue #16896:
URL: https://github.com/apache/airflow/issues/16896#issuecomment-936992647


   closed by https://github.com/apache/airflow/pull/17719


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #18773: updates pipeline_timeout CloudDataFusionStartPipelineOperator

[GitBox]

ephraimbuddy commented on a change in pull request #18773:
URL: https://github.com/apache/airflow/pull/18773#discussion_r723621050



##
File path: airflow/providers/google/cloud/operators/datafusion.py
##
@@ -826,10 +826,8 @@ def __init__(
 
 if success_states:
 self.success_states = success_states
-self.pipeline_timeout = pipeline_timeout

Review comment:
   I mean in the hook for the operator, we should use the timeout. There’s 
a method in the hook that uses timeout, we should change it to accept the 
timeout we have here




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (5306a60 -> cfc2e1b)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 5306a60  Bump pre-commits: `black` and `pyupgrade` (#18782)
 add cfc2e1b  Always run at least one backfill from BackfillJob (#18742)

No new revisions were added by this update.

Summary of changes:
 airflow/cli/commands/dag_command.py| 10 +-
 airflow/jobs/backfill_job.py   | 16 +++-
 airflow/models/dag.py  |  6 +-
 tests/cli/commands/test_dag_command.py |  2 ++
 4 files changed, 27 insertions(+), 7 deletions(-)

[GitHub] [airflow] kaxil merged pull request #18742: Always run at least one backfill from BackfillJob

[GitBox]

kaxil merged pull request #18742:
URL: https://github.com/apache/airflow/pull/18742


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil closed issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"

[GitBox]

kaxil closed issue #18473:
URL: https://github.com/apache/airflow/issues/18473


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (43f334f -> 5306a60)

[kaxilnaik]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 43f334f  Move docker decorator example dag to docker provider (#18739)
 add 5306a60  Bump pre-commits: `black` and `pyupgrade` (#18782)

No new revisions were added by this update.

Summary of changes:
 .pre-commit-config.yaml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

[GitHub] [airflow] kaxil merged pull request #18782: Bump pre-commits: `black` and `pyupgrade`

[GitBox]

kaxil merged pull request #18782:
URL: https://github.com/apache/airflow/pull/18782


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] GuidoTournois opened a new pull request #18784: Make a separate hook for interacting with the Pagerduty Events API

[GitBox]

GuidoTournois opened a new pull request #18784:
URL: https://github.com/apache/airflow/pull/18784


   In order to make use of the PagerDuty Events API, you need to provide two 
API keys in the PagerdutyHook, eventhough one of them is not required. 
   
   1) token: used to access the general Pagerduty rest API. Set in __init__ and 
can't be None
   2) routing_key: used to access the Events API.
   
   
   The proposed setup is two have two separate hooks, one for accessing the 
general Pagerduty API, the other hook for interacting with the EventsAPI. 
   
   However, in order to be back compatible the original hook still contains 
code to interact with both API's, albeit a bit confusing. At least, to me.
   
   
   closes: #18771 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] xinbinhuang edited a comment on issue #18111: HeadObject error when trying to upload file to S3

[GitBox]

xinbinhuang edited a comment on issue #18111:
URL: https://github.com/apache/airflow/issues/18111#issuecomment-936953895


   > I want it to be able to upload a file in any case (file already exists or 
not), I know about the `replace` flag, but I also want to cover the case when 
the file doesn't exists.
   > 
   > My suggestion is to catch the error and allow the upload, without adding 
another error. It will be something like, updating:
   > 
   > ```python
   > except ClientError as e:
   > if e.response["ResponseMetadata"]["HTTPStatusCode"] == 404:
   > return False
   > else:
   > raise e
   > ```
   > 
   > to:
   > 
   > ```python
   > except ClientError as e:
   > if e.response["ResponseMetadata"]["HTTPStatusCode"] IN (404, 
403):
   > return False
   > else:
   > raise e
   > ```
   > 
   > shortest PR ever  I will try to submit one next week.
   
   I believe the `replace=True` flag will just upload the file regardless if 
the key exists or not.  We don't call the `check_for_key` function when 
`replace=True`. Have you tried it?
   
   IMO, It's not reasonable for us to swallow the permission error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] xinbinhuang commented on issue #18111: HeadObject error when trying to upload file to S3

[GitBox]

xinbinhuang commented on issue #18111:
URL: https://github.com/apache/airflow/issues/18111#issuecomment-936953895


   > I want it to be able to upload a file in any case (file already exists or 
not), I know about the `replace` flag, but I also want to cover the case when 
the file doesn't exists.
   > 
   > My suggestion is to catch the error and allow the upload, without adding 
another error. It will be something like, updating:
   > 
   > ```python
   > except ClientError as e:
   > if e.response["ResponseMetadata"]["HTTPStatusCode"] == 404:
   > return False
   > else:
   > raise e
   > ```
   > 
   > to:
   > 
   > ```python
   > except ClientError as e:
   > if e.response["ResponseMetadata"]["HTTPStatusCode"] IN (404, 
403):
   > return False
   > else:
   > raise e
   > ```
   > 
   > shortest PR ever  I will try to submit one next week.
   
   I believe the `replace` flag will just upload the file regardless if the key 
exists or not. Have you tried it? We don't call the `check_for_key` function 
when `replace=True`.
   
   IMO, It's not reasonable for us to swallow the permission error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #18783: Fix deprecated default for ``fab_logging_level`` to ``WARNING``

[GitBox]

github-actions[bot] commented on pull request #18783:
URL: https://github.com/apache/airflow/pull/18783#issuecomment-936938273


   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest main at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #18780: Update changelog with more changes since 2.2.0b2

[GitBox]

github-actions[bot] commented on pull request #18780:
URL: https://github.com/apache/airflow/pull/18780#issuecomment-936932850


   The PR is likely ready to be merged. No tests are needed as no important 
environment files, nor python files were modified by it. However, committers 
might decide that full test matrix is needed and add the 'full tests needed' 
label. Then you should rebase it to the latest main or amend the last commit of 
the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #18783: Fix deprecated default for ``fab_logging_level`` to ``WARNING``

[GitBox]

kaxil opened a new pull request #18783:
URL: https://github.com/apache/airflow/pull/18783


   `WARN` is deprecated in favor of `WARNING`
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #18782: Bump pre-commits: `black` and `pyupgrade`

[GitBox]

github-actions[bot] commented on pull request #18782:
URL: https://github.com/apache/airflow/pull/18782#issuecomment-936902413


   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest main at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org