[GitHub] [airflow] potiuk commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
potiuk commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295454820 ## airflow/cli/cli_parser.py: ## Review Comment: I am very strong for just hard-failing it. If the one who develops executor wants to have a new command,

[GitHub] [airflow] potiuk commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
potiuk commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295454820 ## airflow/cli/cli_parser.py: ## Review Comment: I am very strong for just hard-failing it. If the one who develops executor wants to have a new command,

[airflow] branch v2-7-stable updated (84a5e464dc -> c08c82e9dd)

2023-08-15 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-7-stable in repository https://gitbox.apache.org/repos/asf/airflow.git from 84a5e464dc Update RELEASE_NOTES.rst add eda4bc4b87 Enable test connection in UI if explicitly set

[GitHub] [airflow] ephraimbuddy merged pull request #33390: Sync v2-7-stable with v2-7-test to release 2.7.0

2023-08-15 Thread via GitHub
ephraimbuddy merged PR #33390: URL: https://github.com/apache/airflow/pull/33390 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@air

[GitHub] [airflow] potiuk commented on pull request #31066: Support naming customization on helm chart resources

2023-08-15 Thread via GitHub
potiuk commented on PR #31066: URL: https://github.com/apache/airflow/pull/31066#issuecomment-1680047336 🎉 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [airflow] moiseenkov opened a new pull request, #33429: Fix system test example_cloud_storage_transfer_service_aws

2023-08-15 Thread via GitHub
moiseenkov opened a new pull request, #33429: URL: https://github.com/apache/airflow/pull/33429 - Migrate and fix system test `example_cloud_storage_transfer_service_aws` - Update documentation -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [airflow] potiuk commented on a diff in pull request #33355: Add Python Virtualenv Operator Caching

2023-08-15 Thread via GitHub
potiuk commented on code in PR #33355: URL: https://github.com/apache/airflow/pull/33355#discussion_r1295118289 ## airflow/operators/python.py: ## @@ -606,7 +613,39 @@ def _prepare_venv(self, venv_path: Path) -> None: index_urls=self.index_urls, ) +de

[GitHub] [airflow] potiuk commented on a diff in pull request #33355: Add Python Virtualenv Operator Caching

2023-08-15 Thread via GitHub
potiuk commented on code in PR #33355: URL: https://github.com/apache/airflow/pull/33355#discussion_r1295118289 ## airflow/operators/python.py: ## @@ -606,7 +613,39 @@ def _prepare_venv(self, venv_path: Path) -> None: index_urls=self.index_urls, ) +de

[GitHub] [airflow] potiuk commented on a diff in pull request #33355: Add Python Virtualenv Operator Caching

2023-08-15 Thread via GitHub
potiuk commented on code in PR #33355: URL: https://github.com/apache/airflow/pull/33355#discussion_r1295118289 ## airflow/operators/python.py: ## @@ -606,7 +613,39 @@ def _prepare_venv(self, venv_path: Path) -> None: index_urls=self.index_urls, ) +de

[airflow] branch main updated (7dd0d84d14 -> 86f9f95be5)

2023-08-15 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 7dd0d84d14 Add typing to concrete taskflow examples (#33417) add 86f9f95be5 Revert "Remove airflow user from `Dock

[GitHub] [airflow] potiuk merged pull request #33419: Revert "Remove airflow user from `Dockerfile.ci` (#31970)"

2023-08-15 Thread via GitHub
potiuk merged PR #33419: URL: https://github.com/apache/airflow/pull/33419 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on pull request #33419: Revert "Remove airflow user from `Dockerfile.ci` (#31970)"

2023-08-15 Thread via GitHub
potiuk commented on PR #33419: URL: https://github.com/apache/airflow/pull/33419#issuecomment-1680036940 Merging for now - there is a discussion on how maybe we could leverage the "root" user for other kinds of debugging in slack https://apache-airflow.slack.com/archives/CQ9QHSFQX/p16920852

[GitHub] [airflow] alexbegg commented on pull request #33427: Add sql_hook_params parameter to S3ToSqlOperator

2023-08-15 Thread via GitHub
alexbegg commented on PR #33427: URL: https://github.com/apache/airflow/pull/33427#issuecomment-1680034122 @uranusjr This PR retains how `S3ToSqlOperator ` accesses `BaseHook`, but adds a way to pass `hook_params` parameter to `BaseHook.get_hook`, which then calls `Connection.get_hook` (whi

[GitHub] [airflow] alexbegg commented on pull request #33425: Add sql_hook_params parameter to SqlToS3Operator

2023-08-15 Thread via GitHub
alexbegg commented on PR #33425: URL: https://github.com/apache/airflow/pull/33425#issuecomment-1680029397 I have now split the `S3ToSqlOperator` changes into its own PR: https://github.com/apache/airflow/pull/33427, it needed a different approach since it uses `BaseHook.get_hook` instead o

[airflow] branch main updated: Add typing to concrete taskflow examples (#33417)

2023-08-15 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 7dd0d84d14 Add typing to concrete taskflow exampl

[GitHub] [airflow] uranusjr merged pull request #33417: Add typing to concrete taskflow examples

2023-08-15 Thread via GitHub
uranusjr merged PR #33417: URL: https://github.com/apache/airflow/pull/33417 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] alexbegg opened a new pull request, #33427: Add sql_hook_params parameter to S3ToSqlOperator

2023-08-15 Thread via GitHub
alexbegg opened a new pull request, #33427: URL: https://github.com/apache/airflow/pull/33427 Adding `sql_hook_params` parameter to `SqlToS3Operator`. This will allow you to pass extra config params to the underlying SQL hook. This uses the same "sql_hook_params" parameter name as alr

[airflow] branch main updated: Fix handling of default value and serialization of Param class (#33141)

2023-08-15 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 489ca14946 Fix handling of default value and

[GitHub] [airflow] hussein-awala merged pull request #33141: Fix handling of default value and serialization of Param class

2023-08-15 Thread via GitHub
hussein-awala merged PR #33141: URL: https://github.com/apache/airflow/pull/33141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[GitHub] [airflow] potiuk commented on pull request #33422: Make sure impersonation tests use a temporary `AIRFLOW_HOME`

2023-08-15 Thread via GitHub
potiuk commented on PR #33422: URL: https://github.com/apache/airflow/pull/33422#issuecomment-1680007971 > Well, sqlite test failure it is a bit expected, because impersonate user doesn't have access to `/root/airflow/sqlite/airflow.db` 🤔 Should not it be using the database in the tem

[airflow] branch constraints-2-7 updated: Updating constraints. Github run id:5874754672

2023-08-15 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-2-7 in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-2-7 by this push: new 6f6cca375c Updating const

[GitHub] [airflow] Bowrna opened a new issue, #33426: Statsd metrics dagrun.schedule_delay sends time in different format against the seconds specified in docs

2023-08-15 Thread via GitHub
Bowrna opened a new issue, #33426: URL: https://github.com/apache/airflow/issues/33426 ### What do you see as an issue? For the airflow statsd metric received at the statsd client, the metric `dagrun.schedule_delay` described in airflow docs says it sends the seconds of delay between

[airflow] 01/01: fixup! Update RELEASE_NOTES.rst

2023-08-15 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch v2-7-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit c08c82e9dd0e4aabaa5121519819a636df635210 Author: Ephraim Anierobi AuthorDate: Mon Aug 14 12:16:57 2023 +010

[airflow] branch v2-7-test updated (997c749856 -> c08c82e9dd)

2023-08-15 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-7-test in repository https://gitbox.apache.org/repos/asf/airflow.git omit 997c749856 Apply suggestions from code review omit b741d389ba fixup! Update RELEASE_NOTES.rst ne

[airflow] branch v2-7-test updated (b741d389ba -> 997c749856)

2023-08-15 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-7-test in repository https://gitbox.apache.org/repos/asf/airflow.git from b741d389ba fixup! Update RELEASE_NOTES.rst add 997c749856 Apply suggestions from code review No new

[airflow] 01/02: Improve graph nesting logic (#33421)

2023-08-15 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch v2-7-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 57114c653891ac2cb5015a8f9e5011ebb9723220 Author: Brent Bovenzi AuthorDate: Wed Aug 16 00:38:46 2023 -0400

[airflow] 02/02: fixup! Update RELEASE_NOTES.rst

2023-08-15 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch v2-7-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit b741d389ba0830d3d83913386101dc09484e37ae Author: Ephraim Anierobi AuthorDate: Mon Aug 14 12:16:57 2023 +010

[airflow] branch v2-7-test updated (206699d059 -> b741d389ba)

2023-08-15 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch v2-7-test in repository https://gitbox.apache.org/repos/asf/airflow.git omit 206699d059 fixup! Update RELEASE_NOTES.rst new 57114c6538 Improve graph nesting logic (#33421)

[airflow] branch main updated: Improve graph nesting logic (#33421)

2023-08-15 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 6547b8182f Improve graph nesting logic (#3

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
vandonr-amz commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295362936 ## airflow/cli/cli_parser.py: ## @@ -65,6 +65,22 @@ ALL_COMMANDS_DICT: dict[str, CLICommand] = {sp.name: sp for sp in airflow_commands} +# Check if sub-comm

[GitHub] [airflow] uranusjr commented on a diff in pull request #33417: Add typing to concrete taskflow examples

2023-08-15 Thread via GitHub
uranusjr commented on code in PR #33417: URL: https://github.com/apache/airflow/pull/33417#discussion_r1295360220 ## docs/shared/template-examples/taskflow.rst: ## @@ -17,8 +17,12 @@ .. code-block:: python +from airflow.models.taskinstance import TaskInstance +from

[GitHub] [airflow] uranusjr commented on a diff in pull request #33413: Add read only validation to read only fields

2023-08-15 Thread via GitHub
uranusjr commented on code in PR #33413: URL: https://github.com/apache/airflow/pull/33413#discussion_r1295358300 ## airflow/www/forms.py: ## @@ -124,21 +124,28 @@ class DagRunEditForm(DynamicForm): We don't actually want to allow editing, so everything is read-only here.

[GitHub] [airflow] Lee-W commented on a diff in pull request #33405: respect "soft_fail" argument when running BatchSensor in deferrable mode

2023-08-15 Thread via GitHub
Lee-W commented on code in PR #33405: URL: https://github.com/apache/airflow/pull/33405#discussion_r1295349323 ## airflow/providers/amazon/aws/sensors/batch.py: ## @@ -115,7 +115,15 @@ def execute_complete(self, context: Context, event: dict[str, Any]) -> None: Relies

[GitHub] [airflow] uranusjr commented on a diff in pull request #33405: respect "soft_fail" argument when running BatchSensor in deferrable mode

2023-08-15 Thread via GitHub
uranusjr commented on code in PR #33405: URL: https://github.com/apache/airflow/pull/33405#discussion_r1295347917 ## airflow/providers/amazon/aws/sensors/batch.py: ## @@ -115,7 +115,15 @@ def execute_complete(self, context: Context, event: dict[str, Any]) -> None: Reli

[GitHub] [airflow] Lee-W commented on a diff in pull request #33403: Respect "soft_fail" for core async sensors

2023-08-15 Thread via GitHub
Lee-W commented on code in PR #33403: URL: https://github.com/apache/airflow/pull/33403#discussion_r1295335522 ## airflow/triggers/temporal.py: ## @@ -34,18 +35,29 @@ class DateTimeTrigger(BaseTrigger): The provided datetime MUST be in UTC. """ -def __init__(self

[GitHub] [airflow] Lee-W commented on a diff in pull request #33405: respect "soft_fail" argument when running BatchSensor in deferrable mode

2023-08-15 Thread via GitHub
Lee-W commented on code in PR #33405: URL: https://github.com/apache/airflow/pull/33405#discussion_r1295333987 ## airflow/providers/amazon/aws/sensors/batch.py: ## @@ -115,7 +115,15 @@ def execute_complete(self, context: Context, event: dict[str, Any]) -> None: Relies

[GitHub] [airflow] uranusjr commented on pull request #33425: Add sql_hook_params to S3ToSql and SqlToS3 operators

2023-08-15 Thread via GitHub
uranusjr commented on PR #33425: URL: https://github.com/apache/airflow/pull/33425#issuecomment-1679891886 Because you changed how the BaseHook is accessed and it breaks the mock. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [airflow] uranusjr commented on a diff in pull request #33405: respect "soft_fail" argument when running BatchSensor in deferrable mode

2023-08-15 Thread via GitHub
uranusjr commented on code in PR #33405: URL: https://github.com/apache/airflow/pull/33405#discussion_r1295331003 ## airflow/providers/amazon/aws/sensors/batch.py: ## @@ -115,7 +115,15 @@ def execute_complete(self, context: Context, event: dict[str, Any]) -> None: Reli

[GitHub] [airflow] uranusjr commented on a diff in pull request #33403: Respect "soft_fail" for core async sensors

2023-08-15 Thread via GitHub
uranusjr commented on code in PR #33403: URL: https://github.com/apache/airflow/pull/33403#discussion_r1295330532 ## airflow/triggers/temporal.py: ## @@ -34,18 +35,29 @@ class DateTimeTrigger(BaseTrigger): The provided datetime MUST be in UTC. """ -def __init__(s

[GitHub] [airflow] Lee-W commented on a diff in pull request #33405: respect "soft_fail" argument when running BatchSensor in deferrable mode

2023-08-15 Thread via GitHub
Lee-W commented on code in PR #33405: URL: https://github.com/apache/airflow/pull/33405#discussion_r1295327281 ## airflow/providers/amazon/aws/sensors/__init__.py: ## Review Comment: Sounds good. I just move it to `airflow/providers/amazon/aws/sensors/utils.py` -- This

[GitHub] [airflow] Lee-W commented on a diff in pull request #33403: Respect "soft_fail" for core async sensors

2023-08-15 Thread via GitHub
Lee-W commented on code in PR #33403: URL: https://github.com/apache/airflow/pull/33403#discussion_r1295321179 ## airflow/triggers/temporal.py: ## @@ -34,18 +35,29 @@ class DateTimeTrigger(BaseTrigger): The provided datetime MUST be in UTC. """ -def __init__(self

[GitHub] [airflow] alexbegg commented on pull request #33425: Add sql_hook_params to S3ToSql and SqlToS3 operators

2023-08-15 Thread via GitHub
alexbegg commented on PR #33425: URL: https://github.com/apache/airflow/pull/33425#issuecomment-1679857091 FYI, I added a test to tests/providers/amazon/aws/transfers/test_s3_to_sql.py that is passing, but the other 2 tests in that file are failing. The code for those tests has not changed

[GitHub] [airflow] uranusjr commented on a diff in pull request #33403: Respect "soft_fail" for core async sensors

2023-08-15 Thread via GitHub
uranusjr commented on code in PR #33403: URL: https://github.com/apache/airflow/pull/33403#discussion_r1295291332 ## airflow/triggers/temporal.py: ## @@ -34,18 +35,29 @@ class DateTimeTrigger(BaseTrigger): The provided datetime MUST be in UTC. """ -def __init__(s

[GitHub] [airflow] uranusjr commented on a diff in pull request #33403: Respect "soft_fail" for core async sensors

2023-08-15 Thread via GitHub
uranusjr commented on code in PR #33403: URL: https://github.com/apache/airflow/pull/33403#discussion_r1295290027 ## airflow/triggers/temporal.py: ## @@ -34,18 +35,29 @@ class DateTimeTrigger(BaseTrigger): The provided datetime MUST be in UTC. """ -def __init__(s

[GitHub] [airflow] uranusjr commented on a diff in pull request #33405: respect "soft_fail" argument when running BatchSensor in deferrable mode

2023-08-15 Thread via GitHub
uranusjr commented on code in PR #33405: URL: https://github.com/apache/airflow/pull/33405#discussion_r1295289373 ## airflow/providers/amazon/aws/sensors/__init__.py: ## Review Comment: Better put this in somewhere like `utils.py`; importing `__init__` in child modules can

[GitHub] [airflow] thesuperzapper commented on pull request #31829: Disable celery `task_acks_late`

2023-08-15 Thread via GitHub
thesuperzapper commented on PR #31829: URL: https://github.com/apache/airflow/pull/31829#issuecomment-1679827878 @hussein-awala have you had any chance to look into it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [airflow] uranusjr commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
uranusjr commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295279424 ## airflow/cli/cli_parser.py: ## @@ -65,6 +65,22 @@ ALL_COMMANDS_DICT: dict[str, CLICommand] = {sp.name: sp for sp in airflow_commands} +# Check if sub-command

[GitHub] [airflow] Lee-W commented on pull request #33403: Respect "soft_fail" for core async sensors

2023-08-15 Thread via GitHub
Lee-W commented on PR #33403: URL: https://github.com/apache/airflow/pull/33403#issuecomment-1679821158 > Let's wait #33424 before deciding if we need to merge this one. Hi, I just checked #33424. We might still need this one, as the exceptions are raised in the `execute` method befor

[GitHub] [airflow] alexbegg opened a new pull request, #33425: Adding sql_hook_params to S3ToSql and SqlToS3 operators

2023-08-15 Thread via GitHub
alexbegg opened a new pull request, #33425: URL: https://github.com/apache/airflow/pull/33425 Adding `sql_hook_params` parameter to `S3ToSqlOperator` and `SqlToS3Operator`. This will allow you to pass extra config params to the underlying SQL hook. This uses the same "sql_hook_params

[GitHub] [airflow] github-actions[bot] closed pull request #31206: Add default value to data columns in SQLtoS3Operator

2023-08-15 Thread via GitHub
github-actions[bot] closed pull request #31206: Add default value to data columns in SQLtoS3Operator URL: https://github.com/apache/airflow/pull/31206 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] github-actions[bot] closed pull request #31637: Respect `AUTOCOMMIT` in `session_parameters` of `SnowflakeHook`

2023-08-15 Thread via GitHub
github-actions[bot] closed pull request #31637: Respect `AUTOCOMMIT` in `session_parameters` of `SnowflakeHook` URL: https://github.com/apache/airflow/pull/31637 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [airflow] jedcunningham commented on pull request #31066: Support naming customization on helm chart resources

2023-08-15 Thread via GitHub
jedcunningham commented on PR #31066: URL: https://github.com/apache/airflow/pull/31066#issuecomment-1679782581 Thanks @fgalind1! Congrats on your first commit 🎉 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #31066: Support naming customization on helm chart resources

2023-08-15 Thread via GitHub
boring-cyborg[bot] commented on PR #31066: URL: https://github.com/apache/airflow/pull/31066#issuecomment-1679782203 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.

[airflow] branch main updated: Support naming customization on helm chart resources (#31066)

2023-08-15 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 1a828f393b Support naming customization on h

[GitHub] [airflow] jedcunningham merged pull request #31066: Support naming customization on helm chart resources

2023-08-15 Thread via GitHub
jedcunningham merged PR #31066: URL: https://github.com/apache/airflow/pull/31066 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
vandonr-amz commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295252828 ## airflow/cli/cli_parser.py: ## Review Comment: > Yeah, that's what I meant, raise an exception. ok :p So, yeah, that'd be good if an executor int

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
vandonr-amz commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295251837 ## airflow/cli/cli_parser.py: ## Review Comment: there is `log.fatal` which is marked deprecated in favor of `log.critical`, the highest level of logging. I

[GitHub] [airflow] jedcunningham commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
jedcunningham commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295251461 ## airflow/cli/cli_parser.py: ## Review Comment: Yeah, that's what I meant, raise an exception. -- This is an automated message from the Apache Git Ser

[GitHub] [airflow] o-nikolas commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
o-nikolas commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295241845 ## airflow/cli/cli_parser.py: ## Review Comment: To me fatal means abort the whole thing and exit, there's no fatal log level. _But_ if that's indeed not what

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
vandonr-amz commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295240828 ## airflow/cli/cli_parser.py: ## Review Comment: I think Jed is not talking about an exception but more about the level of logging to use (i.e. use somethin

[GitHub] [airflow] o-nikolas commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
o-nikolas commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295239347 ## airflow/cli/cli_parser.py: ## Review Comment: Yeah, that's a tough one. I see both sides. If a user really wants their cli command to have a name already u

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
vandonr-amz commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295222030 ## airflow/cli/cli_parser.py: ## Review Comment: Yeah, I wasn't sure... It's not preventing airflow from working, but it could be something needing attentio

[airflow] branch main updated: Remove uploading to test pypi for releases (#33412)

2023-08-15 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 23dfa7ffed Remove uploading to test pypi for

[GitHub] [airflow] jedcunningham merged pull request #33412: Remove uploading to test pypi for releases

2023-08-15 Thread via GitHub
jedcunningham merged PR #33412: URL: https://github.com/apache/airflow/pull/33412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[GitHub] [airflow] jedcunningham commented on a diff in pull request #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
jedcunningham commented on code in PR #33423: URL: https://github.com/apache/airflow/pull/33423#discussion_r1295217210 ## airflow/cli/cli_parser.py: ## Review Comment: I wonder if instead of just warning, we should make it fatal? -- This is an automated message from the

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vandonr-amz commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295210703 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from fla

[GitHub] [airflow] hussein-awala opened a new pull request, #33424: Create a new method used to resume the task in order to implement specific logic for operators

2023-08-15 Thread via GitHub
hussein-awala opened a new pull request, #33424: URL: https://github.com/apache/airflow/pull/33424 related: #33405 related: #33196 This PR creates a new method used by the TI to resume the execution, and moves handling the triggers errors to this method. In this way we

[GitHub] [airflow] vandonr-amz opened a new pull request, #33423: Add a mechanism to warn if executors override existing CLI commands

2023-08-15 Thread via GitHub
vandonr-amz opened a new pull request, #33423: URL: https://github.com/apache/airflow/pull/33423 Currently, if an executor redefines and existing command, I think it silently overrides the existing one, which can be an issue. I'm in the process of adding one more way to add commands, so d

[GitHub] [airflow] jedcunningham commented on a diff in pull request #32724: feat(chart): Add existing secret support for PGBouncer metrics exporter.

2023-08-15 Thread via GitHub
jedcunningham commented on code in PR #32724: URL: https://github.com/apache/airflow/pull/32724#discussion_r1295190150 ## chart/templates/secrets/pgbouncer-stats-secret.yaml: ## Review Comment: I agree there isn't much point, but if the env var uses the config... the secre

[GitHub] [airflow] hussein-awala commented on a diff in pull request #33406: fix(TimeSensorAsync): use DAG timezone

2023-08-15 Thread via GitHub
hussein-awala commented on code in PR #33406: URL: https://github.com/apache/airflow/pull/33406#discussion_r1295148713 ## airflow/sensors/time_sensor.py: ## @@ -64,7 +64,7 @@ def __init__(self, *, target_time, **kwargs): self.target_time = target_time aware_t

[GitHub] [airflow] hussein-awala commented on pull request #33413: Add read only validation to read only fields

2023-08-15 Thread via GitHub
hussein-awala commented on PR #33413: URL: https://github.com/apache/airflow/pull/33413#issuecomment-1679659825 Actually it is not related to this change, and it was fixed by #33416, I merge main -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[airflow] branch main updated (60df70526a -> 88b8ba9861)

2023-08-15 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 60df70526a Add parameter to pass role ARN to `GlueJobOperator ` (#33408) add 88b8ba9861 Update gcp.rst (#33

[GitHub] [airflow] hussein-awala merged pull request #33420: Update gcp.rst

2023-08-15 Thread via GitHub
hussein-awala merged PR #33420: URL: https://github.com/apache/airflow/pull/33420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[GitHub] [airflow] hussein-awala commented on pull request #33412: Remove uploading to test pypi for releases

2023-08-15 Thread via GitHub
hussein-awala commented on PR #33412: URL: https://github.com/apache/airflow/pull/33412#issuecomment-1679658187 The failed test is not related to this change, and it was fixed by #33416 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] hussein-awala commented on pull request #33413: Add read only validation to read only fields

2023-08-15 Thread via GitHub
hussein-awala commented on PR #33413: URL: https://github.com/apache/airflow/pull/33413#issuecomment-1679648730 Could you check the failed static checks? Here is the doc: https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst -- This is an automated message from the Apache Git

[GitHub] [airflow] vincbeck commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295119967 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from flask

[airflow] branch main updated (cc360b73c9 -> 60df70526a)

2023-08-15 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from cc360b73c9 Fixing static checks for FabAirflowSecurityManagerOverride (#33416) add 60df70526a Add parameter to

[GitHub] [airflow] vincbeck merged pull request #33408: Add parameter to pass role ARN to `GlueJobOperator `

2023-08-15 Thread via GitHub
vincbeck merged PR #33408: URL: https://github.com/apache/airflow/pull/33408 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vandonr-amz commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295115914 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from fla

[GitHub] [airflow] hussein-awala commented on a diff in pull request #33420: Update gcp.rst

2023-08-15 Thread via GitHub
hussein-awala commented on code in PR #33420: URL: https://github.com/apache/airflow/pull/33420#discussion_r1295123482 ## docs/apache-airflow-providers-google/connections/gcp.rst: ## @@ -27,7 +27,7 @@ The Google Cloud connection type enables the Google Cloud Integrations. Auth

[GitHub] [airflow] potiuk commented on pull request #33355: Add Python Virtualenv Operator Caching

2023-08-15 Thread via GitHub
potiuk commented on PR #33355: URL: https://github.com/apache/airflow/pull/33355#issuecomment-1679615177 > I propose I could add a note to the docs. Yes. Note in the docs should be sufficient (avoid modifying the venv after it is created or similar ) . checking venv context is not a g

[GitHub] [airflow] fdemiane commented on pull request #32606: Add `CloudBatchHook` and operators

2023-08-15 Thread via GitHub
fdemiane commented on PR #32606: URL: https://github.com/apache/airflow/pull/32606#issuecomment-1679624577 Hello @potiuk, I need a reviewer to review and accept the pull request. @ahidalgob already reviewed it. I would appreciate your assistance. Thanks! -- This is an automated mess

[GitHub] [airflow] hussein-awala commented on issue #33402: Scheduler fails to remove pods in Completed state

2023-08-15 Thread via GitHub
hussein-awala commented on issue #33402: URL: https://github.com/apache/airflow/issues/33402#issuecomment-1679598748 Actually we set the resource version to 0 on failure: ```python except Exception: self.log.exception("Unknown error in KubernetesJobWatcher

[GitHub] [airflow] potiuk commented on a diff in pull request #33355: Add Python Virtualenv Operator Caching

2023-08-15 Thread via GitHub
potiuk commented on code in PR #33355: URL: https://github.com/apache/airflow/pull/33355#discussion_r1295118289 ## airflow/operators/python.py: ## @@ -606,7 +613,39 @@ def _prepare_venv(self, venv_path: Path) -> None: index_urls=self.index_urls, ) +de

[GitHub] [airflow] vincbeck commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295078542 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from flask

[GitHub] [airflow] vincbeck commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295115248 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from flask

[GitHub] [airflow] vincbeck commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295081179 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -81,3 +148,35 @@ def get_url_user_profile(self) -> str | None: if not self.security_manager.user_vie

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vandonr-amz commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295115914 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from fla

[GitHub] [airflow] yvesmarinho commented on issue #32727: airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator not filter file

2023-08-15 Thread via GitHub
yvesmarinho commented on issue #32727: URL: https://github.com/apache/airflow/issues/32727#issuecomment-1679588811 please give me more days and i will come back with the test result -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[GitHub] [airflow] hussein-awala commented on issue #33402: Scheduler fails to remove pods in Completed state

2023-08-15 Thread via GitHub
hussein-awala commented on issue #33402: URL: https://github.com/apache/airflow/issues/33402#issuecomment-1679565485 I have two questions: 1. I see in the log: ``` Deleted pod: TaskInstanceKey(dag_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', task_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', run_

[GitHub] [airflow] bbovenzi opened a new pull request, #33421: Improve graph nesting logic

2023-08-15 Thread via GitHub
bbovenzi opened a new pull request, #33421: URL: https://github.com/apache/airflow/pull/33421 Fix how we handle nested groups to make sure edge labels inside of groups are correctly positions and that nested nodes with the same label don't mess with rendering. | Before | Af

[GitHub] [airflow] Dan-Git-Rep opened a new pull request, #33420: Update gcp.rst

2023-08-15 Thread via GitHub
Dan-Git-Rep opened a new pull request, #33420: URL: https://github.com/apache/airflow/pull/33420 three options to connect to GCP not the initial specified two --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://gith

[GitHub] [airflow] potiuk commented on pull request #33419: Revert "Remove airflow user from `Dockerfile.ci` (#31970)"

2023-08-15 Thread via GitHub
potiuk commented on PR #33419: URL: https://github.com/apache/airflow/pull/33419#issuecomment-1679539083 It turned out that airflow user is useful for SSH debugging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] ferruzzi merged pull request #33416: Fixing static checks for FabAirflowSecurityManagerOverride

2023-08-15 Thread via GitHub
ferruzzi merged PR #33416: URL: https://github.com/apache/airflow/pull/33416 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] Taragolis opened a new pull request, #33422: Make sure impersonation tests use a temporary `AIRFLOW_HOME`

2023-08-15 Thread via GitHub
Taragolis opened a new pull request, #33422: URL: https://github.com/apache/airflow/pull/33422 Continuation of #30313 ``` The impersonation tests of ours modify permissions for the AIRFLOW_HOME recursively. This is quite a bad idea when the tests are executed inside

[airflow] branch main updated (3766ab07d7 -> cc360b73c9)

2023-08-15 Thread ferruzzi
This is an automated email from the ASF dual-hosted git repository. ferruzzi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 3766ab07d7 More complete fix for dev release scripts to filter commit for PR (#33418) add cc360b73c9 Fixing sta

[GitHub] [airflow] vincbeck commented on a diff in pull request #33213: Implement `is_authorized()` in auth manager

2023-08-15 Thread via GitHub
vincbeck commented on code in PR #33213: URL: https://github.com/apache/airflow/pull/33213#discussion_r1295078542 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -17,13 +17,37 @@ # under the License. from __future__ import annotations +import itertools + from flask

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #33420: Update gcp.rst

2023-08-15 Thread via GitHub
boring-cyborg[bot] commented on PR #33420: URL: https://github.com/apache/airflow/pull/33420#issuecomment-1679559348 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] potiuk merged pull request #33418: More complete fix for dev release scripts to filter commit for PR

2023-08-15 Thread via GitHub
potiuk merged PR #33418: URL: https://github.com/apache/airflow/pull/33418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] shahar1 commented on pull request #33419: Revert "Remove airflow user from `Dockerfile.ci` (#31970)"

2023-08-15 Thread via GitHub
shahar1 commented on PR #33419: URL: https://github.com/apache/airflow/pull/33419#issuecomment-1679545805 It might be worth adding documentation of where/how it's being used to avoid (non-)accidental deletion for the next time :) -- This is an automated message from the Apache Git Service

  1   2   >