[GitHub] [airflow] syedahsn commented on a diff in pull request #32513: EMR serverless Create/Start/Stop/Delete Application deferrable mode

2023-07-20 Thread via GitHub
syedahsn commented on code in PR #32513: URL: https://github.com/apache/airflow/pull/32513#discussion_r1270293579 ## airflow/providers/amazon/aws/operators/emr.py: ## @@ -1332,16 +1391,46 @@ def execute(self, context: Context) -> None: self.log.info("Stopping applicatio

[GitHub] [airflow] syedahsn commented on a diff in pull request #32513: EMR serverless Create/Start/Stop/Delete Application deferrable mode

2023-07-20 Thread via GitHub
syedahsn commented on code in PR #32513: URL: https://github.com/apache/airflow/pull/32513#discussion_r1270291295 ## airflow/providers/amazon/aws/operators/emr.py: ## @@ -1079,6 +1100,35 @@ def execute(self, context: Context) -> str | None: ) return applica

[GitHub] [airflow] potiuk commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-20 Thread via GitHub
potiuk commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1270242735 ## docs/providers-configurations-ref.rst: ## Review Comment: Yes. I can likely use include. Good point. -- This is an automated message from the Apache Git S

[GitHub] [airflow] potiuk commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-20 Thread via GitHub
potiuk commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1270242124 ## docs/apache-airflow/configurations-ref.rst: ## @@ -32,6 +32,13 @@ generated using the secret key has a short expiry time though - make sure that t that you run ai

[GitHub] [airflow] potiuk commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-20 Thread via GitHub
potiuk commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1270241742 ## airflow/cli/commands/config_command.py: ## @@ -48,6 +49,14 @@ def show_config(args): def get_value(args): """Get one value from configuration.""" +# whi

[GitHub] [airflow] MaicoTimmerman commented on pull request #28626: Introduce logging integration for Apache WebHDFS

2023-07-20 Thread via GitHub
MaicoTimmerman commented on PR #28626: URL: https://github.com/apache/airflow/pull/28626#issuecomment-1645002215 Not stale, but I've been short on time. Will pick this up in couple of weeks. -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [airflow] potiuk commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-20 Thread via GitHub
potiuk commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1270215447 ## airflow/cli/commands/config_command.py: ## @@ -48,6 +49,14 @@ def show_config(args): def get_value(args): """Get one value from configuration.""" +# whi

[airflow] branch constraints-main updated: Updating constraints. Github run id:5618049881

2023-07-20 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 49f96e9b9e Updating con

[GitHub] [airflow] Lee-W commented on issue #32732: airflow.providers.amazong.aws.hooks.base_aws.BaseSessionFactory feeds synchronous credentials to aiobotocore when using `assume_role`

2023-07-20 Thread via GitHub
Lee-W commented on issue #32732: URL: https://github.com/apache/airflow/issues/32732#issuecomment-1644940068 Format the example code for readbility ```python from datetime import timedelta from airflow.decorators import dag from airflow.providers.amazon.aws.operators.batch

[GitHub] [airflow] merobi-hub opened a new pull request, #32735: improve performance of pr script in dev/stats

2023-07-20 Thread via GitHub
merobi-hub opened a new pull request, #32735: URL: https://github.com/apache/airflow/pull/32735 Due to the endpoint used to get PRs from the GitHub API in the `get_contributor_stats` script in dev/stats, which does not accept dates or date ranges, the script errors out with a rate-limit exc

[GitHub] [airflow] shubham22 commented on pull request #32733: FIX: deferrable operators now use AioCredentials

2023-07-20 Thread via GitHub
shubham22 commented on PR #32733: URL: https://github.com/apache/airflow/pull/32733#issuecomment-1644916224 @raphaelauv @syedahsn and @vincbeck - can you please prioritize reviewing this? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [airflow] phanikumv merged pull request #32645: Refactor Sqlalchemy queries to 2.0 style (Part 6)

2023-07-20 Thread via GitHub
phanikumv merged PR #32645: URL: https://github.com/apache/airflow/pull/32645 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[airflow] branch main updated (15d42b4320 -> c7c0deecb4)

2023-07-20 Thread phanikumv
This is an automated email from the ASF dual-hosted git repository. phanikumv pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 15d42b4320 Reduce default for max TIs per query, enforce <= parallelism (#32572) add c7c0deecb4 Refactor Sqlal

[GitHub] [airflow] MatthieuBlais opened a new pull request, #32734: Add support for arrays of different data types in the Trigger Form UI

2023-07-20 Thread via GitHub
MatthieuBlais opened a new pull request, #32734: URL: https://github.com/apache/airflow/pull/32734 ### Description The new Trigger Form UI released in 2.6 is handy but doesn't support `array` params of data types other than string. For more flexibility, this PR proposes to generate a

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #32733: FIX: deferrable operators now use AioCredentials

2023-07-20 Thread via GitHub
boring-cyborg[bot] commented on PR #32733: URL: https://github.com/apache/airflow/pull/32733#issuecomment-1644867596 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] rishi-kulkarni opened a new pull request, #32733: FIX: deferrable operators now use AioCredentials

2023-07-20 Thread via GitHub
rishi-kulkarni opened a new pull request, #32733: URL: https://github.com/apache/airflow/pull/32733 Closes airflow.providers.amazong.aws.hooks.base_aws.BaseSessionFactory feeds synchronous credentials to aiobotocore when using `assume_role` #32732 -- This is an automated message f

[GitHub] [airflow] boring-cyborg[bot] commented on issue #32732: airflow.providers.amazong.aws.hooks.base_aws.BaseSessionFactory feeds synchronous credentials to aiobotocore when using `assume_role`

2023-07-20 Thread via GitHub
boring-cyborg[bot] commented on issue #32732: URL: https://github.com/apache/airflow/issues/32732#issuecomment-1644854034 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[GitHub] [airflow] rishi-kulkarni opened a new issue, #32732: airflow.providers.amazong.aws.hooks.base_aws.BaseSessionFactory feeds synchronous credentials to aiobotocore when using `assume_role`

2023-07-20 Thread via GitHub
rishi-kulkarni opened a new issue, #32732: URL: https://github.com/apache/airflow/issues/32732 ### Apache Airflow version 2.6.3 ### What happened Hi all, I'm having a bit of a problem with aiobotocore and the deferrable AWS Batch Operator. When deferrable is off, everyth

[GitHub] [airflow] github-actions[bot] closed pull request #30331: AIP-44 Migrate MetastoreBackend to Internal API

2023-07-20 Thread via GitHub
github-actions[bot] closed pull request #30331: AIP-44 Migrate MetastoreBackend to Internal API URL: https://github.com/apache/airflow/pull/30331 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] github-actions[bot] commented on pull request #28626: Introduce logging integration for Apache WebHDFS

2023-07-20 Thread via GitHub
github-actions[bot] commented on PR #28626: URL: https://github.com/apache/airflow/pull/28626#issuecomment-1644809436 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] github-actions[bot] commented on pull request #29536: AIP-44 Print usages of Session.query, Session.merge

2023-07-20 Thread via GitHub
github-actions[bot] commented on PR #29536: URL: https://github.com/apache/airflow/pull/29536#issuecomment-1644809409 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #32731: [Models] [Postgres] Check if the dynamically-added index is in the table schema before adding

2023-07-20 Thread via GitHub
boring-cyborg[bot] commented on PR #32731: URL: https://github.com/apache/airflow/pull/32731#issuecomment-1644783906 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] ohaibbq opened a new pull request, #32731: [Models] [Postgres] Check if the dynamically-added index is in the table schema before adding

2023-07-20 Thread via GitHub
ohaibbq opened a new pull request, #32731: URL: https://github.com/apache/airflow/pull/32731 We are utilizing the new `dag.test()` functionality with a Postgres-backed SQLAlchemy engine/session. We use the `airflow.utils.initdb()` and `airflow.utils.resetdb()` functions for test case setup/

[GitHub] [airflow] vijayasarathib opened a new pull request, #32730: Documentation Update to enhance readability

2023-07-20 Thread via GitHub
vijayasarathib opened a new pull request, #32730: URL: https://github.com/apache/airflow/pull/32730 readability change --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst

[GitHub] [airflow] xuganyu96 commented on pull request #32707: Catch arbitrary exception from run_job to prevent zombie scheduler

2023-07-20 Thread via GitHub
xuganyu96 commented on PR #32707: URL: https://github.com/apache/airflow/pull/32707#issuecomment-1644770990 @uranusjr Thank you for the feedback. I have added a test case that mocked `airflow.jobs.job.run_job` to simulate the situation where `run_job` throws some Exception, then tes

[GitHub] [airflow] jedcunningham commented on a diff in pull request #30259: Add a cache to Variable.get when called at dag parsing time

2023-07-20 Thread via GitHub
jedcunningham commented on code in PR #30259: URL: https://github.com/apache/airflow/pull/30259#discussion_r1270064267 ## docs/apache-airflow/best-practices.rst: ## @@ -200,10 +200,12 @@ Some cases of dynamic DAG generation are described in the :doc:`howto/dynamic-da Airflow V

[GitHub] [airflow] jedcunningham commented on pull request #30259: Add a cache to Variable.get when called at dag parsing time

2023-07-20 Thread via GitHub
jedcunningham commented on PR #30259: URL: https://github.com/apache/airflow/pull/30259#issuecomment-1644767177 > Or do you think we should go zero -> all in one step ? Yes, I feel if we do it it should be a shared cache across components so things operate off one TTL and are consiste

[GitHub] [airflow] uranusjr commented on a diff in pull request #32716: task instance listener: call on_task_instance_running after rendering templates

2023-07-20 Thread via GitHub
uranusjr commented on code in PR #32716: URL: https://github.com/apache/airflow/pull/32716#discussion_r1269967131 ## airflow/models/taskinstance.py: ## @@ -1601,7 +1596,8 @@ def _register_dataset_changes(self, *, session: Session) -> None: session=session,

[GitHub] [airflow] jedcunningham commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-20 Thread via GitHub
jedcunningham commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1269955978 ## airflow/configuration.py: ## @@ -1644,6 +1765,61 @@ def remove_all_read_configurations(self): for section in self.sections(): self.remo

[GitHub] [airflow] jedcunningham commented on a diff in pull request #32604: Allow configuration to be contributed by providers

2023-07-20 Thread via GitHub
jedcunningham commented on code in PR #32604: URL: https://github.com/apache/airflow/pull/32604#discussion_r1269951356 ## airflow/__main__.py: ## @@ -45,6 +45,13 @@ def main(): parser = cli_parser.get_parser() argcomplete.autocomplete(parser) args = parser.parse_a

[GitHub] [airflow] apallerlamudi commented on pull request #31885: Allow a destination folder to be provided

2023-07-20 Thread via GitHub
apallerlamudi commented on PR #31885: URL: https://github.com/apache/airflow/pull/31885#issuecomment-1644599892 @stiak Does this mean I need to wait for composer to upgrade their version of package to 10.4.0 for the apache-airflow-providers-google package? -- This is an automated message

[GitHub] [airflow] uranusjr commented on pull request #32707: Catch arbitrary exception from run_job to prevent zombie scheduler

2023-07-20 Thread via GitHub
uranusjr commented on PR #32707: URL: https://github.com/apache/airflow/pull/32707#issuecomment-1644581063 This needs a test case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[GitHub] [airflow] potiuk commented on pull request #32604: Allow configuration to be contributed by providers

2023-07-20 Thread via GitHub
potiuk commented on PR #32604: URL: https://github.com/apache/airflow/pull/32604#issuecomment-1644578831 All is good except ARM builds (which I am working on) . Definitely ready for review -- This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [airflow] uranusjr commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-20 Thread via GitHub
uranusjr commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1269942234 ## tests/auh/managers/test_base_auth_manager.py: ## @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

[GitHub] [airflow] uranusjr commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-20 Thread via GitHub
uranusjr commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1269942234 ## tests/auh/managers/test_base_auth_manager.py: ## @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

[GitHub] [airflow] vincbeck commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-20 Thread via GitHub
vincbeck commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1269940187 ## tests/auh/managers/test_base_auth_manager.py: ## @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

[GitHub] [airflow] uranusjr commented on pull request #32318: Order triggers by - TI priority_weight when assign unassigned triggers

2023-07-20 Thread via GitHub
uranusjr commented on PR #32318: URL: https://github.com/apache/airflow/pull/32318#issuecomment-1644574086 Do we still want/need to deal with NULL values in the ORDER BY clause? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] uranusjr commented on a diff in pull request #32525: Allow auth managers to override the security manager

2023-07-20 Thread via GitHub
uranusjr commented on code in PR #32525: URL: https://github.com/apache/airflow/pull/32525#discussion_r1269937325 ## tests/auh/managers/test_base_auth_manager.py: ## @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agr

[GitHub] [airflow] vincbeck commented on pull request #32697: Handle login by auth managers

2023-07-20 Thread via GitHub
vincbeck commented on PR #32697: URL: https://github.com/apache/airflow/pull/32697#issuecomment-1644561606 > #32525 Thanks @uranusjr for the review! If you have few seconds/minutes, could you review #32525? You added some comments there that I addressed. Thanks :) -- This is an aut

[GitHub] [airflow] uranusjr merged pull request #32572: Reduce config's default for max TIs per query & enforce <= parallelism

2023-07-20 Thread via GitHub
uranusjr merged PR #32572: URL: https://github.com/apache/airflow/pull/32572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: Reduce default for max TIs per query, enforce <= parallelism (#32572)

2023-07-20 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 15d42b4320 Reduce default for max TIs per query,

[GitHub] [airflow] uranusjr commented on a diff in pull request #32697: Handle login by auth managers

2023-07-20 Thread via GitHub
uranusjr commented on code in PR #32697: URL: https://github.com/apache/airflow/pull/32697#discussion_r1269926816 ## airflow/auth/managers/fab/fab_auth_manager.py: ## @@ -43,3 +46,16 @@ def get_user_name(self) -> str: def is_logged_in(self) -> bool: """Return wheth

[GitHub] [airflow] uranusjr commented on a diff in pull request #32697: Handle login by auth managers

2023-07-20 Thread via GitHub
uranusjr commented on code in PR #32697: URL: https://github.com/apache/airflow/pull/32697#discussion_r1269920366 ## airflow/auth/managers/fab/auth/anonymous_user.py: ## @@ -0,0 +1,48 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor lice

[GitHub] [airflow] dstandish merged pull request #32169: Add initial docs for setup / teardown

2023-07-20 Thread via GitHub
dstandish merged PR #32169: URL: https://github.com/apache/airflow/pull/32169 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[airflow] branch main updated: Add initial docs for setup / teardown (#32169)

2023-07-20 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new d70fecfaf6 Add initial docs for setup / teardown

[airflow] branch main updated: [bugfix] fix AWS triggers where deserialization would crash if region was not specified (#32729)

2023-07-20 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 75ed3bc3f8 [bugfix] fix AWS triggers where deseri

[GitHub] [airflow] vincbeck merged pull request #32729: [bugfix] fix AWS triggers where deserialization would crash if region was not specified

2023-07-20 Thread via GitHub
vincbeck merged PR #32729: URL: https://github.com/apache/airflow/pull/32729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] vandonr-amz commented on pull request #30259: Add a cache to Variable.get when called at dag parsing time

2023-07-20 Thread via GitHub
vandonr-amz commented on PR #30259: URL: https://github.com/apache/airflow/pull/30259#issuecomment-1644453632 I happy to see that this has not fallen to the bottom of the abyss yet :) I'll resolve conflicts & add code to include connections as well, in case something good happens. -- T

[GitHub] [airflow] eladkal commented on pull request #29602: Change config max tis per query

2023-07-20 Thread via GitHub
eladkal commented on PR #29602: URL: https://github.com/apache/airflow/pull/29602#issuecomment-168832 Closed in favor of https://github.com/apache/airflow/pull/32572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [airflow] eladkal closed pull request #29602: Change config max tis per query

2023-07-20 Thread via GitHub
eladkal closed pull request #29602: Change config max tis per query URL: https://github.com/apache/airflow/pull/29602 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

[GitHub] [airflow] dstandish commented on pull request #32249: Schedulers are issuing abrupt pod deletes when there is a delay in schedulers' heartbeat

2023-07-20 Thread via GitHub
dstandish commented on PR #32249: URL: https://github.com/apache/airflow/pull/32249#issuecomment-168058 Ah yes thanks @jedcunningham yeah tried this out locally (manually watching pods and messing with them). i guess the idea is to use the pod's deletion_timestamp metadata attr

[GitHub] [airflow] potiuk commented on issue #32271: Task instance state `upstream_failed` is missing from metrics counter `ti.finish...`

2023-07-20 Thread via GitHub
potiuk commented on issue #32271: URL: https://github.com/apache/airflow/issues/32271#issuecomment-1644433809 Yeah. I think more than anything we need clarification on what completed is. Maybe that should be you r PR @johnson-jay-l ? -- This is an automated message from the Apache Git Ser

[airflow] branch main updated (99b8a90346 -> 8e67546660)

2023-07-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 99b8a90346 Filtering and ordering results of DataprocListBatchesOperator (#32500) add 8e67546660 Fix DataformCrea

[GitHub] [airflow] potiuk merged pull request #32599: Fix DataformCreateWorkflowInvocationOperator system test

2023-07-20 Thread via GitHub
potiuk merged PR #32599: URL: https://github.com/apache/airflow/pull/32599 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29055: [AIP-51] Executors vending CLI commands

2023-07-20 Thread via GitHub
o-nikolas commented on code in PR #29055: URL: https://github.com/apache/airflow/pull/29055#discussion_r1269833340 ## airflow/executors/celery_kubernetes_executor.py: ## @@ -231,3 +231,7 @@ def send_callback(self, request: CallbackRequest) -> None: if not self.callback_

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29055: [AIP-51] Executors vending CLI commands

2023-07-20 Thread via GitHub
o-nikolas commented on code in PR #29055: URL: https://github.com/apache/airflow/pull/29055#discussion_r1269832478 ## airflow/executors/executor_loader.py: ## @@ -184,6 +192,31 @@ def validate_database_executor_compatibility(cls, executor: type[BaseExecutor]) if engine

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29055: [AIP-51] Executors vending CLI commands

2023-07-20 Thread via GitHub
o-nikolas commented on code in PR #29055: URL: https://github.com/apache/airflow/pull/29055#discussion_r1269832478 ## airflow/executors/executor_loader.py: ## @@ -184,6 +192,31 @@ def validate_database_executor_compatibility(cls, executor: type[BaseExecutor]) if engine

[GitHub] [airflow] potiuk commented on pull request #30259: Add a cache to Variable.get when called at dag parsing time

2023-07-20 Thread via GitHub
potiuk commented on PR #30259: URL: https://github.com/apache/airflow/pull/30259#issuecomment-1644404485 Yep. I also still think there is an advantage in having that feature in. We have it basically ready, we could ship it in 2.7.0. It's optional and I do not see how it could really cause a

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29055: [AIP-51] Executors vending CLI commands

2023-07-20 Thread via GitHub
o-nikolas commented on code in PR #29055: URL: https://github.com/apache/airflow/pull/29055#discussion_r1269825704 ## airflow/executors/executor_loader.py: ## @@ -184,6 +192,31 @@ def validate_database_executor_compatibility(cls, executor: type[BaseExecutor]) if engine

[GitHub] [airflow] dstandish commented on pull request #32249: Schedulers are issuing abrupt pod deletes when there is a delay in schedulers' heartbeat

2023-07-20 Thread via GitHub
dstandish commented on PR #32249: URL: https://github.com/apache/airflow/pull/32249#issuecomment-1644397309 hey @dirrao, thanks for the very helpful explanation and nice fix for this. i was hoping maybe we could figure out a way to not watch all scheduler's pods... but it seems there may n

[GitHub] [airflow] o-nikolas commented on a diff in pull request #29055: [AIP-51] Executors vending CLI commands

2023-07-20 Thread via GitHub
o-nikolas commented on code in PR #29055: URL: https://github.com/apache/airflow/pull/29055#discussion_r1269819841 ## airflow/executors/celery_kubernetes_executor.py: ## @@ -231,3 +231,7 @@ def send_callback(self, request: CallbackRequest) -> None: if not self.callback_

[GitHub] [airflow] potiuk closed issue #32499: Add filtering to DataprocListBatchesOperator

2023-07-20 Thread via GitHub
potiuk closed issue #32499: Add filtering to DataprocListBatchesOperator URL: https://github.com/apache/airflow/issues/32499 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] potiuk merged pull request #32500: Filtering and ordering results of DataprocListBatchesOperator

2023-07-20 Thread via GitHub
potiuk merged PR #32500: URL: https://github.com/apache/airflow/pull/32500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated (3c14753b03 -> 99b8a90346)

2023-07-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 3c14753b03 Fix BigQueryGetDataOperator where project_id is not being respected in deferrable mode (#32488) add 99

[airflow] branch main updated: Fix BigQueryGetDataOperator where project_id is not being respected in deferrable mode (#32488)

2023-07-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3c14753b03 Fix BigQueryGetDataOperator where projec

[GitHub] [airflow] potiuk merged pull request #32488: Fix BigQueryGetDataOperator where project_id is not being respected in deferrable mode

2023-07-20 Thread via GitHub
potiuk merged PR #32488: URL: https://github.com/apache/airflow/pull/32488 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] ephraimbuddy merged pull request #32687: Refactor setup/teardown ctx mgr to operate freely with other task definitions

2023-07-20 Thread via GitHub
ephraimbuddy merged PR #32687: URL: https://github.com/apache/airflow/pull/32687 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@air

[GitHub] [airflow] potiuk commented on issue #32379: Handling of the DagRun callback log should be visible on the UI somewhere

2023-07-20 Thread via GitHub
potiuk commented on issue #32379: URL: https://github.com/apache/airflow/issues/32379#issuecomment-1644375572 Sure - go for it. I think even if for some reason we won't want to merge it quickly, I think it is a good opportunity to do some second-pair-of-eyes-and hands in that part. Recently

[GitHub] [airflow] potiuk commented on pull request #32604: Allow configuration to be contributed by providers

2023-07-20 Thread via GitHub
potiuk commented on PR #32604: URL: https://github.com/apache/airflow/pull/32604#issuecomment-1644367502 I also did one more thing - since we are modifying "providers" documentation quite heavily here, I noticed we had a lot of old/missing/outdated/badly structured documentation (for exampl

[GitHub] [airflow] mobuchowski closed pull request #31293: openlineage, bigquery: add openlineage method support for BigQueryExecuteQueryOperator

2023-07-20 Thread via GitHub
mobuchowski closed pull request #31293: openlineage, bigquery: add openlineage method support for BigQueryExecuteQueryOperator URL: https://github.com/apache/airflow/pull/31293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] dstandish commented on a diff in pull request #32249: Schedulers are issuing abrupt pod deletes when there is a delay in schedulers' heartbeat

2023-07-20 Thread via GitHub
dstandish commented on code in PR #32249: URL: https://github.com/apache/airflow/pull/32249#discussion_r1269791216 ## airflow/executors/kubernetes_executor_utils.py: ## @@ -143,6 +146,13 @@ def _run( self.log.debug("Event: %s had an event of type %s", task.metadata

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32729: [bugfix] fix AWS triggers where deserialization would crash if region was not specified

2023-07-20 Thread via GitHub
vandonr-amz commented on code in PR #32729: URL: https://github.com/apache/airflow/pull/32729#discussion_r126975 ## airflow/providers/amazon/aws/triggers/rds.py: ## Review Comment: those triggers were merged yesterday, so no trouble in rearranging the arguments, it's n

[GitHub] [airflow] vandonr-amz opened a new pull request, #32729: [bugfix] fix AWS triggers where deserialization would crash if region was not specified

2023-07-20 Thread via GitHub
vandonr-amz opened a new pull request, #32729: URL: https://github.com/apache/airflow/pull/32729 the region is not serialized if it's None, so if a trigger has a non-optional region and we pass `None` to it manually, it doesn't get serialized, and the code crashes when trying to rebuild the

[GitHub] [airflow] phanikumv commented on a diff in pull request #32645: Refactor Sqlalchemy queries to 2.0 style (Part 6)

2023-07-20 Thread via GitHub
phanikumv commented on code in PR #32645: URL: https://github.com/apache/airflow/pull/32645#discussion_r1269777849 ## airflow/models/serialized_dag.py: ## @@ -144,23 +145,19 @@ def write_dag( # If Yes, does nothing # If No or the DAG does not exists, updates /

[GitHub] [airflow] potiuk closed issue #32429: SLA Logical Date is incorrect, SLA Miss Table missing entries for DAGs with catchup=true

2023-07-20 Thread via GitHub
potiuk closed issue #32429: SLA Logical Date is incorrect, SLA Miss Table missing entries for DAGs with catchup=true URL: https://github.com/apache/airflow/issues/32429 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[airflow] branch main updated: Add deferrable mode to ExternalTaskSensor (#29260)

2023-07-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new dda3dcdcfc Add deferrable mode to ExternalTaskSenso

[GitHub] [airflow] potiuk merged pull request #29260: Add deferrable mode to ExternalTaskSensor

2023-07-20 Thread via GitHub
potiuk merged PR #29260: URL: https://github.com/apache/airflow/pull/29260 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] vincbeck commented on pull request #32664: Add endpoint_url in test_connection

2023-07-20 Thread via GitHub
vincbeck commented on PR #32664: URL: https://github.com/apache/airflow/pull/32664#issuecomment-1644313378 > @vincbeck After making the mentioned change, the test for `test_hook_connection_test_failed` in `test/providers/amazon/aws/hooks/test_base_aws.py` started asserting an error due to

[GitHub] [airflow] potiuk merged pull request #32722: Fixup system test for DataprocSubmitJobOperator (Hadoop job)

2023-07-20 Thread via GitHub
potiuk merged PR #32722: URL: https://github.com/apache/airflow/pull/32722 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Fixup docstring for deprecated DataprocSubmitHiveJobOperator (#32723)

2023-07-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new e8287734cb Fixup docstring for deprecated Dataproc

[GitHub] [airflow] potiuk merged pull request #32723: Fixup system test for DataprocSubmitJobOperator (Hive job)

2023-07-20 Thread via GitHub
potiuk merged PR #32723: URL: https://github.com/apache/airflow/pull/32723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Fixup docs and optimize system test for DataprocSubmitJobOperator (Hadoop job) (#32722)

2023-07-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8b7ae76026 Fixup docs and optimize system test for

[GitHub] [airflow] phanikumv commented on a diff in pull request #32645: Refactor Sqlalchemy queries to 2.0 style (Part 6)

2023-07-20 Thread via GitHub
phanikumv commented on code in PR #32645: URL: https://github.com/apache/airflow/pull/32645#discussion_r1269760734 ## airflow/models/serialized_dag.py: ## @@ -379,7 +376,10 @@ def get_latest_version_hash_and_updated_datetime( :param session: ORM Session :return

[GitHub] [airflow] phanikumv commented on a diff in pull request #32645: Refactor Sqlalchemy queries to 2.0 style (Part 6)

2023-07-20 Thread via GitHub
phanikumv commented on code in PR #32645: URL: https://github.com/apache/airflow/pull/32645#discussion_r1269760272 ## airflow/models/serialized_dag.py: ## @@ -350,7 +347,7 @@ def get_max_last_updated_datetime(cls, session: Session = NEW_SESSION) -> dateti :param sess

[GitHub] [airflow] phanikumv commented on a diff in pull request #32645: Refactor Sqlalchemy queries to 2.0 style (Part 6)

2023-07-20 Thread via GitHub
phanikumv commented on code in PR #32645: URL: https://github.com/apache/airflow/pull/32645#discussion_r1269759624 ## airflow/models/serialized_dag.py: ## @@ -275,7 +272,7 @@ def has_dag(cls, dag_id: str, session: Session = NEW_SESSION) -> bool: :param dag_id: the DAG

[GitHub] [airflow] boring-cyborg[bot] commented on issue #32727: airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator not filter file

2023-07-20 Thread via GitHub
boring-cyborg[bot] commented on issue #32727: URL: https://github.com/apache/airflow/issues/32727#issuecomment-1644289452 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[GitHub] [airflow] yvesmarinho opened a new issue, #32727: airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator not filter file

2023-07-20 Thread via GitHub
yvesmarinho opened a new issue, #32727: URL: https://github.com/apache/airflow/issues/32727 ### Apache Airflow version 2.6.3 ### What happened We were unable to filter the file using part of the name and extension with the match_glob option. I'm using the apache-airfl

[GitHub] [airflow] ieunea1128 commented on pull request #32664: Add endpoint_url in test_connection

2023-07-20 Thread via GitHub
ieunea1128 commented on PR #32664: URL: https://github.com/apache/airflow/pull/32664#issuecomment-1644273768 @vincbeck After making the mentioned change, the test for ```test_hook_connection_test_failed``` in ```test/providers/amazon/aws/hooks/test_base_aws.py``` started asserting an error

[GitHub] [airflow] vincbeck commented on a diff in pull request #32664: Add endpoint_url in test_connection

2023-07-20 Thread via GitHub
vincbeck commented on code in PR #32664: URL: https://github.com/apache/airflow/pull/32664#discussion_r1269706866 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -796,7 +796,11 @@ def test_connection(self): """ try: session = self.get_sess

[GitHub] [airflow] potiuk commented on issue #32721: create-custom-decorator.rst does not describe how/where to setup get_provider_info

2023-07-20 Thread via GitHub
potiuk commented on issue #32721: URL: https://github.com/apache/airflow/issues/32721#issuecomment-1644235503 https://airflow.apache.org/docs/apache-airflow-providers/index.html#how-to-create-your-own-provider -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] brokenjacobs commented on issue #32703: Helm chart 1.10.0 introduces a bug with dagProcess and log groomer sidecar, webserver config invalid

2023-07-20 Thread via GitHub
brokenjacobs commented on issue #32703: URL: https://github.com/apache/airflow/issues/32703#issuecomment-1644228870 Beat me to it, Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [airflow] brokenjacobs closed issue #32703: Helm chart 1.10.0 introduces a bug with dagProcess and log groomer sidecar, webserver config invalid

2023-07-20 Thread via GitHub
brokenjacobs closed issue #32703: Helm chart 1.10.0 introduces a bug with dagProcess and log groomer sidecar, webserver config invalid URL: https://github.com/apache/airflow/issues/32703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[GitHub] [airflow] potiuk closed issue #32720: DAG not visible in Airflow web interface when there are no syntax errors

2023-07-20 Thread via GitHub
potiuk closed issue #32720: DAG not visible in Airflow web interface when there are no syntax errors URL: https://github.com/apache/airflow/issues/32720 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] potiuk commented on pull request #32644: Fix dagProcessor not including webserver-config volume

2023-07-20 Thread via GitHub
potiuk commented on PR #32644: URL: https://github.com/apache/airflow/pull/32644#issuecomment-1644211561 Can you please add a test for that ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] ieunea1128 commented on a diff in pull request #32664: Add endpoint_url in test_connection

2023-07-20 Thread via GitHub
ieunea1128 commented on code in PR #32664: URL: https://github.com/apache/airflow/pull/32664#discussion_r1269684728 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -796,7 +796,11 @@ def test_connection(self): """ try: session = self.get_se

[GitHub] [airflow] ieunea1128 commented on a diff in pull request #32664: Add endpoint_url in test_connection

2023-07-20 Thread via GitHub
ieunea1128 commented on code in PR #32664: URL: https://github.com/apache/airflow/pull/32664#discussion_r1269677374 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -796,7 +796,11 @@ def test_connection(self): """ try: session = self.get_se

[GitHub] [airflow] ieunea1128 commented on a diff in pull request #32664: Add endpoint_url in test_connection

2023-07-20 Thread via GitHub
ieunea1128 commented on code in PR #32664: URL: https://github.com/apache/airflow/pull/32664#discussion_r1269677374 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -796,7 +796,11 @@ def test_connection(self): """ try: session = self.get_se

[GitHub] [airflow] mobuchowski closed pull request #31293: openlineage, bigquery: add openlineage method support for BigQueryExecuteQueryOperator

2023-07-20 Thread via GitHub
mobuchowski closed pull request #31293: openlineage, bigquery: add openlineage method support for BigQueryExecuteQueryOperator URL: https://github.com/apache/airflow/pull/31293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] mikaeld opened a new pull request, #32724: feat(chart): Add existing secret support for PGBouncer metrics exporter.

2023-07-20 Thread via GitHub
mikaeld opened a new pull request, #32724: URL: https://github.com/apache/airflow/pull/32724 This PR adds existing secret support for PGBouncer's metrics exporter sidecar container. While PGBouncer's container supports existing secrets for its configuration through `configSecretName`, the m

[GitHub] [airflow] moiseenkov opened a new pull request, #32723: Fixup system test for DataprocSubmitJobOperator (Hive job)

2023-07-20 Thread via GitHub
moiseenkov opened a new pull request, #32723: URL: https://github.com/apache/airflow/pull/32723 Fix for system test for `DataprocSubmitJobOperator` (Hive job): 1. Refactored system test (AIP-47). 2. Update docstrings for generating more explicit documentation with a deprecation message

  1   2   >