[PR] [WIP] feat: added get_openlineage_facets_on_start function in operator definition [airflow]

2024-06-18 Thread via GitHub
rahul-madaan opened a new pull request, #40290: URL: https://github.com/apache/airflow/pull/40290 Added OpenLineage support for AzureBlobStorageToGCSOperator the operator transfers single file from Azure Blob Storage to Google cloud storage. -- This is an automated message from the

Re: [PR] [WIP] feat: added get_openlineage_facets_on_start function in operator definition [airflow]

2024-06-18 Thread via GitHub
boring-cyborg[bot] commented on PR #40290: URL: https://github.com/apache/airflow/pull/40290#issuecomment-2175413384 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'

Re: [I] List Dag Run broken link to dag_run [airflow]

2024-06-18 Thread via GitHub
raphaelauv commented on issue #40214: URL: https://github.com/apache/airflow/issues/40214#issuecomment-2175472851 you must set the filter first ![image](https://github.com/apache/airflow/assets/10202690/ae43ea28-957b-426d-b74c-f3dbd5c7df3c) I have the problem on google chrome

Re: [I] Number of queries to Airflow database in "DAG File Processing Stats" [airflow]

2024-06-18 Thread via GitHub
raphaelauv commented on issue #40282: URL: https://github.com/apache/airflow/issues/40282#issuecomment-2175479486 AIRFLOW__SECRETS__USE_CACHE is by default at false and yes this feature will no more be usefull when DAG_PARSING_METADATABASE_CALL_FORBIDEN is set to True -- This is an automa

Re: [I] Inconsistency in S3 Transfer Operators [airflow]

2024-06-18 Thread via GitHub
bricefotzo commented on issue #37213: URL: https://github.com/apache/airflow/issues/37213#issuecomment-2175553520 @jroachgolf84 I have added those changes https://github.com/apache/airflow/commit/3da0dcff0779079b3347b685c53e4e7c153ccbb4 -- This is an automated message from the Apache Git

(airflow) branch main updated: Bump ws from 7.5.5 to 7.5.10 in /airflow/www (#40288)

2024-06-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 9208e1174a Bump ws from 7.5.5 to 7.5.10 in /airflow

(airflow) branch dependabot/npm_and_yarn/airflow/www/ws-7.5.10 deleted (was 8275d07563)

2024-06-18 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/airflow/www/ws-7.5.10 in repository https://gitbox.apache.org/repos/asf/airflow.git was 8275d07563 Bump ws from 7.5.5 to 7.5.10 in /airflow/www The revisions th

Re: [I] Number of queries to Airflow database in "DAG File Processing Stats" [airflow]

2024-06-18 Thread via GitHub
VladaZakharova commented on issue #40282: URL: https://github.com/apache/airflow/issues/40282#issuecomment-2175592531 @eladkal @potiuk @kaxil Let's continue here our discussion about adding more logs in our output @raphaelauv Thank you for your ideas, but I think this can be not direct

Re: [PR] Remove flaky test case of deferred instance state for served logs [airflow]

2024-06-18 Thread via GitHub
potiuk commented on PR #40257: URL: https://github.com/apache/airflow/pull/40257#issuecomment-2175577488 > In addition to the comment in the code, could you open an Issue with details so someone might pick it up? Yeah. that's what we usually do. Create an issue and link to it from the

Re: [I] *** Could not read served logs: [Errno -3] Temporary failure in name resolution [airflow]

2024-06-18 Thread via GitHub
potiuk closed issue #40289: *** Could not read served logs: [Errno -3] Temporary failure in name resolution URL: https://github.com/apache/airflow/issues/40289 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Bump ws from 7.5.5 to 7.5.10 in /airflow/www [airflow]

2024-06-18 Thread via GitHub
potiuk merged PR #40288: URL: https://github.com/apache/airflow/pull/40288 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [I] dag callbacks not called on SSHOperator (base operator?) [airflow]

2024-06-18 Thread via GitHub
trlopes1974 commented on issue #40119: URL: https://github.com/apache/airflow/issues/40119#issuecomment-2175589052 @vatsrahul1001 where you able to try this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [I] Tasks with 'none_failed_min_one_success' trigger_rule skipping before Dynamic Task Group is fully expanded [airflow]

2024-06-18 Thread via GitHub
le-chartreux commented on issue #39801: URL: https://github.com/apache/airflow/issues/39801#issuecomment-2175651390 Hello, I am also facing this issue, it will be awesome if someone can find a solution (or at least a workaround). Best regards and thank you for developing Airflow! --

(airflow) branch constraints-main updated: Updating constraints. Github run id:9562176812

2024-06-18 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 7b43fbfffd Updating con

Re: [PR] [WIP] feat: added get_openlineage_facets_on_start function in operator definition [airflow]

2024-06-18 Thread via GitHub
kacpermuda commented on code in PR #40290: URL: https://github.com/apache/airflow/pull/40290#discussion_r1644182454 ## airflow/providers/google/cloud/transfers/azure_blob_to_gcs.py: ## @@ -122,3 +122,21 @@ def execute(self, context: Context) -> str: self.bucket_

Re: [PR] Partially resolve builtin variable shadowing (`A001`) [airflow]

2024-06-18 Thread via GitHub
uranusjr commented on code in PR #39278: URL: https://github.com/apache/airflow/pull/39278#discussion_r1644196493 ## dev/breeze/src/airflow_breeze/commands/developer_commands.py: ## @@ -958,7 +958,7 @@ def down(preserve_volumes: bool, cleanup_mypy_cache: bool, project_name: str

Re: [PR] Partially resolve builtin variable shadowing (`A001`) [airflow]

2024-06-18 Thread via GitHub
Taragolis commented on code in PR #39278: URL: https://github.com/apache/airflow/pull/39278#discussion_r1644212475 ## dev/breeze/src/airflow_breeze/commands/developer_commands.py: ## @@ -958,7 +958,7 @@ def down(preserve_volumes: bool, cleanup_mypy_cache: bool, project_name: st

Re: [PR] Partially resolve builtin variable shadowing (`A001`) [airflow]

2024-06-18 Thread via GitHub
potiuk commented on code in PR #39278: URL: https://github.com/apache/airflow/pull/39278#discussion_r1644210053 ## dev/breeze/src/airflow_breeze/commands/developer_commands.py: ## @@ -958,7 +958,7 @@ def down(preserve_volumes: bool, cleanup_mypy_cache: bool, project_name: str):

[I] execution_date format [airflow]

2024-06-18 Thread via GitHub
HumanIearning opened a new issue, #40292: URL: https://github.com/apache/airflow/issues/40292 ### Apache Airflow version 2.9.2 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? I don't know exactly it's bug, but I think i

Re: [PR] Partially resolve builtin variable shadowing (`A001`) [airflow]

2024-06-18 Thread via GitHub
Taragolis commented on code in PR #39278: URL: https://github.com/apache/airflow/pull/39278#discussion_r1644213182 ## dev/breeze/src/airflow_breeze/commands/developer_commands.py: ## @@ -958,7 +958,7 @@ def down(preserve_volumes: bool, cleanup_mypy_cache: bool, project_name: st

Re: [PR] Partially resolve builtin variable shadowing (`A001`) [airflow]

2024-06-18 Thread via GitHub
potiuk commented on code in PR #39278: URL: https://github.com/apache/airflow/pull/39278#discussion_r1644218293 ## dev/breeze/src/airflow_breeze/commands/developer_commands.py: ## @@ -958,7 +958,7 @@ def down(preserve_volumes: bool, cleanup_mypy_cache: bool, project_name: str):

[PR] Improve scheduler loop by reducing repetative `TI.are_dependencies_met` [airflow]

2024-06-18 Thread via GitHub
ephraimbuddy opened a new pull request, #40293: URL: https://github.com/apache/airflow/pull/40293 TI.are_dependencies_met run over and over even when no changes have happened that would allow it to pass. This causes the scheduler loop to get slower and slower as more blocked TIs pile up.

Re: [I] dag callbacks not called on SSHOperator (base operator?) [airflow]

2024-06-18 Thread via GitHub
vatsrahul1001 commented on issue #40119: URL: https://github.com/apache/airflow/issues/40119#issuecomment-2175890646 @trlopes1974 I am able to repo this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Support disabling helm hooks on extraConfigMaps and extraSecrets [airflow]

2024-06-18 Thread via GitHub
boring-cyborg[bot] commented on PR #40294: URL: https://github.com/apache/airflow/pull/40294#issuecomment-2175892375 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'

[PR] Support disabling helm hooks on extraConfigMaps and extraSecrets [airflow]

2024-06-18 Thread via GitHub
ItaiMeushar opened a new pull request, #40294: URL: https://github.com/apache/airflow/pull/40294 # Description As mentioned in #17447: Currently it is not possible to avoid the default Helm annotations on resources such as ```extraConfigMaps``` and ```extraSecrets``` and there's n

Re: [I] List Dag Run broken link to dag_run [airflow]

2024-06-18 Thread via GitHub
vatsrahul1001 commented on issue #40214: URL: https://github.com/apache/airflow/issues/40214#issuecomment-2175893188 @raphaelauv I am able to repo thus -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[PR] Fix dataform and datastore system tests [airflow]

2024-06-18 Thread via GitHub
VladaZakharova opened a new pull request, #40295: URL: https://github.com/apache/airflow/pull/40295 This PR fixes typos in dataform and datastore system tests --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.c

Re: [PR] Add CloudRunServiceHook and operator (create) [airflow]

2024-06-18 Thread via GitHub
jx2lee commented on code in PR #40008: URL: https://github.com/apache/airflow/pull/40008#discussion_r1644349286 ## airflow/providers/google/cloud/operators/cloud_run.py: ## @@ -353,3 +353,57 @@ def _wait_for_operation(self, operation: operation.Operation): except Excep

[PR] refactor: Make sure xcoms work correctly with MSGraphAsyncOperator with concurrent workers [airflow]

2024-06-18 Thread via GitHub
dabla opened a new pull request, #40297: URL: https://github.com/apache/airflow/pull/40297 This pull request makes sure xcoms work correctly in a multi-threaded environment (e.g. when applying dynamic task mapping with concurrent workers) by taking the map_index into account

Re: [PR] Fix dataform and datastore system tests [airflow]

2024-06-18 Thread via GitHub
eladkal merged PR #40295: URL: https://github.com/apache/airflow/pull/40295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

(airflow) branch main updated: Fix dataform and datastore system tests (#40295)

2024-06-18 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3e4ed12907 Fix dataform and datastore system tests

Re: [PR] refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers [airflow]

2024-06-18 Thread via GitHub
potiuk commented on PR #40297: URL: https://github.com/apache/airflow/pull/40297#issuecomment-2176003482 Is it possible to add unit test for that to avoid regressions ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers [airflow]

2024-06-18 Thread via GitHub
dabla commented on PR #40297: URL: https://github.com/apache/airflow/pull/40297#issuecomment-2176012376 > Is it possible to add unit test for that to avoid regressions ? That would mean a unit test using multiple threads as I need concurrency to reproduce it. Are there any examples o

Re: [PR] refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers [airflow]

2024-06-18 Thread via GitHub
potiuk commented on PR #40297: URL: https://github.com/apache/airflow/pull/40297#issuecomment-2176020553 > That would mean a unit test using multiple threads as I need concurrency to reproduce it. Are there any examples of unit tests using multiple threads? When I search in `tests` th

Re: [I] Tasks with 'none_failed_min_one_success' trigger_rule skipping before Dynamic Task Group is fully expanded [airflow]

2024-06-18 Thread via GitHub
le-chartreux commented on issue #39801: URL: https://github.com/apache/airflow/issues/39801#issuecomment-2176028789 Awesome, thank you @mateuslatrova! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Change the pre-commit installing command to use pipx instead of pyenv [airflow]

2024-06-18 Thread via GitHub
gomtangman commented on code in PR #40266: URL: https://github.com/apache/airflow/pull/40266#discussion_r1644431290 ## contributing-docs/03_contributors_quick_start.rst: ## @@ -466,6 +466,11 @@ on macOS, install via 2. Installing required Python packages +.. note:: + If py

Re: [PR] refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers [airflow]

2024-06-18 Thread via GitHub
dabla commented on PR #40297: URL: https://github.com/apache/airflow/pull/40297#issuecomment-2176062512 > > That would mean a unit test using multiple threads as I need concurrency to reproduce it. Are there any examples of unit tests using multiple threads? > > When I search in `test

Re: [PR] refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers [airflow]

2024-06-18 Thread via GitHub
dabla commented on PR #40297: URL: https://github.com/apache/airflow/pull/40297#issuecomment-2176072150 The problem with Python is that the basic threading doesn't do real multithreading, so I don't know if I will be able to reproduce it as we did encounter the issues with multiple workers

Re: [PR] Implement AND logic checkbox for filtering DAGs [airflow]

2024-06-18 Thread via GitHub
TiDeane commented on PR #39564: URL: https://github.com/apache/airflow/pull/39564#issuecomment-2176174129 I already replied in the issue, but figured it might be good to reply here too for consistency: > @Tristan-Zaleski That's strange, I've just rebased with upstream/main and it see

Re: [PR] refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers [airflow]

2024-06-18 Thread via GitHub
potiuk commented on PR #40297: URL: https://github.com/apache/airflow/pull/40297#issuecomment-2176188833 > The problem with Python is that the basic threading doesn't do real multithreading, so I don't know if I will be able to reproduce it as we did encounter the issues with multiple worke

Re: [PR] Resolve elastic search deprecations in tests [airflow]

2024-06-18 Thread via GitHub
potiuk merged PR #40299: URL: https://github.com/apache/airflow/pull/40299 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Resolve elastic search deprecations in tests (#40299)

2024-06-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new aa10d1ee24 Resolve elastic search deprecations in t

(airflow) branch main updated: refactor: Make sure xcoms work correctly in multi-threaded environment by taking the map_index into account (#40297)

2024-06-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 9b394a7229 refactor: Make sure xcoms work correctly

Re: [PR] refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers [airflow]

2024-06-18 Thread via GitHub
potiuk merged PR #40297: URL: https://github.com/apache/airflow/pull/40297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [PR] Change the pre-commit installing command to use pipx instead of pyenv [airflow]

2024-06-18 Thread via GitHub
potiuk merged PR #40266: URL: https://github.com/apache/airflow/pull/40266 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Change the pre-commit installing command to use pipx instead of pyenv (#40266)

2024-06-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 1f7be6f63e Change the pre-commit installing command

Re: [PR] Change the pre-commit installing command to use pipx instead of pyenv [airflow]

2024-06-18 Thread via GitHub
boring-cyborg[bot] commented on PR #40266: URL: https://github.com/apache/airflow/pull/40266#issuecomment-2176220781 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.

Re: [PR] Update backfill tests to use executor loader [airflow]

2024-06-18 Thread via GitHub
potiuk commented on code in PR #40285: URL: https://github.com/apache/airflow/pull/40285#discussion_r1644554626 ## tests/jobs/test_backfill_job.py: ## @@ -763,15 +778,13 @@ def test_backfill_skip_active_scheduled_dagrun(self, dag_maker, caplog): # since DAG backfill is

Re: [PR] Update backfill tests to use executor loader [airflow]

2024-06-18 Thread via GitHub
potiuk commented on code in PR #40285: URL: https://github.com/apache/airflow/pull/40285#discussion_r1644555723 ## tests/jobs/test_backfill_job.py: ## @@ -79,11 +80,38 @@ def dag_bag(): return DagBag(include_examples=True) -# Patch the MockExecutor into the dict of know

[PR] Revert "refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers" [airflow]

2024-06-18 Thread via GitHub
dabla opened a new pull request, #40300: URL: https://github.com/apache/airflow/pull/40300 Reverts apache/airflow#40297 Still one more fix needed sorry @potiuk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Update backfill tests to use executor loader [airflow]

2024-06-18 Thread via GitHub
potiuk commented on code in PR #40285: URL: https://github.com/apache/airflow/pull/40285#discussion_r1644558818 ## tests/jobs/test_backfill_job.py: ## @@ -139,14 +167,14 @@ def test_unfinished_dag_runs_set_to_failed(self, dag_maker): assert State.FAILED == dag_run.st

[PR] refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers [airflow]

2024-06-18 Thread via GitHub
dabla opened a new pull request, #40301: URL: https://github.com/apache/airflow/pull/40301 This pull request makes sure xcoms work correctly in a multi-threaded environment (e.g. when applying dynamic task mapping with concurrent workers) by taking the map_index into

Re: [PR] add git-sync-ssh secret template [airflow]

2024-06-18 Thread via GitHub
romsharon98 commented on code in PR #39936: URL: https://github.com/apache/airflow/pull/39936#discussion_r1644587717 ## chart/templates/secrets/git-ssh-key-secret.yaml: ## @@ -0,0 +1,34 @@ +{{/* + Licensed to the Apache Software Foundation (ASF) under one + or more contributor l

[I] Handle toctree contains reference to nonexisting document '_api/` errors better when building docs [airflow]

2024-06-18 Thread via GitHub
potiuk opened a new issue, #40302: URL: https://github.com/apache/airflow/issues/40302 ### Body When building documentation (`breeze build-docs`) and you press ctrl-c in-between, the autoapi docs might remain empty and they cause ``` toctree contains reference to nonexisting

Re: [I] Deferrable HttpSensor does not move to Triggerer when using `response_check` [airflow]

2024-06-18 Thread via GitHub
josh-fell commented on issue #40209: URL: https://github.com/apache/airflow/issues/40209#issuecomment-2176299198 Probably what the solution _should_ be. May need to confirm with `HttpOperator` too: https://github.com/apache/airflow/discussions/39597#discussioncomment-9758322 -- Thi

(airflow) branch constraints-main updated: Updating constraints. Github run id:9566765894

2024-06-18 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new f1c6167901 Updating con

Re: [I] execution_date format [airflow]

2024-06-18 Thread via GitHub
raphaelauv commented on issue #40292: URL: https://github.com/apache/airflow/issues/40292#issuecomment-2176325754 yes , you can convert it by ```python from pendulum import DateTime def cast_pendulum_date(date: DateTime) -> str: return date.format("-MM-DD HH:mm:ss

Re: [PR] Resolve appflow deprecations in tests [airflow]

2024-06-18 Thread via GitHub
vincbeck merged PR #40298: URL: https://github.com/apache/airflow/pull/40298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

(airflow) branch main updated (1f7be6f63e -> f5d2745909)

2024-06-18 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 1f7be6f63e Change the pre-commit installing command to use pipx instead of pyenv (#40266) add f5d2745909 Resolv

Re: [PR] Add Amazon Comprehend Document Classifier [airflow]

2024-06-18 Thread via GitHub
vincbeck commented on code in PR #40287: URL: https://github.com/apache/airflow/pull/40287#discussion_r1644637473 ## tests/system/providers/amazon/aws/example_comprehend_document_classifier.py: ## @@ -0,0 +1,234 @@ +# Licensed to the Apache Software Foundation (ASF) under one +#

Re: [PR] Revert "refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers" [airflow]

2024-06-18 Thread via GitHub
potiuk merged PR #40300: URL: https://github.com/apache/airflow/pull/40300 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated (f5d2745909 -> e7052d4244)

2024-06-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from f5d2745909 Resolve appflow deprecations in tests (#40298) add e7052d4244 Revert "refactor: Make sure xcoms work co

Re: [I] Tasks with 'none_failed_min_one_success' trigger_rule skipping before Dynamic Task Group is fully expanded [airflow]

2024-06-18 Thread via GitHub
le-chartreux commented on issue #39801: URL: https://github.com/apache/airflow/issues/39801#issuecomment-2176415067 Hello Airflow Community, While waiting for a resolution to the trigger rule issue, I wanted to share a workaround that I've implemented. While it may not address ever

Re: [PR] refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers [airflow]

2024-06-18 Thread via GitHub
dabla commented on PR #40301: URL: https://github.com/apache/airflow/pull/40301#issuecomment-2176436702 @potiuk Will try to write a unit test which tests the case with map_index, but not in a multi-threaded way but at least then this case will be documented and verified for regression throu

[PR] fix a vulnerability in protobuf for pgbouncer_exporter [airflow]

2024-06-18 Thread via GitHub
andrew-stein-sp opened a new pull request, #40303: URL: https://github.com/apache/airflow/pull/40303 --- pgbouncer_exporter 0.17.0 has been released and provides several updates to the golang dependencies. specifically though, this version updates p

Re: [PR] fix a vulnerability in protobuf for pgbouncer_exporter [airflow]

2024-06-18 Thread via GitHub
boring-cyborg[bot] commented on PR #40303: URL: https://github.com/apache/airflow/pull/40303#issuecomment-2176465547 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'

[PR] AIP 64: Add TI try history to Task Instance Details, Logs, and Gantt chart [airflow]

2024-06-18 Thread via GitHub
bbovenzi opened a new pull request, #40304: URL: https://github.com/apache/airflow/pull/40304 Use the TI History endpoint to expand upon the "log attempts" buttons we had for task logs. First, show the state of each try next to the number. https://github.com/apache/airflow/assets/4

Re: [PR] fix a vulnerability in protobuf for pgbouncer_exporter [airflow]

2024-06-18 Thread via GitHub
andrew-stein-sp commented on PR #40303: URL: https://github.com/apache/airflow/pull/40303#issuecomment-2176491596 @potiuk can this image be built and pushed to dockerhub without requiring an update to the values.yaml's in this PR, or is that a req for merge? -- This is an automated messag

Re: [PR] Remove "experimental" banner for OTel Metrics [airflow]

2024-06-18 Thread via GitHub
ferruzzi commented on PR #40286: URL: https://github.com/apache/airflow/pull/40286#issuecomment-2176491601 I'll get the voting thread going and add the news fragment, thanks folks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[I] scheduler_job_runner starving log additional information [airflow]

2024-06-18 Thread via GitHub
caichenghao1991 opened a new issue, #40305: URL: https://github.com/apache/airflow/issues/40305 ### Description find this log message [2024-06-18T12:18:13.280+] {scheduler_job_runner.py:442} INFO - Not scheduling since there are 0 open slots in pool default_pool A sim

Re: [I] scheduler_job_runner starving log additional information [airflow]

2024-06-18 Thread via GitHub
boring-cyborg[bot] commented on issue #40305: URL: https://github.com/apache/airflow/issues/40305#issuecomment-2176535718 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [PR] add git-sync-ssh secret template [airflow]

2024-06-18 Thread via GitHub
jedcunningham commented on code in PR #39936: URL: https://github.com/apache/airflow/pull/39936#discussion_r1644771857 ## chart/values.schema.json: ## @@ -7409,6 +7409,14 @@ ], "default": null },

Re: [PR] Update backfill tests to use executor loader [airflow]

2024-06-18 Thread via GitHub
o-nikolas commented on code in PR #40285: URL: https://github.com/apache/airflow/pull/40285#discussion_r1644786444 ## tests/jobs/test_backfill_job.py: ## @@ -694,20 +712,21 @@ def test_backfill_run_rescheduled(self, dag_maker): ti.refresh_from_db() ti.set_state

Re: [PR] Update backfill tests to use executor loader [airflow]

2024-06-18 Thread via GitHub
o-nikolas commented on code in PR #40285: URL: https://github.com/apache/airflow/pull/40285#discussion_r1644788180 ## tests/jobs/test_backfill_job.py: ## @@ -139,14 +167,14 @@ def test_unfinished_dag_runs_set_to_failed(self, dag_maker): assert State.FAILED == dag_run

Re: [PR] Update backfill tests to use executor loader [airflow]

2024-06-18 Thread via GitHub
o-nikolas commented on code in PR #40285: URL: https://github.com/apache/airflow/pull/40285#discussion_r1644790025 ## tests/jobs/test_backfill_job.py: ## @@ -79,11 +80,38 @@ def dag_bag(): return DagBag(include_examples=True) -# Patch the MockExecutor into the dict of k

Re: [PR] add git-sync-ssh secret template [airflow]

2024-06-18 Thread via GitHub
romsharon98 commented on code in PR #39936: URL: https://github.com/apache/airflow/pull/39936#discussion_r1644789743 ## chart/values.yaml: ## @@ -2505,6 +2505,9 @@ dags: # and specify the name of the secret below # sshKeySecret: airflow-ssh-secret # +# Or set

Re: [PR] Update backfill tests to use executor loader [airflow]

2024-06-18 Thread via GitHub
o-nikolas commented on code in PR #40285: URL: https://github.com/apache/airflow/pull/40285#discussion_r1644791977 ## tests/jobs/test_backfill_job.py: ## @@ -350,9 +377,8 @@ def test_backfill_respect_max_active_tis_per_dag_limit(self, mock_log, dag_maker ) dag

Re: [I] OpenLineage defaults in airflow.cfg incorrect [airflow]

2024-06-18 Thread via GitHub
boring-cyborg[bot] commented on issue #40306: URL: https://github.com/apache/airflow/issues/40306#issuecomment-2176593808 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[I] OpenLineage defaults in airflow.cfg incorrect [airflow]

2024-06-18 Thread via GitHub
merobi-hub opened a new issue, #40306: URL: https://github.com/apache/airflow/issues/40306 ### Apache Airflow Provider(s) openlineage ### Versions of Apache Airflow Providers apache-airflow-providers-openlineage==1.5.0 ### Apache Airflow version 2.9.2

Re: [I] Handle toctree contains reference to nonexisting document '_api/` errors better when building docs [airflow]

2024-06-18 Thread via GitHub
poorvirohidekar commented on issue #40302: URL: https://github.com/apache/airflow/issues/40302#issuecomment-2176601980 @potiuk I would like to try this. Could you please assign this to me? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Update backfill tests to use executor loader [airflow]

2024-06-18 Thread via GitHub
o-nikolas commented on code in PR #40285: URL: https://github.com/apache/airflow/pull/40285#discussion_r1644815890 ## tests/jobs/test_backfill_job.py: ## @@ -763,15 +778,13 @@ def test_backfill_skip_active_scheduled_dagrun(self, dag_maker, caplog): # since DAG backfill

Re: [PR] Update backfill tests to use executor loader [airflow]

2024-06-18 Thread via GitHub
vincbeck commented on code in PR #40285: URL: https://github.com/apache/airflow/pull/40285#discussion_r1644818369 ## tests/jobs/test_backfill_job.py: ## @@ -694,20 +712,21 @@ def test_backfill_run_rescheduled(self, dag_maker): ti.refresh_from_db() ti.set_state(

Re: [PR] AIP-61 - Scheduler job: main loop and event processing for multi executors [airflow]

2024-06-18 Thread via GitHub
o-nikolas commented on PR #40017: URL: https://github.com/apache/airflow/pull/40017#issuecomment-2176612917 @potiuk @uranusjr any time to have a/another look? @dstandish maybe? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [I] scheduler_job_runner starving log additional information [airflow]

2024-06-18 Thread via GitHub
caichenghao1991 closed issue #40305: scheduler_job_runner starving log additional information URL: https://github.com/apache/airflow/issues/40305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Remove unreachable return of jenkins_job_trigger [airflow]

2024-06-18 Thread via GitHub
pankajkoti commented on code in PR #40233: URL: https://github.com/apache/airflow/pull/40233#discussion_r1644843018 ## airflow/providers/jenkins/operators/jenkins_job_trigger.py: ## @@ -71,7 +71,6 @@ def jenkins_request_with_headers(jenkins_server: Jenkins, req: Request) -> Jen

Re: [PR] add git-sync-ssh secret template [airflow]

2024-06-18 Thread via GitHub
jedcunningham commented on code in PR #39936: URL: https://github.com/apache/airflow/pull/39936#discussion_r1644844509 ## chart/values.yaml: ## @@ -2505,6 +2505,9 @@ dags: # and specify the name of the secret below # sshKeySecret: airflow-ssh-secret # +# Or se

Re: [I] Handle toctree contains reference to nonexisting document '_api/` errors better when building docs [airflow]

2024-06-18 Thread via GitHub
amoghrajesh commented on issue #40302: URL: https://github.com/apache/airflow/issues/40302#issuecomment-2176651166 Feel free to work on this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] revamp was it killed externally question [airflow]

2024-06-18 Thread via GitHub
pankajkoti commented on code in PR #40141: URL: https://github.com/apache/airflow/pull/40141#discussion_r1644860673 ## airflow/jobs/backfill_job_runner.py: ## @@ -313,9 +313,14 @@ def _manage_executor_state( and ti.state in self.STATES_COUNT_AS_RUNNING

Re: [I] execution_date format [airflow]

2024-06-18 Thread via GitHub
shahar1 commented on issue #40292: URL: https://github.com/apache/airflow/issues/40292#issuecomment-2176675271 It seems that @raphaelauv 's snippet could do the work if it's only a matter of formatting. It doesn't seem like a bug, though. Converting into a discussion. -- This is an

Re: [I] execution_date format [airflow]

2024-06-18 Thread via GitHub
shahar1 closed issue #40292: execution_date format URL: https://github.com/apache/airflow/issues/40292 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: c

Re: [I] Add deferrable support to `LambdaInvokeFunctionOperator` [airflow]

2024-06-18 Thread via GitHub
vincbeck commented on issue #40307: URL: https://github.com/apache/airflow/issues/40307#issuecomment-2176723207 Indeed, that'b a cool add-on! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[I] [Providers][OpenLineage] Scheduler shuts down after pickle of OpenLineageListener fails [airflow]

2024-06-18 Thread via GitHub
merobi-hub opened a new issue, #40309: URL: https://github.com/apache/airflow/issues/40309 ### Apache Airflow version 2.9.2 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When running Airflow locally with the latest ve

Re: [I] OpenLineage Provider issue: scheduler shuts down after pickle of OpenLineageListener fails [airflow]

2024-06-18 Thread via GitHub
JDarDagran commented on issue #40309: URL: https://github.com/apache/airflow/issues/40309#issuecomment-2176794224 In general `airflow standalone` doesn't really work with OpenLineage, even with earlier versions (at least that's what I recall). More importantly, this is not reproducible in

Re: [PR] Remove "experimental" banner for OTel Metrics [airflow]

2024-06-18 Thread via GitHub
ferruzzi commented on code in PR #40286: URL: https://github.com/apache/airflow/pull/40286#discussion_r1644947942 ## airflow/www/templates/airflow/dags.html: ## @@ -119,12 +119,6 @@ Click here for more information. {% endcall %} {% endif %} - {% if otel_metrics_o

Re: [PR] [WIP] feat: added get_openlineage_facets_on_start function in operator definition [airflow]

2024-06-18 Thread via GitHub
rahul-madaan commented on code in PR #40290: URL: https://github.com/apache/airflow/pull/40290#discussion_r1645006707 ## airflow/providers/google/cloud/transfers/azure_blob_to_gcs.py: ## @@ -122,3 +122,21 @@ def execute(self, context: Context) -> str: self.bucke

Re: [PR] Add Amazon Comprehend Document Classifier [airflow]

2024-06-18 Thread via GitHub
gopidesupavan commented on code in PR #40287: URL: https://github.com/apache/airflow/pull/40287#discussion_r1645008542 ## tests/system/providers/amazon/aws/example_comprehend_document_classifier.py: ## @@ -0,0 +1,234 @@ +# Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Add Amazon Comprehend Document Classifier [airflow]

2024-06-18 Thread via GitHub
gopidesupavan commented on code in PR #40287: URL: https://github.com/apache/airflow/pull/40287#discussion_r1645011338 ## tests/system/providers/amazon/aws/example_comprehend_document_classifier.py: ## @@ -0,0 +1,234 @@ +# Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Add Amazon Comprehend Document Classifier [airflow]

2024-06-18 Thread via GitHub
gopidesupavan commented on PR #40287: URL: https://github.com/apache/airflow/pull/40287#issuecomment-2176905727 > Just 2 comments but overall fantastic work!! Thank you 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [WIP] feat: added get_openlineage_facets_on_start function in operator definition [airflow]

2024-06-18 Thread via GitHub
rahul-madaan commented on code in PR #40290: URL: https://github.com/apache/airflow/pull/40290#discussion_r1645020677 ## airflow/providers/google/cloud/transfers/azure_blob_to_gcs.py: ## @@ -122,3 +122,21 @@ def execute(self, context: Context) -> str: self.bucke

Re: [PR] AIP-61 - Scheduler job: main loop and event processing for multi executors [airflow]

2024-06-18 Thread via GitHub
dstandish commented on code in PR #40017: URL: https://github.com/apache/airflow/pull/40017#discussion_r1645052393 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1844,6 +1895,58 @@ def _executor_to_tis(self, tis: list[TaskInstance]) -> dict[BaseExecutor, list[T

  1   2   >