Re: [PR] Raising Exception if dag is not in active state while triggering dag [airflow]

2024-03-09 Thread via GitHub
jscheffl commented on PR #37836: URL: https://github.com/apache/airflow/pull/37836#issuecomment-1986796235 > If we do this it must be consistent between CLI and API. > > > I think there were related discussions in the past about how to treat a paused DAG when it is triggered >

Re: [I] ElasticSearch log fetching hook is not granular enough in its index filter [airflow]

2024-03-09 Thread via GitHub
Owen-CH-Leung commented on issue #37999: URL: https://github.com/apache/airflow/issues/37999#issuecomment-1986796655 Ok let me take a look and will revert soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] heartbeat recovery message [airflow]

2024-03-09 Thread via GitHub
Bowrna commented on code in PR #34457: URL: https://github.com/apache/airflow/pull/34457#discussion_r1518546357 ## tests/jobs/test_base_job.py: ## @@ -205,15 +205,17 @@ def test_is_alive_scheduler(self, job_type): assert job.is_alive() is False, "Completed jobs even wit

[PR] Improve logging readability with DockerOperator [airflow]

2024-03-09 Thread via GitHub
lyuyunbo opened a new pull request, #38008: URL: https://github.com/apache/airflow/pull/38008 Related: [discussion](https://github.com/apache/airflow/discussions/29302) When DockerOperator logs are displayed in the Airflow UI, the logs from the container are prefixed with "[asctime UT

Re: [PR] Improve logging readability with DockerOperator [airflow]

2024-03-09 Thread via GitHub
boring-cyborg[bot] commented on PR #38008: URL: https://github.com/apache/airflow/pull/38008#issuecomment-1986846284 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'

Re: [PR] Improve logging readability with DockerOperator [airflow]

2024-03-09 Thread via GitHub
lyuyunbo commented on PR #38008: URL: https://github.com/apache/airflow/pull/38008#issuecomment-1986847704 Hi @potiuk, I've added a PR to address your [previous discussion](https://github.com/apache/airflow/discussions/29302) with [MattiaGallegati](https://github.com/MattiaGallegati).

[PR] Updated version of the multi-tenant diagram [airflow]

2024-03-09 Thread via GitHub
potiuk opened a new pull request, #38009: URL: https://github.com/apache/airflow/pull/38009 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pul

[PR] Use AwsLogsHook when fetching Glue job logs [airflow]

2024-03-09 Thread via GitHub
VincentChantreau opened a new pull request, #38010: URL: https://github.com/apache/airflow/pull/38010 This PR aim to fix a bug in the GlueJobHook that occurs when setting the `verbose` parameter of GlueJobOperator to True. The `print_job_logs` method is instantiating a new boto3 client in

Re: [PR] Use AwsLogsHook when fetching Glue job logs [airflow]

2024-03-09 Thread via GitHub
boring-cyborg[bot] commented on PR #38010: URL: https://github.com/apache/airflow/pull/38010#issuecomment-1986853834 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'

Re: [I] Glue Hook not using aws_conn_id for logs access resulting in Error [airflow]

2024-03-09 Thread via GitHub
VincentChantreau commented on issue #37976: URL: https://github.com/apache/airflow/issues/37976#issuecomment-1986856593 @eladkal I've submitted a PR to address the issue :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Use AwsLogsHook when fetching Glue job logs [airflow]

2024-03-09 Thread via GitHub
Taragolis commented on code in PR #38010: URL: https://github.com/apache/airflow/pull/38010#discussion_r1518573347 ## airflow/providers/amazon/aws/hooks/glue.py: ## @@ -112,6 +112,7 @@ def __init__( kwargs["client_type"] = "glue" super().__init__(*args, **kwa

Re: [PR] ECS Executor retry task bug fix [airflow]

2024-03-09 Thread via GitHub
eladkal merged PR #37979: URL: https://github.com/apache/airflow/pull/37979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

(airflow) branch main updated: Fix bug for ECS Executor where tasks were being skipped if one task failed. (#37979)

2024-03-09 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 06e3197ccb Fix bug for ECS Executor where tasks we

Re: [I] Context variable defaults break tasks where these parameters don't go at the end of the task fuction definition [airflow]

2024-03-09 Thread via GitHub
potiuk commented on issue #38006: URL: https://github.com/apache/airflow/issues/38006#issuecomment-1986870576 Yep known issue. We discussed it with @uranusjr and consensus is to provide better message. Feel free to make PR for it. -- This is an automated message from the Apache Git Servic

Re: [PR] Add usage withing jinja template [airflow]

2024-03-09 Thread via GitHub
eladkal commented on code in PR #35479: URL: https://github.com/apache/airflow/pull/35479#discussion_r1518580893 ## docs/apache-airflow-providers-amazon/connections/aws.rst: ## @@ -240,6 +240,17 @@ With a AWS IAM key pair "password": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLE

Re: [PR] Use AwsLogsHook when fetching Glue job logs [airflow]

2024-03-09 Thread via GitHub
VincentChantreau commented on code in PR #38010: URL: https://github.com/apache/airflow/pull/38010#discussion_r1518586371 ## airflow/providers/amazon/aws/hooks/glue.py: ## @@ -112,6 +112,7 @@ def __init__( kwargs["client_type"] = "glue" super().__init__(*args

Re: [I] Fix existing custom operators' templated fields that include logic checks [airflow]

2024-03-09 Thread via GitHub
shahar1 commented on issue #36484: URL: https://github.com/apache/airflow/issues/36484#issuecomment-1986896702 > I believe all entries are fixed now? Almost, we still have #37519 open. Thanks everyone for helping out :) -- This is an automated message from the Apache Git Service.

Re: [I] Fix existing custom operators' templated fields that include logic checks [airflow]

2024-03-09 Thread via GitHub
eladkal commented on issue #36484: URL: https://github.com/apache/airflow/issues/36484#issuecomment-1986899687 > > I believe all entries are fixed now? > > Almost, we still have #37519 open. Thanks everyone for helping out :) I think it was fixed in another PR? -- This is an

Re: [PR] heartbeat recovery message [airflow]

2024-03-09 Thread via GitHub
potiuk commented on PR #34457: URL: https://github.com/apache/airflow/pull/34457#issuecomment-1986905542 this looks good @Bowrna (except the static check failing) - but I have one small proposal. It took me a bit of time to see why grace_multiplier is not passed to to the method in heartbea

[PR] Fix references to `mypy-airflow` in breeze developer tasks docs [airflow]

2024-03-09 Thread via GitHub
shahar1 opened a new pull request, #38011: URL: https://github.com/apache/airflow/pull/38011 Fixes outdated syntax in the examples for running `mypy-airflow` using breeze. --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https:

Re: [PR] Fix references to `mypy-airflow` in breeze developer tasks docs [airflow]

2024-03-09 Thread via GitHub
potiuk merged PR #38011: URL: https://github.com/apache/airflow/pull/38011 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated (06e3197ccb -> 07024b6460)

2024-03-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 06e3197ccb Fix bug for ECS Executor where tasks were being skipped if one task failed. (#37979) add 07024b6460 Fi

Re: [PR] Fix references to `mypy-airflow` in breeze developer tasks docs [airflow]

2024-03-09 Thread via GitHub
potiuk commented on PR #38011: URL: https://github.com/apache/airflow/pull/38011#issuecomment-1986912878 Nice! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [I] Counterintuitive functionality when viewing mapped TIs from Grid View [airflow]

2024-03-09 Thread via GitHub
lotrias17 commented on issue #34489: URL: https://github.com/apache/airflow/issues/34489#issuecomment-1986913936 Hello, could I get assigned to this issue? Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] DatabricksSQLOperator struggles with parsing the data [airflow]

2024-03-09 Thread via GitHub
potiuk commented on issue #36838: URL: https://github.com/apache/airflow/issues/36838#issuecomment-1986914063 @aru-trackunit - seems that @Joffreybvn cannot reproduce the issue - can you please provide a bit more information as he asked in https://github.com/apache/airflow/issues/36838#issu

Re: [PR] Update use cases page [airflow-site]

2024-03-09 Thread via GitHub
potiuk commented on PR #977: URL: https://github.com/apache/airflow-site/pull/977#issuecomment-1986919423 This looks cool, I already made my comments to the content of the videos and voice-over (being a professional lector I pay attention to some quality problems ;) - and it's corrected now

Re: [PR] Updated version of the multi-tenant diagram [airflow]

2024-03-09 Thread via GitHub
potiuk merged PR #38009: URL: https://github.com/apache/airflow/pull/38009 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated (07024b6460 -> 1690906e1d)

2024-03-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 07024b6460 Fix references to `mypy-airflow` in breeze developer tasks docs (#38011) add 1690906e1d Updated versio

[PR] Workaround optional typing error detected by new typing-docutils [airflow]

2024-03-09 Thread via GitHub
potiuk opened a new pull request, #38012: URL: https://github.com/apache/airflow/pull/38012 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pul

Re: [I] ElasticSearch log fetching hook is not granular enough in its index filter [airflow]

2024-03-09 Thread via GitHub
nsAstro commented on issue #37999: URL: https://github.com/apache/airflow/issues/37999#issuecomment-1986923482 Made a few edits to the main issue. Also updated the name after some feedback from @jedcunningham. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Add colour to logs [airflow]

2024-03-09 Thread via GitHub
eladkal commented on issue #37443: URL: https://github.com/apache/airflow/issues/37443#issuecomment-1986927091 @tirkarthi feel free to raise PR. We need to make sure we don't end up in color clash after https://github.com/apache/airflow/pull/37985 -- This is an automated message from the

Re: [PR] Workaround optional typing error detected by new typing-docutils [airflow]

2024-03-09 Thread via GitHub
potiuk merged PR #38012: URL: https://github.com/apache/airflow/pull/38012 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated (1690906e1d -> 9c9df91c00)

2024-03-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 1690906e1d Updated version of the multi-tenant diagram (#38009) add 9c9df91c00 Workaround optional typing error de

Re: [I] Identify duplicate kubernetes section airflow configuration and mark them as deprecated [airflow]

2024-03-09 Thread via GitHub
eladkal commented on issue #36389: URL: https://github.com/apache/airflow/issues/36389#issuecomment-1986927467 @dirrao are you working on this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Introduce mechanism to support multiple executor configuration [airflow]

2024-03-09 Thread via GitHub
potiuk merged PR #37635: URL: https://github.com/apache/airflow/pull/37635 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated (9c9df91c00 -> 93dfdf0d7f)

2024-03-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 9c9df91c00 Workaround optional typing error detected by new typing-docutils (#38012) add 93dfdf0d7f Introduce mec

(airflow) branch constraints-main updated: Updating constraints. Github run id:8216398300

2024-03-09 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 968278f6cc Updating con

[PR] Fix D105 checks for SSH provider [airflow]

2024-03-09 Thread via GitHub
alizademhdi opened a new pull request, #38013: URL: https://github.com/apache/airflow/pull/38013 ## related issue: #37523 Fix D105 checks for SSH provider ## Checks: - [x] airflow/providers/ssh/hooks/ssh.py -- This is an automated message from the Apache Gi

Re: [PR] Fix D105 checks for SSH provider [airflow]

2024-03-09 Thread via GitHub
boring-cyborg[bot] commented on PR #38013: URL: https://github.com/apache/airflow/pull/38013#issuecomment-1986959187 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'

[PR] Fix bug in validate-operators-init pre-commit [airflow]

2024-03-09 Thread via GitHub
shahar1 opened a new pull request, #38014: URL: https://github.com/apache/airflow/pull/38014 related: #36484, Unfortunately, in PR #33786, I accidentally missed a condition to check that templated fields' assigned values are parameters with the same name. This PR fixe

Re: [PR] Fix bug in validate-operators-init pre-commit [airflow]

2024-03-09 Thread via GitHub
eladkal commented on PR #38014: URL: https://github.com/apache/airflow/pull/38014#issuecomment-1986990125 It looks like more files are failing on the test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] 'None' execution date in a few DAG runs across several others [airflow]

2024-03-09 Thread via GitHub
github-actions[bot] commented on issue #37592: URL: https://github.com/apache/airflow/issues/37592#issuecomment-1987019655 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further

Re: [I] Unable to import dags in dag_bag.py [airflow]

2024-03-09 Thread via GitHub
github-actions[bot] commented on issue #37647: URL: https://github.com/apache/airflow/issues/37647#issuecomment-1987019646 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further

Re: [PR] Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true [airflow]

2024-03-09 Thread via GitHub
github-actions[bot] closed pull request #36628: Fix passing dags.gitSync.sshKeySecret with dags.persistence.enabled=true URL: https://github.com/apache/airflow/pull/36628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Migrate to connexion v3 [airflow]

2024-03-09 Thread via GitHub
sudiptob2 commented on PR #37638: URL: https://github.com/apache/airflow/pull/37638#issuecomment-1987085832 ### Need_Help We've made significant improvements to several unit tests. However, there are still failing test cases. If anyone has any thoughts or pointers regarding the failing t

Re: [PR] Raising Exception if dag is not in active state while triggering dag [airflow]

2024-03-09 Thread via GitHub
nj7 commented on PR #37836: URL: https://github.com/apache/airflow/pull/37836#issuecomment-1987103588 hmm, interesting points, totally agree with arguments. Let me work on these changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Migrate to connexion v3 [airflow]

2024-03-09 Thread via GitHub
potiuk commented on PR #37638: URL: https://github.com/apache/airflow/pull/37638#issuecomment-1987109928 I think that for a lot of the tests just need to creatively look at the way how we initialize things ``` ValueError: The name '/auth/fab/v1' is already registered for a differe

[PR] Context parameters fix [airflow]

2024-03-09 Thread via GitHub
kevinalh opened a new pull request, #38015: URL: https://github.com/apache/airflow/pull/38015 This PR does two things for the Python task decorator: 1. Don't allow a default parameter other than None for context parameters, since this is always replaced by None in 2.8.

Re: [PR] Fix None defaults breaking usage of context arguments in decorated tasks [airflow]

2024-03-09 Thread via GitHub
kevinalh closed pull request #38007: Fix None defaults breaking usage of context arguments in decorated tasks URL: https://github.com/apache/airflow/pull/38007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Fix None defaults breaking usage of context arguments in decorated tasks [airflow]

2024-03-09 Thread via GitHub
kevinalh commented on PR #38007: URL: https://github.com/apache/airflow/pull/38007#issuecomment-1987121667 > I don't think we should change order of parameters. This has a hidden potential of breaking behaviour of underlying functions > > So far we discussed with @uranusjr that better

Re: [PR] Don't allow defaults other than None in context parameters, and improve error message [airflow]

2024-03-09 Thread via GitHub
potiuk commented on code in PR #38015: URL: https://github.com/apache/airflow/pull/38015#discussion_r1518758684 ## airflow/decorators/base.py: ## @@ -208,11 +208,25 @@ def __init__( # since values for those will be provided when the task is run. Since # we're n

Re: [PR] Don't allow defaults other than None in context parameters, and improve error message [airflow]

2024-03-09 Thread via GitHub
potiuk commented on code in PR #38015: URL: https://github.com/apache/airflow/pull/38015#discussion_r1518758684 ## airflow/decorators/base.py: ## @@ -208,11 +208,25 @@ def __init__( # since values for those will be provided when the task is run. Since # we're n

Re: [PR] Add missing attrs / correct bad attrs on TaskInstancePydantic [airflow]

2024-03-09 Thread via GitHub
potiuk commented on code in PR #37854: URL: https://github.com/apache/airflow/pull/37854#discussion_r151853 ## airflow/serialization/pydantic/taskinstance.py: ## @@ -105,12 +105,13 @@ class TaskInstancePydantic(BaseModelPydantic, LoggingMixin): trigger_timeout: Optiona