[GitHub] [airflow] syedahsn commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
syedahsn commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232941301 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -184,12 +238,12 @@ def execute(self, context: Context): ) ) -

[GitHub] [airflow] potiuk commented on issue #31972: Struggling with unique key constrain violations

2023-06-16 Thread via GitHub
potiuk commented on issue #31972: URL: https://github.com/apache/airflow/issues/31972#issuecomment-1595559666 The way it looks - it's a problem of your Database. You likely try to use some active-active DB (not standard Postgres, but something that builds on top of Postgres). Likely

[GitHub] [airflow] github-actions[bot] closed pull request #30892: Fix depends_on_past work for Dynamic tasks

2023-06-16 Thread via GitHub
github-actions[bot] closed pull request #30892: Fix depends_on_past work for Dynamic tasks URL: https://github.com/apache/airflow/pull/30892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [airflow] o-nikolas commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
o-nikolas commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232881332 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -184,12 +238,12 @@ def execute(self, context: Context): ) ) -

[GitHub] [airflow] ksundeepsatya closed issue #31908: execution_date backward compatibility between Airflow1 and Airflow2

2023-06-16 Thread via GitHub
ksundeepsatya closed issue #31908: execution_date backward compatibility between Airflow1 and Airflow2 URL: https://github.com/apache/airflow/issues/31908 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[GitHub] [airflow] ksundeepsatya commented on issue #31908: execution_date backward compatibility between Airflow1 and Airflow2

2023-06-16 Thread via GitHub
ksundeepsatya commented on issue #31908: URL: https://github.com/apache/airflow/issues/31908#issuecomment-1595413134 I would like to apologize the community here. Looks like internally we have forked Airflow1 to modify the behavior of execution date in Airflow1 to set execution to end time

[airflow] branch main updated: Deprecate `wait_for_completion` from `EcsRegisterTaskDefinitionOperator` and `EcsDeregisterTaskDefinitionOperator` (#31884)

2023-06-16 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 41fb9b4c49 Deprecate `wait_for_completion` from `

[GitHub] [airflow] eladkal merged pull request #31884: Deprecate `wait_for_completion` from `EcsRegisterTaskDefinitionOperator` and `EcsDeregisterTaskDefinitionOperator`

2023-06-16 Thread via GitHub
eladkal merged PR #31884: URL: https://github.com/apache/airflow/pull/31884 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

[GitHub] [airflow] eladkal commented on a diff in pull request #31884: remove 'wait_for_completion' from ecs register/deregister operators

2023-06-16 Thread via GitHub
eladkal commented on code in PR #31884: URL: https://github.com/apache/airflow/pull/31884#discussion_r1232857696 ## airflow/providers/amazon/aws/operators/ecs.py: ## @@ -202,53 +202,39 @@ class EcsDeregisterTaskDefinitionOperator(EcsBaseOperator): :param task_definition:

[GitHub] [airflow] ginwakeup commented on issue #31972: Struggling with unique key constrain violations

2023-06-16 Thread via GitHub
ginwakeup commented on issue #31972: URL: https://github.com/apache/airflow/issues/31972#issuecomment-1595377593 Same with scheduler: ``` [2023-06-16T22:14:40.368+] {scheduler_job.py:776} ERROR - Exception when executing SchedulerJob._run_scheduler_loop Traceback (most recen

[GitHub] [airflow] ginwakeup commented on issue #31972: Struggling with unique key constrain violations

2023-06-16 Thread via GitHub
ginwakeup commented on issue #31972: URL: https://github.com/apache/airflow/issues/31972#issuecomment-1595371826 Triggerer exception as soon as it starts: ``` [2023-06-16T22:08:22.757+] {triggerer_job.py:108} ERROR - Exception when executing TriggererJob._run_trigger_loop Traceb

[GitHub] [airflow] syedahsn commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
syedahsn commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232833537 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -184,12 +235,12 @@ def execute(self, context: Context): ) ) -

[GitHub] [airflow] ginwakeup commented on issue #31972: Struggling with unique key constrain violations

2023-06-16 Thread via GitHub
ginwakeup commented on issue #31972: URL: https://github.com/apache/airflow/issues/31972#issuecomment-1595366827 @jens-scheffler-bosch forgive me, that's not the scheduler logs, it's the triggerer. The scheduler errors like this: ``` [2023-06-16T22:01:08.483+] {settings.

[GitHub] [airflow] ginwakeup commented on issue #31972: Struggling with unique key constrain violations

2023-06-16 Thread via GitHub
ginwakeup commented on issue #31972: URL: https://github.com/apache/airflow/issues/31972#issuecomment-1595355293 I've also just tried to run `airflow db reset` but it didn't help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [airflow] ginwakeup commented on issue #31972: Struggling with unique key constrain violations

2023-06-16 Thread via GitHub
ginwakeup commented on issue #31972: URL: https://github.com/apache/airflow/issues/31972#issuecomment-1595354267 @jens-scheffler-bosch More details on the setup: - gitSync pulls DAGS and it runs on a separate pod which mounts a PVC - The PVC is mounted on all the airflow components -

[GitHub] [airflow] jens-scheffler-bosch commented on issue #31972: Struggling with unique key constrain violations

2023-06-16 Thread via GitHub
jens-scheffler-bosch commented on issue #31972: URL: https://github.com/apache/airflow/issues/31972#issuecomment-1595351825 Can you please add some details which Airflow version you are actually using? Else it is a bit shooting in the dark, what the problem might be. Taking a look to

[airflow] branch constraints-main updated: Updating constraints. Github run id:5294084271

2023-06-16 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 7fc2caf69c Updating con

[GitHub] [airflow] jens-scheffler-bosch closed issue #31930: UI: New trigger DAG with config form doesn’t allow valid floats and empty strings

2023-06-16 Thread via GitHub
jens-scheffler-bosch closed issue #31930: UI: New trigger DAG with config form doesn’t allow valid floats and empty strings URL: https://github.com/apache/airflow/issues/31930 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [airflow] jens-scheffler-bosch commented on issue #31930: UI: New trigger DAG with config form doesn’t allow valid floats and empty strings

2023-06-16 Thread via GitHub
jens-scheffler-bosch commented on issue #31930: URL: https://github.com/apache/airflow/issues/31930#issuecomment-1595349319 Thanks for reporting this bug - and as Part 1 is already fixed with a PR let me answer on the others. Regarding Part 2) Empty strings are converted to `null` val

[GitHub] [airflow] ginwakeup opened a new issue, #31972: Struggling with unique key constrain violations

2023-06-16 Thread via GitHub
ginwakeup opened a new issue, #31972: URL: https://github.com/apache/airflow/issues/31972 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Airflow keeps polluting the postgres DB. I cannot understand what causes this but it happ

[GitHub] [airflow] killua1zoldyck commented on pull request #31837: Add realtime container execution logs for BatchOperator

2023-06-16 Thread via GitHub
killua1zoldyck commented on PR #31837: URL: https://github.com/apache/airflow/pull/31837#issuecomment-1595336279 > > I have updated the changes from #31824 @o-nikolas > > Thanks! Unfortunately there are some more conflicts due to some more logging PRs being merged I have update

[GitHub] [airflow] dstandish commented on a diff in pull request #30271: Implement clearing behavior for AIP-52

2023-06-16 Thread via GitHub
dstandish commented on code in PR #30271: URL: https://github.com/apache/airflow/pull/30271#discussion_r1232796844 ## airflow/models/abstractoperator.py: ## @@ -188,6 +188,33 @@ def get_flat_relatives(self, upstream: bool = False) -> Collection[Operator]: return se

[GitHub] [airflow] jhbigler commented on pull request #31950: Trino Hook: Add ability to read JWT from file

2023-06-16 Thread via GitHub
jhbigler commented on PR #31950: URL: https://github.com/apache/airflow/pull/31950#issuecomment-1595324301 I have implemented the suggested changes, open to additional feedback (@phanikumv @uranusjr @utkarsharma2) -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [airflow] jens-scheffler-bosch opened a new pull request, #31971: Use null and not default value if array in trigger form is empty

2023-06-16 Thread via GitHub
jens-scheffler-bosch opened a new pull request, #31971: URL: https://github.com/apache/airflow/pull/31971 This PR is for branch 2.6 releases only. It is already fixed on main/2.7.0++ Not more details on this PR, it is a one-liner, details in the bug ticket closes: #31869 -

[airflow] branch main updated: Replace State usages with strong-typed enums (#31735)

2023-06-16 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new b459af3ee0 Replace State usages with strong-type

[GitHub] [airflow] shahar1 opened a new pull request, #31970: Remove airflow user from `Dockerfile.ci`

2023-06-16 Thread via GitHub
shahar1 opened a new pull request, #31970: URL: https://github.com/apache/airflow/pull/31970 Following discussion #31714 - I'm trying to remove `airflow` user from `Dockerfile.ci` and see how it affects it. --- **^ Add meaningful description above** Read the **[Pull

[GitHub] [airflow] vincbeck commented on a diff in pull request #31939: Create ChimeWebhook Hook to send messages to chime room from Airflow.

2023-06-16 Thread via GitHub
vincbeck commented on code in PR #31939: URL: https://github.com/apache/airflow/pull/31939#discussion_r1232681082 ## airflow/providers/amazon/aws/hooks/chime.py: ## @@ -0,0 +1,113 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

[GitHub] [airflow] vincbeck commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
vincbeck commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232670101 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -59,7 +59,54 @@ FARGATE_FULL_NAME = "AWS Fargate profiles" -class EksCreateClusterOperator(BaseOperator)

[GitHub] [airflow] fgalind1 commented on pull request #31066: Support naming customization on helm chart resources

2023-06-16 Thread via GitHub
fgalind1 commented on PR #31066: URL: https://github.com/apache/airflow/pull/31066#issuecomment-1595202690 airflow folks sorry to bother again, but wonder if this PR could get any traction 😃 -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [airflow] Gollum999 opened a new issue, #31969: Custom Timetable breaks a handful of CLI commands

2023-06-16 Thread via GitHub
Gollum999 opened a new issue, #31969: URL: https://github.com/apache/airflow/issues/31969 ### Apache Airflow version 2.6.1 ### What happened A number of Airflow CLI commands will fail with an error like the one below if a DAG uses a [custom Timetable](https://airflow.ap

[GitHub] [airflow] potiuk commented on issue #31968: Incompatible snowflake-connector-python and pyarrow dependencies

2023-06-16 Thread via GitHub
potiuk commented on issue #31968: URL: https://github.com/apache/airflow/issues/31968#issuecomment-1595189119 Surely. Constraints of airflow do not include pandas extra. As explained in our documentation, when you want to install/upgrede providers independently from the airflow core, you sh

[GitHub] [airflow] o-nikolas commented on pull request #31837: Add realtime container execution logs for BatchOperator

2023-06-16 Thread via GitHub
o-nikolas commented on PR #31837: URL: https://github.com/apache/airflow/pull/31837#issuecomment-1595186649 > I have updated the changes from #31824 @o-nikolas Thanks! Unfortunately there are some more conflicts due to some more logging PRs being merged -- This is an automated mess

[airflow] branch main updated: Various fixes on ECS run task operator (#31838)

2023-06-16 Thread onikolas
This is an automated email from the ASF dual-hosted git repository. onikolas pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new e0f21f43c6 Various fixes on ECS run task operator

[GitHub] [airflow] o-nikolas merged pull request #31838: Various fixes on ECS run task operator

2023-06-16 Thread via GitHub
o-nikolas merged PR #31838: URL: https://github.com/apache/airflow/pull/31838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[GitHub] [airflow] o-nikolas commented on a diff in pull request #31838: Various fixes on ECS run task operator

2023-06-16 Thread via GitHub
o-nikolas commented on code in PR #31838: URL: https://github.com/apache/airflow/pull/31838#discussion_r1232645548 ## airflow/providers/amazon/aws/operators/ecs.py: ## @@ -506,25 +515,24 @@ def execute(self, context, session=None): @AwsBaseHook.retry(should_retry_eni)

[GitHub] [airflow] jhbigler commented on a diff in pull request #31950: Trino Hook: Add ability to read JWT from file

2023-06-16 Thread via GitHub
jhbigler commented on code in PR #31950: URL: https://github.com/apache/airflow/pull/31950#discussion_r1232642277 ## airflow/providers/trino/hooks/trino.py: ## @@ -95,7 +95,12 @@ def get_conn(self) -> Connection: elif db.password: auth = trino.auth.BasicAut

[GitHub] [airflow] dstandish closed pull request #31921: Simplify get_flat_relative_ids logic

2023-06-16 Thread via GitHub
dstandish closed pull request #31921: Simplify get_flat_relative_ids logic URL: https://github.com/apache/airflow/pull/31921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] shahar1 commented on a diff in pull request #31885: Allow a destination folder to be provided

2023-06-16 Thread via GitHub
shahar1 commented on code in PR #31885: URL: https://github.com/apache/airflow/pull/31885#discussion_r1232625943 ## airflow/providers/google/suite/transfers/gcs_to_gdrive.py: ## @@ -63,12 +63,15 @@ class GCSToGoogleDriveOperator(BaseOperator): For example, with prefix `

[GitHub] [airflow] dstandish commented on a diff in pull request #31927: Add different version of chain

2023-06-16 Thread via GitHub
dstandish commented on code in PR #31927: URL: https://github.com/apache/airflow/pull/31927#discussion_r1232629952 ## airflow/models/baseoperator.py: ## @@ -1838,6 +1839,38 @@ def cross_downstream( task.set_downstream(to_tasks) +def chain_sequential(*elements: Depen

[GitHub] [airflow] shahar1 commented on a diff in pull request #31885: Allow a destination folder to be provided

2023-06-16 Thread via GitHub
shahar1 commented on code in PR #31885: URL: https://github.com/apache/airflow/pull/31885#discussion_r1232625943 ## airflow/providers/google/suite/transfers/gcs_to_gdrive.py: ## @@ -63,12 +63,15 @@ class GCSToGoogleDriveOperator(BaseOperator): For example, with prefix `

[airflow] branch main updated: Fix masking nested variable fields (#31964)

2023-06-16 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new e22ce9baed Fix masking nested variable fields

[GitHub] [airflow] hussein-awala closed issue #31873: KubernetesPodOperator doesn't mask variables in Rendered Template that are used as arguments

2023-06-16 Thread via GitHub
hussein-awala closed issue #31873: KubernetesPodOperator doesn't mask variables in Rendered Template that are used as arguments URL: https://github.com/apache/airflow/issues/31873 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] hussein-awala merged pull request #31964: Fix masking nested variable fields

2023-06-16 Thread via GitHub
hussein-awala merged PR #31964: URL: https://github.com/apache/airflow/pull/31964 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[GitHub] [airflow] cbuffett opened a new issue, #31968: Incompatible snowflake-connector-python and pyarrow dependencies

2023-06-16 Thread via GitHub
cbuffett opened a new issue, #31968: URL: https://github.com/apache/airflow/issues/31968 ### Apache Airflow version 2.6.1 ### What happened The 2.6.1-python3.7 constraints specify `pyarrow==9.0.0` and `snowflake-connector-python=3.0.3`. However, version 3.0.0 of `snowf

[GitHub] [airflow] o-nikolas commented on a diff in pull request #31942: Use a waiter in athena hook

2023-06-16 Thread via GitHub
o-nikolas commented on code in PR #31942: URL: https://github.com/apache/airflow/pull/31942#discussion_r1232600792 ## airflow/providers/amazon/aws/hooks/athena.py: ## @@ -229,51 +242,33 @@ def get_query_results_paginator( return paginator.paginate(**result_params)

[airflow] branch main updated: Upgrade ruff to 0.272 (#31966)

2023-06-16 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new fc0e5a4d42 Upgrade ruff to 0.272 (#31966) fc0e5a

[GitHub] [airflow] dstandish merged pull request #31966: Upgrade ruff to 0.272

2023-06-16 Thread via GitHub
dstandish merged PR #31966: URL: https://github.com/apache/airflow/pull/31966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[airflow] branch main updated: Fixed failing main on too fast merged change (#31967)

2023-06-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3003b16998 Fixed failing main on too fast merged ch

[GitHub] [airflow] potiuk merged pull request #31967: Fixed failing main on too fast merged change

2023-06-16 Thread via GitHub
potiuk merged PR #31967: URL: https://github.com/apache/airflow/pull/31967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk opened a new pull request, #31967: Fixed failing main on too fast merged change

2023-06-16 Thread via GitHub
potiuk opened a new pull request, #31967: URL: https://github.com/apache/airflow/pull/31967 Merged #31961 - as a typo one-letter-fix thinking that - well it can't possibly cause any problems. Boy, how was I wrong. --- **^ Add meaningful description above** Rea

[GitHub] [airflow] o-nikolas commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
o-nikolas commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232574760 ## airflow/providers/amazon/aws/triggers/eks.py: ## @@ -33,49 +33,52 @@ class EksCreateFargateProfileTrigger(BaseTrigger): :param cluster_name: The name of t

[GitHub] [airflow] uranusjr commented on pull request #31731: Google provider docstring improvements

2023-06-16 Thread via GitHub
uranusjr commented on PR #31731: URL: https://github.com/apache/airflow/pull/31731#issuecomment-1595061846 FWIW we really should switch away from sphinx-autoapi as well (in favor of `sphinx.ext.autodoc`), I think that’s another blocker for Sphinx 7 and generally a pain to interact with. -

[GitHub] [airflow] o-nikolas commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
o-nikolas commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232571656 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -393,51 +416,41 @@ def __init__( ) -> None: self.cluster_name = cluster_name self.pod

[GitHub] [airflow] syedahsn commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
syedahsn commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232568930 ## airflow/providers/amazon/aws/triggers/eks.py: ## @@ -33,49 +33,52 @@ class EksCreateFargateProfileTrigger(BaseTrigger): :param cluster_name: The name of th

[GitHub] [airflow] syedahsn commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
syedahsn commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232567156 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -393,51 +416,41 @@ def __init__( ) -> None: self.cluster_name = cluster_name self.pod_

[GitHub] [airflow] syedahsn commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
syedahsn commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232565571 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -184,12 +235,12 @@ def execute(self, context: Context): ) ) -

[GitHub] [airflow] syedahsn commented on a diff in pull request #31712: feat(operator): add deferrable attribute to base operator

2023-06-16 Thread via GitHub
syedahsn commented on code in PR #31712: URL: https://github.com/apache/airflow/pull/31712#discussion_r1232564403 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -820,7 +826,7 @@ def get_waiter( self, waiter_name: str, parameters: dict[str, st

[GitHub] [airflow] o-nikolas commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
o-nikolas commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232558015 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -184,12 +235,12 @@ def execute(self, context: Context): ) ) -

[GitHub] [airflow] syedahsn commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
syedahsn commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232545410 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -59,7 +59,54 @@ FARGATE_FULL_NAME = "AWS Fargate profiles" -class EksCreateClusterOperator(BaseOperator)

[airflow] branch main updated: Skip python version check in some pre-commit when loading setup.py (#31965)

2023-06-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 578cbb24fb Skip python version check in some pre-co

[GitHub] [airflow] potiuk merged pull request #31965: Skip python version check in some pre-commit when loading setup.py

2023-06-16 Thread via GitHub
potiuk merged PR #31965: URL: https://github.com/apache/airflow/pull/31965 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk opened a new pull request, #31965: Skip python version check in some pre-commit when loading setup.py

2023-06-16 Thread via GitHub
potiuk opened a new pull request, #31965: URL: https://github.com/apache/airflow/pull/31965 Some providers are excluded for some versions of Python (for example the `apache.hive` provider is excluded on Python 3.11. However this might make some pre-commits confused when running on Python 3.

[GitHub] [airflow] phanikumv commented on pull request #31925: Install sqlalchemy-spanner package into Google provider

2023-06-16 Thread via GitHub
phanikumv commented on PR #31925: URL: https://github.com/apache/airflow/pull/31925#issuecomment-1594949515 > @phanikumv I fixed tests, could you please check this PR one more time? whats the use case behind this change? What problem were you facing -- This is an automated message f

[GitHub] [airflow] phanikumv commented on pull request #31925: Install sqlalchemy-spanner package into Google provider

2023-06-16 Thread via GitHub
phanikumv commented on PR #31925: URL: https://github.com/apache/airflow/pull/31925#issuecomment-1594949521 > @phanikumv I fixed tests, could you please check this PR one more time? whats the use case behind this change? What problem were you facing -- This is an automated message f

[airflow] branch main updated: Fix typo in scheduler error (#31961)

2023-06-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f03f73100e Fix typo in scheduler error (#31961) f03

[GitHub] [airflow] potiuk merged pull request #31961: Fix typo in scheduler error

2023-06-16 Thread via GitHub
potiuk merged PR #31961: URL: https://github.com/apache/airflow/pull/31961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on issue #31873: KubernetesPodOperator doesn't mask variables in Rendered Template that are used as arguments

2023-06-16 Thread via GitHub
potiuk commented on issue #31873: URL: https://github.com/apache/airflow/issues/31873#issuecomment-1594934440 > @potiuk @phanikumv I think we should mask the value of the loaded dict when `deserialize_json=True`, I tested #31964 and it fixed the problem cool! -- This is an automate

[GitHub] [airflow] hussein-awala commented on issue #31873: KubernetesPodOperator doesn't mask variables in Rendered Template that are used as arguments

2023-06-16 Thread via GitHub
hussein-awala commented on issue #31873: URL: https://github.com/apache/airflow/issues/31873#issuecomment-1594915719 @potiuk @phanikumv I think we should mask the value of the loaded dict when `deserialize_json=True`, I tested #31964 and it fixed the problem -- This is an automated messag

[GitHub] [airflow] hussein-awala opened a new pull request, #31964: Fix masking nested variable fields

2023-06-16 Thread via GitHub
hussein-awala opened a new pull request, #31964: URL: https://github.com/apache/airflow/pull/31964 closes: #31873 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-gui

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #31838: Various fixes on ECS run task operator

2023-06-16 Thread via GitHub
vandonr-amz commented on code in PR #31838: URL: https://github.com/apache/airflow/pull/31838#discussion_r1232431989 ## tests/system/providers/amazon/aws/example_ecs.py: ## @@ -85,6 +93,7 @@ def get_region(): asg_name = f"{env_id}-asg" aws_region = get_region() +

[GitHub] [airflow] phanikumv commented on a diff in pull request #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
phanikumv commented on code in PR #31960: URL: https://github.com/apache/airflow/pull/31960#discussion_r1232429182 ## airflow/providers/amazon/aws/operators/eks.py: ## @@ -59,7 +59,54 @@ FARGATE_FULL_NAME = "AWS Fargate profiles" -class EksCreateClusterOperator(BaseOperator

[GitHub] [airflow] phanikumv commented on pull request #31951: Fix a couple Airflow release doc errors

2023-06-16 Thread via GitHub
phanikumv commented on PR #31951: URL: https://github.com/apache/airflow/pull/31951#issuecomment-1594900515 > > can you please mention the context of why we are doing this(in the description), is it because the last time RC packaging wasn't correct ? > > We usually place little contex

[GitHub] [airflow] jedcunningham commented on a diff in pull request #31951: Fix a couple Airflow release doc errors

2023-06-16 Thread via GitHub
jedcunningham commented on code in PR #31951: URL: https://github.com/apache/airflow/pull/31951#discussion_r1232406266 ## dev/README_RELEASE_AIRFLOW.md: ## @@ -583,15 +583,14 @@ Check if sources are the same as in the tag: ```bash cd X.Y.Zrc1 tar -xvzf *-source.tar.gz -pushd

[GitHub] [airflow] jedcunningham commented on a diff in pull request #31951: Fix a couple Airflow release doc errors

2023-06-16 Thread via GitHub
jedcunningham commented on code in PR #31951: URL: https://github.com/apache/airflow/pull/31951#discussion_r1232405802 ## dev/README_RELEASE_AIRFLOW.md: ## @@ -583,15 +583,14 @@ Check if sources are the same as in the tag: ```bash cd X.Y.Zrc1 tar -xvzf *-source.tar.gz -pushd

[GitHub] [airflow] vincbeck commented on a diff in pull request #31939: Create ChimeWebhook Hook to send messages to chime room from Airflow.

2023-06-16 Thread via GitHub
vincbeck commented on code in PR #31939: URL: https://github.com/apache/airflow/pull/31939#discussion_r1232392626 ## airflow/providers/amazon/aws/hooks/chime.py: ## @@ -0,0 +1,104 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

[GitHub] [airflow] potiuk commented on issue #31873: KubernetesPodOperator doesn't mask variables in Rendered Template that are used as arguments

2023-06-16 Thread via GitHub
potiuk commented on issue #31873: URL: https://github.com/apache/airflow/issues/31873#issuecomment-1594867356 > Hi @phanikumv , thank you for your comment. > > I tried to implement the use of masking the secret on DAG level, but that still doesn't work: > > ``` > @task >

[GitHub] [airflow] vincbeck commented on a diff in pull request #31942: Use a waiter in athena hook

2023-06-16 Thread via GitHub
vincbeck commented on code in PR #31942: URL: https://github.com/apache/airflow/pull/31942#discussion_r1232381091 ## airflow/providers/amazon/aws/hooks/athena.py: ## @@ -38,8 +41,7 @@ class AthenaHook(AwsBaseHook): Provide thick wrapper around :external+boto3:py:class:

[GitHub] [airflow] potiuk opened a new issue, #31963: Upgrade Sphinx to version 7 to improve documentation errors

2023-06-16 Thread via GitHub
potiuk opened a new issue, #31963: URL: https://github.com/apache/airflow/issues/31963 ### Body We should upgrade Sphinx to version 7 Sphinx 6 has known issues with informing about various errors in our documentation and it becomes REALLY painful. Quite often contributors are

[GitHub] [airflow] MaksYermak commented on pull request #31925: Install sqlalchemy-spanner package into Google provider

2023-06-16 Thread via GitHub
MaksYermak commented on PR #31925: URL: https://github.com/apache/airflow/pull/31925#issuecomment-1594817462 @phanikumv I fixed tests, could you please check this PR one more time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [airflow] Gollum999 commented on issue #31945: Transaction leak when using `inlets`

2023-06-16 Thread via GitHub
Gollum999 commented on issue #31945: URL: https://github.com/apache/airflow/issues/31945#issuecomment-1594804866 Just confirmed that this reproduces on 2.6.1 as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #31134: Add support to trigger Azure Functions from Airflow

2023-06-16 Thread via GitHub
ephraimbuddy commented on code in PR #31134: URL: https://github.com/apache/airflow/pull/31134#discussion_r1232336950 ## tests/providers/microsoft/azure/operators/test_azure_functions.py: ## @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or mor

[GitHub] [airflow] vincbeck commented on a diff in pull request #31838: Various fixes on ECS run task operator

2023-06-16 Thread via GitHub
vincbeck commented on code in PR #31838: URL: https://github.com/apache/airflow/pull/31838#discussion_r1232316815 ## tests/system/providers/amazon/aws/example_ecs.py: ## @@ -85,6 +93,7 @@ def get_region(): asg_name = f"{env_id}-asg" aws_region = get_region() +log

[GitHub] [airflow] jherrmannNetfonds commented on pull request #31798: fix spark-kubernetes-operator compatibility

2023-06-16 Thread via GitHub
jherrmannNetfonds commented on PR #31798: URL: https://github.com/apache/airflow/pull/31798#issuecomment-1594756521 thx for reopening. @blcksrx fyi I am testing my local version from line 123 with ``` try: self.log.info(line) except HTTPError as e: self.log.warn(e

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #31961: Fix typo in scheduler error

2023-06-16 Thread via GitHub
boring-cyborg[bot] commented on PR #31961: URL: https://github.com/apache/airflow/pull/31961#issuecomment-1594694209 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] getaaron opened a new pull request, #31961: Fix typo in scheduler error

2023-06-16 Thread via GitHub
getaaron opened a new pull request, #31961: URL: https://github.com/apache/airflow/pull/31961 An error message uses “its” (possessive pronoun) when it means “it’s” (contraction of “it is”. This pull request resolves the issue by inserting an apostrophe between the second and third chara

[GitHub] [airflow] NikitasB commented on issue #31873: KubernetesPodOperator doesn't mask variables in Rendered Template that are used as arguments

2023-06-16 Thread via GitHub
NikitasB commented on issue #31873: URL: https://github.com/apache/airflow/issues/31873#issuecomment-1594678732 Sorry, I closed this by accident. I'm still curious on a solution because if the token can't be masked, I can't deploy the DAG to production. -- This is an automated message fr

[GitHub] [airflow] ephraimbuddy commented on pull request #31772: Refactor Sqlalchemy queries to 2.0 styles (Part 2)

2023-06-16 Thread via GitHub
ephraimbuddy commented on PR #31772: URL: https://github.com/apache/airflow/pull/31772#issuecomment-1594663021 What do you think about this latest commit: https://github.com/apache/airflow/pull/31772/commits/cad7874e6e36df2e426d9ab5a8909e780e595d10 -- This is an automated message from the

[GitHub] [airflow] potiuk commented on a diff in pull request #31731: Google provider docstring improvements

2023-06-16 Thread via GitHub
potiuk commented on code in PR #31731: URL: https://github.com/apache/airflow/pull/31731#discussion_r1232237312 ## airflow/providers/google/cloud/hooks/bigquery.py: ## @@ -1716,10 +1690,10 @@ def run_load( by one or more columns. BigQuery supports clustering for bot

[GitHub] [airflow] potiuk commented on a diff in pull request #31731: Google provider docstring improvements

2023-06-16 Thread via GitHub
potiuk commented on code in PR #31731: URL: https://github.com/apache/airflow/pull/31731#discussion_r1232236952 ## airflow/providers/google/cloud/hooks/bigquery.py: ## @@ -618,10 +610,10 @@ def create_external_table( :param labels: A dictionary containing labels for the

[GitHub] [airflow] serrovsky commented on a diff in pull request #31955: chore(script): add option to delete logs folders on clean-up

2023-06-16 Thread via GitHub
serrovsky commented on code in PR #31955: URL: https://github.com/apache/airflow/pull/31955#discussion_r1232235026 ## scripts/docker/clean-logs.sh: ## @@ -21,19 +21,30 @@ set -euo pipefail readonly DIRECTORY="${AIRFLOW_HOME:-/usr/local/airflow}" readonly RETENTION="${AIRFLOW

[GitHub] [airflow] potiuk commented on pull request #31798: fix spark-kubernetes-operator compatibility

2023-06-16 Thread via GitHub
potiuk commented on PR #31798: URL: https://github.com/apache/airflow/pull/31798#issuecomment-1594625593 Sorry, Sorry, it was **PURELY** accidental :). I usually write briefly why I am closing stuff (sometimes maybe in a harsh way but I usually tell why ;) -- This is an automated message

[GitHub] [airflow] jherrmannNetfonds commented on pull request #31798: fix spark-kubernetes-operator compatibility

2023-06-16 Thread via GitHub
jherrmannNetfonds commented on PR #31798: URL: https://github.com/apache/airflow/pull/31798#issuecomment-1594608832 Please consider reopening the PR, since 2.6.x came with braking changes for the `SparkKubernetesOperator` compared to 2.5.x which would be "fixed" in this PR. With 2.6.x the `

[GitHub] [airflow] blcksrx commented on pull request #31798: fix spark-kubernetes-operator compatibility

2023-06-16 Thread via GitHub
blcksrx commented on PR #31798: URL: https://github.com/apache/airflow/pull/31798#issuecomment-1594577291 @potiuk Why closing PR? With a try catch it would be done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [airflow] potiuk commented on pull request #31798: fix spark-kubernetes-operator compatibility

2023-06-16 Thread via GitHub
potiuk commented on PR #31798: URL: https://github.com/apache/airflow/pull/31798#issuecomment-1594573389 > > > @jherrmannNetfonds Good catch buddy. Since with `watch=True` this operator is responsible for the execution, logs and states of the task, I believe in that case the pod/job should

[GitHub] [airflow] potiuk closed pull request #31798: fix spark-kubernetes-operator compatibility

2023-06-16 Thread via GitHub
potiuk closed pull request #31798: fix spark-kubernetes-operator compatibility URL: https://github.com/apache/airflow/pull/31798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] syedahsn opened a new pull request, #31960: Refactor Eks Create Cluster Operator code

2023-06-16 Thread via GitHub
syedahsn opened a new pull request, #31960: URL: https://github.com/apache/airflow/pull/31960 The `EksCreateClusterOperator` is capable of capable of creating a Fargate Profile or Nodegroup that is associated with the newly created cluster. The code for creating the Fargate Profile and Node

[GitHub] [airflow] potiuk commented on pull request #31798: fix spark-kubernetes-operator compatibility

2023-06-16 Thread via GitHub
potiuk commented on PR #31798: URL: https://github.com/apache/airflow/pull/31798#issuecomment-1594572974 > > @jherrmannNetfonds Good catch buddy. Since with `watch=True` this operator is responsible for the execution, logs and states of the task, I believe in that case the pod/job should be

[GitHub] [airflow] raphaelauv opened a new pull request, #31959: bump : git-sync v4

2023-06-16 Thread via GitHub
raphaelauv opened a new pull request, #31959: URL: https://github.com/apache/airflow/pull/31959 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [airflow] potiuk commented on issue #31957: Airflow Observability Improvement Request

2023-06-16 Thread via GitHub
potiuk commented on issue #31957: URL: https://github.com/apache/airflow/issues/31957#issuecomment-1594564595 Feel free to add those metrics. Assigned you. Note that we already have "open-telemetry" added in Airlfow `main` and it's being worked on and improved, so you might want to cc: @fer

[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #31772: Refactor Sqlalchemy queries to 2.0 styles (Part 2)

2023-06-16 Thread via GitHub
ephraimbuddy commented on code in PR #31772: URL: https://github.com/apache/airflow/pull/31772#discussion_r1232107795 ## airflow/models/dag.py: ## @@ -3518,7 +3541,7 @@ def deactivate_deleted_dags(cls, alive_dag_filelocs: list[str], session=NEW_SESS """ log.de

  1   2   >