Re: [PR] Fix stacklevel in warnings.warn into the providers [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on code in PR #36831: URL: https://github.com/apache/airflow/pull/36831#discussion_r1454889462 ## airflow/providers/google/cloud/links/dataproc.py: ## @@ -76,10 +80,6 @@ class DataprocLink(BaseOperatorLink): This link is deprecated. """ -wa

Re: [PR] Fix assignment of template field in `__init__` in `vertex_ai` [airflow]

2024-01-17 Thread via GitHub
romsharon98 commented on code in PR #36789: URL: https://github.com/apache/airflow/pull/36789#discussion_r1454899438 ## airflow/providers/google/cloud/operators/vertex_ai/custom_job.py: ## @@ -445,9 +445,19 @@ def __init__( self, *, command: Sequence[s

Re: [PR] Fix stacklevel in warnings.warn into the providers [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on code in PR #36831: URL: https://github.com/apache/airflow/pull/36831#discussion_r1454905728 ## airflow/providers/google/cloud/links/dataproc.py: ## @@ -76,10 +80,6 @@ class DataprocLink(BaseOperatorLink): This link is deprecated. """ -wa

Re: [PR] add impersonation_chain support when calling Bigquery Operators in deferrable mode [airflow]

2024-01-17 Thread via GitHub
m1racoli commented on code in PR #36341: URL: https://github.com/apache/airflow/pull/36341#discussion_r1454943542 ## airflow/providers/google/cloud/hooks/bigquery.py: ## @@ -3247,6 +3259,36 @@ def _format_schema_for_description(schema: dict) -> list: return description

Re: [PR] add impersonation_chain support when calling Bigquery Operators in deferrable mode [airflow]

2024-01-17 Thread via GitHub
m1racoli commented on code in PR #36341: URL: https://github.com/apache/airflow/pull/36341#discussion_r1454943542 ## airflow/providers/google/cloud/hooks/bigquery.py: ## @@ -3247,6 +3259,36 @@ def _format_schema_for_description(schema: dict) -> list: return description

Re: [PR] avoid retrying after KubernetesPodOperator has been marked as failed [airflow]

2024-01-17 Thread via GitHub
Lee-W commented on code in PR #36749: URL: https://github.com/apache/airflow/pull/36749#discussion_r1454952681 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -826,7 +842,11 @@ def on_kill(self) -> None: } if self.termination_grace_period i

[PR] Fix deprecations into the GCP Dataproc links [airflow]

2024-01-17 Thread via GitHub
Taragolis opened a new pull request, #36834: URL: https://github.com/apache/airflow/pull/36834 Related: - #31895 - https://github.com/apache/airflow/pull/36831#discussion_r1454659677 `warnings.warn` raised every time when module `airflow.providers.google.cloud.li

Re: [I] Status of testing of Apache Airflow 2.8.1rc1 [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on issue #36808: URL: https://github.com/apache/airflow/issues/36808#issuecomment-1895362241 - https://github.com/apache/airflow/pull/36281 Check different component under `pendulum 3` (default in constraints) and pendulum 2.1.2, can't find any noticeable side effe

Re: [PR] add impersonation_chain support when calling Bigquery Operators in deferrable mode [airflow]

2024-01-17 Thread via GitHub
m1racoli commented on code in PR #36341: URL: https://github.com/apache/airflow/pull/36341#discussion_r1454943542 ## airflow/providers/google/cloud/hooks/bigquery.py: ## @@ -3247,6 +3259,36 @@ def _format_schema_for_description(schema: dict) -> list: return description

Re: [PR] add impersonation_chain support when calling Bigquery Operators in deferrable mode [airflow]

2024-01-17 Thread via GitHub
m1racoli commented on code in PR #36341: URL: https://github.com/apache/airflow/pull/36341#discussion_r1455033407 ## airflow/providers/google/cloud/hooks/bigquery.py: ## @@ -3247,6 +3259,36 @@ def _format_schema_for_description(schema: dict) -> list: return description

Re: [PR] avoid retrying after KubernetesPodOperator has been marked as failed [airflow]

2024-01-17 Thread via GitHub
Lee-W commented on code in PR #36749: URL: https://github.com/apache/airflow/pull/36749#discussion_r1455038360 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -573,10 +575,17 @@ def execute_sync(self, context: Context): self.pod, istio_enabled, self

[PR] add Lineas & Infofarm to the INTHEWILD.MD file [airflow]

2024-01-17 Thread via GitHub
kobethuwis opened a new pull request, #36835: URL: https://github.com/apache/airflow/pull/36835 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelin

[I] SqlSensor doesn't support BiqQuery using google_cloud_platform connection [airflow]

2024-01-17 Thread via GitHub
nenkie76 opened a new issue, #36836: URL: https://github.com/apache/airflow/issues/36836 ### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.7.3 ### What happened? `SqlSensor` from

Re: [I] ShortCircuitOperator does not properly retry when python_callable exits unexpectedly [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on issue #36809: URL: https://github.com/apache/airflow/issues/36809#issuecomment-1895499822 I check on the `main` and this kind of bug exists. However it not only affects `ShortCircuitOperator`, I reproduced on regular task flow operators ```python from __fu

[I] Title: Error When Using Multiple DataSets as Triggers in Apache AirFlow [airflow]

2024-01-17 Thread via GitHub
SoftDed opened a new issue, #36837: URL: https://github.com/apache/airflow/issues/36837 ### Apache Airflow version 2.8.0 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When using two or more DataSets as triggers, an er

Re: [I] Title: Error When Using Multiple DataSets as Triggers in Apache AirFlow [airflow]

2024-01-17 Thread via GitHub
boring-cyborg[bot] commented on issue #36837: URL: https://github.com/apache/airflow/issues/36837#issuecomment-1895502218 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [PR] add Lineas & Infofarm to the INTHEWILD.MD file [airflow]

2024-01-17 Thread via GitHub
kobethuwis closed pull request #36835: add Lineas & Infofarm to the INTHEWILD.MD file URL: https://github.com/apache/airflow/pull/36835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Deprecate AutoMLTrainModelOperator for Vision and Video [airflow]

2024-01-17 Thread via GitHub
VladaZakharova commented on PR #36473: URL: https://github.com/apache/airflow/pull/36473#issuecomment-1895523166 @eladkal Hi! Can you please check the PR? Thanks! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] add Lineas & Infofarm to the INTHEWILD.MD file [airflow]

2024-01-17 Thread via GitHub
Taragolis merged PR #36835: URL: https://github.com/apache/airflow/pull/36835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

(airflow) branch main updated: add Lineas & Infofarm to the INTHEWILD.MD file (#36835)

2024-01-17 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 50b719f7c5 add Lineas & Infofarm to the INTHEWIL

[I] DatabricksSQLOperator struggles with parsing the data [airflow]

2024-01-17 Thread via GitHub
aru-trackunit opened a new issue, #36838: URL: https://github.com/apache/airflow/issues/36838 ### Apache Airflow Provider(s) databricks ### Versions of Apache Airflow Providers The error has not been present in version `apache-airflow-providers-databricks==4.7.0` I up

Re: [PR] add deferrable mode to RedshiftDataOperator [airflow]

2024-01-17 Thread via GitHub
Lee-W commented on code in PR #36586: URL: https://github.com/apache/airflow/pull/36586#discussion_r1455238325 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -201,3 +217,56 @@ def get_table_primary_key( break return pk_columns or None +

Re: [PR] add deferrable mode to RedshiftDataOperator [airflow]

2024-01-17 Thread via GitHub
Lee-W commented on code in PR #36586: URL: https://github.com/apache/airflow/pull/36586#discussion_r1455237630 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -201,3 +217,56 @@ def get_table_primary_key( break return pk_columns or None +

[I] DatabricksSQLOperator struggles with parsing the data [airflow]

2024-01-17 Thread via GitHub
aru-trackunit opened a new issue, #36839: URL: https://github.com/apache/airflow/issues/36839 ### Apache Airflow Provider(s) databricks ### Versions of Apache Airflow Providers The error has not been present in version apache-airflow-providers-databricks==4.7.0 I upgr

Re: [I] DatabricksSQLOperator struggles with parsing the data [airflow]

2024-01-17 Thread via GitHub
aru-trackunit commented on issue #36838: URL: https://github.com/apache/airflow/issues/36838#issuecomment-1895586081 There is also another ticket that is related to the similar code path: https://github.com/apache/airflow/issues/36839 -- This is an automated message from the Apache Git

Re: [PR] Secret masker ignores passwords with special chars [airflow]

2024-01-17 Thread via GitHub
aritra24 commented on PR #36692: URL: https://github.com/apache/airflow/pull/36692#issuecomment-1895595089 Added a secret_masker test which tests that the logs do get redacted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

(airflow) branch main updated: Correcting the ETIMEDOUT error code and its usages (#36830)

2024-01-17 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 7a814f1403 Correcting the ETIMEDOUT error code and

Re: [PR] Correcting the ETIMEDOUT error code and its usages [airflow]

2024-01-17 Thread via GitHub
potiuk merged PR #36830: URL: https://github.com/apache/airflow/pull/36830 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[PR] Add deferrable param in FileSensor [airflow]

2024-01-17 Thread via GitHub
pankajastro opened a new pull request, #36840: URL: https://github.com/apache/airflow/pull/36840 Look like we have trigger implementation for the filesystem but the defer mechanism is missing in the sensor. this PR introduce `deferrable` param in FileSensor.

[PR] Use lax `atlassian-python-api` limitation [airflow]

2024-01-17 Thread via GitHub
Taragolis opened a new pull request, #36841: URL: https://github.com/apache/airflow/pull/36841 related: https://github.com/apache/airflow/pull/36815 Locally run tests with `3.41.7` and error gone. Keep `beautifulsoup4` in provider requirements just in case because it

(airflow) branch constraints-main updated: Updating constraints. Github run id:7554974126

2024-01-17 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 8c14950e64 Updating con

Re: [PR] Implement AthenaSQLHook [airflow]

2024-01-17 Thread via GitHub
flolas commented on PR #36171: URL: https://github.com/apache/airflow/pull/36171#issuecomment-1895715074 @eladkal Tests seems good now 👍 , can you review again the code? I made some changes to address @Taragolis comments. -- This is an automated message from the Apache Git Service. To res

[I] Triggering DAG with parameters is mandatory when show_trigger_form_if_no_params is enabled [airflow]

2024-01-17 Thread via GitHub
csp33 opened a new issue, #36843: URL: https://github.com/apache/airflow/issues/36843 ### Apache Airflow version 2.8.0 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When setting `show_trigger_form_if_no_params` to Tru

Re: [PR] Fix stacklevel in warnings.warn into the providers [airflow]

2024-01-17 Thread via GitHub
potiuk commented on code in PR #36831: URL: https://github.com/apache/airflow/pull/36831#discussion_r1455632802 ## airflow/providers/celery/executors/celery_executor_utils.py: ## @@ -88,7 +88,8 @@ def _get_celery_app() -> Celery: "Change it to `airflow.providers.ce

Re: [PR] Fix stacklevel in warnings.warn into the providers [airflow]

2024-01-17 Thread via GitHub
potiuk commented on code in PR #36831: URL: https://github.com/apache/airflow/pull/36831#discussion_r1455634916 ## airflow/providers/microsoft/azure/hooks/adx.py: ## @@ -142,7 +142,9 @@ def warn_if_collison(key, backcompat_key): warnings.warn(

Re: [PR] Deprecate AutoMLTrainModelOperator for Vision and Video [airflow]

2024-01-17 Thread via GitHub
potiuk merged PR #36473: URL: https://github.com/apache/airflow/pull/36473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Deprecate AutoMLTrainModelOperator for Vision and Video (#36473)

2024-01-17 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 437d4e4467 Deprecate AutoMLTrainModelOperator for V

Re: [PR] Fix successful Apache Druid task submissions reported as failed [airflow]

2024-01-17 Thread via GitHub
listik commented on PR #36813: URL: https://github.com/apache/airflow/pull/36813#issuecomment-1895892147 > Could you please add a unit test covering it? Hey @potiuk, I thought [this unit test](https://github.com/apache/airflow/blob/7a814f1403752d42d309f79010d9214083eda762/tests/provid

Re: [PR] Fix stacklevel in warnings.warn into the providers [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on PR #36831: URL: https://github.com/apache/airflow/pull/36831#issuecomment-1895920129 Missing `stacklevel` could be prohibited resolved by https://docs.astral.sh/ruff/rules/no-explicit-stacklevel/ by I do not any way how to ban `stacklevel=1` rather than write somethin

Re: [PR] Add deferrable param in FileSensor [airflow]

2024-01-17 Thread via GitHub
Lee-W commented on code in PR #36840: URL: https://github.com/apache/airflow/pull/36840#discussion_r1455735439 ## airflow/sensors/filesystem.py: ## @@ -75,3 +92,20 @@ def poke(self, context: Context): if files: return True return Fa

Re: [PR] Implement AthenaSQLHook [airflow]

2024-01-17 Thread via GitHub
eladkal commented on PR #36171: URL: https://github.com/apache/airflow/pull/36171#issuecomment-1896051684 :tada: :tada::tada: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

(airflow) branch main updated: AthenaSqlHook implementation (#36171)

2024-01-17 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 666127238b AthenaSqlHook implementation (#36171) 6

Re: [I] Add AthenaSqlHook based on PyAthena [airflow]

2024-01-17 Thread via GitHub
eladkal closed issue #34823: Add AthenaSqlHook based on PyAthena URL: https://github.com/apache/airflow/issues/34823 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Implement AthenaSQLHook [airflow]

2024-01-17 Thread via GitHub
eladkal merged PR #36171: URL: https://github.com/apache/airflow/pull/36171 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

Re: [PR] Fix deprecations into the GCP Dataproc links [airflow]

2024-01-17 Thread via GitHub
potiuk commented on code in PR #36834: URL: https://github.com/apache/airflow/pull/36834#discussion_r1455909489 ## airflow/providers/google/cloud/links/dataproc.py: ## @@ -159,6 +168,13 @@ def get_link( else "" ) +def __attrs_post_init__(self): Revie

Re: [I] Generate a mermaid diagram for the database ERD schema [airflow]

2024-01-17 Thread via GitHub
potiuk commented on issue #36842: URL: https://github.com/apache/airflow/issues/36842#issuecomment-1896097998 Just a comment - small problem with that is that we would have to find a way how t generate this mermaid diagram from our model or the DB. The current ERD diagram is generated strai

Re: [I] SqlSensor doesn't support BiqQuery using google_cloud_platform connection [airflow]

2024-01-17 Thread via GitHub
potiuk closed issue #36836: SqlSensor doesn't support BiqQuery using google_cloud_platform connection URL: https://github.com/apache/airflow/issues/36836 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] Update faq.rst with (hopefully) clearer description of start_date [airflow]

2024-01-17 Thread via GitHub
RNHTTR opened a new pull request, #36846: URL: https://github.com/apache/airflow/pull/36846 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)*

(airflow) branch constraints-main updated: Updating constraints. Github run id:7558014654

2024-01-17 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new c67f126146 Updating con

[PR] Create GKEStartJobOperator and KubernetesJobOperator [airflow]

2024-01-17 Thread via GitHub
MaksYermak opened a new pull request, #36847: URL: https://github.com/apache/airflow/pull/36847 In this PR I have created operators for starting Kubernetes Job in clusters(ordinary and GKE). --- **^ Add meaningful description above** Read the **[Pull Request

Re: [I] Copy button on a Dag run's config copies only one run config [airflow]

2024-01-17 Thread via GitHub
boring-cyborg[bot] commented on issue #36848: URL: https://github.com/apache/airflow/issues/36848#issuecomment-1896167838 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[I] Copy button on a Dag run's config copies only one run config [airflow]

2024-01-17 Thread via GitHub
anteverse opened a new issue, #36848: URL: https://github.com/apache/airflow/issues/36848 ### Apache Airflow version 2.8.0 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Hi, I believe I hit a case where the run config

Re: [PR] Fix deprecations into the GCP Dataproc links [airflow]

2024-01-17 Thread via GitHub
hussein-awala merged PR #36834: URL: https://github.com/apache/airflow/pull/36834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

(airflow) branch main updated: Fix deprecations into the GCP Dataproc links (#36834)

2024-01-17 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 1ea623a834 Fix deprecations into the GCP Data

Re: [PR] Change default `parquet_row_group_size` in `BaseSQLToGCSOperator` [airflow]

2024-01-17 Thread via GitHub
renzepost commented on PR #36817: URL: https://github.com/apache/airflow/pull/36817#issuecomment-1896177270 Ah, got it! I've added a more verbose description in the changelog. Let me know if I missed anything or need to change the wording. -- This is an automated message from the Apache G

Re: [PR] Create GKEStartJobOperator and KubernetesJobOperator [airflow]

2024-01-17 Thread via GitHub
eladkal commented on PR #36847: URL: https://github.com/apache/airflow/pull/36847#issuecomment-1896193155 There was a discussion about this capability in https://github.com/apache/airflow/issues/17490 -- This is an automated message from the Apache Git Service. To respond to the message,

[PR] feat: full support for google credentials in gcloud-aio clients [airflow]

2024-01-17 Thread via GitHub
m1racoli opened a new pull request, #36849: URL: https://github.com/apache/airflow/pull/36849 The class CredentialsToken implements the ability to generate access tokens to be used in gcloud-aio clients from Google credentials objects provided by instances of Google Cloud hooks. With

Re: [PR] add impersonation_chain support when calling Bigquery Operators in deferrable mode [airflow]

2024-01-17 Thread via GitHub
m1racoli commented on PR #36341: URL: https://github.com/apache/airflow/pull/36341#issuecomment-1896215415 Based on my review comments I've taken another attempt on this in #36849. That one should make it easier to use credentials from hooks in gcloud-aio clients without much extra work.

Re: [PR] Create GKEStartJobOperator and KubernetesJobOperator [airflow]

2024-01-17 Thread via GitHub
MaksYermak commented on PR #36847: URL: https://github.com/apache/airflow/pull/36847#issuecomment-1896225619 > Could you merge/rebase main? Sure, I have rebased the branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Change default `parquet_row_group_size` in `BaseSQLToGCSOperator` [airflow]

2024-01-17 Thread via GitHub
potiuk commented on PR #36817: URL: https://github.com/apache/airflow/pull/36817#issuecomment-1896225993 Nice -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Fix webserver always redirecting to home page if user was not logged in [airflow]

2024-01-17 Thread via GitHub
vincbeck merged PR #36833: URL: https://github.com/apache/airflow/pull/36833 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

Re: [I] Webserver redirects to main page when user is not logged in [airflow]

2024-01-17 Thread via GitHub
vincbeck closed issue #36832: Webserver redirects to main page when user is not logged in URL: https://github.com/apache/airflow/issues/36832 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

(airflow) branch main updated (1ea623a834 -> 3266e94a4d)

2024-01-17 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 1ea623a834 Fix deprecations into the GCP Dataproc links (#36834) add 3266e94a4d Fix webserver always redirecting

Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]

2024-01-17 Thread via GitHub
vincbeck commented on PR #36828: URL: https://github.com/apache/airflow/pull/36828#issuecomment-1896244392 > > Can you add the test cases for this? > > Absolutely, I just have to find out how to test `on_kill`... It's on my todo list. Will dig into the AWS mock setup... You can

Re: [PR] Add extra operator links for EMR Serverless [airflow]

2024-01-17 Thread via GitHub
dacort commented on PR #34225: URL: https://github.com/apache/airflow/pull/34225#issuecomment-1896255911 > How does this work in deferrable mode? `persist_links` is called before the `self.defer` call is triggered for `EmrServerlessStartJobTrigger`. I'm not too familiar with deferrabl

Re: [PR] Index optimized fast path to avoid slower groupby queryplan [airflow]

2024-01-17 Thread via GitHub
dstandish commented on code in PR #33242: URL: https://github.com/apache/airflow/pull/33242#discussion_r1456148944 ## airflow/models/dag.py: ## @@ -3068,21 +3068,45 @@ def bulk_write_to_db( # Skip these queries entirely if no DAGs can be scheduled to save time.

[PR] check query status before SnowflakeSqlApiOperator execute in deferrable mode [airflow]

2024-01-17 Thread via GitHub
vatsrahul1001 opened a new pull request, #36850: URL: https://github.com/apache/airflow/pull/36850 This PR addresses checking the status of the query before deferring the task to the triggerer. --- **^ Add meaningful description above** Read the **[Pu

Re: [PR] feat: add Yandex Cloud Lockbox secrets backend [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on code in PR #36449: URL: https://github.com/apache/airflow/pull/36449#discussion_r1456058203 ## airflow/providers/yandex/utils/user_agent.py: ## @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license ag

Re: [I] Copy button on a Dag run's config copies only one run config [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on issue #36848: URL: https://github.com/apache/airflow/issues/36848#issuecomment-1896313580 I guess it similar to https://github.com/apache/airflow/issues/34846 and might be even fixed by https://github.com/apache/airflow/pull/34904 -- This is an automated message fro

Re: [I] Copy button on a Dag run's config copies only one run config [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on issue #36848: URL: https://github.com/apache/airflow/issues/36848#issuecomment-1896327316 Well nope, it still persists in current `main` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] Use lax `atlassian-python-api` limitation [airflow]

2024-01-17 Thread via GitHub
Taragolis merged PR #36841: URL: https://github.com/apache/airflow/pull/36841 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

(airflow) branch main updated: Use lax `atlassian-python-api` limitation (#36841)

2024-01-17 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new c76d6e58f0 Use lax `atlassian-python-api` limita

Re: [PR] add deferrable mode to RedshiftDataOperator [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on code in PR #36586: URL: https://github.com/apache/airflow/pull/36586#discussion_r1456241727 ## airflow/providers/amazon/aws/triggers/redshift_data.py: ## @@ -0,0 +1,92 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

Re: [I] Generate a mermaid diagram for the database ERD schema [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on issue #36842: URL: https://github.com/apache/airflow/issues/36842#issuecomment-1896394352 I've found this project https://github.com/tedivm/paracelsus, which is pretty new and unfortunately provide only CLI -- This is an automated message from the Apache Git Service

Re: [PR] Add deferrable param in FileSensor [airflow]

2024-01-17 Thread via GitHub
pankajastro commented on code in PR #36840: URL: https://github.com/apache/airflow/pull/36840#discussion_r1456276483 ## airflow/sensors/filesystem.py: ## @@ -75,3 +92,20 @@ def poke(self, context: Context): if files: return True ret

Re: [PR] Index optimized fast path to avoid slower groupby queryplan [airflow]

2024-01-17 Thread via GitHub
joshowen commented on code in PR #33242: URL: https://github.com/apache/airflow/pull/33242#discussion_r1456302037 ## airflow/models/dag.py: ## @@ -3068,21 +3068,45 @@ def bulk_write_to_db( # Skip these queries entirely if no DAGs can be scheduled to save time.

Re: [PR] Change default `parquet_row_group_size` in `BaseSQLToGCSOperator` [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on code in PR #36817: URL: https://github.com/apache/airflow/pull/36817#discussion_r1456304434 ## airflow/providers/google/CHANGELOG.rst: ## @@ -26,6 +26,14 @@ Changelog - +The default value of ``parquet_row_group_size`` in ``BaseSQLToGCSOperator`

Re: [PR] Change default `parquet_row_group_size` in `BaseSQLToGCSOperator` [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on code in PR #36817: URL: https://github.com/apache/airflow/pull/36817#discussion_r1456304434 ## airflow/providers/google/CHANGELOG.rst: ## @@ -26,6 +26,14 @@ Changelog - +The default value of ``parquet_row_group_size`` in ``BaseSQLToGCSOperator`

Re: [PR] Index optimized fast path to avoid slower groupby queryplan [airflow]

2024-01-17 Thread via GitHub
joshowen commented on code in PR #33242: URL: https://github.com/apache/airflow/pull/33242#discussion_r1456302037 ## airflow/models/dag.py: ## @@ -3068,21 +3068,45 @@ def bulk_write_to_db( # Skip these queries entirely if no DAGs can be scheduled to save time.

Re: [PR] Index optimized fast path to avoid slower groupby queryplan [airflow]

2024-01-17 Thread via GitHub
joshowen commented on code in PR #33242: URL: https://github.com/apache/airflow/pull/33242#discussion_r1456302037 ## airflow/models/dag.py: ## @@ -3068,21 +3068,45 @@ def bulk_write_to_db( # Skip these queries entirely if no DAGs can be scheduled to save time.

Re: [PR] Index optimized fast path to avoid slower groupby queryplan [airflow]

2024-01-17 Thread via GitHub
dstandish commented on PR #33242: URL: https://github.com/apache/airflow/pull/33242#issuecomment-1896456594 hi @joshowen i moved the query builder to a function and added some testing for the query structure. please take a look. i changed the structure of the query slightly so that we don

Re: [PR] Add deferrable functionality to the AirbyteJobSensor and AirbyteTriggerSyncOperator [airflow]

2024-01-17 Thread via GitHub
pankajastro merged PR #36780: URL: https://github.com/apache/airflow/pull/36780 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airf

(airflow) branch main updated: Add deferrable functionality to the AirbyteJobSensor and AirbyteTriggerSyncOperator (#36780)

2024-01-17 Thread pankaj
This is an automated email from the ASF dual-hosted git repository. pankaj pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3806a63bfe Add deferrable functionality to the Airb

Re: [PR] Add deferrable functionality to the AirbyteJobSensor and AirbyteTriggerSyncOperator [airflow]

2024-01-17 Thread via GitHub
boring-cyborg[bot] commented on PR #36780: URL: https://github.com/apache/airflow/pull/36780#issuecomment-1896509986 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.

Re: [I] The task is stuck in a queued state forever in case of pod launch errors [airflow]

2024-01-17 Thread via GitHub
RNHTTR commented on issue #36403: URL: https://github.com/apache/airflow/issues/36403#issuecomment-1896554983 Hey @dirrao -- Wondering if you had a chance to open a PR for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] add deferrable mode to RedshiftDataOperator [airflow]

2024-01-17 Thread via GitHub
syedahsn commented on code in PR #36586: URL: https://github.com/apache/airflow/pull/36586#discussion_r1456412652 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -201,3 +216,54 @@ def get_table_primary_key( break return pk_columns or Non

Re: [PR] add deferrable mode to RedshiftDataOperator [airflow]

2024-01-17 Thread via GitHub
syedahsn commented on code in PR #36586: URL: https://github.com/apache/airflow/pull/36586#discussion_r1456419985 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -201,3 +216,54 @@ def get_table_primary_key( break return pk_columns or Non

Re: [I] Pgbouncer not set properly upon using data.metadataSecretName [airflow]

2024-01-17 Thread via GitHub
cdallaire81 commented on issue #33924: URL: https://github.com/apache/airflow/issues/33924#issuecomment-1896615681 I'm also stuck on this problem, any updates? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

(airflow) branch constraints-main updated: Updating constraints. Github run id:7560833867

2024-01-17 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 2483a1a5df Updating con

[PR] Fallback to default value if `[aws] cloudwatch_task_handler_json_serializer` not set [airflow]

2024-01-17 Thread via GitHub
Taragolis opened a new pull request, #36851: URL: https://github.com/apache/airflow/pull/36851 Pretty interesting bug initially reported in [Slack](https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1705515217540559) I was able to reproduce it locally if for some

Re: [PR] Fallback to default value if `[aws] cloudwatch_task_handler_json_serializer` not set [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on PR #36851: URL: https://github.com/apache/airflow/pull/36851#issuecomment-1896665726 Update tests, just figure out that `conf_vars` remove parameter if value set to `None` so my initial hack not required -- This is an automated message from the Apache Git Service. T

Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]

2024-01-17 Thread via GitHub
syedahsn commented on code in PR #36828: URL: https://github.com/apache/airflow/pull/36828#discussion_r1456470327 ## airflow/providers/amazon/aws/operators/ec2.py: ## @@ -183,22 +183,36 @@ def execute(self, context: Context): MaxCount=self.max_count, **

Re: [PR] Fallback to default value if `[aws] cloudwatch_task_handler_json_serializer` not set [airflow]

2024-01-17 Thread via GitHub
vincbeck commented on PR #36851: URL: https://github.com/apache/airflow/pull/36851#issuecomment-1896702643 > LGTM However, we should investigate to know why it doesn't load the default value Agree, why the condition [here](https://github.com/apache/airflow/blob/main/airflow/configura

Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]

2024-01-17 Thread via GitHub
AchimGaedkeLynker commented on code in PR #36828: URL: https://github.com/apache/airflow/pull/36828#discussion_r1456505130 ## airflow/providers/amazon/aws/operators/ec2.py: ## @@ -183,22 +183,36 @@ def execute(self, context: Context): MaxCount=self.max_count,

Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]

2024-01-17 Thread via GitHub
AchimGaedkeLynker commented on code in PR #36828: URL: https://github.com/apache/airflow/pull/36828#discussion_r1456505130 ## airflow/providers/amazon/aws/operators/ec2.py: ## @@ -183,22 +183,36 @@ def execute(self, context: Context): MaxCount=self.max_count,

Re: [PR] EC2 CreateInstance: terminate instances in on_kill [airflow]

2024-01-17 Thread via GitHub
AchimGaedkeLynker commented on code in PR #36828: URL: https://github.com/apache/airflow/pull/36828#discussion_r1456505130 ## airflow/providers/amazon/aws/operators/ec2.py: ## @@ -183,22 +183,36 @@ def execute(self, context: Context): MaxCount=self.max_count,

Re: [I] Support split_statements AthenaOperator [Amazon provider] [airflow]

2024-01-17 Thread via GitHub
flolas closed issue #34583: Support split_statements AthenaOperator [Amazon provider] URL: https://github.com/apache/airflow/issues/34583 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [WIP] Fix vault gcp auth [airflow]

2024-01-17 Thread via GitHub
fpopic commented on PR #35855: URL: https://github.com/apache/airflow/pull/35855#issuecomment-1896885029 reopen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [PR] Fallback to default value if `[aws] cloudwatch_task_handler_json_serializer` not set [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on PR #36851: URL: https://github.com/apache/airflow/pull/36851#issuecomment-1896936483 That is a good question, and unfortunetly I do not have an answer yet. I've just create simple snippet for reproduce it locally and debug purpouse ```python import temp

Re: [PR] Fallback to default value if `[aws] cloudwatch_task_handler_json_serializer` not set [airflow]

2024-01-17 Thread via GitHub
Taragolis commented on PR #36851: URL: https://github.com/apache/airflow/pull/36851#issuecomment-1897034776 This one load default providers config ```python with conf.make_sure_configuration_loaded(True): return conf._default_values._sections ```

  1   2   >