[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #32331: Only update crawler tags if present in config dict

2023-07-04 Thread via GitHub
utkarsharma2 commented on code in PR #32331: URL: https://github.com/apache/airflow/pull/32331#discussion_r1251581477 ## tests/providers/amazon/aws/hooks/test_glue_crawler.py: ## @@ -160,6 +160,18 @@ def test_remove_all_tags(self, mock_get_conn): ResourceArn=self.cr

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #32331: Only update crawler tags if present in config dict

2023-07-04 Thread via GitHub
utkarsharma2 commented on code in PR #32331: URL: https://github.com/apache/airflow/pull/32331#discussion_r1251581477 ## tests/providers/amazon/aws/hooks/test_glue_crawler.py: ## @@ -160,6 +160,18 @@ def test_remove_all_tags(self, mock_get_conn): ResourceArn=self.cr

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #32319: Make ODBCHook result JSON-serializable

2023-07-04 Thread via GitHub
utkarsharma2 commented on code in PR #32319: URL: https://github.com/apache/airflow/pull/32319#discussion_r1251604306 ## airflow/providers/odbc/hooks/odbc.py: ## @@ -65,6 +67,7 @@ def __init__( dsn: str | None = None, connect_kwargs: dict | None = None,

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #32319: Make ODBCHook result JSON-serializable

2023-07-04 Thread via GitHub
utkarsharma2 commented on code in PR #32319: URL: https://github.com/apache/airflow/pull/32319#discussion_r1251604927 ## tests/providers/odbc/hooks/test_odbc.py: ## @@ -19,31 +19,55 @@ import json import logging +from collections import namedtuple from unittest import mock

[GitHub] [airflow] Adaverse commented on a diff in pull request #32331: Only update crawler tags if present in config dict

2023-07-04 Thread via GitHub
Adaverse commented on code in PR #32331: URL: https://github.com/apache/airflow/pull/32331#discussion_r1251606258 ## tests/providers/amazon/aws/hooks/test_glue_crawler.py: ## @@ -160,6 +160,18 @@ def test_remove_all_tags(self, mock_get_conn): ResourceArn=self.crawle

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #32319: Make ODBCHook result JSON-serializable

2023-07-04 Thread via GitHub
utkarsharma2 commented on code in PR #32319: URL: https://github.com/apache/airflow/pull/32319#discussion_r1251608982 ## tests/providers/odbc/hooks/test_odbc.py: ## @@ -235,3 +259,15 @@ def test_sqlalchemy_scheme_extra(self): hook = self.get_hook(conn_params=dict(extra

[GitHub] [airflow] nathadfield commented on pull request #31414: Fix future DagRun rarely triggered by race conditions when max_active_runs reached its upper limit.

2023-07-04 Thread via GitHub
nathadfield commented on PR #31414: URL: https://github.com/apache/airflow/pull/31414#issuecomment-1619678941 @doiken Just picking up the thread here, your PR still has some failed static tests. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] uranusjr commented on a diff in pull request #31852: [Webserver] Use custom_operator_name instead of task_type in UI for taskinstance object

2023-07-04 Thread via GitHub
uranusjr commented on code in PR #31852: URL: https://github.com/apache/airflow/pull/31852#discussion_r1251616707 ## airflow/models/taskinstance.py: ## @@ -617,6 +619,11 @@ def prev_attempted_tries(self) -> int: def next_try_number(self) -> int: return self._try_nu

[GitHub] [airflow] uranusjr commented on a diff in pull request #31852: [Webserver] Use custom_operator_name instead of task_type in UI for taskinstance object

2023-07-04 Thread via GitHub
uranusjr commented on code in PR #31852: URL: https://github.com/apache/airflow/pull/31852#discussion_r1251617378 ## airflow/models/taskinstance.py: ## @@ -617,6 +619,11 @@ def prev_attempted_tries(self) -> int: def next_try_number(self) -> int: return self._try_nu

[GitHub] [airflow] phanikumv opened a new pull request, #32339: Refactor Sqlalchemy queries to 2.0 style (Part 4)

2023-07-04 Thread via GitHub
phanikumv opened a new pull request, #32339: URL: https://github.com/apache/airflow/pull/32339 This is a continuation of the effort to refactor the queries to sqlalchemy 2.0 related: https://github.com/apache/airflow/issues/28723 --- **^ Add meaningful descri

[GitHub] [airflow] Lee-W commented on pull request #31712: Add default_deferrable config

2023-07-04 Thread via GitHub
Lee-W commented on PR #31712: URL: https://github.com/apache/airflow/pull/31712#issuecomment-1619850204 @potiuk Sorry for taking so long. I added some descriptions to the [Deferrable Operators & Triggers](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferri

[GitHub] [airflow] raphaelauv commented on a diff in pull request #32310: Update python.rst with multi-Scheduler LocalExecutor info

2023-07-04 Thread via GitHub
raphaelauv commented on code in PR #32310: URL: https://github.com/apache/airflow/pull/32310#discussion_r1251739035 ## docs/apache-airflow/core-concepts/executor/local.rst: ## @@ -47,3 +47,12 @@ Arguably, :class:`~airflow.executors.sequential_executor.SequentialExecutor` cou p

[GitHub] [airflow] zbstof opened a new pull request, #32340: Update docs in databricks.py - we use 2.1 now

2023-07-04 Thread via GitHub
zbstof opened a new pull request, #32340: URL: https://github.com/apache/airflow/pull/32340 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for m

[GitHub] [airflow] uranusjr commented on pull request #32340: Update docs in databricks.py - we use 2.1 now

2023-07-04 Thread via GitHub
uranusjr commented on PR #32340: URL: https://github.com/apache/airflow/pull/32340#issuecomment-1619907544 Maybe we should drop the `api/X.Y` prefix entirely instead? This feels impossible to keep fresh indefinitely. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [airflow] potiuk commented on pull request #32340: Update docs in databricks.py - we use 2.1 now

2023-07-04 Thread via GitHub
potiuk commented on PR #32340: URL: https://github.com/apache/airflow/pull/32340#issuecomment-1619948239 > Maybe we should drop the `api/X.Y` prefix entirely instead? This feels impossible to keep fresh indefinitely. Yep. Good idea. -- This is an automated message from the Apache G

[GitHub] [airflow] Lee-W commented on a diff in pull request #31925: Install sqlalchemy-spanner package into Google provider

2023-07-04 Thread via GitHub
Lee-W commented on code in PR #31925: URL: https://github.com/apache/airflow/pull/31925#discussion_r1251815031 ## airflow/providers/google/cloud/hooks/spanner.py: ## @@ -70,6 +85,34 @@ def _get_client(self, project_id: str) -> Client: ) return self._client

[GitHub] [airflow] Cricktom commented on issue #32324: DAG run fails when leaf node task is skipped.

2023-07-04 Thread via GitHub
Cricktom commented on issue #32324: URL: https://github.com/apache/airflow/issues/32324#issuecomment-1619974269 @ephraimbuddy could you please suggest that what might be causing this? I am executing exactly the same DAG mentioned above but getting DAG run failure. Is there any configuratio

[GitHub] [airflow] Lee-W commented on a diff in pull request #32319: Make ODBCHook result JSON-serializable

2023-07-04 Thread via GitHub
Lee-W commented on code in PR #32319: URL: https://github.com/apache/airflow/pull/32319#discussion_r1251834471 ## airflow/providers/odbc/hooks/odbc.py: ## @@ -211,3 +215,9 @@ def get_sqlalchemy_connection( engine = self.get_sqlalchemy_engine(engine_kwargs=engine_kwargs)

[GitHub] [airflow] phanikumv closed pull request #32339: Refactor Sqlalchemy queries to 2.0 style (Part 4)

2023-07-04 Thread via GitHub
phanikumv closed pull request #32339: Refactor Sqlalchemy queries to 2.0 style (Part 4) URL: https://github.com/apache/airflow/pull/32339 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] potiuk opened a new pull request, #32341: Retry transactions on occasional deadlocks for rendered fields

2023-07-04 Thread via GitHub
potiuk opened a new pull request, #32341: URL: https://github.com/apache/airflow/pull/32341 This is a follow-up on #18616 where we introduced retries on the occassional deadlocks when rendered task fields have been deleted by parallel threads (this is not a real deadlock, it's because MySQL

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31596: Add Deferrable switch to SnowflakeSqlApiOperator

2023-07-04 Thread via GitHub
utkarsharma2 commented on code in PR #31596: URL: https://github.com/apache/airflow/pull/31596#discussion_r1251846823 ## airflow/providers/snowflake/hooks/snowflake_sql_api.py: ## @@ -254,3 +246,30 @@ def get_sql_api_query_status(self, query_id: str) -> dict[str, str | list[str

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31596: Add Deferrable switch to SnowflakeSqlApiOperator

2023-07-04 Thread via GitHub
utkarsharma2 commented on code in PR #31596: URL: https://github.com/apache/airflow/pull/31596#discussion_r1251847977 ## airflow/providers/snowflake/hooks/snowflake_sql_api.py: ## @@ -254,3 +246,30 @@ def get_sql_api_query_status(self, query_id: str) -> dict[str, str | list[str

[GitHub] [airflow] potiuk merged pull request #32317: Typo fix in `redirects.txt` in Amazon providers doc

2023-07-04 Thread via GitHub
potiuk merged PR #32317: URL: https://github.com/apache/airflow/pull/32317 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: typo fix (#32317)

2023-07-04 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 702209898e typo fix (#32317) 702209898e is describe

[GitHub] [airflow] Lee-W commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-07-04 Thread via GitHub
Lee-W commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1251857603 ## airflow/providers/databricks/operators/databricks.py: ## @@ -24,6 +24,8 @@ from logging import Logger from typing import TYPE_CHECKING, Any, Sequence +from datab

[GitHub] [airflow] pankajastro commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-07-04 Thread via GitHub
pankajastro commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1251869555 ## airflow/providers/databricks/hooks/databricks.py: ## @@ -135,9 +137,27 @@ def __init__( ) -> None: super().__init__(databricks_conn_id, timeout_s

[GitHub] [airflow] potiuk commented on issue #32295: Broken url for the source code pointing to example_dags under providers

2023-07-04 Thread via GitHub
potiuk commented on issue #32295: URL: https://github.com/apache/airflow/issues/32295#issuecomment-1620030406 Sure. It looks like it is. Needs a bit of investigation where it comes from but likely post-processing the HTML pages is indeed something that might be needed -- This is an autom

[GitHub] [airflow] pankajastro commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-07-04 Thread via GitHub
pankajastro commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1245638521 ## airflow/providers/databricks/operators/databricks.py: ## @@ -24,6 +24,8 @@ from logging import Logger from typing import TYPE_CHECKING, Any, Sequence +from

[GitHub] [airflow] phanikumv commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-07-04 Thread via GitHub
phanikumv commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1251874101 ## airflow/providers/databricks/operators/databricks.py: ## @@ -160,6 +162,142 @@ def get_link( return XCom.get_value(key=XCOM_RUN_PAGE_URL_KEY, ti_key=ti_

[GitHub] [airflow] pankajastro commented on a diff in pull request #32221: Add `DatabricksJobsCreateOperator`

2023-07-04 Thread via GitHub
pankajastro commented on code in PR #32221: URL: https://github.com/apache/airflow/pull/32221#discussion_r1245638521 ## airflow/providers/databricks/operators/databricks.py: ## @@ -24,6 +24,8 @@ from logging import Logger from typing import TYPE_CHECKING, Any, Sequence +from

[GitHub] [airflow] ephraimbuddy commented on issue #32158: on_*_callback unexpected behavior with `functools.partial`

2023-07-04 Thread via GitHub
ephraimbuddy commented on issue #32158: URL: https://github.com/apache/airflow/issues/32158#issuecomment-1620052480 > I just tested it on main branch, the tasks have the state `success`. > > However, I have the callback exception in the task log, and from Airflow documentation: >

[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #31852: [Webserver] Use custom_operator_name instead of task_type in UI for taskinstance object

2023-07-04 Thread via GitHub
utkarsharma2 commented on code in PR #31852: URL: https://github.com/apache/airflow/pull/31852#discussion_r1251891591 ## airflow/models/taskinstance.py: ## @@ -617,6 +619,11 @@ def prev_attempted_tries(self) -> int: def next_try_number(self) -> int: return self._tr

[GitHub] [airflow] potiuk merged pull request #32334: Add proxy_user template check

2023-07-04 Thread via GitHub
potiuk merged PR #32334: URL: https://github.com/apache/airflow/pull/32334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Add proxy_user template check (#32334)

2023-07-04 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 1c1dbd872d Add proxy_user template check (#32334) 1

[GitHub] [airflow] potiuk commented on pull request #32334: Add proxy_user template check

2023-07-04 Thread via GitHub
potiuk commented on PR #32334: URL: https://github.com/apache/airflow/pull/32334#issuecomment-1620100166 cc: @eladkal -> you might want to add this one too to the upcoming wave of providers if there is still time -- This is an automated message from the Apache Git Service. To respond to t

[GitHub] [airflow] potiuk commented on issue #31764: Logs disappearing when a task completes as failed and is up to retry

2023-07-04 Thread via GitHub
potiuk commented on issue #31764: URL: https://github.com/apache/airflow/issues/31764#issuecomment-1620126605 cc: @dstandish - I think you need to chime-in here, I am not sure what the right if should be here :) -- This is an automated message from the Apache Git Service. To respond to th

[GitHub] [airflow] avinashpandeshwar commented on issue #32106: GCSToBigQueryOperator and BigQueryToGCSOperator do not respect their project_id arguments

2023-07-04 Thread via GitHub
avinashpandeshwar commented on issue #32106: URL: https://github.com/apache/airflow/issues/32106#issuecomment-1620143493 @bhagany I have corrected my PR https://github.com/apache/airflow/pull/32232 so that hook's project_id is the fallback for both storage and compute of the bq_to_gcs and g

[GitHub] [airflow] avinashpandeshwar commented on pull request #32232: Fixing Issue - Provided project_id parameter not getting used to subm…

2023-07-04 Thread via GitHub
avinashpandeshwar commented on PR #32232: URL: https://github.com/apache/airflow/pull/32232#issuecomment-1620161120 @eladkal The PR now includes fixes to both bq_to_gcs and gcs_to_bq ( fixed in multiple places within the operator - regular table, external table, deferred runs, max_id_key ca

[GitHub] [airflow] potiuk opened a new pull request, #32343: Revert "Refactor Sqlalchemy queries to 2.0 style (Part 3) (#32177)"

2023-07-04 Thread via GitHub
potiuk opened a new pull request, #32343: URL: https://github.com/apache/airflow/pull/32343 This reverts commit 1065687ec6df2b9b3557e38a67e71f835796427f. --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache

[GitHub] [airflow] potiuk closed pull request #32343: Revert "Refactor Sqlalchemy queries to 2.0 style (Part 3) (#32177)"

2023-07-04 Thread via GitHub
potiuk closed pull request #32343: Revert "Refactor Sqlalchemy queries to 2.0 style (Part 3) (#32177)" URL: https://github.com/apache/airflow/pull/32343 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] tirkarthi opened a new issue, #32344: datetime.utcnow and datetime.utcfromtimestamp are deprecated in Python 3.12

2023-07-04 Thread via GitHub
tirkarthi opened a new issue, #32344: URL: https://github.com/apache/airflow/issues/32344 ### Apache Airflow version 2.6.2 ### What happened The usage of `datetime.utcnow` is deprecated in Python 3.12 and it can be replaced with `airflow.utils.timezone.utcnow` and a simi

[GitHub] [airflow] tirkarthi commented on issue #32344: datetime.utcnow and datetime.utcfromtimestamp are deprecated in Python 3.12

2023-07-04 Thread via GitHub
tirkarthi commented on issue #32344: URL: https://github.com/apache/airflow/issues/32344#issuecomment-1620181391 I am working on a PR for this and will create it once complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [airflow] potiuk commented on pull request #32343: Revert "Refactor Sqlalchemy queries to 2.0 style (Part 3) (#32177)"

2023-07-04 Thread via GitHub
potiuk commented on PR #32343: URL: https://github.com/apache/airflow/pull/32343#issuecomment-1620194560 Until it is somehow mitigated, just "full tests needed" shoudl be applied in cases like this one (sqlalchemy refactor) to test also mssql -- This is an automated message from the Apach

[GitHub] [airflow] JDarDagran opened a new pull request, #32345: postgres: deprecate `runtime_parameters` in favor of options in `hook_params`

2023-07-04 Thread via GitHub
JDarDagran opened a new pull request, #32345: URL: https://github.com/apache/airflow/pull/32345 Currently you need to pass `runtime_parameters` argument to explicitly set e.g. search_path in PostgresOperator. The solution IMO is not that good. It introduces use of `psycopg2.SQL` which is

[GitHub] [airflow] potiuk commented on pull request #32343: Revert "Refactor Sqlalchemy queries to 2.0 style (Part 3) (#32177)"

2023-07-04 Thread via GitHub
potiuk commented on PR #32343: URL: https://github.com/apache/airflow/pull/32343#issuecomment-1620205519 OK. It passed the tests that were failing in main. Merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [airflow] potiuk merged pull request #32343: Revert "Refactor Sqlalchemy queries to 2.0 style (Part 3) (#32177)"

2023-07-04 Thread via GitHub
potiuk merged PR #32343: URL: https://github.com/apache/airflow/pull/32343 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Revert "Refactor Sqlalchemy queries to 2.0 style (Part 3) (#32177)" (#32343)

2023-07-04 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3c3c337edd Revert "Refactor Sqlalchemy queries to 2

[GitHub] [airflow] potiuk commented on pull request #32177: Refactor Sqlalchemy queries to 2.0 style (Part 3)

2023-07-04 Thread via GitHub
potiuk commented on PR #32177: URL: https://github.com/apache/airflow/pull/32177#issuecomment-1620207244 As discussed in https://github.com/apache/airflow/pull/32343 - this one is reverted and needs to be reapplied with "full tests needed" :(. -- This is an automated message from the Apa

[GitHub] [airflow] potiuk commented on pull request #32339: Refactor Sqlalchemy queries to 2.0 style (Part 4)

2023-07-04 Thread via GitHub
potiuk commented on PR #32339: URL: https://github.com/apache/airflow/pull/32339#issuecomment-1620208515 applied "full tests needed" to that one as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] juhai commented on a diff in pull request #32278: Allow creating unique pod and hostname for KubernetesPodOperator from task job_id

2023-07-04 Thread via GitHub
juhai commented on code in PR #32278: URL: https://github.com/apache/airflow/pull/32278#discussion_r1252014543 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -158,6 +158,8 @@ class KubernetesPodOperator(BaseOperator): suffix if random_name_suffix is True)

[GitHub] [airflow] ephraimbuddy closed issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs

2023-07-04 Thread via GitHub
ephraimbuddy closed issue #28499: "Dependency already registered for DAG" warnings with tasks with multiple outputs URL: https://github.com/apache/airflow/issues/28499 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [airflow] amoghrajesh commented on a diff in pull request #31663: Logging from all containers in KubernetesOperatorPod

2023-07-04 Thread via GitHub
amoghrajesh commented on code in PR #31663: URL: https://github.com/apache/airflow/pull/31663#discussion_r1252030400 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -342,6 +348,9 @@ def __init__( self.cluster_context = cluster_context self.reattach

[GitHub] [airflow] pankajastro commented on a diff in pull request #32323: refresh connection if an exception is caught in "AzureDataFactory"

2023-07-04 Thread via GitHub
pankajastro commented on code in PR #32323: URL: https://github.com/apache/airflow/pull/32323#discussion_r1252031539 ## airflow/providers/microsoft/azure/triggers/data_factory.py: ## @@ -68,24 +70,38 @@ def serialize(self) -> tuple[str, dict[str, Any]]: async def run(self)

[GitHub] [airflow] pankajastro commented on a diff in pull request #32323: refresh connection if an exception is caught in "AzureDataFactory"

2023-07-04 Thread via GitHub
pankajastro commented on code in PR #32323: URL: https://github.com/apache/airflow/pull/32323#discussion_r1252034037 ## airflow/providers/microsoft/azure/triggers/data_factory.py: ## @@ -68,24 +70,38 @@ def serialize(self) -> tuple[str, dict[str, Any]]: async def run(self)

[GitHub] [airflow] mobuchowski closed pull request #32345: postgres: deprecate `runtime_parameters` in favor of options in `hook_params`

2023-07-04 Thread via GitHub
mobuchowski closed pull request #32345: postgres: deprecate `runtime_parameters` in favor of options in `hook_params` URL: https://github.com/apache/airflow/pull/32345 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [airflow] AlexisBRENON opened a new issue, #32346: Sub-hour scheduling without catchup result in unaligned data intervals

2023-07-04 Thread via GitHub
AlexisBRENON opened a new issue, #32346: URL: https://github.com/apache/airflow/issues/32346 ### Apache Airflow version 2.6.2 ### What happened I setup a DAG to run every 15 minutes, with a start date aligned on hour start, to have it scheduled with data interval like:

[GitHub] [airflow] potiuk commented on issue #32344: datetime.utcnow and datetime.utcfromtimestamp are deprecated in Python 3.12

2023-07-04 Thread via GitHub
potiuk commented on issue #32344: URL: https://github.com/apache/airflow/issues/32344#issuecomment-1620299500 Just be warned. We are using pendulum in some places. And in some not. This might have rather undesireable side effects when you replace those deprecated. Not sure how but I know @u

[GitHub] [airflow] potiuk merged pull request #32052: Disable default allowing the testing of connections in UI, API and CLI

2023-07-04 Thread via GitHub
potiuk merged PR #32052: URL: https://github.com/apache/airflow/pull/32052 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Disable default allowing the testing of connections in UI, API and CLI (#32052)

2023-07-04 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new e4c3ecf8ce Disable default allowing the testing of

[GitHub] [airflow] uranusjr commented on issue #32344: datetime.utcnow and datetime.utcfromtimestamp are deprecated in Python 3.12

2023-07-04 Thread via GitHub
uranusjr commented on issue #32344: URL: https://github.com/apache/airflow/issues/32344#issuecomment-1620319427 I talked to Paul Ganssle a while ago and these two specifically have few caveats, you basically need to use the version without `utc` and make sure to always attach an appropriate

[GitHub] [airflow] patgarz commented on issue #32158: on_*_callback unexpected behavior with `functools.partial`

2023-07-04 Thread via GitHub
patgarz commented on issue #32158: URL: https://github.com/apache/airflow/issues/32158#issuecomment-1620323789 Hey @ephraimbuddy , I concur with @hussein-awala --- it is fixed on `main`. I was only able to reproduce in 2.6.0, 2.6.1, and 2.6.2. Testing using `breeze` and the main I no longer

[GitHub] [airflow] JDarDagran opened a new pull request, #32347: Change default schema behaviour in SQLParser.

2023-07-04 Thread via GitHub
JDarDagran opened a new pull request, #32347: URL: https://github.com/apache/airflow/pull/32347 This PR changes where default database schema is used. It's no longer passed to SQL parser, instead it's used as fallback when any other method for finding schema fails (including calling informa

[GitHub] [airflow] potiuk commented on a diff in pull request #32309: Hide sensitive values from extra in connection edit form

2023-07-04 Thread via GitHub
potiuk commented on code in PR #32309: URL: https://github.com/apache/airflow/pull/32309#discussion_r1252087193 ## airflow/www/views.py: ## @@ -4819,8 +4835,21 @@ def process_form(self, form, is_created): # value isn't an empty string. i

[GitHub] [airflow] proofrock commented on issue #32332: Installation: pydantic.errors.PydanticUserError: A non-annotated attribute was detected: `dag_id = `.

2023-07-04 Thread via GitHub
proofrock commented on issue #32332: URL: https://github.com/apache/airflow/issues/32332#issuecomment-1620329508 I'll do that, @potiuk. Thanks for being such a nice project and people, it's really meaningful the way you treated me in this case, even if it was my fault (in good faith, of cou

[GitHub] [airflow] potiuk commented on issue #32158: on_*_callback unexpected behavior with `functools.partial`

2023-07-04 Thread via GitHub
potiuk commented on issue #32158: URL: https://github.com/apache/airflow/issues/32158#issuecomment-1620339430 > I assume that means it's fixed, but I suppose it could also represent a bug that's somehow added during packaging. You can run: ``` breeze release-management prepa

[GitHub] [airflow] potiuk commented on issue #32332: Installation: pydantic.errors.PydanticUserError: A non-annotated attribute was detected: `dag_id = `.

2023-07-04 Thread via GitHub
potiuk commented on issue #32332: URL: https://github.com/apache/airflow/issues/32332#issuecomment-1620342707 It's nice to hear warm words like that :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[airflow] branch main updated (e4c3ecf8ce -> d1625188ff)

2023-07-04 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from e4c3ecf8ce Disable default allowing the testing of connections in UI, API and CLI (#32052) add d1625188ff Remove

[GitHub] [airflow] potiuk merged pull request #32300: Remove outdated comment from chart values in the workers KEDA conf section

2023-07-04 Thread via GitHub
potiuk merged PR #32300: URL: https://github.com/apache/airflow/pull/32300 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] amoghrajesh commented on pull request #31615: Defining priorityClasses for airflow services and workers

2023-07-04 Thread via GitHub
amoghrajesh commented on PR #31615: URL: https://github.com/apache/airflow/pull/31615#issuecomment-1620348165 @hussein-awala the unit tests are ready. The PR is ready for review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] Lee-W commented on a diff in pull request #32323: refresh connection if an exception is caught in "AzureDataFactory"

2023-07-04 Thread via GitHub
Lee-W commented on code in PR #32323: URL: https://github.com/apache/airflow/pull/32323#discussion_r1252107812 ## airflow/providers/microsoft/azure/triggers/data_factory.py: ## @@ -68,24 +70,38 @@ def serialize(self) -> tuple[str, dict[str, Any]]: async def run(self) -> Asy

[GitHub] [airflow] vincbeck opened a new issue, #32348: Move sync-perm command

2023-07-04 Thread via GitHub
vincbeck opened a new issue, #32348: URL: https://github.com/apache/airflow/issues/32348 It has been brought up by Jarek in this [discussion](https://github.com/apache/airflow/discussions/32187#discussioncomment-6323078). I dont have much context on it but it seems we need to move this comm

[GitHub] [airflow] rzjfr opened a new pull request, #32349: Chart: add container lifecycle hooks

2023-07-04 Thread via GitHub
rzjfr opened a new pull request, #32349: URL: https://github.com/apache/airflow/pull/32349 This PR attempts to add possibility to add [lifecycle](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/) to containers. There are some use cases which could utiliz

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #32349: Chart: add container lifecycle hooks

2023-07-04 Thread via GitHub
boring-cyborg[bot] commented on PR #32349: URL: https://github.com/apache/airflow/pull/32349#issuecomment-1620411253 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] flinz commented on issue #32283: EmptyOperator in dynamically mapped TaskGroups does not respect upstream dependencies

2023-07-04 Thread via GitHub
flinz commented on issue #32283: URL: https://github.com/apache/airflow/issues/32283#issuecomment-1620413678 @pragnareddye here is a more stable test, that will fail reliably if the order of execution is not respected in each branch. ```python import time from datetime import dateti

[GitHub] [airflow] vincbeck commented on issue #32211: AIP-56 - Documentation

2023-07-04 Thread via GitHub
vincbeck commented on issue #32211: URL: https://github.com/apache/airflow/issues/32211#issuecomment-1620414399 Happy to. Though, I was thinking we should work on this task later in the project once most of the code will be written, it might be easier. What do you think? -- This is an au

[GitHub] [airflow] vincbeck merged pull request #32322: Bug fix GCSToS3Operator: avoid `ValueError` when `replace=False` with files already in S3

2023-07-04 Thread via GitHub
vincbeck merged PR #32322: URL: https://github.com/apache/airflow/pull/32322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: Bug fix GCSToS3Operator: avoid `ValueError` when `replace=False` with files already in S3 (#32322)

2023-07-04 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 575bf2f040 Bug fix GCSToS3Operator: avoid `ValueE

[GitHub] [airflow] vincbeck merged pull request #32305: Fix get_conn test for ImpalaHook

2023-07-04 Thread via GitHub
vincbeck merged PR #32305: URL: https://github.com/apache/airflow/pull/32305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: feat: Use 'use_ssl' parameter instead of 'ssl' + Use attributes of the patched function (#32305)

2023-07-04 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 836bdb29c0 feat: Use 'use_ssl' parameter instead

[GitHub] [airflow] mobuchowski merged pull request #32347: Change default schema behaviour in SQLParser.

2023-07-04 Thread via GitHub
mobuchowski merged PR #32347: URL: https://github.com/apache/airflow/pull/32347 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airf

[airflow] branch main updated: Change default schema behaviour in SQLParser. (#32347)

2023-07-04 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 65fad4affc Change default schema behaviour in

[GitHub] [airflow] pierrejeambrun commented on pull request #32303: Replace re with re2

2023-07-04 Thread via GitHub
pierrejeambrun commented on PR #32303: URL: https://github.com/apache/airflow/pull/32303#issuecomment-1620480956 @bolkedebruin Commit message has been updated with your suggestion. Explicit re2 import are now made when re2 is used. -- This is an automated message from the Apache Git

[GitHub] [airflow] patgarz commented on issue #32158: on_*_callback unexpected behavior with `functools.partial`

2023-07-04 Thread via GitHub
patgarz commented on issue #32158: URL: https://github.com/apache/airflow/issues/32158#issuecomment-1620511233 Thanks; I did some additional testing and I am confident that the issue is resolved in >=2.7: - Using `breeze` with `v2-6-test` branch produced results consistent with Airfl

[GitHub] [airflow] phanikumv opened a new pull request, #32350: Refactor Sqlalchemy queries to 2.0 style (Part 3)

2023-07-04 Thread via GitHub
phanikumv opened a new pull request, #32350: URL: https://github.com/apache/airflow/pull/32350 This is a continuation of the effort to refactor the queries to sqlalchemy 2.0 (Re-applying the revert done in https://github.com/apache/airflow/pull/32343 by applying the label `full tests neede

[GitHub] [airflow] Joffreybvn commented on a diff in pull request #32319: Make ODBCHook result JSON-serializable

2023-07-04 Thread via GitHub
Joffreybvn commented on code in PR #32319: URL: https://github.com/apache/airflow/pull/32319#discussion_r1252223599 ## airflow/providers/odbc/hooks/odbc.py: ## @@ -211,3 +215,9 @@ def get_sqlalchemy_connection( engine = self.get_sqlalchemy_engine(engine_kwargs=engine_kw

[GitHub] [airflow] ephraimbuddy opened a new pull request, #32351: Support setting dependencies for tasks called outside TaskGroup context manager

2023-07-04 Thread via GitHub
ephraimbuddy opened a new pull request, #32351: URL: https://github.com/apache/airflow/pull/32351 Currently, you must instantiate a classic operator or call a decorated operator inside the context manager before it will link up with the context manager. For example, tasks 1 and 2 below will

[GitHub] [airflow] Joffreybvn commented on a diff in pull request #32319: Make ODBCHook result JSON-serializable

2023-07-04 Thread via GitHub
Joffreybvn commented on code in PR #32319: URL: https://github.com/apache/airflow/pull/32319#discussion_r1252224880 ## airflow/providers/common/sql/hooks/sql.py: ## @@ -372,6 +372,15 @@ def run( else: return results +def _make_serializable(self, resul

[GitHub] [airflow] Joffreybvn commented on a diff in pull request #32319: Make ODBCHook result JSON-serializable

2023-07-04 Thread via GitHub
Joffreybvn commented on code in PR #32319: URL: https://github.com/apache/airflow/pull/32319#discussion_r1252223599 ## airflow/providers/odbc/hooks/odbc.py: ## @@ -211,3 +215,9 @@ def get_sqlalchemy_connection( engine = self.get_sqlalchemy_engine(engine_kwargs=engine_kw

[GitHub] [airflow] Joffreybvn commented on a diff in pull request #32319: Make ODBCHook result JSON-serializable

2023-07-04 Thread via GitHub
Joffreybvn commented on code in PR #32319: URL: https://github.com/apache/airflow/pull/32319#discussion_r1252228253 ## tests/providers/odbc/hooks/test_odbc.py: ## @@ -235,3 +259,15 @@ def test_sqlalchemy_scheme_extra(self): hook = self.get_hook(conn_params=dict(extra=j

[airflow] branch constraints-main updated: Updating constraints. Github run id:5456548032

2023-07-04 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 0e1f1059ac Updating con

[GitHub] [airflow] SridharCR commented on issue #32211: AIP-56 - Documentation

2023-07-04 Thread via GitHub
SridharCR commented on issue #32211: URL: https://github.com/apache/airflow/issues/32211#issuecomment-1620550143 Yeah sure, makes sense! will work on this, once the base code is merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [airflow] ferruzzi commented on issue #32164: Metrics - Encrypted OTel Endpoint?

2023-07-04 Thread via GitHub
ferruzzi commented on issue #32164: URL: https://github.com/apache/airflow/issues/32164#issuecomment-1620591786 No rush at all, thanks for taking it on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [airflow] vandonr-amz opened a new pull request, #32352: deprecate arbitrary parameter passing to RDS hook

2023-07-04 Thread via GitHub
vandonr-amz opened a new pull request, #32352: URL: https://github.com/apache/airflow/pull/32352 The way RDS operators are built allows passing arbitrary parameters to the hook via a kwarg named `hook_params`. However this is not documented anywhere, and the usefulness of the feature looks

[GitHub] [airflow] ferruzzi commented on issue #12771: Instrument Airflow with Opentracing/Opentelemetry

2023-07-04 Thread via GitHub
ferruzzi commented on issue #12771: URL: https://github.com/apache/airflow/issues/12771#issuecomment-1620596591 Airflow 2.7 will have full support for OTel metrics at feature parity with StatsD. Traces, Spans, and Logging are coming at a future date. If you have a dev branch checked

[airflow-site] branch add-announcement-for-dennis created (now 42063cbcf4)

2023-07-04 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch add-announcement-for-dennis in repository https://gitbox.apache.org/repos/asf/airflow-site.git at 42063cbcf4 Add Dennis' announcement This branch includes the following new commits:

[airflow-site] 01/01: Add Dennis' announcement

2023-07-04 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch add-announcement-for-dennis in repository https://gitbox.apache.org/repos/asf/airflow-site.git commit 42063cbcf42c2de397db6faf5a8fb965be49df54 Author: Jarek Potiuk AuthorDate: Tue Jul 4 20:15:38

[GitHub] [airflow-site] potiuk opened a new pull request, #826: Add Dennis' announcement

2023-07-04 Thread via GitHub
potiuk opened a new pull request, #826: URL: https://github.com/apache/airflow-site/pull/826 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

[GitHub] [airflow-site] potiuk merged pull request #826: Add Dennis' announcement

2023-07-04 Thread via GitHub
potiuk merged PR #826: URL: https://github.com/apache/airflow-site/pull/826 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

[airflow-site] branch main updated: Add Dennis' announcement (#826)

2023-07-04 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/main by this push: new e416fe9da5 Add Dennis' announcement (#826) e41

[GitHub] [airflow-client-java] harry1989 commented on pull request #1: WIP: airflow java client

2023-07-04 Thread via GitHub
harry1989 commented on PR #1: URL: https://github.com/apache/airflow-client-java/pull/1#issuecomment-1620610340 @potiuk I'm interested in this MR, we need this SDK to integrate with our applications,I'm willing to contribute, please let me know if there is anything that I can assist with.

  1   2   >