[GitHub] [airflow] vioao opened a new pull request, #32795: DruidHook add SQL-based task support

2023-07-24 Thread via GitHub
vioao opened a new pull request, #32795: URL: https://github.com/apache/airflow/pull/32795 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for mo

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #32795: DruidHook add SQL-based task support

2023-07-24 Thread via GitHub
boring-cyborg[bot] commented on PR #32795: URL: https://github.com/apache/airflow/pull/32795#issuecomment-1647353312 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] dzhigimont commented on pull request #32701: Fix logic of trigger rule one failed inside mapped group

2023-07-24 Thread via GitHub
dzhigimont commented on PR #32701: URL: https://github.com/apache/airflow/pull/32701#issuecomment-1647371922 > Why are the upstream tasks calculated incorrectly before the task is expanded? Task expansion happens on the task, why would it affect the upstream tasks? Apologies

[GitHub] [airflow] moiseenkov opened a new pull request, #32796: Add system test and docs for DataprocSubmitJobOperator (Trino job)

2023-07-24 Thread via GitHub
moiseenkov opened a new pull request, #32796: URL: https://github.com/apache/airflow/pull/32796 Add system test and docs for DataprocSubmitJobOperator (Trino job) Related issues: https://github.com/apache/airflow/issues/32407, https://github.com/apache/airflow/issues/22430, https://g

[GitHub] [airflow] bkossakowska opened a new pull request, #32797: Fix CloudFirestoreExportDatabaseOperator system test

2023-07-24 Thread via GitHub
bkossakowska opened a new pull request, #32797: URL: https://github.com/apache/airflow/pull/32797 Fix CloudFirestoreExportDatabaseOperator system test --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/ai

[GitHub] [airflow] bkossakowska closed pull request #32797: Fix CloudFirestoreExportDatabaseOperator system test

2023-07-24 Thread via GitHub
bkossakowska closed pull request #32797: Fix CloudFirestoreExportDatabaseOperator system test URL: https://github.com/apache/airflow/pull/32797 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [airflow] moiseenkov opened a new pull request, #32798: Add sustem test and docs for DataprocSubmitJobOperator (Presto job)

2023-07-24 Thread via GitHub
moiseenkov opened a new pull request, #32798: URL: https://github.com/apache/airflow/pull/32798 Add system test and docs for DataprocSubmitJobOperator (Presto job) Related issues: https://github.com/apache/airflow/issues/32407, https://github.com/apache/airflow/issues/22430, https://

[GitHub] [airflow] bbovenzi commented on a diff in pull request #31806: New gantt tab

2023-07-24 Thread via GitHub
bbovenzi commented on code in PR #31806: URL: https://github.com/apache/airflow/pull/31806#discussion_r1271930553 ## airflow/www/static/js/dag/details/index.tsx: ## @@ -134,6 +153,12 @@ const Details = ({ openGroupIds, onToggleGroups, hoveredTaskState }: Props) => { [setSe

[GitHub] [airflow] pateash commented on a diff in pull request #32795: DruidHook add SQL-based task support

2023-07-24 Thread via GitHub
pateash commented on code in PR #32795: URL: https://github.com/apache/airflow/pull/32795#discussion_r1271933402 ## airflow/providers/apache/druid/hooks/druid.py: ## @@ -96,14 +98,20 @@ def submit_indexing_job(self, json_index_spec: dict[str, Any] | str) -> None: req

[GitHub] [airflow] pateash commented on a diff in pull request #32795: DruidHook add SQL-based task support

2023-07-24 Thread via GitHub
pateash commented on code in PR #32795: URL: https://github.com/apache/airflow/pull/32795#discussion_r1271931468 ## airflow/providers/apache/druid/hooks/druid.py: ## @@ -96,14 +98,20 @@ def submit_indexing_job(self, json_index_spec: dict[str, Any] | str) -> None: req

[GitHub] [airflow] bbovenzi commented on pull request #31806: New gantt tab

2023-07-24 Thread via GitHub
bbovenzi commented on PR #31806: URL: https://github.com/apache/airflow/pull/31806#issuecomment-1647490534 > Looks great! Tested locally and seems nice. 🎉 > > I just have issues scrolling the gantt. (scroll bars are not working for me), also I have vertical scroll when there is plenty

[GitHub] [airflow] Lee-W commented on a diff in pull request #32786: Add TabularVttsSensor

2023-07-24 Thread via GitHub
Lee-W commented on code in PR #32786: URL: https://github.com/apache/airflow/pull/32786#discussion_r1271939625 ## airflow/providers/tabular/provider.yaml: ## @@ -31,6 +31,7 @@ versions: dependencies: - apache-airflow>=2.4.0 + - pyiceberg==0.4.0 Review Comment: Wonderi

[GitHub] [airflow] vioao commented on pull request #32795: DruidHook add SQL-based task support

2023-07-24 Thread via GitHub
vioao commented on PR #32795: URL: https://github.com/apache/airflow/pull/32795#issuecomment-1647498017 > As they're using different endpoints, I wonder whether we should create a new hook and operator instead of overriding the parameter. @Lee-W I am not sure about this. But I th

[GitHub] [airflow] Lee-W commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
Lee-W commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1271951686 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -53,6 +53,7 @@ def execute_query( database: str, sql: str | list[str], cluster_

[GitHub] [airflow] bbovenzi commented on a diff in pull request #32350: Refactor Sqlalchemy queries to 2.0 style (Part 3)

2023-07-24 Thread via GitHub
bbovenzi commented on code in PR #32350: URL: https://github.com/apache/airflow/pull/32350#discussion_r1271975386 ## airflow/www/views.py: ## @@ -2293,15 +2300,17 @@ def clear(self, *, session: Session = NEW_SESSION): # Lock the related dag runs to prevent from po

[GitHub] [airflow-client-python] almost-online opened a new pull request, #89: Update task_instance.py

2023-07-24 Thread via GitHub
almost-online opened a new pull request, #89: URL: https://github.com/apache/airflow-client-python/pull/89 Apache Airflow Version: [v2.6.3](https://pypi.python.org/pypi/apache-airflow/2.6.3) Fixing a `trigger` and a `triggerer_job` properties as nullable -- This is an autom

[GitHub] [airflow] pepijngriffioen commented on issue #30883: BranchPythonOperator skips downstream tasks for all mapped instances in TaskGroup mapping

2023-07-24 Thread via GitHub
pepijngriffioen commented on issue #30883: URL: https://github.com/apache/airflow/issues/30883#issuecomment-1647558256 Thanks for picking up this fix. Unfortunately we are experiencing the same problem in airflow 2.6.3, with expanding a taskgroup that contains a branch operator. I am not re

[GitHub] [airflow] ivica-k commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
ivica-k commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272000469 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -53,6 +53,7 @@ def execute_query( database: str, sql: str | list[str], cluste

[GitHub] [airflow] uranusjr commented on pull request #32798: Add sustem test and docs for DataprocSubmitJobOperator (Presto job)

2023-07-24 Thread via GitHub
uranusjr commented on PR #32798: URL: https://github.com/apache/airflow/pull/32798#issuecomment-1647579649 Can you amend and force-push to fix the typoe in the commit message? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[GitHub] [airflow] uranusjr commented on a diff in pull request #32781: Override base log folder by using task handler's base_log_folder

2023-07-24 Thread via GitHub
uranusjr commented on code in PR #32781: URL: https://github.com/apache/airflow/pull/32781#discussion_r1272016495 ## tests/utils/test_serve_logs.py: ## @@ -22,7 +22,9 @@ import jwt import pytest import time_machine +from py import path Review Comment: Please use `tmp_path

[GitHub] [airflow] Lee-W commented on pull request #32795: DruidHook add SQL-based task support

2023-07-24 Thread via GitHub
Lee-W commented on PR #32795: URL: https://github.com/apache/airflow/pull/32795#issuecomment-1647591195 @vioao Or maybe we can make the new hook a subclass of the original one? WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[GitHub] [airflow] Lee-W commented on a diff in pull request #32779: Fix hardcoded container name in remote logging option for Azure Blob Storage

2023-07-24 Thread via GitHub
Lee-W commented on code in PR #32779: URL: https://github.com/apache/airflow/pull/32779#discussion_r1272026282 ## airflow/providers/microsoft/azure/log/wasb_task_handler.py: ## @@ -148,7 +146,7 @@ def _read_remote_logs(self, ti, try_number, metadata=None) -> tuple[list[str], l

[GitHub] [airflow] monik-a opened a new issue, #32799: Tableau - problem when fetching job status

2023-07-24 Thread via GitHub
monik-a opened a new issue, #32799: URL: https://github.com/apache/airflow/issues/32799 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened We’e refreshing Tableau dashboards with Airflow and the below set of actions creates issues

[GitHub] [airflow] Adaverse commented on a diff in pull request #32779: Fix hardcoded container name in remote logging option for Azure Blob Storage

2023-07-24 Thread via GitHub
Adaverse commented on code in PR #32779: URL: https://github.com/apache/airflow/pull/32779#discussion_r1272050947 ## airflow/providers/microsoft/azure/log/wasb_task_handler.py: ## @@ -148,7 +146,7 @@ def _read_remote_logs(self, ti, try_number, metadata=None) -> tuple[list[str],

[GitHub] [airflow] vioao commented on pull request #32795: DruidHook add SQL-based task support

2023-07-24 Thread via GitHub
vioao commented on PR #32795: URL: https://github.com/apache/airflow/pull/32795#issuecomment-1647638560 @Lee-W I thought about it. But it also needs to change the current `DruidOperator` or add a new one. At the same time, the current `DruidHook` also needs to be reformed(eg. extract th

[GitHub] [airflow] amir-brightdata opened a new issue, #32800: Issue with fetching log file via download button

2023-07-24 Thread via GitHub
amir-brightdata opened a new issue, #32800: URL: https://github.com/apache/airflow/issues/32800 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Every time i try to press the download button in a specific task log page, the enti

[GitHub] [airflow] Adaverse commented on a diff in pull request #32779: Fix hardcoded container name in remote logging option for Azure Blob Storage

2023-07-24 Thread via GitHub
Adaverse commented on code in PR #32779: URL: https://github.com/apache/airflow/pull/32779#discussion_r1272050947 ## airflow/providers/microsoft/azure/log/wasb_task_handler.py: ## @@ -148,7 +146,7 @@ def _read_remote_logs(self, ti, try_number, metadata=None) -> tuple[list[str],

[GitHub] [airflow] pankajastro commented on pull request #32000: Fix DAG test command stuck in infinite loop

2023-07-24 Thread via GitHub
pankajastro commented on PR #32000: URL: https://github.com/apache/airflow/pull/32000#issuecomment-1647650586 > I think we need a test for this Hi @uranusjr I can see some tests for dag test feature https://github.com/apache/airflow/blob/main/tests/models/test_dag.py#L1891-L1985 but

[airflow] branch load-plugins-from-providers updated (b78c4ceca8 -> e6bd5213ec)

2023-07-24 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a change to branch load-plugins-from-providers in repository https://gitbox.apache.org/repos/asf/airflow.git omit b78c4ceca8 plugins: load plugins from providers add 8b7ae76026 Fixup docs and optimize

[airflow] 01/01: plugins: load plugins from providers

2023-07-24 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a commit to branch load-plugins-from-providers in repository https://gitbox.apache.org/repos/asf/airflow.git commit e6bd5213ec80bb5fc39b21986a6a67a9c233ade1 Author: Maciej Obuchowski AuthorDate: Tue Jun 27 14:

[GitHub] [airflow] uranusjr commented on pull request #32701: Fix logic of trigger rule one failed inside mapped group

2023-07-24 Thread via GitHub
uranusjr commented on PR #32701: URL: https://github.com/apache/airflow/pull/32701#issuecomment-1647662076 Thanks for the very detailed explanation, that makes sense now! I think the fix is on the right track, but not at the right place. We have a separate dep that checks for this scenario

[GitHub] [airflow] ephraimbuddy opened a new pull request, #32801: Fix failing dagrun query due to a refactor

2023-07-24 Thread via GitHub
ephraimbuddy opened a new pull request, #32801: URL: https://github.com/apache/airflow/pull/32801 There was a mistake in refactoring this query but the code had no test coverage. This commit fixes it and adds a test to cover this part of the code -- This is an automated message fr

[GitHub] [airflow] phanikumv commented on a diff in pull request #32350: Refactor Sqlalchemy queries to 2.0 style (Part 3)

2023-07-24 Thread via GitHub
phanikumv commented on code in PR #32350: URL: https://github.com/apache/airflow/pull/32350#discussion_r1272102429 ## airflow/www/views.py: ## @@ -2293,15 +2300,17 @@ def clear(self, *, session: Session = NEW_SESSION): # Lock the related dag runs to prevent from p

[GitHub] [airflow] uranusjr opened a new pull request, #32802: Ensure BaseOperator in MappedTaskGroup is expanded

2023-07-24 Thread via GitHub
uranusjr opened a new pull request, #32802: URL: https://github.com/apache/airflow/pull/32802 This applies MappedTaskIsExpanded to BaseOperator (as well as MappedOperator), so it is checked to be expanded if it's in a mapped task group. Fix #30333 Fix #30334 Close #32701

[GitHub] [airflow] VladaZakharova commented on pull request #32487: Migrating google dataflow example dags to system tests

2023-07-24 Thread via GitHub
VladaZakharova commented on PR #32487: URL: https://github.com/apache/airflow/pull/32487#issuecomment-1647701412 Hi! This system test is in our priority list now, so it is important to prepare this test in he nearest future. What do you think about my team taking this ticket? -- This

[GitHub] [airflow] antonio-antuan commented on pull request #32802: Ensure BaseOperator in MappedTaskGroup is expanded

2023-07-24 Thread via GitHub
antonio-antuan commented on PR #32802: URL: https://github.com/apache/airflow/pull/32802#issuecomment-1647705874 awesome! I'll check it within a couple of days -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [airflow] uranusjr merged pull request #32801: Fix failing dagrun query due to sqlalchemy 2.0 refactor

2023-07-24 Thread via GitHub
uranusjr merged PR #32801: URL: https://github.com/apache/airflow/pull/32801 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: Fix failing dagrun query due to sqlalchemy 2.0 refactor (#32801)

2023-07-24 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 0339f94a50 Fix failing dagrun query due to sqlalc

[GitHub] [airflow] uranusjr commented on a diff in pull request #32692: plugins: load plugins from providers

2023-07-24 Thread via GitHub
uranusjr commented on code in PR #32692: URL: https://github.com/apache/airflow/pull/32692#discussion_r1272122138 ## airflow/plugins_manager.py: ## @@ -267,6 +274,27 @@ def load_plugins_from_plugin_directory(): import_errors[file_path] = str(e) +def load_provide

[GitHub] [airflow] raycarter commented on pull request #26766: Allow xcom sidecar container image to be configurable

2023-07-24 Thread via GitHub
raycarter commented on PR #26766: URL: https://github.com/apache/airflow/pull/26766#issuecomment-1647718951 Hello, is this PR in any version released? I can't find the changes in the current version 2.6.3 or in the main-branch. -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] moiseenkov commented on pull request #32798: Add system test and docs for DataprocSubmitJobOperator (Presto job)

2023-07-24 Thread via GitHub
moiseenkov commented on PR #32798: URL: https://github.com/apache/airflow/pull/32798#issuecomment-1647720094 > Can you amend and force-push to fix the typoe in the commit message? Hi @uranusjr , thanks for noticing. Fixed. -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] uranusjr commented on pull request #32000: Fix DAG test command stuck in infinite loop

2023-07-24 Thread via GitHub
uranusjr commented on PR #32000: URL: https://github.com/apache/airflow/pull/32000#issuecomment-1647722231 I think building a DAG with `dag_maker` and then call `test` would work. The test should be annotated with `@pytest.mark.execution_timeout` since we also want to test for the infinite

[GitHub] [airflow] phanikumv merged pull request #32689: Add missing execute_complete method for `DatabricksRunNowOperator`

2023-07-24 Thread via GitHub
phanikumv merged PR #32689: URL: https://github.com/apache/airflow/pull/32689 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

[GitHub] [airflow] amoghrajesh commented on pull request #32487: Migrating google dataflow example dags to system tests

2023-07-24 Thread via GitHub
amoghrajesh commented on PR #32487: URL: https://github.com/apache/airflow/pull/32487#issuecomment-1647723772 Sure. If it's a priority, go ahead and take it over. Thanks @VladaZakharova -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[airflow] branch main updated (0339f94a50 -> 6313e52932)

2023-07-24 Thread phanikumv
This is an automated email from the ASF dual-hosted git repository. phanikumv pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 0339f94a50 Fix failing dagrun query due to sqlalchemy 2.0 refactor (#32801) add 6313e52932 Add missing execute

[GitHub] [airflow] VladaZakharova commented on pull request #32487: Migrating google dataflow example dags to system tests

2023-07-24 Thread via GitHub
VladaZakharova commented on PR #32487: URL: https://github.com/apache/airflow/pull/32487#issuecomment-1647737634 > Sure. If it's a priority, go ahead and take it over. > > Thanks @VladaZakharova :) To be able to create a service account, you should have project in gcp. If yo

[GitHub] [airflow] NilsJPWerner commented on pull request #32792: Adjust graph zoom based on selected task

2023-07-24 Thread via GitHub
NilsJPWerner commented on PR #32792: URL: https://github.com/apache/airflow/pull/32792#issuecomment-1647767495 This looks awesome! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #32803: [FIX] update xcom argument for bash operator

2023-07-24 Thread via GitHub
boring-cyborg[bot] commented on PR #32803: URL: https://github.com/apache/airflow/pull/32803#issuecomment-1647774681 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution

[GitHub] [airflow] mayanktri opened a new pull request, #32803: [FIX] update xcom argument for bash operator

2023-07-24 Thread via GitHub
mayanktri opened a new pull request, #32803: URL: https://github.com/apache/airflow/pull/32803 xcom push argument is incorrect in version 2.6.3. xcom_push should be replaced with do_xcom_push in BashOperator. -- This is an automated message from the Apache Git Service. To respond to t

[airflow] branch constraints-main updated: Updating constraints. Github run id:5644224142

2023-07-24 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 476db0a013 Updating con

[GitHub] [airflow] dzhigimont commented on pull request #32802: Ensure BaseOperator in MappedTaskGroup is expanded

2023-07-24 Thread via GitHub
dzhigimont commented on PR #32802: URL: https://github.com/apache/airflow/pull/32802#issuecomment-1647809044 @uranusjr Hmm, it doesn't work in my env, you can use the test from my PR https://github.com/apache/airflow/pull/32701 `test_upstream_in_mapped_group_with_one_failed_rule_task` to ch

[GitHub] [airflow] Owen-CH-Leung commented on a diff in pull request #32781: Override base log folder by using task handler's base_log_folder

2023-07-24 Thread via GitHub
Owen-CH-Leung commented on code in PR #32781: URL: https://github.com/apache/airflow/pull/32781#discussion_r1272198290 ## tests/utils/test_serve_logs.py: ## @@ -22,7 +22,9 @@ import jwt import pytest import time_machine +from py import path Review Comment: Thanks. I've us

[GitHub] [airflow] gecko655 opened a new issue, #32804: `DagRun.find()` fails when given `execution_date` obtained from `context.get('execution_date')` directly

2023-07-24 Thread via GitHub
gecko655 opened a new issue, #32804: URL: https://github.com/apache/airflow/issues/32804 ### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened ### Airflow version 2.5.3+composer (The latest available airflow version in Cloud Compos

[GitHub] [airflow] raycarter opened a new issue, #32805: Docker image of xcom_sidecar is not configurable

2023-07-24 Thread via GitHub
raycarter opened a new issue, #32805: URL: https://github.com/apache/airflow/issues/32805 ### Description The iamge of xcom_sidecar is not cunfigurable (code: [xcom_sidecar.py#36](https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py#L36))

[airflow] branch main updated: improve performance of pr script in dev/stats (#32735)

2023-07-24 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 6f75a8a917 improve performance of pr script in

[GitHub] [airflow] mobuchowski merged pull request #32735: improve performance of pr script in dev/stats

2023-07-24 Thread via GitHub
mobuchowski merged PR #32735: URL: https://github.com/apache/airflow/pull/32735 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airf

[GitHub] [airflow] vincbeck commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
vincbeck commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272269208 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -95,6 +100,12 @@ def execute_query( statement_id = resp["Id"] +if kwargs["ClusterI

[GitHub] [airflow] mobuchowski commented on a diff in pull request #32716: task instance listener: call on_task_instance_running after rendering templates

2023-07-24 Thread via GitHub
mobuchowski commented on code in PR #32716: URL: https://github.com/apache/airflow/pull/32716#discussion_r1272290431 ## airflow/models/taskinstance.py: ## @@ -1651,16 +1646,23 @@ def signal_handler(signum, frame): ) # Run pre_execute callback -

[GitHub] [airflow] mobuchowski commented on a diff in pull request #32716: task instance listener: call on_task_instance_running after rendering templates

2023-07-24 Thread via GitHub
mobuchowski commented on code in PR #32716: URL: https://github.com/apache/airflow/pull/32716#discussion_r1272290431 ## airflow/models/taskinstance.py: ## @@ -1651,16 +1646,23 @@ def signal_handler(signum, frame): ) # Run pre_execute callback -

[GitHub] [airflow] Lee-W opened a new pull request, #32806: fix(providers/databricks): remove the execute method from to-be-deprecated DatabricksRunNowDeferrableOperator

2023-07-24 Thread via GitHub
Lee-W opened a new pull request, #32806: URL: https://github.com/apache/airflow/pull/32806 As mentioned in #27837, we should do something similar to DatabricksRunOperator instead of running the original executing logic in DatabricksRunNowDeferrableOperator This is due to a bug that causes i

[GitHub] [airflow] RNHTTR commented on a diff in pull request #32709: Enable pools to consider deferred tasks

2023-07-24 Thread via GitHub
RNHTTR commented on code in PR #32709: URL: https://github.com/apache/airflow/pull/32709#discussion_r1272305349 ## docs/apache-airflow/authoring-and-scheduling/deferring.rst: ## @@ -20,7 +20,7 @@ Deferrable Operators & Triggers Standard :doc:`Operators ` and :doc:`Sensors <.

[GitHub] [airflow] Lee-W commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
Lee-W commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272312032 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -95,6 +100,12 @@ def execute_query( statement_id = resp["Id"] +if kwargs["ClusterIden

[GitHub] [airflow] Lee-W commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
Lee-W commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272312032 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -95,6 +100,12 @@ def execute_query( statement_id = resp["Id"] +if kwargs["ClusterIden

[GitHub] [airflow] Lee-W commented on a diff in pull request #32779: Fix hardcoded container name in remote logging option for Azure Blob Storage

2023-07-24 Thread via GitHub
Lee-W commented on code in PR #32779: URL: https://github.com/apache/airflow/pull/32779#discussion_r1272312846 ## airflow/providers/microsoft/azure/log/wasb_task_handler.py: ## @@ -148,7 +146,7 @@ def _read_remote_logs(self, ti, try_number, metadata=None) -> tuple[list[str], l

[GitHub] [airflow] vincbeck commented on a diff in pull request #28900: Convert DagFileProcessor.execute_callbacks to Internal API

2023-07-24 Thread via GitHub
vincbeck commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1272322945 ## airflow/models/dag.py: ## @@ -1465,11 +1510,22 @@ def get_dagrun( raise TypeError("You must provide either the execution_date or the run_id")

[GitHub] [airflow] vincbeck merged pull request #32787: Add `ExternalBranchPythonOperator`

2023-07-24 Thread via GitHub
vincbeck merged PR #32787: URL: https://github.com/apache/airflow/pull/32787 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated (6f75a8a917 -> d05e42e5d2)

2023-07-24 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 6f75a8a917 improve performance of pr script in dev/stats (#32735) add d05e42e5d2 Add `ExternalBranchPythonOperat

[GitHub] [airflow] karakanb commented on pull request #32705: add dag_run_ids and task_ids filter for the batch task instance API endpoint

2023-07-24 Thread via GitHub
karakanb commented on PR #32705: URL: https://github.com/apache/airflow/pull/32705#issuecomment-1648008286 @uranusjr done, assumed it'd go in 2.6.4, please take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [airflow] moiseenkov opened a new pull request, #32807: Fix Dataproc system tests

2023-07-24 Thread via GitHub
moiseenkov opened a new pull request, #32807: URL: https://github.com/apache/airflow/pull/32807 1. Fixed Dataproc system tests - made sure that each test creates objects with unique names, so they could run in parallel with no conflicts. 3. Removed unused variables. 4. Refactored sligh

[GitHub] [airflow] amoghrajesh opened a new pull request, #32808: Adding support for disabling token mounts using automountServiceAccountToken

2023-07-24 Thread via GitHub
amoghrajesh opened a new pull request, #32808: URL: https://github.com/apache/airflow/pull/32808 To control mounting of the serviceAccount's API credentials onto pods, automountServiceAccountToken property can be used. The changes can either be in serviceAccount definitions

[airflow] 01/01: plugins: load plugins from providers

2023-07-24 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a commit to branch load-plugins-from-providers in repository https://gitbox.apache.org/repos/asf/airflow.git commit ef4ece9f9e2c038200f77d5dc9e10709ca42efd5 Author: Maciej Obuchowski AuthorDate: Tue Jun 27 14:

[airflow] branch load-plugins-from-providers updated (e6bd5213ec -> ef4ece9f9e)

2023-07-24 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a change to branch load-plugins-from-providers in repository https://gitbox.apache.org/repos/asf/airflow.git omit e6bd5213ec plugins: load plugins from providers new ef4ece9f9e plugins: load plugins f

[GitHub] [airflow] mobuchowski commented on a diff in pull request #32692: plugins: load plugins from providers

2023-07-24 Thread via GitHub
mobuchowski commented on code in PR #32692: URL: https://github.com/apache/airflow/pull/32692#discussion_r1272367426 ## airflow/providers_manager.py: ## @@ -1024,6 +1039,20 @@ def _discover_config(self) -> None: if provider.data.get("config"): self.

[GitHub] [airflow] ivica-k commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
ivica-k commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272370482 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -53,6 +53,7 @@ def execute_query( database: str, sql: str | list[str], cluste

[GitHub] [airflow] ivica-k commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
ivica-k commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272371045 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -95,6 +100,12 @@ def execute_query( statement_id = resp["Id"] +if kwargs["ClusterId

[GitHub] [airflow] ivica-k commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
ivica-k commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272371737 ## tests/providers/amazon/aws/hooks/test_redshift_data.py: ## @@ -39,22 +41,53 @@ def test_conn_attribute(self): @mock.patch("airflow.providers.amazon.aws.hooks

[GitHub] [airflow] Lee-W commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
Lee-W commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272382745 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -60,13 +60,17 @@ def execute_query( with_event: bool = False, wait_for_completion: bool

[GitHub] [airflow] mikaeld commented on a diff in pull request #32724: feat(chart): Add existing secret support for PGBouncer metrics exporter.

2023-07-24 Thread via GitHub
mikaeld commented on code in PR #32724: URL: https://github.com/apache/airflow/pull/32724#discussion_r1272393274 ## chart/templates/secrets/pgbouncer-stats-secret.yaml: ## Review Comment: IMO there is no point in having a custom key with the included chart secret. I believ

[GitHub] [airflow] mikaeld commented on a diff in pull request #32724: feat(chart): Add existing secret support for PGBouncer metrics exporter.

2023-07-24 Thread via GitHub
mikaeld commented on code in PR #32724: URL: https://github.com/apache/airflow/pull/32724#discussion_r1272402622 ## chart/values.schema.json: ## @@ -5341,6 +5341,22 @@ ], "$ref": "#/definitions/io.k8s.api.core.v1.Resourc

[GitHub] [airflow] vincbeck commented on pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
vincbeck commented on PR #32785: URL: https://github.com/apache/airflow/pull/32785#issuecomment-1648116560 Besides the nits mentioned by @Lee-W , it looks good to me :) Good job! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] mikaeld commented on a diff in pull request #32724: feat(chart): Add existing secret support for PGBouncer metrics exporter.

2023-07-24 Thread via GitHub
mikaeld commented on code in PR #32724: URL: https://github.com/apache/airflow/pull/32724#discussion_r1272402622 ## chart/values.schema.json: ## @@ -5341,6 +5341,22 @@ ], "$ref": "#/definitions/io.k8s.api.core.v1.Resourc

[GitHub] [airflow] ivica-k commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
ivica-k commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272412389 ## airflow/providers/amazon/aws/operators/redshift_data.py: ## @@ -55,6 +58,7 @@ class RedshiftDataOperator(BaseOperator): template_fields = ( "cluste

[GitHub] [airflow] ivica-k commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
ivica-k commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272412389 ## airflow/providers/amazon/aws/operators/redshift_data.py: ## @@ -55,6 +58,7 @@ class RedshiftDataOperator(BaseOperator): template_fields = ( "cluste

[GitHub] [airflow] ivica-k commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
ivica-k commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272416092 ## airflow/providers/amazon/aws/hooks/redshift_data.py: ## @@ -60,13 +60,17 @@ def execute_query( with_event: bool = False, wait_for_completion: boo

[GitHub] [airflow] ivica-k commented on a diff in pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
ivica-k commented on code in PR #32785: URL: https://github.com/apache/airflow/pull/32785#discussion_r1272416728 ## airflow/providers/amazon/aws/operators/redshift_data.py: ## @@ -55,6 +58,7 @@ class RedshiftDataOperator(BaseOperator): template_fields = ( "cluste

[GitHub] [airflow] vincbeck commented on pull request #32756: move user details view to fab auth manager

2023-07-24 Thread via GitHub
vincbeck commented on PR #32756: URL: https://github.com/apache/airflow/pull/32756#issuecomment-1648147600 It looks good! Nice! Some unit tests are failing but besides that I like it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [airflow] eladkal commented on pull request #31843: Add ExternalBranchPythonOperator

2023-07-24 Thread via GitHub
eladkal commented on PR #31843: URL: https://github.com/apache/airflow/pull/31843#issuecomment-1648162930 Resolved in https://github.com/apache/airflow/pull/32787 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] eladkal closed pull request #31843: Add ExternalBranchPythonOperator

2023-07-24 Thread via GitHub
eladkal closed pull request #31843: Add ExternalBranchPythonOperator URL: https://github.com/apache/airflow/pull/31843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

[GitHub] [airflow] bhagany commented on a diff in pull request #30053: fix setting project_id for gs to bq and bq to gs

2023-07-24 Thread via GitHub
bhagany commented on code in PR #30053: URL: https://github.com/apache/airflow/pull/30053#discussion_r1272466593 ## airflow/providers/google/cloud/transfers/bigquery_to_gcs.py: ## @@ -193,7 +193,7 @@ def _submit_job( return hook.insert_job( configuration=

[GitHub] [airflow] Sighery commented on issue #32657: Increase connections HTTP login length to 5000 characters

2023-07-24 Thread via GitHub
Sighery commented on issue #32657: URL: https://github.com/apache/airflow/issues/32657#issuecomment-1648206389 @jens-scheffler-bosch @potiuk So, tried it today [in my fork](https://github.com/Sighery/airflow/pull/1). I think I managed to set up the required CI checks correctly. I couldn't g

[GitHub] [airflow] sunank200 commented on a diff in pull request #31350: openlineage, gcs: add openlineage methods for GcsToGcsOperator

2023-07-24 Thread via GitHub
sunank200 commented on code in PR #31350: URL: https://github.com/apache/airflow/pull/31350#discussion_r1272477650 ## airflow/providers/google/cloud/transfers/gcs_to_gcs.py: ## @@ -464,7 +466,37 @@ def _copy_single_object(self, hook, source_object, destination_object):

[GitHub] [airflow] ferruzzi commented on issue #32271: Task instance state `upstream_failed` is missing from metrics counter `ti.finish...`

2023-07-24 Thread via GitHub
ferruzzi commented on issue #32271: URL: https://github.com/apache/airflow/issues/32271#issuecomment-1648233538 Interesting, yeah, I see where you are coming from but I think I agree with hussain's comment. If you are using the default trigger rules and the upstream task failed, then this

[GitHub] [airflow] syedahsn opened a new pull request, #32809: Deferrable mode for Sqs Sensor

2023-07-24 Thread via GitHub
syedahsn opened a new pull request, #32809: URL: https://github.com/apache/airflow/pull/32809 This PR adds the deferrable option to the SQS sensor. I refactored the existing sensor to pull out some of the code that processes the boto API response so that it could be used by the Trigger,

[GitHub] [airflow] jedcunningham commented on a diff in pull request #32775: Add pre-Airflow-2-7 hardcoded defaults for config for older providers

2023-07-24 Thread via GitHub
jedcunningham commented on code in PR #32775: URL: https://github.com/apache/airflow/pull/32775#discussion_r1272529966 ## airflow/configuration.py: ## @@ -1929,6 +1942,18 @@ def create_default_config_parser(configuration_description: dict[str, dict[str, return parser +

[GitHub] [airflow] vincbeck merged pull request #32785: adds support for querying Redshift Serverless clusters

2023-07-24 Thread via GitHub
vincbeck merged PR #32785: URL: https://github.com/apache/airflow/pull/32785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: Add support for querying Redshift Serverless clusters (#32785)

2023-07-24 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8012c9fce6 Add support for querying Redshift Serv

[GitHub] [airflow] vincbeck merged pull request #32664: Add endpoint_url in test_connection

2023-07-24 Thread via GitHub
vincbeck merged PR #32664: URL: https://github.com/apache/airflow/pull/32664 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: Add endpoint_url in test_connection (#32664)

2023-07-24 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 282854b55f Add endpoint_url in test_connection (#

[GitHub] [airflow] vincbeck closed issue #32280: RedshiftDataOperator: Add support for Redshift serverless clusters

2023-07-24 Thread via GitHub
vincbeck closed issue #32280: RedshiftDataOperator: Add support for Redshift serverless clusters URL: https://github.com/apache/airflow/issues/32280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [airflow] yquaziii commented on issue #32793: Airflow2 : When skipping by BranchPythonOperator with taskgroup, all downstream tasks is skipping.

2023-07-24 Thread via GitHub
yquaziii commented on issue #32793: URL: https://github.com/apache/airflow/issues/32793#issuecomment-1648305491 Are you using depends_on_past=True in sub_dag.? if that's not the case then the next task group should have executed. -- This is an automated message from the Apache Git Servic

  1   2   3   >