[GitHub] [airflow] potiuk commented on issue #29648: Optionally exclude task deferral time from overall runtime of the task

2023-02-20 Thread via GitHub
potiuk commented on issue #29648: URL: https://github.com/apache/airflow/issues/29648#issuecomment-1438000722 I am inclined to close it as "won't do" :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test

2023-02-20 Thread via GitHub
Taragolis commented on PR #29616: URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437988917 Something new 😮 ```console airflow-scheduler_1 | airflow-scheduler_1 | BACKEND=redis airflow-scheduler_1 | DB_HOST=redis airflow-scheduler_1 | DB_PORT=637

[GitHub] [airflow] eladkal commented on issue #29607: Status of testing Providers that were prepared on February 18, 2023

2023-02-20 Thread via GitHub
eladkal commented on issue #29607: URL: https://github.com/apache/airflow/issues/29607#issuecomment-1437961577 > #29552 Tested and it works. From my perspective there are 2 issues. > > * proper documentation on how to use `drive_folder` and `folder_id` arguments together (as they can

[GitHub] [airflow] Lee-W closed pull request #29649: Livy operator async (fixing failed test in PR 29047)

2023-02-20 Thread via GitHub
Lee-W closed pull request #29649: Livy operator async (fixing failed test in PR 29047) URL: https://github.com/apache/airflow/pull/29649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] Lee-W commented on pull request #29649: Livy operator async (fixing failed test in PR 29047)

2023-02-20 Thread via GitHub
Lee-W commented on PR #29649: URL: https://github.com/apache/airflow/pull/29649#issuecomment-1437959795 Thanks @potiuk . I'll close this one and rebase on #29047 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] potiuk commented on pull request #29649: Livy operator async (fixing failed test in PR 29047)

2023-02-20 Thread via GitHub
potiuk commented on PR #29649: URL: https://github.com/apache/airflow/pull/29649#issuecomment-1437955076 Please rebase. I fixed some "queue" stuck for our CI builds and you need to rebase to re-run tests. -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [airflow] potiuk commented on pull request #29644: Remove <2.0.0 limit on google-cloud-bigtable

2023-02-20 Thread via GitHub
potiuk commented on PR #29644: URL: https://github.com/apache/airflow/pull/29644#issuecomment-1437952272 > @uranusjr think I got everything. Last test failed due to 2 hour queue timeout. Yeah - we had some "blocked" build queue - I unblocked it now. Rebasing to re-run. -- This is

[GitHub] [airflow] aru-trackunit commented on issue #29607: Status of testing Providers that were prepared on February 18, 2023

2023-02-20 Thread via GitHub
aru-trackunit commented on issue #29607: URL: https://github.com/apache/airflow/issues/29607#issuecomment-1437952596 #29552 Tested and it works. From my perspective there are 2 issues. - proper documentation on how to use `drive_folder` and `folder_id` arguments together (as they can

[GitHub] [airflow] r-richmond commented on pull request #29644: Remove <2.0.0 limit on google-cloud-bigtable

2023-02-20 Thread via GitHub
r-richmond commented on PR #29644: URL: https://github.com/apache/airflow/pull/29644#issuecomment-1437945839 @uranusjr think I got everything. Last test failed due to 2 hour queue timeout. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

[airflow-ci-infra] branch main updated: Update docker runner to 302.1

2023-02-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git The following commit(s) were added to refs/heads/main by this push: new 80680f8 Update docker runner to 302.1 806

[GitHub] [airflow] potiuk closed issue #29621: Fix adding annotations for dag persistence PVC

2023-02-20 Thread via GitHub
potiuk closed issue #29621: Fix adding annotations for dag persistence PVC URL: https://github.com/apache/airflow/issues/29621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[GitHub] [airflow] potiuk closed issue #28745: annotations in logs pvc

2023-02-20 Thread via GitHub
potiuk closed issue #28745: annotations in logs pvc URL: https://github.com/apache/airflow/issues/28745 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on issue #25297: on_failure_callback is not called when task is terminated externally

2023-02-20 Thread via GitHub
potiuk commented on issue #25297: URL: https://github.com/apache/airflow/issues/25297#issuecomment-1437923131 @seub - please provide a reproducible case and logs if you would like to report it. It might be something specific for your case, and to be brutally honest it's much easier for a re

[GitHub] [airflow] potiuk commented on issue #29648: Optionally exclude task deferral time from overall runtime of the task

2023-02-20 Thread via GitHub
potiuk commented on issue #29648: URL: https://github.com/apache/airflow/issues/29648#issuecomment-1437918635 > As an alternative and maybe simpler feature it'd also suit my use case if we could make it so that we are optionally able to "reset" the execution timer when the task resumes from

[GitHub] [airflow] andrewgodwin commented on issue #29648: Optionally exclude task deferral time from overall runtime of the task

2023-02-20 Thread via GitHub
andrewgodwin commented on issue #29648: URL: https://github.com/apache/airflow/issues/29648#issuecomment-1437917495 Yes, the changes required here are non-trivial - you'd need some sort of extra column to track cumulative runtime and then have the workers add values to that whenever they fi

[GitHub] [airflow] potiuk commented on pull request #29616: Refactor docker-compose quick start test

2023-02-20 Thread via GitHub
potiuk commented on PR #29616: URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437911170 > That is more interesting. I've seen before that statics checks sometimes failed with particular this version of docker 20.10.23+azure-2 and didn't seen that this happen in docker witho

[GitHub] [airflow] uranusjr commented on a diff in pull request #29616: Refactor docker-compose quick start test

2023-02-20 Thread via GitHub
uranusjr commented on code in PR #29616: URL: https://github.com/apache/airflow/pull/29616#discussion_r1112580793 ## docker_tests/test_docker_compose_quick_start.py: ## @@ -114,53 +104,60 @@ def wait_for_terminal_dag_state(dag_id, dag_run_id): break -def test_tr

[GitHub] [airflow] potiuk commented on pull request #29616: Refactor docker-compose quick start test

2023-02-20 Thread via GitHub
potiuk commented on PR #29616: URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437905137 Closed/Reopened to re-run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] potiuk closed pull request #29616: Refactor docker-compose quick start test

2023-02-20 Thread via GitHub
potiuk closed pull request #29616: Refactor docker-compose quick start test URL: https://github.com/apache/airflow/pull/29616 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] rkarish closed pull request #29034: Add functionality for operators to template all eligible fields (apac…

2023-02-20 Thread via GitHub
rkarish closed pull request #29034: Add functionality for operators to template all eligible fields (apac… URL: https://github.com/apache/airflow/pull/29034 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] pshrivastava27 commented on issue #29432: Jinja templating doesn't work with container_resources when using dymanic task mapping with Kubernetes Pod Operator

2023-02-20 Thread via GitHub
pshrivastava27 commented on issue #29432: URL: https://github.com/apache/airflow/issues/29432#issuecomment-1437887986 > @jose-lpa using `Variable.get()` in the dag script is not recommended because the `DagFileProcessor` process the script each X minutes, and it loads this variable from the

[GitHub] [airflow] uranusjr commented on a diff in pull request #29625: Aggressively cache entry points in process

2023-02-20 Thread via GitHub
uranusjr commented on code in PR #29625: URL: https://github.com/apache/airflow/pull/29625#discussion_r1112560468 ## airflow/utils/entry_points.py: ## @@ -28,26 +30,33 @@ log = logging.getLogger(__name__) +EPnD = Tuple[metadata.EntryPoint, metadata.Distribution] -def entr

[GitHub] [airflow] amoghrajesh commented on issue #29322: DAG list, sorting lost when switching page

2023-02-20 Thread via GitHub
amoghrajesh commented on issue #29322: URL: https://github.com/apache/airflow/issues/29322#issuecomment-1437874203 @bbovenzi I need some help understanding how to fix this issue. Please refer to my last comment -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] amoghrajesh commented on issue #29621: Fix adding annotations for dag persistence PVC

2023-02-20 Thread via GitHub
amoghrajesh commented on issue #29621: URL: https://github.com/apache/airflow/issues/29621#issuecomment-1437873701 @potiuk we can close this issue. The PR is merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [airflow] amoghrajesh commented on issue #28745: annotations in logs pvc

2023-02-20 Thread via GitHub
amoghrajesh commented on issue #28745: URL: https://github.com/apache/airflow/issues/28745#issuecomment-1437873467 @potiuk we can close this. It has been merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [airflow] Lee-W opened a new pull request, #29649: Livy operator async (fixing failed test in PR 29047)

2023-02-20 Thread via GitHub
Lee-W opened a new pull request, #29649: URL: https://github.com/apache/airflow/pull/29649 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more infor

[GitHub] [airflow] seub commented on issue #25297: on_failure_callback is not called when task is terminated externally

2023-02-20 Thread via GitHub
seub commented on issue #25297: URL: https://github.com/apache/airflow/issues/25297#issuecomment-1437822040 @potiuk It's the same issue, and it's easy to reproduce: just interrupt a task manually (in the UI) and observe that `on_failure_callback` is not called. Please try and let me know if

[GitHub] [airflow] josh-fell commented on issue #29607: Status of testing Providers that were prepared on February 18, 2023

2023-02-20 Thread via GitHub
josh-fell commented on issue #29607: URL: https://github.com/apache/airflow/issues/29607#issuecomment-1437817901 #29565 looks good. Thanks for organizing @eladkal! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [airflow] joshuaghezzi commented on issue #29648: Optionally exclude task deferral time from overall runtime of the task

2023-02-20 Thread via GitHub
joshuaghezzi commented on issue #29648: URL: https://github.com/apache/airflow/issues/29648#issuecomment-1437787575 As an alternative and maybe simpler feature it'd also suit my use case if we could make it so that we are optionally able to "reset" the execution timer when the task resumes

[GitHub] [airflow] hussein-awala commented on pull request #29498: add missing read for K8S config file from conn in deferred `KubernetesPodOperator`

2023-02-20 Thread via GitHub
hussein-awala commented on PR #29498: URL: https://github.com/apache/airflow/pull/29498#issuecomment-1437703231 > Hi! May i ask in which format you will pass the config file to trigger? So it will be just a file passed as a parameter to trigger? Or how? @VladaZakharova - Yes, I pass t

[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-20 Thread via GitHub
pgagnon commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1112410526 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._sess

[GitHub] [airflow] s0neq commented on a diff in pull request #29635: YandexCloud provider: support Yandex SDK feature "endpoint"

2023-02-20 Thread via GitHub
s0neq commented on code in PR #29635: URL: https://github.com/apache/airflow/pull/29635#discussion_r1112407914 ## airflow/providers/yandex/hooks/yandex.py: ## @@ -122,7 +127,8 @@ def __init__( self.connection = self.get_connection(self.connection_id) self.extra

[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-20 Thread via GitHub
Taragolis commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1112407307 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._se

[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-20 Thread via GitHub
pgagnon commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1112403111 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._sess

[GitHub] [airflow] Taragolis commented on a diff in pull request #29635: YandexCloud provider: support Yandex SDK feature "endpoint"

2023-02-20 Thread via GitHub
Taragolis commented on code in PR #29635: URL: https://github.com/apache/airflow/pull/29635#discussion_r1112404383 ## airflow/providers/yandex/hooks/yandex.py: ## @@ -122,7 +127,8 @@ def __init__( self.connection = self.get_connection(self.connection_id) self.e

[GitHub] [airflow] s0neq commented on a diff in pull request #29635: YandexCloud provider: support Yandex SDK feature "endpoint"

2023-02-20 Thread via GitHub
s0neq commented on code in PR #29635: URL: https://github.com/apache/airflow/pull/29635#discussion_r1112403729 ## airflow/providers/yandex/hooks/yandex.py: ## @@ -122,7 +127,8 @@ def __init__( self.connection = self.get_connection(self.connection_id) self.extra

[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-20 Thread via GitHub
pgagnon commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1112403111 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._sess

[GitHub] [airflow] github-actions[bot] commented on issue #20267: can't delete airflow connection from CLI when it has bad fernet key

2023-02-20 Thread via GitHub
github-actions[bot] commented on issue #20267: URL: https://github.com/apache/airflow/issues/20267#issuecomment-1437689864 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activit

[GitHub] [airflow] github-actions[bot] commented on issue #20588: Health check path is not configured properly if AIRFLOW__WEBSERVER__BASE_URL is set via ENV variable

2023-02-20 Thread via GitHub
github-actions[bot] commented on issue #20588: URL: https://github.com/apache/airflow/issues/20588#issuecomment-1437689838 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activit

[GitHub] [airflow] github-actions[bot] commented on issue #20821: Airflow can not schedule task in new subDAG

2023-02-20 Thread via GitHub
github-actions[bot] commented on issue #20821: URL: https://github.com/apache/airflow/issues/20821#issuecomment-1437689819 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activit

[GitHub] [airflow] github-actions[bot] commented on issue #20949: Spark Submit Operator reporting job failure. Not able to poll the successful completion of Spark Job.

2023-02-20 Thread via GitHub
github-actions[bot] commented on issue #20949: URL: https://github.com/apache/airflow/issues/20949#issuecomment-1437689787 This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activit

[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-20 Thread via GitHub
Taragolis commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1112397450 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._se

[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-20 Thread via GitHub
Taragolis commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1112397450 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._se

[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-20 Thread via GitHub
Taragolis commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1112397450 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._se

[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-20 Thread via GitHub
pgagnon commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1112391709 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._sess

[GitHub] [airflow] hussein-awala commented on issue #29621: Fix adding annotations for dag persistence PVC

2023-02-20 Thread via GitHub
hussein-awala commented on issue #29621: URL: https://github.com/apache/airflow/issues/29621#issuecomment-1437660008 @potiuk we can close it since #29622 is megred. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[airflow] branch revert-29408-docker-compose-change-example updated (9da3771744 -> 37aef76496)

2023-02-20 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch revert-29408-docker-compose-change-example in repository https://gitbox.apache.org/repos/asf/airflow.git discard 9da3771744 Revert "Improve health checks in example docker-compose and clarif

[GitHub] [airflow] Taragolis commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-20 Thread via GitHub
Taragolis commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1112384223 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._se

[airflow] branch main updated: Don't push secret in XCOM in BigQueryCreateDataTransferOperator (#29348)

2023-02-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f51742d20b Don't push secret in XCOM in BigQueryCr

[GitHub] [airflow] potiuk merged pull request #29348: Don't push secret in XCOM in BigQueryCreateDataTransferOperator

2023-02-20 Thread via GitHub
potiuk merged PR #29348: URL: https://github.com/apache/airflow/pull/29348 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk closed issue #29209: BigQueryCreateDataTransferOperator will log AWS credentials when transferring from S3

2023-02-20 Thread via GitHub
potiuk closed issue #29209: BigQueryCreateDataTransferOperator will log AWS credentials when transferring from S3 URL: https://github.com/apache/airflow/issues/29209 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [airflow] potiuk commented on issue #29648: Optionally exclude task deferral time from overall runtime of the task

2023-02-20 Thread via GitHub
potiuk commented on issue #29648: URL: https://github.com/apache/airflow/issues/29648#issuecomment-1437654137 Maybe, but this would not be accurate - there are lots of nuances - when the task start running and when it is already deferred (both bringing back and deferring tasks take time). P

[airflow] branch main updated: Update google cloud dlp package and adjust hook and operators (#29234)

2023-02-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3dbcf99d20 Update google cloud dlp package and adju

[GitHub] [airflow] potiuk merged pull request #29234: Update google cloud dlp package and adjust hook and operators

2023-02-20 Thread via GitHub
potiuk merged PR #29234: URL: https://github.com/apache/airflow/pull/29234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on issue #29643: Add google-cloud-run python client to GCP requirements

2023-02-20 Thread via GitHub
potiuk commented on issue #29643: URL: https://github.com/apache/airflow/issues/29643#issuecomment-1437649040 Feel free to attempt it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[airflow] branch main updated (5835b08e8b -> 901774718c)

2023-02-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 5835b08e8b Adding possibility for annotations in logs pvc (#29270) add 901774718c Fix adding annotations for dag p

[GitHub] [airflow] potiuk merged pull request #29622: Fix adding annotations for dag persistence PVC

2023-02-20 Thread via GitHub
potiuk merged PR #29622: URL: https://github.com/apache/airflow/pull/29622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Adding possibility for annotations in logs pvc (#29270)

2023-02-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 5835b08e8b Adding possibility for annotations in lo

[GitHub] [airflow] potiuk merged pull request #29270: Adding possibility for annotations in logs pvc

2023-02-20 Thread via GitHub
potiuk merged PR #29270: URL: https://github.com/apache/airflow/pull/29270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on issue #28647: Intermitent log on deferrable operator

2023-02-20 Thread via GitHub
potiuk commented on issue #28647: URL: https://github.com/apache/airflow/issues/28647#issuecomment-1437642318 cc: @andrewgodwin maybe that rings a bell for you ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [airflow] potiuk commented on pull request #29094: Add `max_active_tis_per_dagrun` for Dynamic Task Mapping

2023-02-20 Thread via GitHub
potiuk commented on PR #29094: URL: https://github.com/apache/airflow/pull/29094#issuecomment-1437640955 I think this one is great - though I do not understand all the details of mapped tasks and concurrency there. @uranusjr @ashb - maybe you can take a look, sounds like a great feature of

[airflow] branch main updated (ee0a56a2ca -> 6ef5ba9104)

2023-02-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from ee0a56a2ca Trigger Class migration to internal API (#29099) add 6ef5ba9104 Refactor Dataproc Trigger (#29364) No

[GitHub] [airflow] potiuk merged pull request #29364: Refactor Dataproc Trigger

2023-02-20 Thread via GitHub
potiuk merged PR #29364: URL: https://github.com/apache/airflow/pull/29364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on issue #25060: Airflow scheduler crashes due to 'duplicate key value violates unique constraint "task_instance_pkey"'

2023-02-20 Thread via GitHub
potiuk commented on issue #25060: URL: https://github.com/apache/airflow/issues/25060#issuecomment-1437625953 @uranusjr - wasn't the "null" mapped task instance fixed since 2.4.3 ? I cannot find it easily -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] pankajastro commented on pull request #29348: Don't push secret in XCOM in BigQueryCreateDataTransferOperator

2023-02-20 Thread via GitHub
pankajastro commented on PR #29348: URL: https://github.com/apache/airflow/pull/29348#issuecomment-1437626057 > > And I guess we need to add tests for that, for avoid situation that somehow it return in the future. > > Agree Added here 4eade3237e25f92095d74b76cd1f16f99945c842

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29648: Optionally exclude task deferral time from overall runtime of the task

2023-02-20 Thread via GitHub
boring-cyborg[bot] commented on issue #29648: URL: https://github.com/apache/airflow/issues/29648#issuecomment-1437605958 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] joshuaghezzi opened a new issue, #29648: Optionally exclude task deferral time from overall runtime of the task

2023-02-20 Thread via GitHub
joshuaghezzi opened a new issue, #29648: URL: https://github.com/apache/airflow/issues/29648 ### Description Ability to optionally exclude the time that the task was deferred from the overall execution time of the task. ### Use case/motivation I've created a custom opera

[GitHub] [airflow] pgagnon commented on a diff in pull request #29623: Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity

2023-02-20 Thread via GitHub
pgagnon commented on code in PR #29623: URL: https://github.com/apache/airflow/pull/29623#discussion_r1112360505 ## airflow/providers/amazon/aws/hooks/base_aws.py: ## @@ -312,19 +312,35 @@ def _get_web_identity_credential_fetcher( base_session = self.basic_session._sess

[GitHub] [airflow] potiuk commented on issue #29432: Jinja templating doesn't work with container_resources when using dymanic task mapping with Kubernetes Pod Operator

2023-02-20 Thread via GitHub
potiuk commented on issue #29432: URL: https://github.com/apache/airflow/issues/29432#issuecomment-1437596856 Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

[GitHub] [airflow] potiuk commented on issue #29621: Fix adding annotations for dag persistence PVC

2023-02-20 Thread via GitHub
potiuk commented on issue #29621: URL: https://github.com/apache/airflow/issues/29621#issuecomment-1437596195 Feel free to fix them @amoghrajesh . Assigned you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [airflow] potiuk commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3

2023-02-20 Thread via GitHub
potiuk commented on issue #29393: URL: https://github.com/apache/airflow/issues/29393#issuecomment-1437595099 cc: @dstandish - maybe you can comment on how it changes in 2.6 (and whether the problem is going to be fixed there). -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] potiuk commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters

2023-02-20 Thread via GitHub
potiuk commented on PR #29580: URL: https://github.com/apache/airflow/pull/29580#issuecomment-1437592624 Yep. I thouthg about it and this seems like non-obvious great way of solving the "cost" problem connected with secret backends. -- This is an automated message from the Apache Git Serv

[GitHub] [airflow] potiuk commented on pull request #29406: Add Fail Fast feature for DAGs

2023-02-20 Thread via GitHub
potiuk commented on PR #29406: URL: https://github.com/apache/airflow/pull/29406#issuecomment-1437591152 @ashb @uranusjr - can you think of some hidden side-effects. Seems like this implementation of "fail-fast" is simple and might just work - unless I missed something. -- This is an aut

[GitHub] [airflow] Taragolis commented on a diff in pull request #29195: Fixing Task Duration view in case of manual DAG runs only (#22015)

2023-02-20 Thread via GitHub
Taragolis commented on code in PR #29195: URL: https://github.com/apache/airflow/pull/29195#discussion_r1112342301 ## airflow/models/dag.py: ## @@ -112,6 +113,8 @@ log = logging.getLogger(__name__) +_T = TypeVar("_T") + Review Comment: ```suggestion ``` ##

[GitHub] [airflow] potiuk commented on a diff in pull request #29406: Add Fail Fast feature for DAGs

2023-02-20 Thread via GitHub
potiuk commented on code in PR #29406: URL: https://github.com/apache/airflow/pull/29406#discussion_r1112342672 ## airflow/models/taskinstance.py: ## @@ -172,6 +172,19 @@ def set_current_context(context: Context) -> Generator[Context, None, None]: ) +def stop_a

[GitHub] [airflow] potiuk commented on a diff in pull request #29406: Add Fail Fast feature for DAGs

2023-02-20 Thread via GitHub
potiuk commented on code in PR #29406: URL: https://github.com/apache/airflow/pull/29406#discussion_r1112342504 ## airflow/models/taskinstance.py: ## @@ -172,6 +172,19 @@ def set_current_context(context: Context) -> Generator[Context, None, None]: ) +def stop_a

[GitHub] [airflow] potiuk commented on pull request #29518: Google Providers - Fix _MethodDefault deepcopy failure

2023-02-20 Thread via GitHub
potiuk commented on PR #29518: URL: https://github.com/apache/airflow/pull/29518#issuecomment-1437581881 @IKholopov - can you please split this one in two PRs: 1) introduce base class 2) adding deepcopy That will be much better when it comes to historical look at the changes.

[GitHub] [airflow] potiuk commented on a diff in pull request #29518: Google Providers - Fix _MethodDefault deepcopy failure

2023-02-20 Thread via GitHub
potiuk commented on code in PR #29518: URL: https://github.com/apache/airflow/pull/29518#discussion_r1112340761 ## airflow/providers/google/cloud/operators/cloud_base.py: ## @@ -0,0 +1,34 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] potiuk commented on a diff in pull request #29518: Google Providers - Fix _MethodDefault deepcopy failure

2023-02-20 Thread via GitHub
potiuk commented on code in PR #29518: URL: https://github.com/apache/airflow/pull/29518#discussion_r1112340285 ## airflow/providers/google/cloud/operators/cloud_base.py: ## @@ -0,0 +1,34 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] EKami commented on issue #13311: pendulum.tz.zoneinfo.exceptions.InvalidTimezone

2023-02-20 Thread via GitHub
EKami commented on issue #13311: URL: https://github.com/apache/airflow/issues/13311#issuecomment-1437579039 Nope, sorry guys, I gave up on Airflow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] Taragolis opened a new pull request, #29647: Revert "Improve health checks in example docker-compose and clarify usage"

2023-02-20 Thread via GitHub
Taragolis opened a new pull request, #29647: URL: https://github.com/apache/airflow/pull/29647 Just in case check is same error happen as described here #29616 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[airflow] 01/01: Revert "Improve health checks in example docker-compose and clarify usage (#29408)"

2023-02-20 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch revert-29408-docker-compose-change-example in repository https://gitbox.apache.org/repos/asf/airflow.git commit 9da37717449f5267b4ef8c4962cb89f6fa5f102c Author: Andrey Anshin AuthorDate: Tue F

[airflow] branch revert-29408-docker-compose-change-example created (now 9da3771744)

2023-02-20 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch revert-29408-docker-compose-change-example in repository https://gitbox.apache.org/repos/asf/airflow.git at 9da3771744 Revert "Improve health checks in example docker-compose and clarif

[GitHub] [airflow] raphaelauv commented on issue #29432: Jinja templating doesn't work with container_resources when using dymanic task mapping with Kubernetes Pod Operator

2023-02-20 Thread via GitHub
raphaelauv commented on issue #29432: URL: https://github.com/apache/airflow/issues/29432#issuecomment-1437573499 @potiuk could we add this one to https://github.com/apache/airflow/milestone/68 ? -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [airflow] hussein-awala commented on issue #29432: Jinja templating doesn't work with container_resources when using dymanic task mapping with Kubernetes Pod Operator

2023-02-20 Thread via GitHub
hussein-awala commented on issue #29432: URL: https://github.com/apache/airflow/issues/29432#issuecomment-1437570393 @jose-lpa using `Variable.get()` in the dag script is not recommended because the `DagFileProcessor` process the script each X minutes, and it loads this variable from the DB

[GitHub] [airflow] potiuk commented on issue #29315: AIP-44 Migrate BaseJob.heartbeat to InternalAPI

2023-02-20 Thread via GitHub
potiuk commented on issue #29315: URL: https://github.com/apache/airflow/issues/29315#issuecomment-1437565241 Assigned you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[airflow] branch main updated (69babdcf74 -> ee0a56a2ca)

2023-02-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 69babdcf74 Use provided `USE_AIRFLOW_VERSION` env in verify providers step (#29081) add ee0a56a2ca Trigger Class

[GitHub] [airflow] potiuk closed issue #28613: AIP-44 Migrate Trigger class to Internal API

2023-02-20 Thread via GitHub
potiuk closed issue #28613: AIP-44 Migrate Trigger class to Internal API URL: https://github.com/apache/airflow/issues/28613 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] potiuk merged pull request #29099: Trigger Class migration to internal API

2023-02-20 Thread via GitHub
potiuk merged PR #29099: URL: https://github.com/apache/airflow/pull/29099 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] potiuk commented on pull request #29348: Don't push secret in XCOM in BigQueryCreateDataTransferOperator

2023-02-20 Thread via GitHub
potiuk commented on PR #29348: URL: https://github.com/apache/airflow/pull/29348#issuecomment-1437563814 > And I guess we need to add tests for that, for avoid situation that somehow it return in the future. Agree -- This is an automated message from the Apache Git Service. To resp

[airflow] branch main updated (54c4d590ca -> 69babdcf74)

2023-02-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 54c4d590ca fix clear dag run openapi spec responses by adding additional return type (#29600) add 69babdcf74 Use

[GitHub] [airflow] potiuk merged pull request #29081: Use provided `USE_AIRFLOW_VERSION` env in verify providers step

2023-02-20 Thread via GitHub
potiuk merged PR #29081: URL: https://github.com/apache/airflow/pull/29081 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[GitHub] [airflow] Taragolis commented on issue #29640: NoBoundaryInMultipartDefect raised using S3Hook

2023-02-20 Thread via GitHub
Taragolis commented on issue #29640: URL: https://github.com/apache/airflow/issues/29640#issuecomment-1437551366 Just for confirmation do you have same problem with same version of `boto3` and `botocore` if you call [S3.Client.download_file](https://boto3.amazonaws.com/v1/documentation/api/

[GitHub] [airflow] potiuk commented on a diff in pull request #29495: Change permissions of config/password files created by airflow

2023-02-20 Thread via GitHub
potiuk commented on code in PR #29495: URL: https://github.com/apache/airflow/pull/29495#discussion_r1112322251 ## airflow/configuration.py: ## @@ -1545,6 +1548,12 @@ def initialize_config() -> AirflowConfigParser: return local_conf +def make_group_other_inaccessible(fi

[GitHub] [airflow] potiuk commented on a diff in pull request #29495: Change permissions of config/password files created by airflow

2023-02-20 Thread via GitHub
potiuk commented on code in PR #29495: URL: https://github.com/apache/airflow/pull/29495#discussion_r1112320830 ## airflow/configuration.py: ## @@ -1545,6 +1548,12 @@ def initialize_config() -> AirflowConfigParser: return local_conf +def make_group_other_inaccessible(fi

[GitHub] [airflow] Taragolis commented on issue #29640: NoBoundaryInMultipartDefect raised using S3Hook

2023-02-20 Thread via GitHub
Taragolis commented on issue #29640: URL: https://github.com/apache/airflow/issues/29640#issuecomment-1437543247 > shouldn't be such an exception :) `¯\_(ツ)_/¯` https://github.com/boto/botocore/issues/2608 -- This is an automated message from the Apache Git Service. To respond to th

[GitHub] [airflow] steren commented on a diff in pull request #28525: Add CloudRunExecuteJobOperator

2023-02-20 Thread via GitHub
steren commented on code in PR #28525: URL: https://github.com/apache/airflow/pull/28525#discussion_r1112314074 ## airflow/providers/google/cloud/operators/cloud_run.py: ## @@ -0,0 +1,115 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] steren commented on a diff in pull request #28525: Add CloudRunExecuteJobOperator

2023-02-20 Thread via GitHub
steren commented on code in PR #28525: URL: https://github.com/apache/airflow/pull/28525#discussion_r1112314074 ## airflow/providers/google/cloud/operators/cloud_run.py: ## @@ -0,0 +1,115 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] Taragolis commented on pull request #29489: Fix failing multiple-output-inference tests for Python 3.8+

2023-02-20 Thread via GitHub
Taragolis commented on PR #29489: URL: https://github.com/apache/airflow/pull/29489#issuecomment-1437535260 > Then we will fix it there :) That why I mention this PR, might be it would save some time when it time come to Python 3.11 -- This is an automated message from the Apac

[airflow] branch main updated: fix clear dag run openapi spec responses by adding additional return type (#29600)

2023-02-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 54c4d590ca fix clear dag run openapi spec responses

  1   2   3   >